Image processing method, device, and storage medium

Abstract

Embodiments of the present application provide an image processing method, a device, and a storage medium. In the embodiments of the present application, target object detection is performed on a fused image to be processed to obtain image position information of a target object; an image of the target object is acquired on the basis of the image position information of the target object; and then a transparent background layer comprising the image of the target object is generated on the basis of the image of the target object and the image position information of the target object. Therefore, image materials for image design are acquired from existing fused images by performing automatic layer separation on the existing fused images, thereby enriching the image materials, and improving the diversity of the image materials.

Description

Image processing method, device and storage medium Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, and storage medium. Background Art

[0002] With the continuous advancement of image processing technology, various types of fused images are emerging. A fused image is a term used in contrast to natural images captured by a camera. Fused images, also known as composite images, are images composed of different elements or layers, such as e-commerce images, posters, advertising designs, and photos with beautification layers added using photo editing software.

[0003] The aesthetics of fused images are crucial. For example, in the e-commerce sector, a high-quality image can attract more consumers and increase sales. However, during the creation of fused images, problems often arise, such as the uniformity of e-commerce image styles, insufficient image diversity, and a lack of creativity. These issues can negatively impact the quality of product images and the user experience, ultimately impacting sales. To ensure differentiation and diversity in fused images, designers need a sufficient supply of image resources.

[0004] Summary of the Invention

[0005] At least some embodiments of the present application provide an image processing method, device, and storage medium for realizing layer separation of a fused image, thereby providing image materials for the design and production of the fused image.

[0006] The present invention provides an image processing method, including:

[0007] Performing target object detection on the fused image to be processed to determine image position information of the target object;

[0008] Acquiring an image of the target object according to the image position information of the target object and the fused image to be processed;

[0009] A transparent background layer including the image of the target object is generated according to the image position information of the target object and the image of the target object; the transparent background layer including the target object has a transparent background except for the image of the target object.

[0010] The embodiment of the present application further provides a computing device, comprising: a memory and a processor; wherein the memory is configured to store a computer program to be processed;

[0011] The processor is coupled to the memory and configured to execute the computer program to perform the steps in the above image processing method.

[0012] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned image processing method.

[0013] In an embodiment of the present application, target object detection is performed on the fused image to be processed to obtain image position information of the target object; and based on the image position information of the target object, an image of the target object is obtained; thereafter, a transparent background layer containing the image of the target object is generated based on the image of the target object and the image position information of the target object, thereby achieving automatic layer separation of the existing fused image, obtaining image materials for image design from the existing fused image, enriching the image materials, and helping to improve the diversity of the image materials. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0015] FIG1 is a schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application;

[0016] FIG2 is a schematic diagram of a fused image and layers of the fused image provided in an embodiment of the present application;

[0017] FIG3a is an architecture diagram of an image processing system provided in an embodiment of the present application;

[0018] FIG3 b is a schematic diagram of a specific process of layer separation using the image processing method provided in an embodiment of the present application;

[0019] FIG4 is a schematic diagram of a fused image and a transparent background layer of a target object provided in an embodiment of the present application;

[0020] FIG5 is a schematic diagram of the framework and working principle of the open target detection model provided in an embodiment of the present application;

[0021] FIG6 is a schematic diagram of an image segmentation process of a subject object provided in an embodiment of the present application;

[0022] FIG7 is a schematic diagram of a text image segmentation process provided by an embodiment of the present application;

[0023] FIG8 is a schematic diagram showing the effect of erasing text on a fused image to be processed according to an embodiment of the present application;

[0024] FIG9 is a schematic diagram of the template frame separation effect provided in an embodiment of the present application;

[0025] FIG10 is a schematic diagram of the template element separation effect provided in an embodiment of the present application;

[0026] FIG11 is a schematic diagram of the separation effect of each layer provided in an embodiment of the present application;

[0027] FIG12 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] In order to enrich the image material library, in some embodiments of the present application, target object detection is performed on the fused image to be processed to obtain the image position information of the target object; and based on the image position information of the target object, the image of the target object is obtained; then, based on the image of the target object and the image position information of the target object, a transparent background layer of the target object is generated, thereby achieving automatic layer separation of the existing fused image, obtaining image materials for image design from the existing fused image, enriching the image materials, and helping to improve the diversity of the image materials.

[0030] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0031] It should be noted that the same reference numerals denote the same objects in the following drawings and embodiments, and therefore, once an object is defined in one drawing or embodiment, it does not need to be further discussed in the subsequent drawings and embodiments.

[0032] FIG1 is a flow chart of an image processing method provided in an embodiment of the present application. As shown in FIG1 , the image processing method mainly includes:

[0033] 101. Perform target object detection on the fused image to be processed to determine image position information of the target object.

[0034] 102. Acquire an image of the target object according to the image position information of the target object and the fused image to be processed.

[0035] 103. Generate a transparent background layer containing the image of the target object based on the image position information of the target object and the image of the target object.

[0036] In this embodiment, the term "fused image" is used primarily in contrast to natural images. A fused image, also known as a composite image, is an image composed of different elements or layers. Layers are like films containing elements such as text or graphics, stacked one on top of another in sequence to form the final page.

[0037] A fused image can be created by adding some retouching layers to a natural image through artistic design. Of course, fused images can also be created by artists or designers by combining image materials, and so on. A natural image refers to an image captured directly by an image acquisition device of a specific object. The specific object captured by the image acquisition device is the subject in the natural image. For example, a raw photo of a product captured by a camera is a natural image. The product is the subject.

[0038] The main object in a fused image is determined by the design purpose or function of the fused image. For example, a fused image designed to promote a specific object may include an image of the promoted object itself, as well as promotional text and other information. The main object of this fused image is the promoted object. For example, a fused image may be an e-commerce image. E-commerce images are images created to display product information in e-commerce. They typically include product images and text, but may also include other elements such as spokesperson images and image templates. For example, images created on online shopping platforms (such as websites and applications) to display products and pricing information. Since the function of an e-commerce image is to display product information, the main object of the e-commerce image is the product itself, but it can also be something else. In the e-commerce image shown in Figure 2, the snack packaging is the main object. Of course, the main object of a product image can also be objects used to assist in displaying the product, such as a product spokesperson or a pet. For example, if the product is pet food, the main object of the product image may include not only the pet food itself but also the pet itself.

[0039] For another example, the fused image may be a photo with some beautification layers added using photo editing software, and the main object of the original photo is the main object of the fused image.

[0040] The part of the fused image other than the image of the main object is called the image background, or the main object background. For example, the product background on the far right in Figure 2 is the part other than the image of the main object.

[0041] A fused image consists of multiple layers. Multiple refers to two or more layers. The specific number of layers is determined by the source material used to create the fused image. For example, as shown in Figure 2, the fused image is an e-commerce image. E-commerce images typically include some or all of the following: product layer, text layer, template border layer, and template element layer. Template borders and template elements are both image materials within the image template.

[0042] Image templates are used to combine with the main object's image, modify the main object's image background, and / or add decorative elements to the main object's image. Template borders are used to modify the main object's image, generally serving as the frame of the fused image. Template elements are also used to modify the main object's image, also known as decorative elements. They generally include patterns, graphics with specific shapes and / or colors, designs, and / or text with specific fonts, colors, and / or styles, and floating boxes.

[0043] In this embodiment, the fused image to be processed includes at least two layers and may be an original fused image. The original fused image refers to a fused image that has not undergone layer separation, and may be a fused image that has not undergone layer separation using the flowchart shown in FIG1 , such as the e-commerce image on the far left of FIG2 . As shown in the "Image Source" section of FIG3a , the original fused image may be a fused image obtained from an application. The application may be, but is not limited to, an application (APP), a mini-program, a client, or a website.

[0044] Of course, the fused image to be processed can also be an image comprising at least two layers obtained by separating the layers of the original fused image. For example, a fused image can be obtained by separating the main object layer from the original fused image; or a fused image can be obtained by separating the text layer from the original fused image. The images shown from left to right in Figure 2 can all serve as fused images to be processed.

[0045] In this embodiment of the present application, to enrich the image library of fused images, in step 101, target object detection may be performed on the fused image to be processed to determine the image location information of the target object. The target object is determined by the target of the layer separation. Different layer separation targets result in different target objects.

[0046] For example, as shown in the application layer in Figure 3a, if layer separation is performed to separate the main object layer in the fused image to be processed, the target object is the main object. For another example, if layer separation is performed to separate the text layer in the fused image to be processed, the target object is the text. For another example, if layer separation is performed to separate the template border layer in the fused image to be processed, the target object is the template border. For another example, if layer separation is performed to separate the template elements in the fused image to be processed, the target object is the template elements, and so on.

[0047] In some embodiments, the target object can be determined by manual interaction. As shown in the interactive module in FIG3a , the user can select the target object on the fused image to be processed by clicking on it, or by drawing a frame to select the target object.

[0048] In practical applications, when performing object detection on an image, a rectangular detection frame is typically used to mark the target object contained in the image. In this embodiment, the spatial information of the rectangular detection frame can be used to represent the image position information of the target object. The spatial information of the rectangular detection frame is the image position information of the rectangular detection frame on the fused image to be processed, which can reflect the spatial distribution of the target object in the image. Accordingly, the image position information of the target object may include: the center position and size of the rectangular detection frame. Optionally, the center position of the rectangular detection frame can be represented by the coordinates of the center of the rectangular detection frame in the fused image to be processed, and the size of the rectangular detection frame can be represented by the width and height of the rectangular detection frame. Accordingly, the image position information of the target object can be expressed as (x, y, w, h). Where (x, y) represents the coordinates of the center of the rectangular detection frame in the fused image to be processed, and w and h represent the width and height of the rectangular detection frame, respectively. Alternatively, the image position information of the target object may include: the vertex coordinates of the rectangular detection frame, i.e., the coordinates of the four vertices of the rectangular detection frame in the fused image to be processed.

[0049] Of course, the image location information of the target object can also be displayed. Specifically, as shown in the second figure in Figure 4, a rectangular detection frame of the target object is displayed on the fused image to be processed. In some embodiments, the rectangular detection frame of the target object can also be adjusted through manual intervention. For example, as shown in the interactive module in Figure 3a, the displayed rectangular detection frame can be adjusted by scaling the rectangular detection frame (corresponding to the "zoom" in Figure 3a).

[0050] Because the target object's image position information represents the target object's position within the fused image to be processed, in step 102, an image of the target object can be acquired based on the target object's image position information and the fused image to be processed. In some embodiments, the target object's image position information can be used to segment the target object from the fused image to be processed. The specific implementation of this step will be described in detail in the following embodiments and will not be elaborated upon here.

[0051] After acquiring the image of the target object, in step 103, a transparent background layer containing the image of the target object can be generated based on the image position information of the target object and the image of the target object. The image position information of the target object is used to indicate the size of the transparent background layer of the target object, and the image of the target object is used to provide the foreground image of the transparent background layer of the target object. As shown in FIG4 , the image framed by the rectangular detection frame is the image of the target object. The rightmost sub-image in FIG4 is the transparent background layer containing the image of the target object. The image of the target object itself is the foreground image of the transparent background layer of the target object, and the area of ​​the transparent background layer containing the image of the target object, excluding the image of the target object itself, is a transparent background.

[0052] Because the transparent background layer of the image containing the target object is transparent, it can be stored as an image material in the material library. This allows fusion image designers, such as graphic designers, to directly obtain the target object's transparent background layer from the material library and overlay it on the image to be designed to create a new fusion image.

[0053] In this embodiment, target object detection is performed on the fused image to be processed to obtain image position information of the target object; and based on the image position information of the target object, an image of the target object is obtained; thereafter, a transparent background layer containing the image of the target object is generated based on the image of the target object and the image position information of the target object, thereby achieving automatic layer separation of the existing fused image and obtaining image materials for image design from the existing fused image, enriching the image materials and helping to improve the diversity of the image materials.

[0054] On the other hand, by using existing fused images to obtain image materials, there is no need for designers to manually create image materials, which can reduce the cost of obtaining image materials, including but not limited to: time cost and labor cost.

[0055] The specific implementation of the above steps 102 and 103 will be exemplified below in combination with the layer separation of different target objects.

[0056] In some embodiments, the target object is the main object contained in the fused image to be processed. The fused image to be processed is generally the original fused image. For an original fused image containing both text and the main object, the main object layer is generally separated from the original fused image first, followed by the text layer. This is primarily because the main object may contain text information, which does not constitute the text layer that needs to be separated in this application. Therefore, for the original fused image, the target object can be the main object contained in the original fused image.

[0057] Accordingly, in the above step 102, combined with the "subject object detection" of the "model layer" in Figure 3a and Figure 3b, the target object detection on the fused image to be processed can be implemented as follows: using the subject object detection model, subject object detection is performed on the original fused image to determine the image position information of the subject object, that is, the image position information of the subject object in the original fused image.

[0058] In the embodiments of the present application, the specific implementation form of the subject object detection model is not limited. Optionally, the subject object detection model can be a convolutional neural network (CNN) model, a deep neural network (DNN) model, a recurrent neural network (RNN) model, a fully convolutional neural network (FCN) model, or a self-attention model. The self-attention model can be a transformer model or a Swin-Transformer model.

[0059] The aforementioned subject detection model can be used to detect subjects in training image samples used during the subject detection model training phase. However, the detection accuracy for subjects that were not included in the model training phase is poor, and it may even be incapable of detecting subjects that were not included in the model training phase. However, in real-world applications, subjects come in a wide variety of types, and even within the same category, there are tens of thousands of different types. For example, e-commerce images contain tens of thousands, or even hundreds of millions, of different products. If images of each product were used to train the subject detection model, the time and computational cost of model training would be extremely high.

[0060] To reduce the time and computational complexity of the model training phase, this embodiment introduces a mechanism for detecting arbitrary targets based on textual cues. Accordingly, in step 102, target object detection on the fused image to be processed can also be implemented by: obtaining textual description information specifying the target object to be detected; and performing subject object detection on the original fused image based on the textual description information to determine the image position information of the subject object contained in the original fused image, i.e., the image position information of the subject object in the original fused image.

[0061] For example, the original fused image is an e-commerce image, and the text description information may be "detect all products". Then, all products in the e-commerce image can be detected based on the text description information of "detect" all products to obtain the image location information of all products contained in the e-commerce image.

[0062] The above mechanism of detecting arbitrary targets with text prompts can be implemented using an open target detection model, such as the Grounding Dino model.

[0063] As shown in Figure 5, the open target detection model has a dual encoder and single decoder structure, including: an image backbone network (Image Backbone) for image feature extraction, a text backbone network (Text Backbone) for text feature extraction, and a feature enhancer (Feature Enhancer) for fusing image features and text features. In addition, the open target detection model also includes: a language-guided query selection module and a decoding module for the target detection bounding box (Bbox). Among them, the language-guided query selection module is used for query initialization and is mainly used to screen image features that are more similar to the text features of the input text description information. Because the open target detection model incorporates text information for specifying the target object to be detected, it can support open set object detection. After extensive experiments, the inventors of this application found that the open target detection model has higher detection accuracy for subject objects that did not participate in model training than the other neural network models mentioned above. That is, the open target detection model can also achieve high accuracy in detecting subject objects that did not participate in model training, achieving better target detection results.

[0064] For example, in the field of e-commerce, there are tens of thousands, or even hundreds of millions, of e-commerce product categories. By using an open object detection model, it is possible to detect with high accuracy the products used in the model training phase, and also to detect other products that did not participate in the model training with high accuracy.

[0065] Based on the above-mentioned open object detection model, as shown in Figure 5, when performing subject object detection on the original fused image based on the text description information, the original fused image and the text description information can be input into the open object detection model. In the open object detection model, image feature extraction is performed on the original fused image to obtain the image features of the original fused image; text feature extraction is performed on the text description information to obtain the text features of the text description information; the text features of the text description information are used to obtain target image features that are compatible with the text features from the image features of the original fused image; and the image position information of the subject object is determined based on the target image features.

[0066] Specifically, as shown in FIG5 , in the open target detection model, the image backbone network is used to extract image features from the original fused image to obtain image features of the original fused image; the text backbone network is used to extract text features from the text description information to obtain text features of the text description information; further, the feature enhancer can be used to fuse the image features and text features to obtain fused image features and fused text features. In the language-guided query selection module, the fused text features can be used to filter the fused image features to obtain target image features whose similarity to the fused text features meets the set similarity requirements. The similarity requirements can be implemented as: the similarity to the fused text features is greater than or equal to the set similarity threshold; or, the similarity to the fused text features is ranked in the top N. Wherein, N is a positive integer.

[0067] Furthermore, the target image features can be input into the Bbox decoding module, which decodes the target image features to obtain the image position information of the main object contained in the original fused image. Figure 5 shows the image position information of the main object using the center coordinates and width and height (x, y, w, h) of the rectangular detection box, but this is not intended to be limiting.

[0068] After determining the image position information of the subject object, the image of the target object can be obtained based on the subject object's image position information and the original fused image. Specifically, combining the "Subject Object Segmentation" of the "Model Layer" in Figure 3a, Figure 3b, and Figure 6, the original fused image (i.e., Figure 6 (a)) can be segmented based on the subject object's image position information to obtain a mask image of the subject object, as shown in Figure 6 (b).

[0069] Specifically, the image position information of the main object can be used as prompt information (Prompt), and the original fused image can be used as the image to be segmented to input the image segmentation model; in the image segmentation model, the original fused image and the image position information of the main object are encoded respectively to obtain the image feature vector of the original fused image and the prompt feature vector of the image position information of the main object; further, the image feature vector of the original fused image and the prompt feature vector of the image position information of the main object can be mask mapped to obtain a mask map of the main object.

[0070] In the embodiments of the present application, the specific implementation form of the image segmentation model is not limited. In some embodiments, the image segmentation model can be a segmentation-anything model (SAM). The SAM model can complete the segmentation of the subject object in a zero-shot learning situation.

[0071] The SAM model may include an image encoder, a prompt word encoder, and a mask decoder. The image encoder encodes the original fused image to obtain an image feature vector; the prompt word encoder encodes the image position information of the subject object to obtain a prompt feature vector; and the mask decoder maps the image feature vector of the original fused image and the prompt feature vector of the subject object's image position information to a mask, predicting the subject object's mask image to obtain the subject object's mask image.

[0072] Furthermore, combining the "Main Object Cutout" in the "Model Layer" in Figure 3a, Figure 3b, and Figure 6, the original fused image can be cut out based on the mask image of the main object to obtain an image of the main object, as shown in Figure 6(c). Specifically, the area covered by the mask image of the main object is cut out from the original fused image to obtain the image of the main object.

[0073] The image obtained by subtracting the image of the subject from the original fused image is defined as the first fused image, as shown in Figure 6(d). The area corresponding to the original image of the subject in the first fused image becomes a hole area, as shown in the black area in Figure 6(d). Therefore, in this embodiment, image restoration can be performed on the hole area of ​​the first fused image to obtain a second fused image, as shown in Figure 6(e). The background color of the hole area in the restored second fused image is consistent with the background color surrounding the hole area.

[0074] Optionally, an image generation model can be used to repair the hole area of ​​the first fused image to obtain a second fused image. The image generation model can be a visual dilation model, such as a stable dilation (Stable Diffusion) model. Among them, the Stable Diffusion model is an image generation model based on the latent diffusion model (Latent Diffusion Model), which has functions such as generating an image based on a text prompt as input (text2img), generating a new image based on an input image and a corresponding text prompt (img2img), and image inpainting. In this embodiment, the image inpainting function of the Stable Diffusion model is mainly used.

[0075] Based on the image of the subject object obtained by segmenting the original fused image, step 103 can be implemented as follows: determining the size of the subject object's transparent background layer based on the image position information of the subject object; and generating a transparent background layer having the aforementioned size and containing the subject object's image based on the subject object's image, as the transparent background layer of the subject object's image, thereby separating the subject object layer. The foreground image of the transparent background layer containing the subject object's image is the subject object's image; and the background image of the transparent background layer containing the subject object's image is transparent, i.e., the area of ​​the transparent background layer other than the subject object's image is a transparent background.

[0076] Furthermore, the transparent background layer of the image containing the subject object and the subject object's attribute information can be used as image materials for persistent storage. For example, the transparent background layer of the image containing the subject object and the subject object's attribute information can be used as image materials and stored in a material library. The subject object's attribute information refers to the feature information used to reflect the subject object in the original fused image and the subject object's inherent attribute information. The feature information of the subject object in the original fused image includes: the subject object's image position information, the subject object's color, and the image resolution. The subject object's inherent attribute information includes: the subject object's name, category, etc.

[0077] Depending on the subject object, the subject object's inherent attribute information may also include other information. For example, if the subject object is an animal, the inherent attribute information may also include the product model and manufacturer. For another example, if the subject object is an animal, the inherent attribute information may also include the animal's breed, hobbies, etc.

[0078] In addition to separating the main object layer, the image processing method provided in the embodiment of the present application can also be used to separate the text layer. The specific implementation method of separating the text layer is exemplified below.

[0079] In the embodiment of separating the text layer, the target object is text. The fused image to be processed in the image processing method shown in Figure 1 can be the above-mentioned original fused image (such as the image shown in Figure (a) in Figure 6), or it can be the target fused image after the layer of the main object is separated from the original fused image (such as the image shown in Figure (e) in Figure 6). The target fused image can be the second fused image of the above-mentioned embodiment, that is, the second fused image obtained by performing image repair on the hole area of ​​the first fused image as shown in Figure (e) in Figure 6. The first fused image is the image obtained after the image of the main object is removed from the original fused image.

[0080] Accordingly, in combination with the "text detection" of the "model layer" in Figure 3a and Figure 3b, the above-mentioned step 101 of performing target object detection on the fused image to be processed can be implemented as: using the text detection model to perform text detection on the fused image to be processed to determine the image position information of the text contained in the fused image to be processed.

[0081] In the embodiments of the present application, the specific implementation form of the text detection model is not limited. Optionally, the text detection model may be a CNN model, a DNN model, an RNN model, an FCN model or a self-attention model. Of course, the text detection model may be a text detection network model, such as a deep bidirectional network (DBNet) model or a DBNet++ model. Among them, the DBNet++ model introduces a multi-level feature map aggregation (Adaptive Scale Fusion, ASF) module based on the DBNet model. Features of different scales are processed by the ASF module to obtain better fusion features. Both the DBNet model and the DBNet++ model can effectively detect texts of various shapes and directions, and have good performance in detection speed and accuracy.

[0082] The DBNet and DBNet++ models can detect text regions in the fused image to be processed and output the image location information of the text contained in the fused image to be processed, as well as the size of the text region. The image location information of the text contained in the fused image to be processed is represented by the image coordinates of the text region in the fused image to be processed.

[0083] When using the DBNet model or the DBNet++ model to perform text detection on a fused image to be processed, the fused image to be processed can be input into the DBNet model or the DBNet++ model. In the DBNet model or the DBNet++ model, the fused image to be processed is first preprocessed. Specifically, the fused image to be processed is subjected to preprocessing operations such as scaling and normalization to make the preprocessed fused image suitable for input into a CNN for training and prediction. Feature extraction can then be performed on the preprocessed fused image. Specifically, a CNN can be used to extract image features from the preprocessed fused image. Furthermore, feature fusion can be performed on the extracted image features. Specifically, the extracted image features can be fused using a bidirectional long short-term memory (LSTM) network to obtain fused image features. For the DBNet++ model, the ASF module can also be used to fuse image features of different scales to obtain fused image features.

[0084] Furthermore, the fused image features can be transformed to suit subsequent text classification and regression tasks. Specifically, the fused image features can be transformed using a 1x1 convolutional layer to suit subsequent classification and regression tasks. Using a 1x1 convolutional layer can reduce or increase the number of channels to meet the requirements of the subsequent network.

[0085] Furthermore, the fused image features can be used for text classification. Specifically, a classification head can be used to classify each feature point to determine whether it belongs to a text region. The classification head typically consists of multiple (two or more) convolutional layers and a fully connected layer to convert the fused image features into a binary classification output (text or non-text).

[0086] Furthermore, a regression head can be used to regress each text detection box to determine its image position and size information. The regression head typically consists of multiple (two or more) convolutional layers and a fully connected layer, which converts the fused features into a four-tuple vector output. The four-tuple can be represented by the coordinates of the top left corner, width, and height of the text detection box.

[0087] The image position information and size information of the text detection frame are the image position information and size information of the text area of ​​the fused image to be processed, that is, the image position information and size information of the text contained in the fused image to be processed.

[0088] Accordingly, step 102 of acquiring an image of the target object based on the image position information of the target object and the fused image to be processed can be implemented by acquiring an image of the text from the fused image to be processed based on the image position information of the text. Specifically, an image of the text region can be segmented from the fused image to be processed based on the image position information of the text.

[0089] For example, as shown in Figure 7 (b), the image position information of the text is represented by the image position information and size information of the text detection box in the text area; based on the image position information of the text detection box, the image of the text can be segmented from the fused image to be processed, as shown in Figure 7 (c).

[0090] In some embodiments of the present application, the above-described method for segmenting the image of the main object from the original fused image can be used to segment the image of text from the fused image to be processed. However, in actual applications, the size of the text in the fused image is much smaller than the main object. This may result in the inability to accurately obtain a mask image for the text when segmenting the image of the text from the fused image to be processed using the above-described method for segmenting the image of the main object from the original fused image because the text font is too small, resulting in text segmentation errors and low accuracy in text layer separation.

[0091] To improve the accuracy of text layer separation, in this embodiment, as shown in "Text Recognition" in Figure 3b, based on the text image obtained in the above embodiment, text recognition can be performed on the text image to determine the text content contained in the fused image to be processed. In this embodiment of the application, a text recognition model can be used to perform text recognition on the text image to obtain the text content contained in the fused image to be processed.

[0092] Optionally, the text recognition model can be a CNN model, a DNN model, an RNN model, an FCN model, or a Convolutional Recurrent Neural Network (CRNN) model. The inventors of this application have found that the CRNN model outperforms other models in terms of Chinese language recognition accuracy. Therefore, if the fused image to be processed contains Chinese characters, the CRNN model is preferred.

[0093] Furthermore, by combining the "Text Erase" function in the "Model Layer" in Figure 3a, Figure 3b, and Figure 8, the text contained in the fused image to be processed can be erased based on the text image, resulting in the fused image after the text is erased, as shown in Figure 8(b). Figure 8(a) shows the fused image to be processed.

[0094] In the embodiments of the present application, the specific implementation form of erasing the text contained in the fused image to be processed is not limited. In some embodiments, the text contained in the fused image to be processed can be erased using a latent diffusion model. The latent diffusion model is a model based on the latent diffusion model algorithm, which can be used for various image tasks such as image restoration, layout image generation, semantic image generation, and image super-resolution. In this embodiment, the latent diffusion model is used for text erasure, and the intermediate representation result can be obtained by step-by-step denoising in a latent representation space, and then decoded into a complete generated image. Since the latent representation space can significantly reduce the amount of calculation compared to the previous diffusion model based on pixel space, the task of erasing text can be completed within a few seconds.

[0095] In order to enable the latent scaling model to be used for text erasure, it is necessary to train the latent scaling model to enable it to have the text erasure function. The following is an exemplary description of the training process of the latent scaling model.

[0096] When training the latent expansion model, a first sample image, a partial image of text contained in the first sample image, and a second sample image after the text is erased from the first sample image may be obtained. The first sample image may be a fused image. Furthermore, with minimization of the loss function as a training objective, the first sample image and the partial image of text contained in the first sample image are used as training samples to train the initial model corresponding to the latent diffusion model, thereby obtaining the latent diffusion model.

[0097] The loss function is determined by the difference between the predicted sample image after the text is erased from the first sample image output by the model training and the second sample image. The difference between the predicted sample image after the text is erased from the first sample image output by the model training and the second sample image can be represented by the difference in pixel values ​​between the predicted sample image corresponding to the first sample image and the same second sample image. Accordingly, the loss function can be expressed as:

[0098] In the above formula (1), K represents the total number of first sample images, k represents the kth first sample image, and k = 0, 1, ..., K. (X-1) represents the maximum horizontal coordinate of the pixel in the first sample image and the second sample image; (Y-1) represents the maximum vertical coordinate of the pixel in the first sample image and the second sample image. The first sample image and the second sample image are both images of X*Y pixels. (x, y) represents the pixel coordinates of each pixel point in the first sample image and the second sample image. x = 0, 1, ..., (X-1); x = 0, 1, ..., (Y-1).

[0099] Among them, p k(x,y) represents the pixel value of the pixel point with pixel coordinates (x, y) in the second sample image corresponding to the kth first sample image; p k(x,y) The pixel value of the pixel point with pixel coordinates (x, y) in the predicted sample image corresponding to the k-th first sample image.

[0100] The loss functions shown in the above embodiments are for illustrative purposes only and are not intended to be limiting.

[0101] Based on the trained latent scaling model, the image of the text obtained from the fused image to be processed and the fused image to be processed can be input into the latent scaling model. In the latent scaling model, the fused image to be processed can be encoded into low-dimensional latent data. Then, using the image of the text as noise, the image of the text is erased from the low-dimensional latent data to obtain low-dimensional latent data of the image after the text is erased. Furthermore, the low-dimensional latent data of the image after the text is erased can be decoded to obtain a fused image after the text is erased, thereby erasing the text contained in the fused image from the fused image to be processed.

[0102] Furthermore, a target transparent background layer can be added at the image position indicated by the image position information of the text; and based on the text image and the text content contained in the identified fused image to be processed, the text content contained in the fused image to be processed is added to the text transparent background layer, and the text style in the text image is transferred to the text added to the target transparent background layer, so as to obtain a transparent background layer containing the text contained in the fused image to be processed, thereby achieving the separation of the text layer. In this way, the style of the text in the separated transparent background layer containing the text is consistent with the text style in the text image.

[0103] The style of a text is used to reflect the appearance characteristics of the text, including but not limited to: the format of the text (such as one or more of the font, size, font shape and color) and the spacing between characters, etc. Multiple refers to two or more types.

[0104] Optionally, the image of the text and the text content contained in the fused image to be processed can be input into the text style transfer model; in the text style transfer model, the text content contained in the fused image to be processed can be added to the target transparent background layer; and the style of the text content added to the target transparent background layer can be set to the style of the text in the above-mentioned text image.

[0105] Among them, the text style transfer model can be a text generative adversarial network (TextGAN) model, a cycle generative adversarial network (Cycle GAN) or other GAN models.

[0106] The above embodiment identifies the text contained in the fused image to be processed, determining the text content within the fused image to be processed. Furthermore, the text within the fused image to be processed is erased, and a target transparent background layer is added to the text area within the fused image to be processed. The text content and style within the fused image to be processed are then transferred to the target transparent background layer, thereby achieving separation of the text from the transparent background layer. Compared to image segmentation techniques for achieving text layer separation, this embodiment eliminates the need to determine a text mask image and eliminates the need to focus on text size or outline boundaries. Therefore, it helps improve the accuracy of text layer separation.

[0107] After the text layer is separated, the transparent background layer containing the text and the attribute information of the text can be used as image materials for persistent storage. For example, the transparent background layer containing the text and the attribute information of the text can be used as image materials and stored in a material library, etc. The attribute information of the text refers to the information used to reflect the appearance characteristics of the main object in the original fused image. The attribute information of the text may include: image position information of the text, style of the text, spacing between characters of the text, image resolution, etc. For the specific content of the style of the text, please refer to the relevant description of the above embodiment, which will not be repeated here.

[0108] In some embodiments of the present application, the original fused image also includes a template border. To further enrich the image material, the template border can also be separated into layers. Due to the different shapes of template borders, it is impossible to detect the template border of the fused image using object detection methods.

[0109] In order to achieve the separation of the template border layer, in this embodiment, the position of the main object is first excluded from the original fused image. This is because the inventors of this application found through analysis and research on the fused image that for a fused image with a template border, the image of the main object in the fused image will cause certain interference to the separation of the template border. Therefore, the target fused image used to separate the template border can be the image obtained after separating the layer of the main object from the above-mentioned original image, that is, the second fused image shown in the above embodiment. The second fused image is obtained by image restoration of the hole area of ​​the first fused image; the first fused image is the image after the image of the main object is removed from the original fused image. Of course, the target fused image can also be the image obtained after separating the layer of the main object and the text layer from the original image.

[0110] The inventors of the present application have discovered that there is a certain relative positional relationship between the image position of the main object of the fused image and the image position of the template frame of the fused image. Therefore, the target fused image can be segmented according to the image position information of the main object to obtain a mask image containing the template frame of the target fused image.

[0111] Optionally, the image position information of the subject object can be used as a prompt, and the target fused image can be used as the image to be segmented. In the image segmentation model, the target fused image and the image position information of the subject object are encoded separately to obtain an image feature vector of the target fused image and a prompt feature vector of the subject object's image position information. Furthermore, mask mapping can be performed on the image feature vector of the target fused image and the prompt feature vector of the subject object's image position information to obtain a mask image containing a template bounding box of the target fused image. The image position information of the subject object is used to indicate the image region to exclude the subject object during mask mapping.

[0112] Furthermore, the mask image of the template frame used by the target fused image can be screened out from the mask image containing the template frame of the target fused image according to the image position information of the main object.

[0113] Specifically, as shown in Figure 9 (a), based on the upper and lower boundaries in the image position information of the main object, a first mask image whose midpoint is higher than the upper boundary of the main object and a second mask image whose midpoint is lower than the lower boundary of the main object can be screened out from the mask image of the template frame containing the target fusion image; and based on the left and right boundaries in the image position information of the main object, a third mask image located to the left of the left boundary and a fourth mask image located to the right of the right boundary can be screened out from the mask image of the template frame containing the target fusion image.

[0114] Furthermore, the above-mentioned first mask image, second mask image, third mask image and fourth mask image can be connected to obtain the mask image of the template border used by the target fusion image (corresponding to the "border detection" of the "model layer" in Figure 3a and the "template border detection and segmentation" in Figure 3b).

[0115] Furthermore, the mask image of the template border used in the target fusion image can be used to separate the transparent background layer containing the template border from the target fusion image (corresponding to the "template border segmentation" of the "model layer" in Figure 3a and the "template border detection segmentation" in Figure 3b), as shown in Figure 9 (b).

[0116] After separating the transparent background layer containing the template border, the transparent background layer containing the template border and the attribute information of the template border can be used as image materials for persistent storage. For example, the transparent background layer containing the template border and the attribute information of the template border can be used as image materials and stored in a material library. The attribute information of the template border refers to information used to reflect the appearance characteristics of the main object in the original fused image. The attribute information of the template border may include: the image position information of the template border, the color of the template border, the composition, the layout, and the image resolution.

[0117] In some embodiments of the present application, the fused image may also include some template elements. Embodiments of the present application may also provide methods for separating template element layers. Template elements may exist on any transparent background layer separated from the original fused image. For example, in the original fused image shown in FIG10 , template elements exist on layers such as the subject object layer and the template border layer. Therefore, template elements can be separated from any transparent background layer separated from the original fused image. For ease of description, any transparent background layer separated from the original fused image is defined as the target layer.

[0118] In this embodiment, in combination with the "element detection" of the "model layer" in FIG3a and FIG3b, the template element detection model can be used to perform template element detection on the target layer to determine the image position information of the template element, that is, the image position information of the template element in the target layer.

[0119] The target element detection model can be a CNN model, a DNN model, an RNN model, an FCN model, a self-attention model, or an open object detection model. Of course, a You Only Look Once (YOLO) model is one that can identify the category and location of objects in an image with just one look. For example, any of the YOLO models from V1 to V5 can be used.

[0120] Furthermore, combined with the “template element segmentation” of the “model layer” in FIG3a and FIG3b , the target layer can be segmented according to the image position information of the template element to obtain the mask image of the template element.

[0121] Specifically, the image position information of the template element can be used as prompt information (Prompt), and the target layer can be used as the image to be segmented. In the image segmentation model, the image position information of the target layer and the template element are respectively encoded to obtain an image feature vector of the target layer and a prompt feature vector of the image position information of the template element. Furthermore, mask mapping can be performed on the image feature vector of the target layer and the prompt feature vector of the image position information of the template element to obtain a mask map of the template element. The image segmentation model can be a SAM model. The specific implementation method can be referred to the relevant content of the subject-object layer separation mentioned above and will not be repeated here.

[0122] Furthermore, combining the "Template Element Cutout" in the "Model Layer" in Figure 3a and Figure 3b, the target layer can be cut out based on the template element's mask to obtain an image of the template element. Specifically, based on the template element's mask, the area covered by the template element's mask is cut out from the target layer to obtain an image of the template element.

[0123] Furthermore, a transparent background layer including the template element can be generated based on the image position information of the template element and the image of the template element. In the transparent background layer including the template element, the area other than the template element is a transparent background.

[0124] Specifically, the size of the transparent background layer of the template element is determined based on the image position information of the template element. The size of the transparent background layer of the template element is the same as the size of the image area determined by the image position information of the template element. Furthermore, based on the image of the template element, a transparent background layer having the aforementioned size and containing the image of the template element can be generated as the transparent background layer containing the template element, thereby achieving separation of the template element layer. The foreground image of the transparent background layer containing the template element is the image of the template element; the background image of the transparent background layer containing the template element is transparent.

[0125] Furthermore, the transparent background layer containing the template element and the attribute information of the template element can be used as image material for persistent storage. For example, the transparent background layer containing the template element and the attribute information of the template element can be used as image material and stored in the material library. The attribute information of the template element refers to the appearance characteristics of the template element in the target layer. The attribute information of the template element includes: the image location information of the template element, the shape, color, layout style, and image resolution of the template element.

[0126] In an embodiment of the present application, as shown in FIG3 b , after separating the layers of each object from the original fused image, the background style of the original fused image may be classified, and attribute information of the background layer of the original fused image may be saved.

[0127] For each transparent background layer separated from the original fused image, the attribute information of each transparent background layer and the objects corresponding to each transparent background layer can be used as image material for persistent storage. The objects corresponding to each transparent background layer include, but are not limited to, one or more of the following: subject objects, text, template borders, and template elements. For details on the attribute information of the objects corresponding to each transparent background layer, please refer to the relevant content of the above embodiment and will not be repeated here.

[0128] For each transparent background layer, super-resolution processing can be performed to obtain transparent background layers of different resolutions. Super-resolution processing refers to reconstructing a high-resolution image from a low-resolution image.

[0129] Optionally, a super-resolution algorithm can be used to super-resolution each transparent background layer to obtain transparent background layers of different resolutions. Super-resolution algorithms include but are not limited to: interpolation algorithms, edge-based methods, statistics-based methods, or deep learning-based methods. Among them, the interpolation algorithm increases the resolution of the image by interpolating the pixels of the low-resolution image. The edge-based method uses the edge information in the image to reconstruct the high-resolution image. For example, the edge information of the low-resolution image is extracted using an edge detection algorithm, and then mapped to the high-resolution image to obtain a high-resolution image. The statistics-based method is based on statistical modeling of the relationship between pixels in the image. For example, the low-resolution image is learned using a high-resolution image in the example library, and then the low-resolution image is converted into a high-resolution image using a statistical model. The deep learning-based method achieves high-quality super-resolution reconstruction by using a neural network model to learn the mapping relationship between low-resolution images and high-resolution images.

[0130] In some embodiments, each transparent background layer may be subjected to super-resolution processing, and the resulting transparent background layers of different resolutions may be persistently stored. Of course, in other embodiments, as shown in the "Super-resolution" in the "Interaction Module" of FIG3a , the user may issue a super-resolution processing request through the interaction module. The super-resolution processing request includes the requested image resolution. Alternatively, in response to the user's super-resolution processing request, the above-mentioned operation of super-resolution processing of the transparent background layer may be performed to obtain a transparent background layer of an image resolution that meets the user's super-resolution processing request.

[0131] The above-described embodiments of separating the layers of each object and the order of separation are for illustrative purposes only and do not imply that the fused image includes all layers, nor does it imply that the fused image includes the layers shown in the above-described embodiments. The following uses the original fused image shown in FIG11 as an example for a detailed description.

[0132] As shown in Figure 11, the original fused image is a product image, including the product "headphones," text, a template border, and template elements. As shown in Figure 11, the layer separation method proposed in the above embodiment can be used to separate the original product image on the left side of Figure 11 into the transparent background layers on the right side, including the product layer, text layer, template element layer, and template border layer. Each layer and its attribute information can be preserved. By recombining the transparent background layers, the original product image on the left side can be restored.

[0133] In the embodiments of the present application, the image processing method proposed in the above embodiments can be used to separate the layers of various fused images to obtain the transparent background layers of each object contained in the fused image, thereby realizing automated layer separation, avoiding the tedious and repetitive work of manual image cutting, and reducing the time for manual processing. At the same time, automated layer separation can ensure that the visual effects of the original fused image are fully presented on the subsequent editing page, thereby improving development efficiency and aesthetics. Through reasonable cropping and super-resolution processing, the separated layers can also achieve adaptation to different resolutions, thereby improving the compatibility and usability of the image material.

[0134] Through the image processing method provided in the embodiment of the present application, a large number of fused images can be separated into layers to obtain a large amount of image materials, which can provide rich image materials for the subsequent production of fused images (such as e-commerce images, advertising images or posters, etc.), thereby improving the production efficiency of fused images and reducing production costs.

[0135] It should be noted that the execution entity of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution entity of steps 101 and 102 can be device A; for another example, the execution entity of step 101 can be device A, and the execution entity of step 102 can be device B; and so on.

[0136] In addition, some of the processes described in the above embodiments and the accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The sequence numbers of the operations, such as 101, 102, etc., are used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.

[0137] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned image processing method.

[0138] Figure 12 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. As shown in Figure 12, the computing device mainly includes: a memory 12a and a processor 12b; wherein the memory 12a is configured to store computer programs;

[0139] The processor 12b is coupled to the memory 12a and is configured to execute a computer program for: performing target object detection on the fused image to be processed to determine image position information of the target object; obtaining an image of the target object based on the image position information of the target object and the fused image to be processed; and generating a transparent background layer containing the image of the target object based on the image position information of the target object and the image of the target object. The transparent background layer containing the target object, excluding the image of the target object, is transparent.

[0140] In some embodiments, the fused image to be processed is an original fused image. The target object is the main object contained in the original fused image. The original fused image refers to a fused image that has not undergone layer separation. When performing target object detection on the fused image to be processed, the processor 12b is configured to: obtain textual description information specifying the target object to be detected; and perform main object detection on the original fused image based on the textual description information to determine image position information of the main object.

[0141] Furthermore, when the processor 12b performs subject object detection on the original fused image based on the text description information, it is configured to: input the original fused image and the text description information into an open target detection model; in the open target detection model, perform image feature extraction on the original fused image to obtain image features of the original fused image; perform text feature extraction on the text description information to obtain text features of the text description information; use the text features of the text description information to obtain target image features that are adapted to the text features from the image features of the original fused image; and determine the image position information of the subject object based on the target image features.

[0142] Optionally, when the processor 12b obtains the image of the target object based on the image position information of the target object and the fused image to be processed, it is set to: perform image segmentation on the original fused image according to the image position information of the main object to obtain a mask map of the main object; and perform cutout processing on the original fused image according to the mask map of the main object to obtain an image of the main object.

[0143] Accordingly, when the processor 12b generates a transparent background layer containing the image of the target object based on the image position information of the target object and the image of the target object, it is configured to: determine the first size of the transparent background layer of the main object based on the image position information of the main object; and generate a transparent background layer with a first size and containing the image of the main object based on the image of the main object.

[0144] Optionally, when the processor 12b performs image segmentation on the original fused image based on the image position information of the target object, it is configured as follows: the image position information of the main object is used as prompt information, and the original fused image is used as the image to be segmented to input the image segmentation model; in the image segmentation model, the image position information of the original fused image and the main object are respectively encoded to obtain the image feature vector of the original fused image and the prompt feature vector of the image position information of the main object; the image feature vector of the original fused image and the prompt feature vector of the image position information of the main object are mask mapped to obtain a mask map of the main object.

[0145] Optionally, the processor 12b is further configured to: perform image restoration on the hole area of ​​the first fused image to obtain a second fused image; the first fused image is an image obtained by subtracting the image of the main object from the original fused image.

[0146] In some embodiments, the fused image to be processed is the original fused image or the target fused image obtained by separating the layer of the main object from the original fused image; the target object is text. Accordingly, when performing target object detection on the fused image to be processed, the processor 12b is configured to: perform text detection on the fused image to be processed using a text detection model to determine image position information of the text contained in the fused image to be processed. Accordingly, when acquiring an image of the target object based on the image position information of the target object and the fused image to be processed, the processor 12b is configured to: acquire an image of the text from the fused image to be processed based on the image position information of the text.

[0147] Optionally, when the processor 12b generates a transparent background layer containing the image of the target object based on the image position information of the target object and the image of the target object, it is configured to: perform text recognition on the image of the text to determine the text content contained in the fused image to be processed; erase the text contained in the fused image to be processed based on the image of the text; add a target transparent background layer at the image position indicated by the image position information of the text; and add the text content contained in the fused image to be processed on the target transparent background layer based on the image and text content of the text, and transfer the text style in the image of the text to the text added on the target transparent background layer to obtain a transparent background layer containing the text contained in the fused image to be processed.

[0148] Furthermore, when the processor 12b erases the text contained in the fused image to be processed based on the image of the text, it is configured to: input the image of the text and the fused image to be processed into a latent diffusion model; in the latent diffusion model, encode the fused image to be processed into low-dimensional latent data; use the image of the text as noise, erase the image of the text in the low-dimensional latent data to obtain the low-dimensional latent data after the image of the text is erased; further, decode the low-dimensional latent data after the image of the text is erased to obtain the fused image after the text is erased.

[0149] Optionally, the processor 12b is further configured to: before inputting the image of the text and the fused image to be processed into the latent diffusion model, obtain a first sample image, a local image of the text contained in the first sample image, and a second sample image after the text is erased from the first sample image; with minimization of the loss function as the training goal, and with the first sample image and the local image of the text contained in the first sample image as training samples, perform model training on the initial model corresponding to the latent diffusion model to obtain the latent diffusion model; wherein the loss function is determined by the difference between the predicted sample image after the text is erased from the first sample image output by the model training, and the second sample image.

[0150] Furthermore, when the processor 12b adds the text content contained in the fused image to be processed on the target transparent background layer based on the image and text content of the text, and transfers the text style in the image of the text to the text added on the target transparent background layer, it is set to: input the image and text content of the text into the text style transfer model; in the text style transfer model, add the text content contained in the fused image to be processed on the target transparent background layer; and set the style of the text content added on the target transparent background layer to the style of the text in the image of the text.

[0151] In other embodiments, the processor 12b is further configured to: perform image segmentation on the target fused image according to the image position information of the main object to obtain a mask image of the template border containing the target fused image; the target fused image is an image obtained after separating the layer of the main object from the original fused image; according to the image position information of the main object, filter out the mask image of the template border used by the target fused image from the mask image of the template border containing the target fused image; and separate the transparent background layer of the template border from the target fused image according to the mask image of the template border used by the target fused image.

[0152] Furthermore, when the processor 12b filters out the mask image of the template frame used by the target fused image from the mask image containing the template frame in the target fused image according to the image position information of the main object, it is set to: according to the upper and lower boundaries in the image position information of the main object, filter out a first mask image whose midpoint is higher than the upper boundary of the main object and a second mask image whose midpoint is lower than the lower boundary of the main object from the mask image containing the template frame; according to the left and right boundaries in the image position information of the main object, filter out a third mask image located on the left side of the left boundary and a fourth mask image located on the right side of the right boundary from the mask image containing the template frame of the target fused image; and connect the first mask image, the second mask image, the third mask image and the fourth mask image to obtain the mask image of the template frame used by the target fused image.

[0153] Optionally, the processor 12b is also configured to: perform template element detection on the target layer using a template element detection model to determine the image position information of the template element; the target layer is any transparent background layer separated from the original fused image; according to the image position information of the template element, perform image segmentation on the target layer to obtain a mask image of the template element; according to the mask image of the template element, perform cutout processing on the target layer to obtain an image of the template element; according to the image position information of the template element and the image of the template element, generate a transparent background layer containing the template element.

[0154] Optionally, the processor 12b is also configured to: store the attribute information of each transparent background layer separated from the original fused image and the objects corresponding to each transparent background layer as image materials for persistent storage; and perform super-resolution processing on each transparent background layer to obtain transparent background layers of different resolutions.

[0155] In some optional embodiments, as shown in Figure 12 , the computing device may further include components such as a communication component 12c and a power supply component 12d. In some embodiments, the computing device may be implemented as a terminal device such as a computer and may further include optional components such as a display component 12e and an audio component 12f. The schematic illustration of some components in Figure 12 does not imply that the computing device includes all components shown in Figure 12 , nor does it imply that the computing device only includes the components shown in Figure 12 .

[0156] The computing device provided in this embodiment obtains the image position information of the target object by performing target object detection on the fused image to be processed; and obtains the image of the target object based on the image position information of the target object; thereafter, a transparent background layer containing the image of the target object is generated based on the image of the target object and the image position information of the target object, thereby achieving automatic layer separation of the existing fused image, obtaining image materials for image design from the existing fused image, enriching the image materials, and helping to improve the diversity of the image materials.

[0157] On the other hand, by using existing fused images to obtain image materials, there is no need for designers to manually create image materials, which can reduce the cost of obtaining image materials, including but not limited to: time cost and labor cost.

[0158] In an embodiment of the present application, the memory is configured to store a computer program and may be configured to store various other data to support operations on the device where it is located. The processor may execute the computer program stored in the memory to implement the corresponding control logic. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0159] In the embodiment of the present application, the processor can be any hardware processing device that can execute the logic of the above method. Optionally, the processor can be a central processing unit (CPU), a graphics processing unit (GPU) or a microcontroller unit (MCU); it can also be a programmable device such as a field programmable gate array (FPGA), a programmable array logic device (PAL), a general array logic device (GAL), a complex programmable logic device (CPLD); or an application specific integrated circuit (ASIC) chip; or an advanced reduced instruction set (RISC) processor (Advanced RISC Machines, ARM) or a system on chip (SoC), etc., but is not limited thereto.

[0160] In an embodiment of the present application, the communication component is configured to facilitate wired or wireless communication between the device in which it is located and other devices. The device in which the communication component is located can access a wireless network based on a communication standard, such as Wireless Fidelity (WiFi), 2G or 3G, 4G, 5G or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can also be based on Near Field Communication (NFC) technology, Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology or other technologies.

[0161] In an embodiment of the present application, the display component may include a liquid crystal display (LCD) and a touch panel (TP). If the display component includes a touch panel, the display component may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0162] In embodiments of the present application, a power supply assembly is configured to provide power to various components of the device in which it is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply assembly is located.

[0163] In an embodiment of the present application, the audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as call mode, recording mode, and voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component further includes a speaker configured to output audio signals. For example, for a device with a language interaction function, voice interaction with a user may be achieved through the audio component.

[0164] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0165] It should also be noted that the descriptions such as "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit "first" and "second" to different types.

[0166] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) that contain computer-usable program code.

[0167] The present application is described with reference to the flow chart and / or block diagram of the method, device (or system) and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0168] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0170] In a typical configuration, a computing device includes one or more processors (eg, CPU, etc.), input / output interfaces, network interfaces, and memory.

[0171] Memory may include non-permanent storage in a computer-readable medium, random-access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0172] Computer storage media is readable storage media, also known as readable media. Readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium configured to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0173] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, commodity, or apparatus comprising the aforementioned elements.

[0174] The above contents are embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should all be included within the scope of the claims of the present application.

Claims

1. An image processing method, comprising: Performing target object detection on the fused image to be processed to determine image position information of the target object; Acquire an image of the target object according to the image position information of the target object and the fused image to be processed; A transparent background layer including the image of the target object is generated according to the image position information of the target object and the image of the target object; the transparent background layer including the target object has a transparent background except for the image of the target object.

2. The method according to claim 1, wherein: The fused image to be processed is an original fused image, and the target object is a main object contained in the original fused image; the original fused image refers to a fused image that has not been layer-separated; The performing target object detection on the fused image to be processed to determine the image position information of the target object includes: Obtain text description information for specifying the target object to be detected; Inputting the original fused image and the text description information into an open object detection model; In the open object detection model, image feature extraction is performed on the original fused image to obtain image features of the original fused image; Performing text feature extraction on the text description information to obtain text features of the text description information; Using the text features of the text description information, obtaining target image features adapted to the text features from the image features of the original fused image; The image position information of the main object is determined according to the target image feature.

3. The method according to claim 2, wherein: The step of acquiring the image of the target object according to the image position information of the target object and the fused image to be processed includes: According to the image position information of the main object, the original fused image is segmented to obtain a mask image of the main object; According to the mask image of the main object, the original fused image is subjected to a matting process to obtain an image of the main object; Correspondingly, generating a transparent background layer including the image of the target object according to the image position information of the target object and the image of the target object includes: Determining a first size of a transparent background layer of the main object according to the image position information of the main object; A transparent background layer having the first size and including the image of the main object is generated according to the image of the main object.

4. The method according to claim 3, wherein: The performing image segmentation on the original fused image according to the image position information of the target object to obtain a mask image of the main object includes: The image position information of the main object is used as the prompt information, and the original fused image is used as the image to be segmented. Enter the image segmentation model; In the image segmentation model, the original fused image and the image position information of the main object are respectively encoded to obtain an image feature vector of the original fused image and a prompt feature vector of the image position information of the main object; Mask mapping is performed on the image feature vector of the original fused image and the prompt feature vector of the image position information of the main object to obtain a mask map of the main object.

5. The method according to claim 3, wherein: The method further comprises: Image restoration is performed on the hole region of the first fused image to obtain a second fused image; the first fused image is an image obtained by subtracting the image of the main object from the original fused image.

6. The method according to claim 1, wherein: The fused image to be processed is an original fused image or a target fused image after the layer of the main object is separated from the original fused image; The target object is text; The performing target object detection on the fused image to be processed to determine the image position information of the target object includes: Performing text detection on the fused image to be processed by using a text detection model to determine image position information of the text contained in the fused image to be processed; The step of acquiring the image of the target object according to the image position information of the target object and the fused image to be processed includes: According to the image position information of the text, acquiring the image of the text from the fused image to be processed; The step of generating a transparent background layer including the image of the target object according to the image position information of the target object and the image of the target object comprises: Performing text recognition on the image of the text to determine the text content contained in the fused image to be processed; According to the image of the text, erasing the text contained in the fused image to be processed; Adding a target transparent background layer at the image position indicated by the image position information of the text; According to the image of the text and the text content, the text content contained in the fused image to be processed is added to the target transparent background layer, and the text style in the image of the text is transferred to the text added to the text transparent background layer to obtain a transparent background layer containing the text contained in the fused image to be processed.

7. The method according to claim 6, wherein: The step of erasing the text contained in the fused image to be processed according to the image of the text includes: Inputting the image of the text and the fused image to be processed into a latent diffusion model; In the latent diffusion model, encoding the fused image to be processed into low-dimensional latent data; Using the image of the text as noise, erasing the image of the text in the low-dimensional latent data to obtain low-dimensional latent data after erasing the image of the text; The low-dimensional latent data after the image with the text erased is decoded to obtain a fused image after the text is erased.

8. The method according to claim 7, wherein: Before inputting the image of the text and the fused image to be processed into the potential diffusion model, the method further includes: Acquire a first sample image, a partial image of text contained in the first sample image, and a second sample image after the text is erased from the first sample image; Taking minimization of the loss function as a training objective, taking the first sample image and a partial image of the text contained in the first sample image as training samples, performing model training on an initial model corresponding to the latent diffusion model to obtain the latent diffusion model; The loss function is determined by the difference between the predicted sample image after the text is erased from the first sample image output by model training and the second sample image.

9. The method according to claim 7, wherein: The step of adding the text content contained in the fused image to be processed to the target transparent background layer according to the image of the text and the text content, and transferring the text style in the image of the text to the text added to the target transparent background layer includes: Inputting the image of the text and the text content into a text style transfer model; In the text style transfer model, the text content contained in the fused image to be processed is added to the target transparent background layer; and the style of the text content added to the target transparent background layer is set to the style of the text in the image of the text.

10. The method according to any one of claims 2 to 9, wherein: Also includes: According to the image position information of the main object, the target fused image is segmented to obtain a mask image containing a template frame of the target fused image; The target fused image is an image obtained by separating the layer of the main object from the original fused image; According to the image position information of the subject object, the mask image of the template frame used by the target fused image is screened out from the mask image containing the template frame of the target fused image; According to the mask image of the template frame used by the target fused image, a transparent background layer including the template frame is separated from the target fused image.

11. The method according to claim 10, wherein: The step of selecting the mask image of the template frame used by the target fused image from the mask image containing the template frame in the target fused image according to the image position information of the subject object comprises: According to the upper and lower boundaries in the image position information of the main object, a first mask image whose midpoint is higher than the upper boundary of the main object and a second mask image whose midpoint is lower than the lower boundary of the main object are screened out from the mask image including the template frame of the target fused image; According to the left and right boundaries in the image position information of the subject object, a third mask image located on the left side of the left boundary and a fourth mask image located on the right side of the right boundary are screened out from the mask image including the template frame of the target fused image; The first mask image, the second mask image, the third mask image and the fourth mask image are connected to obtain a mask image of a template frame used by a target fused image.

12. The method according to claim 10, wherein: Also includes: Performing template element detection on the target layer using the template element detection model to determine image position information of the template element; The target layer is any transparent background layer separated from the original fused image; According to the image position information of the template element, the target layer is segmented to obtain a mask image of the template element; According to the mask image of the template element, the target layer is subjected to a cutout process to obtain an image of the template element; The transparent background layer including the template element is generated according to the image position information of the template element and the image of the template element.

13. The method according to claim 12, wherein: Also includes: The transparent background layers separated from the original fused image and the attribute information of the objects corresponding to the transparent background layers are used as image materials for persistent storage; The transparent background layers are super-resolved to obtain transparent background layers with different resolutions.

14. A computing device comprising: A memory and a processor; wherein the memory is configured to store a computer program; The processor is coupled to the memory and is configured to execute the computer program for performing the steps in the method according to any one of claims 1 to 13.

15. A computer-readable storage medium storing computer instructions, which, when executed by one or more processors, causes the one or more processors to perform the steps of the method according to any one of claims 1 to 13.

Images