A method and system for redrawing the background of commodity images based on multidimensional composite constraints

Abstract

This invention belongs to the field of image processing technology and provides a method and system for redrawing the background of product images based on multidimensional composite constraints. The method involves: extracting the main subject of the original image using a matting model to generate a main subject image containing alpha channels; preprocessing a reference scene image to obtain an optimized scene image, which is then input into a multimodal large model, a depth map model, an IPAdapter model, and a BrushNet model to generate prompts and construct multidimensional composite constraints for background generation using Stable Diffusion; generating a background image that matches the main subject of the original image using the Stable Diffusion model; and compositing the main subject image and the background image to output the final product image. This invention can automatically and accurately generate customized scenes that conform to specified visual parameters, significantly reducing labor and time costs and achieving a high degree of natural visual fusion between products and scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a commodity image background redrawing method and system based on multi-dimensional composite constraints. BACKGROUND

[0002] With the rapid development of e-commerce, it is often necessary to provide customers with the display of the matching effect of commodities in a specific scene, and image processing technology (especially image background processing technology) plays an important role; the existing background redrawing method usually adopts mechanical superposition or basic image fusion algorithm for simple synthesis, which has the following defects: Firstly, the existing background redrawing method cannot realize the automatic matching and natural transition of commodities and background in color, illumination and texture through mechanical image superposition or basic fusion algorithm, resulting in obvious visual fragmentation of the synthesized image; Secondly, the existing background redrawing method cannot dynamically generate customized scenes highly consistent with commodity semantics, style and composition requirements by relying on a preset static background material library, and the background material has great limitations; Thirdly, the existing background redrawing method cannot realize intelligent understanding, automatic generation and pixel-level adaptive fusion of scene content through manual splicing and manual parameter adjustment, and the labor and time cost is high.

[0003] Therefore, we need to develop a commodity image background redrawing method and system based on multi-dimensional composite constraints, which can break away from the dependence on preset backgrounds, automatically and accurately generate customized scenes that meet specified visual parameters, greatly reduce labor and time costs, and realize high natural visual fusion of commodities and scenes. SUMMARY

[0004] The present application aims to provide a commodity image background redrawing method and system based on multi-dimensional composite constraints to solve the problems of insufficient fusion, great limitations and high cost of the existing background redrawing method mentioned in the background.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions: According to one aspect of the present application, a commodity image background redrawing method based on multi-dimensional composite constraints is provided, which comprises the following steps: extracting the main body of the original image through a matting model to generate a main body image containing a transparent channel; preprocessing the reference scene image to obtain an optimized scene image; inputting the optimized scene image into a multi-modal large model to generate a prompt word; The optimized scene graph is input into the depth map model, IPAdapter model and BrushNet model respectively to construct the multi-dimensional composite constraints for generating the background using Stable Diffusion. The prompt words and the multidimensional composite constraints are input into the Stable Diffusion model, and the Stable Diffusion model generates a background image that matches the subject of the original image. The main image and the background image are combined to output the final product image.

[0006] According to another aspect of the present invention, a product image background redrawing system based on multidimensional composite constraints is provided. The system includes: a matting module, a preprocessing module, a prompt word generation module, a composite constraint construction module, a background image generation module, and a product image compositing module. Wherein: The aforementioned image matting module is used to extract the main subject of the original image using a matting model and generate a main subject image containing the alpha channel; The aforementioned preprocessing module is used to preprocess the reference scene image to obtain an optimized scene image; The aforementioned prompt word generation module is used to input the optimized scene graph into the multimodal large model and generate prompt words; The aforementioned composite constraint construction module is used to input the optimized scene graph into the depth map model, IPAdapter model, and BrushNet model respectively to construct multi-dimensional composite constraint conditions for the Stable Diffusion background generation. The aforementioned background image generation module is used to input the prompt word and the multidimensional composite constraint conditions into the Stable Diffusion model, and generate a background image that matches the main body of the original image through the Stable Diffusion model; The aforementioned product image compositing module is used to combine the main image and the background image to output the final product image.

[0007] Based on the aforementioned scheme, the matting model is an encoder-decoder model built on the ViTAE architecture; wherein, the encoder of the matting model includes a global self-attention module, a local convolution module and a multi-scale feature fusion module, and the global self-attention module and the local convolution module are set in parallel.

[0008] Based on the aforementioned scheme, the generation of the main image containing the transparency channel specifically involves: The original image is input into the depth map model, the first depth map feature is extracted, and the region of interest containing the main body of the original image is located and obtained based on the RGB information of the original image and the first depth map feature. The region of interest image is input into the matting model, global and local features are extracted and fused, and a high-precision alpha mask is generated. The high-precision Alpha mask is subjected to morphological dilation to obtain an optimized Alpha mask. The optimized alpha mask is used as a transparency channel and synthesized with the corresponding main area in the original image to generate a main image containing the transparency channel.

[0009] The specific steps for locating and obtaining the region of interest image containing the main body of the original image include: The original image is processed by the depth map model to extract the first depth map features, and image processing is performed based on the first depth map features to obtain an initial foreground depth mask. The RGB information of the original image is concatenated with the initial foreground depth mask to form multimodal input data, which is then input into the matting model to output a visual feature map containing high-level semantic information of the subject. Thresholding or lightweight clustering is performed on the visual feature map to obtain a coarse localization mask for the main semantic region; Perform a logical AND operation between the initial foreground depth mask and the coarse positioning mask to obtain the fused mask; Based on the fusion mask, the corresponding region is cropped from the original image and extended outward by a preset boundary margin to output the region of interest image.

[0010] The specific steps for predicting and generating a high-precision alpha mask include: The region of interest image is input into the matting model; Global features of the main subject in the original image are extracted using the global self-attention module of the encoder in the image matting model. Local features of the main subject in the original image are extracted using the local convolution module of the encoder in the image matting model. The global and local features are fused by the multi-scale feature fusion module of the encoder in the matting model to obtain fused features; The decoder in the matting model predicts and generates a high-precision alpha mask based on the fused features.

[0011] The specific steps of performing morphological dilation on the high-precision alpha mask include: Define a circular structuring element with a preset radius R; Create a blank image with the same size as the high-precision alpha mask; For each pixel position (i, j) in the high-precision alpha mask, align the origin of the circular structuring element to the pixel position, and calculate the maximum value of the alpha value of all pixels within the coverage area of ​​the circular structuring element, Max(i, j). The maximum value Max(i, j) is assigned to the pixel at the same coordinates (i, j) in the blank image; Once the calculation and assignment of each pixel position (i, j) in the high-precision alpha mask are completed, the optimized alpha mask is obtained.

[0012] Based on the aforementioned scheme, the preprocessing of the reference scene image specifically includes: Identify interfering elements in the reference scene image and generate corresponding interference masks; wherein, the interfering elements include text, watermarks, etc. The reference scene image and the interference mask are input into the LaMa model. The LaMa model's fast Fourier convolution is used to perform context-aware texture filling and pixel repair to obtain the optimized scene image.

[0013] Based on the aforementioned solution, the generation of prompt words specifically involves: The optimized scene graph is input into a multimodal large model, and the deep visual features of the optimized scene graph are extracted by the visual encoder of the multimodal large model. The deep visual features are mapped and aligned to the shared semantic space of the multimodal large model to obtain aligned features; wherein, the shared semantic space is learned by pre-training the multimodal large model; Based on the alignment features, a natural language description of the optimized scene graph is generated through the autoregressive generation of the multimodal large model, and the natural language description is used as a prompt word.

[0014] Based on the aforementioned scheme, the multidimensional composite constraint conditions for constructing the Stable Diffusion generation background are specifically as follows: The optimized scene map is input into the depth map model, and the second depth map features of the optimized scene map are extracted as spatial structure constraints. The optimized scene graph is input into the IPAdapter model, and the visual style features of the optimized scene graph are extracted as semantic style constraints. The optimized scene map and the high-precision alpha mask are input into the BrushNet model to extract boundary fusion features as pixel-level boundary constraints. Based on the second depth map features, the visual style features, and the boundary fusion features, a multidimensional composite constraint condition for generating the background using StableDiffusion is constructed.

[0015] Based on the aforementioned scheme, when the prompt word and the multidimensional composite constraint are input into the StableDiffusion model, the text encoder of the StableDiffusion model first encodes the prompt word into a text vector, and then fuses the text vector with the visual style features in the multidimensional composite constraint through a cross-attention mechanism to obtain the text-style constraint. Further, the second depth map feature and boundary fusion feature in the multidimensional composite constraint will be input into the backbone network of the StableDiffusion model to generate the depth-boundary constraint, so that the StableDiffusion model generates a background image that matches the subject of the original image under the joint constraints of the text-style constraint and the depth-boundary constraint.

[0016] As can be seen from the above technical solution, compared with the prior art, the present invention has at least the following advantages and positive effects: (1) The present invention uses a matting model based on the ViTAE architecture to extract the subject of the image, which can accurately process complex edges such as hair of the subject and ensure that the edges of the subject and background are separated smoothly and naturally.

[0017] (2) By constructing multi-dimensional composite constraints to control the generation of the background, the present invention can achieve automatic and accurate matching of the subject and the generated background in terms of pixel-level light and shadow, color and geometric structure, eliminate the visual sense of separation and achieve a high degree of naturalness in visual fusion.

[0018] (3) The present invention inputs the preprocessed reference scene map into the multimodal large model to generate prompt words, and dynamically generates a customized background that matches the subject based on the prompt words and multidimensional composite constraints. It can get rid of the dependence on the preset background template, flexibly and automatically generate backgrounds that meet actual needs, and greatly reduce time and labor costs. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below only involve some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 The flowchart illustrates a method for redrawing the background of a commodity image based on multidimensional composite constraints, provided by an embodiment of the present invention. Figure 2 A flowchart of a method for generating a main image containing a transparency channel, provided by an embodiment of the present invention, is shown. Figure 3 A flowchart of a method for locating a region of interest provided by an embodiment of the present invention is shown; Figure 4 This diagram illustrates a method for constructing multidimensional composite constraints and generating a background image according to an embodiment of the present invention. Figure 5 The diagram shows a schematic representation of a product image background redrawing system based on multidimensional composite constraints provided in an embodiment of the present invention. in, Figure 5 The annotations in the accompanying drawings are explained as follows: 500 - A product image background redrawing system based on multidimensional composite constraints; 501 - Image matting module; 502 - Preprocessing module; 503 - Prompt word generation module; 504 - Composite constraint construction module; 505 - Background image generation module; 506 - Product image synthesis module. Detailed Implementation

[0021] To more clearly explain the purpose, technical solutions, and advantages of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein. On the contrary, these embodiments are provided so that the present invention will be more comprehensive and complete, and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0022] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the invention.

[0023] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0024] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0025] The present invention will now be described in detail with reference to specific embodiments: Example 1

[0026] like Figure 1 As shown in the figure, this embodiment of the invention provides a method for redrawing the background of a product image based on multidimensional composite constraints. The specific steps of this method are as follows: S1: Extract the main subject of the original image using a matting model and generate a main subject image containing the alpha channel; Preferably, in this embodiment, the matting model is based on an encoder-decoder model built on the ViTAE architecture, and its core components include: The encoder, serving as the backbone of the model's feature extraction, incorporates a global self-attention module, a local convolutional module, and a multi-scale feature fusion module. The global self-attention module, based on the Transformer architecture, captures long-range dependencies in the image, extracting the global contour and semantic features of the subject. The local convolutional module, composed of multiple stacked convolutional layers, runs in parallel with the global self-attention module, capturing local details such as image edges and textures. The multi-scale feature fusion module aligns and fuses features from the two parallel modules, outputting multi-scale fused features that combine global semantics with local details. Decoder: Composed of a lightweight convolutional neural network containing multiple upsampling and convolutional layers; its function is to receive the fused features output by the encoder, gradually restore the spatial resolution, and finally predict and generate a high-precision alpha mask pixel by pixel.

[0027] Specifically, in this embodiment, the main subject of the original image is first extracted using a matting model to generate a main subject image containing alpha channels; such as Figure 2 As shown, the specific steps include the following: S201: Input the original image into the depth map model, extract the first depth map features, and based on the RGB information of the original image and the first depth map features, locate and obtain the region of interest image containing the main body of the original image; Specifically, such as Figure 3 As shown, locating the region of interest in an image includes the following steps: S301: The original image is processed by the depth map model to extract the first depth map features, and image processing is performed based on the first depth map features to obtain an initial foreground depth mask; wherein, the image processing includes filtering, thresholding, and morphological processing of the depth map. S302: The RGB information of the original image is concatenated with the initial foreground depth mask to form multimodal input data, which is then input into the matting model to output a visual feature map containing high-level semantic information of the subject. S303: Threshold or lightweight clustering is performed on the visual feature map to obtain a coarse localization mask for the main semantic region; S304: Perform a logical AND operation between the initial foreground depth mask and the coarse positioning mask to obtain a fused mask; wherein, the fused mask is a precise binary mask of the main body region; S305: Based on the fusion mask, crop out the corresponding region from the original image and extend it outward by a preset boundary margin to output the region of interest image.

[0028] S202: Input the image of the region of interest into the matting model, extract and fuse global and local features, and predict and generate a high-precision alpha mask; Further, after obtaining the region of interest through step S201, the region of interest is input into the matting model. The global self-attention module of the encoder in the matting model extracts the global features of the original image subject; at the same time, the local features of the original image subject are extracted through the parallel local convolution module; then, the global features and local features are input together into the multi-scale feature fusion module of the encoder in the matting model, and the global features and local features are fused by the multi-scale feature fusion module to obtain the fused features; finally, the fused features are input into the decoder in the matting model, and the decoder predicts and generates a high-precision alpha mask based on the fused features.

[0029] In this embodiment, the global features capture the outline of the main body, and the local features capture the edge texture details, which can effectively solve the problem of unclear recognition of the edge of the main body in complex backgrounds.

[0030] S203: Perform morphological dilation on the high-precision Alpha mask to obtain an optimized Alpha mask. Furthermore, after obtaining the high-precision alpha mask, the high-precision alpha mask is subjected to morphological dilation processing; specifically including the following steps: Define a circular structuring element with a preset radius R; Create a blank image with the same size as the high-precision alpha mask; For each pixel position (i, j) in the high-precision alpha mask, align the origin of the circular structuring element to the pixel position, and calculate the maximum value of the alpha value of all pixels within the coverage area of ​​the circular structuring element, Max(i, j). The maximum value Max(i, j) is assigned to the pixel at the same coordinates (i, j) in the blank image; Once the calculation and assignment of each pixel position (i, j) in the high-precision alpha mask are completed, the optimized alpha mask is obtained.

[0031] The structuring element is typically a small binary matrix (e.g., 3x3, 5x5), with its center point called the origin. Points with a value of 1 define the shape of the structuring element (e.g., circle, cross, rectangle). In this embodiment, the structuring element is defined as a circle, and the edge of the alpha mask is expanded outward according to a preset radius to ensure coverage of all semi-transparent pixels and fine hair areas. This ensures that the generated transparent channel has a complete and jagged edge, laying the foundation for high-quality compositing in the future.

[0032] S204: The optimized Alpha mask is used as a transparency channel and synthesized with the corresponding main body region in the original image to generate a main body image containing the transparency channel; S2: Preprocess the reference scene graph to obtain the optimized scene graph; Furthermore, a reference scene image is selected according to actual needs, and preprocessed to eliminate background interference; specifically, the preprocessing process is as follows: Identify interfering elements (such as text, watermarks, etc.) in the reference scene image and generate corresponding interference masks; input the reference scene image and the interference mask into the LaMa model, and perform context-aware texture filling and pixel repair through the fast Fourier convolution of the LaMa model to obtain the optimized scene image.

[0033] S3: Input the optimized scene graph into the multimodal large model to generate prompt words; Furthermore, after preprocessing the reference scene image, a natural language description of the scene image is generated using a multimodal large model, which serves as prompts for the subsequent model to generate a background image that meets the requirements. Specifically, generating prompts includes the following steps: The optimized scene graph is input into the multimodal large model, and the deep visual features of the optimized scene graph are extracted by the visual encoder of the multimodal large model. The deep visual features are mapped and aligned to the shared semantic space of the multimodal large model to obtain aligned features; wherein, the shared semantic space is learned by pre-training the multimodal large model; Based on the alignment features, a natural language description of the optimized scene graph is generated through the autoregressive generation of the multimodal large model, and the natural language description is used as a prompt word.

[0034] In this embodiment, the multimodal large model is preferably the Florence-2 model, but it can also be a model such as LLaVA (Large Language and Vision Assistant) or Qwen-VL (Qwen-Vision Language). The specific model can be adjusted according to actual needs, and this embodiment does not impose any restrictions.

[0035] S4: Input the optimized scene map into the depth map model, IPAdapter model and BrushNet model respectively to construct the multi-dimensional composite constraint conditions for generating the background using Stable Diffusion; S5: Input the prompt word and the multidimensional composite constraint together into the Stable Diffusion model, and generate a background image that matches the main body of the original image through the Stable Diffusion model; Furthermore, a multidimensional composite constraint condition controlling background generation is constructed, and based on this constraint condition, a background image matching the subject of the original image is generated using a Stable Diffusion model; wherein, the Stable Diffusion model is pre-tuned on a commercial e-commerce graph dataset. Specifically, as shown... Figure 4 As shown, constructing multidimensional composite constraints and generating a background image includes the following steps: S401: Input the optimized scene map into the depth map model, and extract the second depth map features of the optimized scene map as spatial structure constraints; S402: Input the optimized scene graph into the IPAdapter model, extract the visual style features of the optimized scene graph, and use them as semantic style constraints; S403: Input the optimized scene map and the high-precision alpha mask into the BrushNet model to extract boundary fusion features as pixel-level boundary constraints. S404: Based on the second depth map features, the visual style features, and the boundary fusion features, construct a multi-dimensional composite constraint condition for the Stable Diffusion background generation; S405: Input the prompt words and the multidimensional composite constraints into the Stable Diffusion model; S406: The prompt words are encoded into text vectors by a text encoder using a Stable Diffusion model, and the text vectors are fused with the visual style features in the multidimensional composite constraints through a cross-attention mechanism to obtain text-style condition constraints; S407: The second depth map feature and the boundary fusion feature in the multidimensional composite constraint condition are together entered into the backbone network of the Stable Diffusion model to generate depth-boundary condition constraints; S407: Under the combined constraints of text-style constraints and depth-boundary constraints, generate a background image that matches the subject of the original image using the StableDiffusion model.

[0036] S6: Combine the main image and the background image to output the final product image.

[0037] Furthermore, after generating a background image that matches the main subject of the original image, the main subject image containing the transparency channel generated in step S1 is combined with the background image to obtain and output the final product image.

[0038] The product image background redrawing method described in this embodiment uses a ViTAE-based image matting model to extract the main subject of the image. It can accurately handle complex edges such as hair on the main subject, ensuring that the edges separating the main subject from the background are smooth and natural. By seamlessly erasing interfering elements such as text and watermarks in the reference image, it can avoid the negative impact of inferior reference images on the style transfer of the subsequent generated model. By generating prompt words through a multimodal large model, it constructs multidimensional composite constraints to control the generation of the background. This not only eliminates the dependence on preset background templates and flexibly and automatically generates backgrounds that meet actual needs, greatly reducing time and labor costs, but also enables automatic and accurate matching of the main subject and the generated background in terms of pixel-level light and shadow, color and geometric structure, eliminating the sense of visual disjointedness and achieving a high degree of natural visual fusion. Example 2

[0039] like Figure 5As shown, this embodiment of the invention provides a product image background redrawing system 500 based on multi-dimensional composite constraints. The system includes: a cutout module 501, a preprocessing module 502, a prompt word generation module 503, a composite constraint construction module 504, a background image generation module 505, and a product image compositing module 506; wherein: The image matting module 501 is used to extract the main subject of the original image through the image matting model and generate a main subject image containing the transparency channel; Preferably, in this embodiment, the matting model in the matting module 501 is an encoder-decoder model built based on the ViTAE architecture; wherein, the encoder includes a global self-attention module, a local convolution module, and a multi-scale feature fusion module, and the global self-attention module and the local convolution module are set in parallel. The generation of the main image containing the alpha channel specifically includes the following steps: The original image is input into the depth map model, the first depth map feature is extracted, and the region of interest containing the main body of the original image is located and obtained based on the RGB information of the original image and the first depth map feature. The region of interest image is input into the matting model, global and local features are extracted and fused, and a high-precision alpha mask is generated. The high-precision Alpha mask is subjected to morphological dilation to obtain an optimized Alpha mask. The optimized alpha mask is used as a transparency channel and synthesized with the corresponding main area in the original image to generate a main image containing the transparency channel.

[0040] Preprocessing module 502 is used to preprocess the reference scene image to obtain an optimized scene image; In this embodiment, the preprocessing is performed using the Fast Fourier Convolution technique in the LaMa model, specifically as follows: Identify interfering elements (such as text or watermarks) in the reference scene image and generate corresponding interference masks; The reference scene image and the interference mask are input into the LaMa model, and context-aware texture filling and pixel repair are performed through the fast Fourier convolution of the LaMa model to obtain the optimized scene image.

[0041] The prompt word generation module 503 is used to input the optimized scene graph into the multimodal large model and generate prompt words; Furthermore, after preprocessing the reference scene image, it is input into a multimodal large model to generate a natural language description of the reference scene image, which serves as prompts to guide the model in generating a background image that meets the requirements. Specifically, generating prompts includes the following steps: The optimized scene graph is input into the multimodal large model, and the deep visual features of the optimized scene graph are extracted by the visual encoder of the multimodal large model. The deep visual features are mapped and aligned to the shared semantic space of the multimodal large model to obtain aligned features; wherein, the shared semantic space is learned by pre-training the multimodal large model; Based on the alignment features, a natural language description of the optimized scene graph is generated through the autoregressive generation of the multimodal large model, and the natural language description is used as a prompt word.

[0042] The composite constraint construction module 504 is used to input the optimized scene map into the depth map model, IPAdapter model and BrushNet model respectively to construct the multi-dimensional composite constraint conditions for the Stable Diffusion background generation. Furthermore, the optimized scene graph is processed using a depth map model, an IPAdapter model, and a BrushNet model respectively to construct multi-dimensional composite constraints, specifically including: The optimized scene map is input into the depth map model, and the second depth map features of the optimized scene map are extracted as spatial structure constraints. The optimized scene graph is input into the IPAdapter model, and the visual style features of the optimized scene graph are extracted as semantic style constraints. The optimized scene map and the high-precision alpha mask are input into the BrushNet model to extract boundary fusion features as pixel-level boundary constraints. Based on the second depth map features, the visual style features, and the boundary fusion features, a multidimensional composite constraint condition for generating the background using StableDiffusion is constructed.

[0043] Background image generation module 505 is used to input the prompt word and the multidimensional composite constraint conditions into the Stable Diffusion model, and generate a background image that matches the main body of the original image through the Stable Diffusion model; Furthermore, when the prompt word and the multidimensional composite constraint are input into the Stable Diffusion model, the text encoder of the Stable Diffusion model first encodes the prompt word into a text vector, and then fuses the text vector with the visual style features in the multidimensional composite constraint through a cross-attention mechanism to obtain the text-style constraint. Furthermore, the second depth map feature and boundary fusion feature in the multidimensional composite constraint will be input into the backbone network of the Stable Diffusion model to generate the depth-boundary constraint, so that the Stable Diffusion model generates a background image that matches the subject of the original image under the joint constraints of the text-style constraint and the depth-boundary constraint.

[0044] The product image synthesis module 506 is used to synthesize the main image and the background image to output the final product image.

[0045] Furthermore, after generating a background image that matches the main subject of the original image, the main subject image containing the transparency channel generated in the cutout module 501 is combined with the background image to obtain the final product image, thereby completing the background redrawing of the original product image.

[0046] In this embodiment, a subject image containing an alpha channel is first generated based on the matting module 501. Then, the optimized scene image obtained from the preprocessing module 502 is input into the multimodal large model, as well as into the depth map model, IPAdapter model, and BrushNet model. On the one hand, prompt words are automatically generated, and on the other hand, multidimensional composite constraints are constructed. Then, the generated prompt words and multidimensional composite constraints are input into the background image generation module 505. The background image generation module 505 has a built-in pre-trained Stable Diffusion model. Under the prompt words and multidimensional composite constraints, it can control the Stable Diffusion model to generate a background image that matches the subject of the original image. This eliminates the dependence on preset background templates, greatly reduces time and manual costs, and achieves automatic and accurate matching of the subject and the generated background in terms of pixel-level light and shadow, color, and geometric structure, eliminating visual disjointedness and achieving a high degree of natural visual fusion.

[0047] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims. It should be understood that the invention is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for redrawing the background of a commodity image based on multidimensional composite constraints, characterized in that, Includes the following steps: The main subject of the original image is extracted using a matting model, and a main subject image containing the alpha channel is generated. The reference scene image is preprocessed to obtain the optimized scene image; The optimized scene graph is input into a multimodal large model to generate prompt words; The optimized scene graph is input into the depth map model, IPAdapter model and BrushNet model respectively to construct the multi-dimensional composite constraints for generating the background using Stable Diffusion. The prompt words and the multidimensional composite constraints are input into the Stable Diffusion model, and the Stable Diffusion model generates a background image that matches the subject of the original image. The main image and the background image are combined to output the final product image.

2. The method for redrawing the background of a product image according to claim 1, characterized in that, The matting model is an encoder-decoder model built on the ViTAE architecture; wherein, the encoder of the matting model includes a global self-attention module, a local convolution module and a multi-scale feature fusion module, and the global self-attention module and the local convolution module are set in parallel.

3. The method for redrawing the background of a product image according to claim 1, characterized in that, The generation of the main image containing the alpha channel specifically involves: The original image is input into the depth map model, the first depth map feature is extracted, and the region of interest containing the main body of the original image is located and obtained based on the RGB information of the original image and the first depth map feature. The region of interest image is input into the matting model, global and local features are extracted and fused, and a high-precision alpha mask is generated. The high-precision Alpha mask is subjected to morphological dilation to obtain an optimized Alpha mask. The optimized alpha mask is used as a transparency channel and synthesized with the corresponding main area in the original image to generate a main image containing the transparency channel.

4. The method for redrawing the background of a product image according to claim 3, characterized in that, The process of locating and obtaining the region of interest image containing the main body of the original image specifically involves: The original image is processed by the depth map model to extract the first depth map features, and image processing is performed based on the first depth map features to obtain an initial foreground depth mask. The RGB information of the original image is concatenated with the initial foreground depth mask to form multimodal input data, which is then input into the matting model to output a visual feature map containing high-level semantic information of the subject. Thresholding or lightweight clustering is performed on the visual feature map to obtain a coarse localization mask for the main semantic region; Perform a logical AND operation between the initial foreground depth mask and the coarse positioning mask to obtain the fused mask; Based on the fusion mask, the corresponding region is cropped from the original image and extended outward by a preset boundary margin to output the region of interest image.

5. The method for redrawing the background of a product image according to claim 3, characterized in that, The prediction generates a high-precision alpha mask, specifically as follows: The region of interest image is input into the matting model; Global features of the main subject in the original image are extracted using the global self-attention module of the encoder in the image matting model. Local features of the main subject in the original image are extracted using the local convolution module of the encoder in the image matting model. The global and local features are fused by the multi-scale feature fusion module of the encoder in the matting model to obtain fused features; The decoder in the matting model predicts and generates a high-precision alpha mask based on the fused features.

6. The method for redrawing the background of a product image according to claim 3, characterized in that, The morphological dilation process performed on the high-precision alpha mask specifically involves: Define a circular structuring element with a preset radius R; Create a blank image with the same size as the high-precision alpha mask; For each pixel position (i, j) in the high-precision alpha mask, align the origin of the circular structuring element to the pixel position, and calculate the maximum value of the alpha value of all pixels within the coverage area of ​​the circular structuring element, Max(i, j). The maximum value Max(i, j) is assigned to the pixel at the same coordinates (i, j) in the blank image; Once the calculation and assignment of each pixel position (i, j) in the high-precision alpha mask are completed, the optimized alpha mask is obtained.

7. The method for redrawing the background of a product image according to claim 1, characterized in that, The preprocessing of the reference scene image specifically includes: Identify interfering elements in the reference scene diagram and generate corresponding interference masks; The reference scene image and the interference mask are input into the LaMa model. The LaMa model's fast Fourier convolution is used to perform context-aware texture filling and pixel repair to obtain the optimized scene image.

8. The method for redrawing the background of a product image according to claim 1, characterized in that, The generated prompt words are specifically: The optimized scene graph is input into a multimodal large model, and the deep visual features of the optimized scene graph are extracted by the visual encoder of the multimodal large model. The deep visual features are mapped and aligned to the shared semantic space of the multimodal large model to obtain aligned features; wherein, the shared semantic space is learned by pre-training the multimodal large model; Based on the alignment features, a natural language description of the optimized scene graph is generated through the autoregressive generation of the multimodal large model, and the natural language description is used as a prompt word.

9. The method for redrawing the background of a product image according to claim 1, characterized in that, The multidimensional composite constraint conditions for constructing the StableDiffusion generation background are specifically as follows: The optimized scene map is input into the depth map model, and the second depth map features of the optimized scene map are extracted as spatial structure constraints. The optimized scene graph is input into the IPAdapter model, and the visual style features of the optimized scene graph are extracted as semantic style constraints. The optimized scene map and the high-precision alpha mask are input into the BrushNet model to extract boundary fusion features as pixel-level boundary constraints. Based on the second depth map features, the visual style features, and the boundary fusion features, a multidimensional composite constraint condition for generating the background using StableDiffusion is constructed.

10. A product image background redrawing system based on multidimensional composite constraints, characterized in that, include: The image matting module is used to extract the main subject of the original image using a matting model and generate a main subject image that includes the alpha channel. The preprocessing module is used to preprocess the reference scene image to obtain an optimized scene image; The prompt word generation module is used to input the optimized scene graph into the multimodal large model and generate prompt words; The composite constraint construction module is used to input the optimized scene graph into the depth map model, IPAdapter model and BrushNet model respectively to construct multi-dimensional composite constraint conditions for the Stable Diffusion generated background; The background image generation module is used to input the prompt words and the multidimensional composite constraints into the StableDiffusion model, and generate a background image that matches the main body of the original image through the StableDiffusion model. The product image compositing module is used to combine the main image and the background image to output the final product image.

Images