Product image editing methods, equipment, and storage media

Abstract

This application discloses a method, device, and storage medium for editing product images, relating to the field of image processing technology. The method, based on received image editing instructions, determines the element to be edited and its corresponding target visual attributes; determines the target editing region in the image based on the semantic similarity between the element to be edited and other image elements in the image to be edited; maps the visual features of the target editing region to a latent code space to obtain latent code features; determines the KL divergence constraint values ​​for each feature dimension corresponding to the latent code features based on the target visual attributes; and edits the target editing region based on the target visual attributes and the KL divergence constraint values, achieving local modification while maintaining the overall shape, thus improving the image editing effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to methods, devices and storage media for editing product images. Background Technology

[0002] In e-commerce product image creation, using artificial intelligence (AI) to generate and modify product images is a common image editing task. This includes adjusting material texture and modifying color schemes. Currently, AI-generated product image editing methods primarily use diffusion models or generative adversarial networks to redraw the entire image. While this can achieve style transfer or material replacement, global redrawing resamples every pixel. Pixels at the boundary between edited and non-edited areas may become blurred or distorted due to resampling, lacking precise control over local editing areas. For example, modifying the material of a product locally, such as changing a backpack from canvas to leather, may alter the product's geometric contours, such as distorting the backpack's shape or shifting the zipper position. Therefore, current AI-generated image editing methods have poor local editing performance.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a method for editing product images, aiming to solve the technical problem of how to improve the local editing effect of images.

[0005] To achieve the above objectives, this application proposes a method for editing product images, comprising the following steps:

[0006] Based on the received image editing instructions, determine the element to be edited and the target visual attributes corresponding to the element to be edited;

[0007] The target editing region in the image to be edited is determined based on the semantic similarity between the element to be edited and each image element in the image to be edited.

[0008] The visual features of the target editing region are mapped to the latent code space to obtain the latent code features;

[0009] Based on the target visual attributes, determine the KL divergence constraint values ​​for each feature dimension corresponding to the latent code features;

[0010] The target editing region is edited based on the target visual attributes and the KL divergence constraint value.

[0011] In one embodiment, the step of determining the target editing region in the image to be edited based on the semantic similarity between the element to be edited and each image element of the image to be edited includes:

[0012] Based on the semantic similarity between the element to be edited and each image element of the image to be edited, a semantic association graph corresponding to the image to be edited is determined;

[0013] The visual saliency map and the semantic association map corresponding to the image to be edited are weighted and fused to obtain the heat map corresponding to the image to be edited.

[0014] Based on the heatmap, the target editing area is determined in the image to be edited.

[0015] In one embodiment, the step of determining the target editing region in the image to be edited based on the heatmap includes:

[0016] Identify pixel regions in the heat map whose heat values ​​are greater than a preset heat value, and generate a mask matrix within the pixel regions.

[0017] Based on the first preset rectangular kernel, determine the maximum thermal value of each pixel in the mask matrix within the neighborhood corresponding to the first preset rectangular kernel, and use the maximum thermal value as the first target thermal value of the pixel;

[0018] Based on the second preset rectangular kernel, the minimum first target thermal value of each pixel in the mask matrix within the neighborhood corresponding to the second preset rectangular kernel is determined, and the minimum first target thermal value is used as the second target thermal value of the pixel, wherein the second preset rectangular kernel is greater than the first preset rectangular kernel;

[0019] The target pixel region in the heat map whose second target heat value is greater than the preset heat value is determined, and the target editing region is determined based on the target pixel region.

[0020] In one embodiment, the step of determining the target editing region based on the target pixel region includes:

[0021] A first noise is added to the target pixel region to obtain the original noise map;

[0022] The gradient vector corresponding to the heatmap is determined based on the preset operator kernel;

[0023] When denoising the original noise map, the second noise corresponding to the current denoising step is determined based on the difference between the map to be processed and the heat map corresponding to the current denoising step, the target first noise corresponding to the current denoising step, and the gradient vector, and the map to be processed is denoised based on the second noise.

[0024] The target pixel region obtained after denoising is used as the target editing region.

[0025] In one embodiment, the step of determining the KL divergence constraint values ​​of each feature dimension corresponding to the latent code feature based on the target visual attributes includes:

[0026] Determine the target feature dimensions corresponding to the target visual attributes;

[0027] Based on the type of the target visual attribute, determine the KL divergence constraint value of the target feature dimension;

[0028] Set the KL divergence constraint values ​​of all other feature dimensions besides the target feature dimension to the minimum constraint value.

[0029] In one embodiment, the step of editing the target editing region based on the target visual attributes and the KL divergence constraint value includes:

[0030] Determine the attribute difference between the attribute value corresponding to the target feature dimension and the attribute value corresponding to the target visual attribute;

[0031] The image generation model modifies the attribute values ​​corresponding to the target feature dimensions until the attribute difference is less than or equal to a preset minimum difference, and the KL divergence value corresponding to each dimension of the latent code feature does not exceed the KL divergence constraint value corresponding to each dimension feature, then the edited image is output.

[0032] In one embodiment, after the step of editing the target editing region according to the target visual attributes and the KL divergence constraint value, the method further includes:

[0033] Obtain the illumination transfer equation of the image to be edited, wherein the illumination transfer equation represents the image pixel value as an integral relationship between the preset light source brightness and the bidirectional reflectance distribution function;

[0034] If the element to be edited is a background element, the pixel value of the target editing area after editing is determined according to the light transmission equation.

[0035] In one embodiment, after the step of obtaining the illumination transport equation of the image to be edited, the method further includes:

[0036] If the target visual attribute of the element to be edited includes a material attribute, then determine the bidirectional reflectance distribution function corresponding to the material attribute;

[0037] The pixel values ​​of the pixel region corresponding to the element to be edited are determined based on the bidirectional reflection distribution function corresponding to the material properties.

[0038] In addition, to achieve the above objectives, this application also proposes a product image editing device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the product image editing method described above.

[0039] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the product image editing method described above.

[0040] This application provides a method for editing product images. Based on received image editing instructions, the method determines the element to be edited and its corresponding target visual attributes; based on the semantic similarity between the element to be edited and other image elements in the image to be edited, it determines the target editing region in the image; it maps the visual features of the target editing region to a latent code space to obtain latent code features; based on the target visual attributes, it determines the KL divergence constraint values ​​for each feature dimension corresponding to the latent code features; and based on the target visual attributes and the KL divergence constraint values, it edits the target editing region.

[0041] The above method avoids geometric distortion caused by overall redrawing by accurately locating the target editing area, and limits the scope of attribute modification by preserving the original structural information through latent code mapping and KL divergence constraint values. This allows for local modification while maintaining the overall shape, thus improving the image editing effect. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A flowchart illustrating the method for editing product images in this application (Example 1);

[0045] Figure 2 A flowchart illustrating the second embodiment of the method for editing product images in this application;

[0046] Figure 3 A flowchart illustrating the method for editing product images in this application, as provided in Embodiment 3;

[0047] Figure 4 A flowchart illustrating the fourth embodiment of the method for editing product images in this application;

[0048] Figure 5 This is a schematic diagram of the overall process of editing product images in the embodiments of this application;

[0049] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the product image editing method in this application embodiment.

[0050] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not intended to limit this application.

[0052] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments. It should be noted that all actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection laws and regulations of the country where the application is located, and with authorization from the owner of the corresponding device.

[0053] In e-commerce product image creation, using AI to generate and modify product images is a common image editing task, such as adjusting material texture and modifying color schemes. Currently, AI-generated product image editing methods mainly use diffusion models or generative adversarial networks to redraw the entire image. While this can achieve style transfer or material replacement, it lacks precise control over local areas. For example, when modifying the material of a product, such as changing a backpack from canvas to leather, it may alter the geometric outline of the product, such as distorting the backpack shape or shifting the zipper position. Therefore, current AI-generated product image editing methods have poor local editing effects.

[0054] In view of the above problems, this application proposes a method for editing product images. Based on received image editing instructions, the method determines the element to be edited and its corresponding target visual attributes; based on the semantic similarity between the element to be edited and each image element in the image to be edited, it determines the target editing region in the image to be edited; it maps the visual features of the target editing region to a latent code space to obtain latent code features; based on the target visual attributes, it determines the KL divergence constraint values ​​for each feature dimension corresponding to the latent code features; and based on the target visual attributes and the KL divergence constraint values, it edits the target editing region.

[0055] The above method avoids geometric distortion caused by overall redrawing by accurately locating the target editing area, and limits the scope of attribute modification by preserving the original structural information through latent code mapping and KL divergence constraint values. This allows for local modification while maintaining the overall shape, thus improving the image editing effect.

[0056] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or product image editing device capable of performing the above functions. The following description uses a product image editing device as an example to illustrate this embodiment and the subsequent embodiments.

[0057] Based on this, the first embodiment of this application provides a method for editing product images, referring to... Figure 1 In this embodiment, the method for editing the product image includes steps S10 to S50:

[0058] Step S10: Based on the received image editing instructions, determine the element to be edited and the target visual attributes corresponding to the element to be edited.

[0059] It should be noted that image editing instructions are user-inputted commands to guide local image editing, including one or more combinations of text descriptions, voice commands, sample images, or interactive operations. The element to be edited is the specific object or area in the image that needs modification, such as the main body of a product, a partial component, or the background. The target visual attribute refers to the visual features that need adjustment, including material properties, color properties, lighting properties, or geometric properties.

[0060] Alternatively, a multimodal input channel may be provided through the graphical user interface of the product image editing device, including one or more combinations of text input boxes, voice command modules, and sample image upload modules.

[0061] The system includes several modules: a text input box displaying prompt text indicating the element to be modified and its target visual attributes; a voice command module converting user voice commands into structured text containing the element to be modified and its target visual attributes; and an image upload module receiving user-uploaded reference images, generating pixel-level label maps using a semantic segmentation model, identifying image elements and their corresponding visual attributes, performing feature matching between the reference and editable images, identifying elements in the editable image that differ from the aforementioned elements, designating them as editable elements, and using the visual attributes of these elements in the reference image as target visual attributes.

[0062] Optionally, the product image editing device can directly display the image to be edited and set interactive tools on the image to be edited, including a mask drawing tool and an attribute selection panel.

[0063] The mask drawing tool can include a brush or a rectangle. Users can manually mark the target area as the element to be edited in the image using a brush or rectangle. In response to this marking action, the system displays an attribute selection panel, which shows a drop-down menu or slider for the visual attributes corresponding to the target area. In response to the selection operation received by the attribute selection panel, the target visual attributes of the element to be edited are determined.

[0064] Step S20: Determine the target editing region in the image to be edited based on the semantic similarity between the element to be edited and each image element of the image to be edited.

[0065] Alternatively, image elements in the image to be edited can be identified through edge detection and color clustering.

[0066] For example, the image to be edited is converted into an RGB image. Contour information is extracted from the RGB image using an edge detection algorithm to generate an edge binary map. Color clustering is then performed on the RGB image, dividing each pixel of the image to be edited into a predetermined number of regions based on color similarity. Each pixel is labeled with its corresponding color cluster number, generating a color label map. Based on the edge binary map and the color label map, the image elements in the image to be edited are determined.

[0067] Optionally, the step of determining the image elements in the image to be edited based on the edge binary map and the color label map includes: marking regions with pixel values ​​greater than zero in the edge binary map as edge regions; traversing the color category regions corresponding to each color cluster number in the color label map; and calculating the intersection of each color category region with the edge regions. If the proportion of non-edge region pixels in a color category region is greater than a preset proportion threshold, then the color category region is determined as an expandable region. Adjacent expandable regions are merged, and through a preset filling algorithm, continuous non-edge regions of the same color are merged into a single image element to obtain a merged color-edge fusion map. Each image element in the merged color-edge fusion map corresponds to a unique label.

[0068] Optionally, pre-trained semantic segmentation models can be used to identify image elements in the image to be edited.

[0069] For example, a pre-trained semantic segmentation model is loaded, the image to be edited is scaled to the model's input size, and input into the model for segmentation, generating a pixel-level label map. The label map annotates the image element corresponding to each pixel in the image to be edited and the visual attributes corresponding to that image element. The label map is then mapped to the size of the image to be edited to identify the image elements in the image based on the labels.

[0070] Preferably, taking the HRNet-W48 model as an example, HRNet-W48 employs a multi-resolution parallel convolution branch, which can maintain high-resolution feature maps while fusing low-resolution semantic information to better identify image elements in the image to be edited. For example, the scaled image to be edited is input into the HRNet-W48 model. The HRNet-W48 model extracts at least one resolution feature from the image to be edited, obtaining at least one resolution feature. Then, the extracted resolution feature maps are upsampled to the same size through bilinear interpolation and stitched together to generate a fused feature map. This fused feature map is then convolved to reduce its pixel channels to a preset number of categories, generating the final feature map. After obtaining the feature map, the Softmax function is applied to the channel value of each pixel in the feature map to calculate the category probability corresponding to each pixel. The category index with the highest category probability is taken as the pixel label of each pixel to indicate the image element in the image to be edited.

[0071] Optionally, step S20 includes steps S21 to S23:

[0072] Step S21: Determine the semantic association graph corresponding to the image to be edited based on the semantic similarity between the element to be edited and each image element of the image to be edited.

[0073] Step S22: The visual saliency map and the semantic association map corresponding to the image to be edited are weighted and fused to obtain the heat map corresponding to the image to be edited.

[0074] Step S23: Determine the target editing area in the image to be edited based on the heat map.

[0075] For example, the pixel labels corresponding to each image element are mapped to a unified vocabulary, or a pre-trained Word2Vec model is used to convert the pixel labels or the words corresponding to the pixel labels into word vectors. The similarity between the word vectors corresponding to the image elements and the word vectors corresponding to the elements to be edited is calculated as the semantic association between the elements to be edited and each image element in the image to be edited, thus obtaining the semantic association graph corresponding to the image to be edited.

[0076] In one feasible implementation, the region containing the image element with the highest semantic relevance can be directly selected as the target editing region in the semantic association graph.

[0077] A visual saliency map is a grayscale image generated by quantifying the visual attractiveness of different regions in an image using algorithms. Higher pixel values ​​in the grayscale map indicate that the region is more visually appealing. By fusing visual saliency maps, the target editing area can be made more consistent with human visual habits, avoiding image distortion caused by modifying insignificant areas, and balancing semantic relevance with visual effect.

[0078] Optionally, for each pixel in the semantic association map, its semantic weight is determined based on the semantic association degree of its corresponding image element. Based on the semantic weight of each pixel, the semantic association degree and visual saliency of each pixel are weighted and fused to obtain a heatmap corresponding to the image to be edited. The area containing the image element with the highest heat value in the heatmap is then selected as the target editing area.

[0079] Optionally, the semantic relevance and visual saliency of each pixel can be weighted and fused using the following formula:

[0080]

[0081] Wherein, heat_map is a heatmap, α is the semantic weight, semantic_map is a semantic association map, and saliency_map is a visual saliency map.

[0082] Optionally, the area containing the image element with the highest heat value in the heatmap can be used as the target editing area.

[0083] Step S30: Map the visual features of the target editing region to the latent code space to obtain the latent code features.

[0084] It should be noted that latent code features are vector representations of the visual features of the target editing area compressed into a low-dimensional feature space by the encoder network. Each feature dimension corresponds to specific visual attributes such as material roughness, color distribution, and surface normal direction.

[0085] Understandably, the latent code mapping process encodes different visual attribute features of the target editing area into different feature dimensions, achieving decoupling between different visual attributes of the same image element. When you want to modify a certain visual attribute of an image element, you can achieve precise editing by directly adjusting the specific feature dimension of the latent code feature, while keeping the feature dimensions corresponding to other visual attributes unchanged. This avoids the problem of overall changes caused by modifying local features, which is common in traditional AI-generated product image editing methods.

[0086] Step S40: Determine the KL divergence constraint values ​​for each feature dimension corresponding to the latent code feature based on the target visual attributes.

[0087] It should be noted that KL divergence (KL divergence) is an indicator that quantifies the difference between two probability distributions. It measures the degree of deviation between the distributions before and after modification of latent code features; the smaller the value, the closer the distributions are. In this embodiment, the KL divergence constraint value of the feature dimension refers to the value that limits the KL divergence of the feature dimension. It can be a fixed value or two values ​​used to constrain the range of the KL divergence of that feature dimension, in order to control the difference between the edited latent code feature distribution and the original distribution.

[0088] Optionally, step S40 includes steps S41 to S43:

[0089] Step S41: Determine the target feature dimension corresponding to the target visual attribute.

[0090] For example, when mapping the visual features of the target editing region to latent code features in the latent code space, the encoder encodes the visual attributes corresponding to each pixel within the target editing region into various feature dimensions according to the latent code feature mapping table. Therefore, after determining the target visual attributes, the feature dimensions associated with the target visual attributes can be determined as the target feature dimensions based on the feature dimension-attribute mapping relationship within the encoder.

[0091] Step S42: Determine the KL divergence constraint value of the target feature dimension according to the type of the target visual attribute.

[0092] Understandably, different visual attributes have varying degrees of impact on images. For example, changing material roughness may not significantly affect the overall structure, but excessive color changes can lead to inconsistencies with the surrounding environment. Material attributes may involve more complex texture variations, requiring a certain degree of freedom, while overly loose constraints on color changes can result in excessive color differences. Therefore, divergence constraint values ​​can be pre-set according to the type of different visual attributes. Once the target feature dimension corresponding to the target visual attribute is determined, the KL divergence constraint value corresponding to the target feature dimension is determined based on the type of the target visual attribute.

[0093] Step S43: Set the KL divergence constraint values ​​of all other feature dimensions besides the target feature dimension to the minimum constraint values.

[0094] After determining the KL divergence constraint value corresponding to the target feature dimension, set the KL divergence constraint values ​​of all other feature dimensions to the minimum constraint value. This forces other feature dimensions to maintain a distribution close to the original distribution during the latent code feature adjustment process, ensuring that the editing operation only applies to the target feature dimension and improving the accuracy of local editing.

[0095] Step S50: Edit the target editing region according to the target visual attributes and the KL divergence constraint value.

[0096] Optionally, step S50 includes steps S51 to S52:

[0097] Step S51: Determine the attribute difference between the attribute value corresponding to the target feature dimension and the attribute value corresponding to the target visual attribute.

[0098] Step S52: Modify the attribute values ​​corresponding to the target feature dimensions using the image generation model until the attribute difference is less than or equal to the preset minimum difference, and the KL divergence values ​​corresponding to each dimension of the latent code feature do not exceed the KL divergence constraint values ​​corresponding to each dimension feature, then output the edited image.

[0099] For example, the attribute difference between the attribute values ​​corresponding to the target feature dimensions and the attribute values ​​corresponding to the target visual attributes, as well as the KL divergence constraint values ​​corresponding to each dimension of features in the target editing region, are determined to establish the loss function for the image generation model. The target editing region is then modified using the image generation model, and the optimized modifications are validated using the loss function.

[0100] For example, the latent code features corresponding to the target editing region determined in step S30 are used as input to the image generation model. At this time, the values ​​of each feature dimension in the latent code features have not yet been adjusted. Subsequently, the attribute values ​​corresponding to the elements to be edited in the image to be edited are obtained, and the difference between them and the target visual attributes is calculated to quantify the gap between the current editing step and the target editing effect. Next, a loss function is designed, which consists of two parts: one part is the target attribute error term, that is, the difference between the attribute value corresponding to the element to be edited and the target visual attribute, which is used to quantify the degree of modification completion; the other part is the KL constraint term, which is obtained by accumulating the portion of the KL divergence of each dimension feature that exceeds its corresponding KL divergence constraint value, and is used to quantify the degree of distribution deviation. The above target attribute error term and KL constraint term are weighted and summed through weight coefficients to form the loss function.

[0101] After determining the loss function, gradient descent optimization is initiated, and the gradient of the loss function with respect to each feature dimension is calculated. Preferably, the values ​​of the target feature dimensions are adjusted first to reduce the error of the target attribute. That is, the values ​​of the target feature dimensions are adjusted to reduce the difference between the attribute value corresponding to the element to be edited and the target visual attribute, while suppressing the gradient of non-target feature dimensions to keep their KL divergence below the minimum constraint value.

[0102] After each adjustment, the updated latent code features are input into the generation model to generate the edited image. The attribute values ​​corresponding to the elements to be edited in the edited image are then re-extracted, and the difference between these values ​​and the target visual attributes is verified to meet the preset minimum difference. If not, optimization continues. If, during optimization, the KL divergence of any feature dimension exceeds its corresponding KL divergence constraint value, the adjustment of that feature dimension is suppressed by increasing the weight of the KL constraint term or reducing the optimization step size. After multiple iterations of optimization, when the difference between the elements to be edited and the target visual attributes is less than or equal to the preset minimum difference, and the KL divergence of all feature dimensions meets the constraints, the optimization process terminates, and the final edited image is output. At this point, the visual attributes of the target editing region have been accurately modified, and the distribution of other regions remains consistent with the original image to be edited.

[0103] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S23 also includes steps S231 to S234:

[0104] Step S231: Determine the pixel region in the heat map whose heat value is greater than the preset heat value, and generate a mask matrix in the pixel region.

[0105] In this embodiment, the heat map value refers to the pixel value of each pixel in the heat map. All pixels in the heat map are traversed. If the pixel value of a traversed pixel is greater than a preset heat map value, the corresponding mask matrix position is set to 1; otherwise, it is set to 0.

[0106] Step S232: Based on the first preset rectangular kernel, determine the maximum thermal value of each pixel in the mask matrix within the neighborhood corresponding to the first preset rectangular kernel, and use the maximum thermal value as the first target thermal value of the pixel.

[0107] It should be noted that the first preset rectangular kernel refers to a small rectangular filter kernel, such as a 3×3 rectangular area. Its function is to filter the neighborhood of each pixel in the heatmap by applying the maximum value filter to connect adjacent pixels with higher heatmap values, thereby avoiding scattered areas caused by heatmap noise or incomplete edge detection and improving the continuity and integrity of the target editing area.

[0108] For example, firstly, the size of a first preset rectangular core is selected, which determines the neighborhood range corresponding to each pixel. Then, each pixel in the heatmap is traversed. For each pixel, the heat values ​​of all pixels within its corresponding first preset rectangular core neighborhood are calculated, and the maximum value is selected as the first target heat value for that pixel, generating a first target heat value matrix. Through these steps, areas that might otherwise be ignored due to individual pixel values ​​being lower than the preset heat value are preserved and expanded because higher values ​​exist in their neighborhoods, ultimately forming a more complete target editing area.

[0109] Step S233: Based on the second preset rectangular kernel, determine the minimum first target thermal value of each pixel in the mask matrix within the neighborhood corresponding to the second preset rectangular kernel, and use the minimum first target thermal value as the second target thermal value of the pixel, wherein the second preset rectangular kernel is greater than the first preset rectangular kernel.

[0110] The second preset rectangular kernel is a rectangular filter kernel with a larger size than the first preset rectangular kernel, such as a 5×5 rectangular region. Its function is to perform minimum value filtering on the neighborhood of each pixel in the first target thermal value matrix generated in step S2332.

[0111] Understandably, designing the second preset rectangular core to be larger than the first preset rectangular core enables secondary filtering of the expanded area. This avoids over-connection of areas that may be caused by small core expansion, and prevents area breakage that may be caused by large core filtering, thus achieving a balance between the integrity and accuracy of the target editing area.

[0112] For example, first, the size of the second preset rectangular kernel is selected. Then, each pixel in the first target heat value matrix is ​​traversed. For each pixel, the first target heat value of all pixels in the neighborhood of its corresponding second preset rectangular kernel is calculated, and the minimum value among them is selected as the second target heat value of that pixel. Through the above steps, the edges or noise parts with lower heat values ​​in the larger region that was originally connected due to the expansion of the small kernel will be replaced by smaller heat values ​​in their neighborhood, thereby preserving only the core region with a consistently high heat value.

[0113] Step S234: Determine the target pixel region in the heat map where the second target heat value is greater than the preset heat value, and determine the target editing region based on the target pixel region.

[0114] Optionally, step S234 includes steps S2341 to S2344:

[0115] Step S2341: Add first noise to the target pixel area to obtain the original noise map.

[0116] Understandably, the noise added here is the initial noise from the reverse generation process in the simulated diffusion model, serving as the basic input for subsequent denoising. By confining the noise to the target pixel region, it ensures that noise generation is strongly correlated with the target pixel region, avoiding computational redundancy caused by global noise.

[0117] Step S2342: Determine the gradient vector corresponding to the heat map according to the preset operator kernel.

[0118] It should be noted that the gradient vector reflects the direction and intensity of pixel value changes in the heatmap, pointing in the direction of the fastest increase in heat value, and the magnitude represents the rate of change. This step obtains spatial structure information of the heatmap, providing directional guidance for noise constraints in the subsequent denoising process.

[0119] Optionally, the operator kernel can be the Sobel operator, the Prewitt operator, or the Roberts operator. Taking the Sobel operator as an example, it obtains the gradient components of each pixel in the horizontal and vertical directions and then combines them into a gradient vector.

[0120] Step S2343: When denoising the original noise map, the second noise corresponding to the current denoising step is determined based on the difference between the image to be processed and the heat map corresponding to the current denoising step, the target first noise corresponding to the current denoising step, and the gradient vector, and the image to be processed is denoised based on the second noise.

[0121] During the denoising process, the difference between the image to be processed in the current denoising step and the original heatmap is used to quantify the gap between the current denoising effect and the target heatmap distribution. Simultaneously, a second noise is calculated by combining the target first noise and the gradient vector of the current step. Specifically, the adjustment direction of the second noise is determined by the gradient vector: if the gradient direction of a certain region points to a high-probability region of the heatmap, the noise amount in that direction is reduced; conversely, it is increased. Through this dynamic adjustment, the denoising process can be guided along the direction of high heatmap values, causing the mask edges to gradually shrink or expand towards high-probability regions, ensuring that subsequent latent code feature mapping and attribute editing only apply to the most relevant image regions.

[0122] Step S2344: The target pixel region obtained after denoising is used as the target editing region.

[0123] After multiple iterations of denoising, the noise is gradually constrained to high-probability regions in the heatmap, and the final denoising result is the accurate target editing region. At this point, the mask edges have been aligned with the highly active regions in the heatmap that are strongly correlated with the target editing region through gradient guidance and noise constraints, ensuring that subsequent latent code feature mapping and attribute editing only apply to the most relevant image regions.

[0124] In this embodiment, through the above two neighborhood processing steps, the region that is highly similar in semantics to the image editing command and is visually concentrated can be accurately located, eliminating noise interference and ensuring the accuracy of the target editing region.

[0125] Based on the above embodiments of this application, in the third embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 After step S50, the method for editing the product image further includes steps S60-S70:

[0126] Step S60: Obtain the illumination transfer equation of the image to be edited. The illumination transfer equation represents the image pixel values ​​as an integral relationship between the preset light source brightness and the bidirectional reflectance distribution function.

[0127] It should be noted that the light transport equation is a mathematical model in computer graphics that describes the interaction between light rays and objects. It is used to express the pixel value (I) of an image as an integral of the light source brightness (L), the bidirectional reflectance distribution function (BRDF, denoted as f), and geometric relationships. The expression can be found in the following formula:

[0128]

[0129] Where x is the pixel position, and These represent the incident light and the observation direction, respectively. Let be the angle between the incident light and the normal. Indicates the pixel position x and the direction of incident light. The brightness of the light source; Indicates the pixel position x and the viewing direction. The final brightness is the pixel value; Ω represents the upper hemisphere space, which covers all possible incident light directions.

[0130] For example, by using reverse rendering techniques such as Neural Radiation Field (NeRF) or Differentiable Rendering, light source parameters such as light source direction, light source intensity, and light source color, as well as material BRDF parameters such as diffuse reflection coefficient, specular reflection coefficient, and roughness, can be determined from the image to be edited.

[0131] Optionally, if the element to be edited is a background element, the lighting distribution of the background area usually involves ambient lighting or global illumination, and the lighting transfer equation of the background area can be decomposed as follows:

[0132]

[0133] In this context, `I_background` represents the light transfer equation for the background area, `L_env` is the ambient light intensity, and `f_env` is the ambient light reflection function of the background material; `L_direct` is the direct light intensity, and `f_direct` is the direct light reflection function. Direct light intensity refers to the brightness of an object's surface when direct light from a specific light source, such as the sun, a light bulb, or a flash, strikes it. Ambient light intensity refers to the sum of all indirect light in the surrounding environment, including reflected light from the sky, walls, the ground, or other objects. When the AI ​​generates the initial image to be edited, the user can control the AI ​​to generate an image with a specific lighting distribution through commands.

[0134] Step S70: If the element to be edited is a background element, determine the pixel value of the target editing area after editing according to the light transmission equation.

[0135] When the element to be edited is a background element, such as changing the background wall color or adjusting the background brightness, the original light transmission equation used when generating the image can be used to ensure that the edited background and foreground elements, such as people and objects, are kept in the same lighting, avoiding light banding or shadow misalignment. For example, if it is necessary to adjust the background brightness, such as to enhance the ambient light, the brightness can be adjusted by updating the light source parameter "L_env".

[0136] Based on the above embodiments of this application, in the fourth embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 After step S60, the method for editing the product image further includes steps S80-S90:

[0137] Step S80: If the target visual attribute of the element to be edited includes a material attribute, then determine the bidirectional reflectance distribution function corresponding to the material attribute.

[0138] It should be noted that the bidirectional reflectance distribution function (BRF) is a mathematical model describing the characteristics of light reflected from a material surface. It is used to correlate the brightness of outgoing light with the brightness of incoming light, where the brightness of outgoing light can be represented by pixel values. If the target visual attribute of the element to be edited includes material attributes, the BRF model corresponding to that material attribute can be directly selected, such as the Cook-Torrance model for metals and the Oren-Nayar model for rough surfaces. If the material is extracted from a reference image, the parameters of the corresponding BRF can be determined from the reference image using reverse rendering techniques such as differentiable path tracing.

[0139] The parameters of the bidirectional reflectance distribution function include at least a diffuse reflection term and a specular reflection term. The diffuse reflection term describes the material's ability to scatter ambient light and is typically related to the material's color and diffuse reflectance coefficient. The specular reflection term describes the material's ability to reflect direct light specularly and is typically related to roughness, Fresnel coefficient, and normal distribution function.

[0140] Step S90: Determine the pixel value of the pixel region corresponding to the element to be edited based on the bidirectional reflection distribution function corresponding to the material property.

[0141] The illumination transfer equation obtained in step S60 is combined with the bidirectional reflectance distribution function corresponding to the aforementioned material properties to calculate the pixel values ​​of the target editing area. This ensures that the edited material visually matches the real material while maintaining consistent pixel values ​​with the scene lighting conditions in the original image to be edited.

[0142] For example, refer to Figure 5 , Figure 5 This application provides a schematic diagram of the overall process for editing product images in an embodiment of the present application. Specifically:

[0143] First, the system obtains the user's natural language image editing instructions, such as "change the backpack material from canvas to leather and replace the background with a desert." Then, the CosXL_Edit instruction decoupler parses the user's input instructions. This decoupler is responsible for extracting key verbs and nouns from the instructions to determine the elements to be edited and their target visual attributes. The parsed instructions are then processed by a semantic type recognition module, which determines the type of editing the user intends based on the extracted key verbs and nouns, including subject modification, attribute modification, or background replacement. If the recognition result is subject modification, such as modifying the main area of ​​a product, the Florence2 model can be used to locate the product area. Florence2 is a visual model used to identify visually salient areas in the image to be edited and locate the elements to be modified. If the recognition result is attribute modification, such as material replacement, SAM2 (Segment Anything Model 2) is used to generate a material mask. SAM2 is used to generate a mask for the target editing area in the image to be edited, allowing for local editing. If the recognition result is background replacement, scene semantic segmentation is performed to identify and separate the background area. After performing specific editing operations, KL divergence constraints are used to ensure consistency in lighting, texture, and other aspects of the edited image. Optionally, a local diffusion algorithm can be used to redraw the edited area to smooth the transition and optimize the editing effect. Finally, the image processed by the above steps is output.

[0144] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the editing method of the product images in this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0145] This application provides a product image editing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the product image editing method in Embodiment 1 above.

[0146] The following is for reference. Figure 6 The diagram illustrates a structural schematic of a product image editing device suitable for implementing embodiments of this application. The product image editing device in the embodiments of this application may include, but is not limited to, mobile terminals such as laptops, tablets (PADs), and fixed terminals such as desktop computers. Figure 6 The product image editing device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0147] like Figure 6As shown, the product image editing device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the product image editing device. The processing unit 1001, the read-only memory 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the product image editing device to communicate wirelessly or wiredly with other devices to exchange data. Although product image editing devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0148] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0149] The product image editing device provided in this application, employing the product image editing method described in the above embodiments, can solve the technical problem of how to improve the local editing effect of images. Compared with the prior art, the beneficial effects of the product image editing device provided in this application are the same as those of the product image editing method described in the above embodiments, and other technical features of the product image editing device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0150] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0151] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0152] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the product image editing method described in the above embodiments.

[0153] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0154] The aforementioned computer-readable storage medium may be included in a product image editing device; or it may exist independently and not be assembled into a product image editing device.

[0155] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a product image editing device, enable the product image editing device to write computer program code for performing the operations of this application in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, or as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0158] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described product image editing method, and can solve the technical problem of how to improve the local editing effect of an image. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the product image editing method provided in the above embodiments, and will not be repeated here.

[0159] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the product image editing method described above.

[0160] The computer program product provided in this application solves the technical problem of how to improve the local editing effect of images. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the product image editing method provided in the above embodiments, and will not be repeated here.

[0161] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An editing method of a product picture, characterized by, The method comprises the following steps: According to the received picture editing instruction, determine the to-be-edited element and the target visual attribute corresponding to the to-be-edited element; According to the semantic similarity between the to-be-edited element and each picture element of the to-be-edited picture, determine the semantic association graph corresponding to the to-be-edited picture; Weighted fusion of the visual saliency map corresponding to the to-be-edited picture and the semantic association graph is obtained. According to the heat map, determine the target editing area in the to-be-edited picture; Map the visual features of the target editing area to the latent code space to obtain the latent code features; Determine the target feature dimension corresponding to the target visual attribute; According to the type of the target visual attribute, determine the KL divergence constraint value of the target feature dimension; Set the KL divergence constraint value of each other feature dimension except the target feature dimension to the minimum constraint value; Determine the attribute difference between the attribute value corresponding to the target feature dimension and the attribute value corresponding to the target visual attribute; Modify the attribute value corresponding to the target feature dimension through the picture generation model until the attribute difference is less than or equal to the preset minimum difference, and the KL divergence value corresponding to each dimension feature of the latent code feature does not exceed the KL divergence constraint value corresponding to each dimension feature.

2. The product picture editing method of claim 1, wherein, The step of determining the target editing area in the to-be-edited picture according to the heat map comprises: Determine the pixel area in the heat map where the heat value is greater than the preset heat value, and generate a mask matrix in the pixel area; According to a first preset rectangular kernel, determine the maximum heat value of each pixel in the mask matrix in the neighborhood corresponding to the first preset rectangular kernel, and take the maximum heat value as the first target heat value of the pixel; According to a second preset rectangular kernel, determine the minimum first target heat value of each pixel in the mask matrix in the neighborhood corresponding to the second preset rectangular kernel, and take the minimum first target heat value as the second target heat value of the pixel, wherein the second preset rectangular kernel is greater than the first preset rectangular kernel; Determine the target pixel area in the heat map where the second target heat value is greater than the preset heat value, and determine the target editing area according to the target pixel area.

3. The product picture editing method of claim 2, wherein, The step of determining the target editing area according to the target pixel area comprises: Add first noise to the target pixel area to obtain an original noise map; Determine the gradient vector corresponding to the heat map according to a preset operator kernel; When denoising the original noise map, determine the second noise corresponding to the current denoising step according to the difference between the to-be-processed map corresponding to the current denoising step and the heat map, the target first noise corresponding to the current denoising step, and the gradient vector, and denoise the to-be-processed map according to the second noise; Take the target pixel area obtained after denoising as the target editing area.

4. The product picture editing method of claim 1, wherein, The step of modifying the attribute value corresponding to the target feature dimension by the picture generation model until the attribute difference is less than or equal to a preset minimum difference and each dimension feature of the latent code feature corresponds to a KL divergence value that does not exceed a KL divergence constraint value corresponding to each dimension feature, and then outputting the edited picture further comprises: obtaining a light transmission equation of the picture to be edited, the light transmission equation representing a pixel value of the picture as an integral relationship of a preset light source brightness and a bidirectional reflectance distribution function; if the element to be edited is a background element, determining a pixel value of the target editing area after editing according to the light transmission equation.

5. The product picture editing method of claim 4, wherein, The step of obtaining the light transmission equation of the picture to be edited further comprises: if the target visual attribute of the element to be edited includes a material attribute, determining a bidirectional reflectance distribution function corresponding to the material attribute; determining a pixel value of a pixel area corresponding to the element to be edited according to the bidirectional reflectance distribution function corresponding to the material attribute.

6. An editing apparatus of a product picture, characterized by comprising: The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the product picture editing method according to any one of claims 1 to 5.

7. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the product picture editing method according to any one of claims 1 to 5.

Images