Image adjustment method, device, readable storage medium and program product

Abstract

This application relates to an image adjustment method, apparatus, computer device, computer-readable storage medium, and computer program product. The method includes: acquiring a reference image, an original image, and a style cue for a target region; matching the target region in the reference image with the style cue; extracting features from the target region in the reference image using a cross-attention mechanism to obtain attention decoupling features; and denoising the original image based on a combination of the features from the style cue and the attention decoupling features to obtain an adjusted image; wherein the target region in the adjusted image matches the style cue. This method enables image adjustment based on unpaired data, ensuring processing efficiency.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image adjustment method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of image processing technology, text-guided image editing technology has emerged. Image editing technology, capable of precise manipulation of images through natural language prompts, has attracted significant attention from both academia and industry, opening up new possibilities for adaptive image processing and user-defined editing. The text-to-image DiffusionTransformer (T2IDiT) model, based on DiffusionModels and the Transformer architecture, requires model tuning based on high-quality paired data.

[0003] However, in real-world applications, obtaining high-quality pairing data presents numerous challenges, which significantly limits the applicability of this model. Summary of the Invention

[0004] Based on this, this application provides an image adjustment method, apparatus, computer device, computer-readable storage medium, and computer program product, which can perform image adjustment based on unpaired data and ensure processing efficiency.

[0005] On the one hand, this application provides an image adjustment method, including:

[0006] Obtain a reference image, an original image, and style hints for the target region; the target region in the reference image matches the style hints.

[0007] Based on the cross-attention mechanism, feature extraction is performed on the target area in the reference image to obtain attention decoupling features;

[0008] Based on the combined features of the style cue features and the attention decoupling features, the original image is denoised to obtain the adjusted image;

[0009] The target area of ​​the adjusted image matches the style prompt.

[0010] In one embodiment, the step of extracting features from the target region in the reference image according to the cross-attention mechanism to obtain attention decoupling features includes:

[0011] Decoupling query features are determined based on the target region mask of the reference image;

[0012] Decoupling key features and decoupling value features are determined based on the features of the reference image;

[0013] Based on the decoupling query features, attention features are extracted from the decoupling key features and the decoupling value features to obtain attention decoupling features.

[0014] In one embodiment, the step of extracting attention features from the decoupling key features and the decoupling value features based on the decoupling query features to obtain attention decoupling features includes:

[0015] The decoupling dot product value is obtained by performing a dot product between the decoupling query feature and the decoupling key feature.

[0016] Based on the number of feature dimensions contained in the decoupling key features, the decoupling dot product value is scaled to obtain the attention decoupling weight coefficient;

[0017] Based on the attention decoupling weight coefficients, the decoupling value features are weighted to obtain the attention decoupling features.

[0018] In one embodiment, the cross-attention mechanism includes an N-layer cross-attention network;

[0019] The step of extracting features from the target region in the reference image according to the cross-attention mechanism to obtain attention decoupling features includes:

[0020] Based on the i-th layer cross-attention network, feature extraction is performed on the target part in the reference image to obtain the i-th attention decoupling feature to be adjusted;

[0021] The i-th attention decoupling feature to be adjusted is linearly transformed to obtain the i-th linearly transformed feature.

[0022] Based on the characteristics after the i-th linear transformation, the (i-1)-th attention decoupling feature is adjusted to obtain the i-th attention decoupling feature; where i is an integer greater than or equal to 2, and i is less than or equal to N; when i equals N, the i-th attention decoupling feature is the N-th attention decoupling feature;

[0023] The combined features based on the style cues and the attention decoupling features are used to denoise the original image to obtain the adjusted image, including:

[0024] Based on the combined features of the style cue features and the Nth attention decoupling features, the original image is denoised to obtain the adjusted image.

[0025] In one embodiment, the combined features based on the style cue features and the attention decoupling features are used to denoise the original image to obtain an adjusted image, including:

[0026] Based on the (k-1)th combined feature of the style hint feature and the attention decoupling feature, determine the (k-1)th fusion query feature;

[0027] The (k-1)th feature of the original image is used as the (k-1)th fusion key feature and the (k-1)th fusion value feature;

[0028] Based on the self-attention feature of the (k-1)th feature of the original image, the (k-1)th feature of the original image is adjusted to obtain the kth feature of the original image;

[0029] Based on the (k-1)th fusion query feature, attention features are extracted from the (k-1)th fusion key feature and the (k-1)th fusion value feature. Then, based on the extracted k-th fusion attention feature, the (k-1)th combined feature is adjusted to obtain the k-th combined feature; k is an integer greater than or equal to 2 and less than or equal to M.

[0030] The Mth combined feature is fused with the Mth feature of the original image to obtain the fused feature;

[0031] The fused features are denoised based on the Mth combined features to obtain the denoised features.

[0032] Based on the denoised features, the adjusted image is obtained.

[0033] In one embodiment, obtaining the adjusted image based on the denoised features includes:

[0034] Based on the first weight parameter, the denoised features are adjusted to obtain the gating weights corresponding to the denoised features;

[0035] Based on the second weight parameter, the denoised features are adjusted to obtain the gating query features, the gating key features, and the gating value features corresponding to the denoised features.

[0036] Based on the gating query feature corresponding to the denoised feature, attention feature extraction is performed on the gating key feature and the gating value feature corresponding to the denoised feature to obtain the attention feature to be adjusted corresponding to the denoised feature.

[0037] Based on the gating weights corresponding to the denoised features, the attention features to be adjusted corresponding to the denoised features are adjusted to obtain the gating adjusted features.

[0038] The gating-adjusted features are then adjusted, and the denoised features are further adjusted to obtain the target fusion features;

[0039] An adjusted image is generated based on the target fusion features.

[0040] In one embodiment, generating the adjusted image based on the target fusion features includes:

[0041] Based on the target fusion features, determine the new denoised features;

[0042] Based on the new denoised features, the step of obtaining the gating weights is returned to obtain the new target fusion features;

[0043] An adjusted image is generated based on the new target fusion features that satisfy the condition for stopping adjustment.

[0044] On the one hand, this application also provides an image adjustment device, including:

[0045] The acquisition module is used to acquire a reference image, an original image, and style hints for the target area; the target area in the reference image matches the style hints.

[0046] The extraction module is used to extract features from the target area in the reference image according to the cross-attention mechanism to obtain attention decoupling features;

[0047] A denoising module is used to denoise the original image based on the combined features of the style cue features and the attention decoupling features to obtain an adjusted image; wherein the target part of the adjusted image matches the style cue.

[0048] On the one hand, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0049] Obtain a reference image, an original image, and style hints for the target region; the target region in the reference image matches the style hints.

[0050] Based on the cross-attention mechanism, feature extraction is performed on the target area in the reference image to obtain attention decoupling features;

[0051] Based on the combined features of the style cue features and the attention decoupling features, the original image is denoised to obtain the adjusted image;

[0052] The target area of ​​the adjusted image matches the style prompt.

[0053] On the one hand, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0054] Obtain a reference image, an original image, and style hints for the target region; the target region in the reference image matches the style hints.

[0055] Based on the cross-attention mechanism, feature extraction is performed on the target area in the reference image to obtain attention decoupling features;

[0056] Based on the combined features of the style cue features and the attention decoupling features, the original image is denoised to obtain the adjusted image;

[0057] The target area of ​​the adjusted image matches the style prompt.

[0058] On the one hand, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0059] Obtain a reference image, an original image, and style hints for the target region; the target region in the reference image matches the style hints.

[0060] Based on the cross-attention mechanism, feature extraction is performed on the target area in the reference image to obtain attention decoupling features;

[0061] Based on the combined features of the style cue features and the attention decoupling features, the original image is denoised to obtain the adjusted image;

[0062] The target area of ​​the adjusted image matches the style prompt.

[0063] The aforementioned image adjustment method, apparatus, computer equipment, computer-readable storage medium, and computer program product utilize a reference image to provide image-dimensional information for the target area, while style cues provide text-dimensional information for the original image. Furthermore, following a cross-attention mechanism, attention decoupling features are obtained, decoupling the target area from other information in the reference image. These attention decoupling features separate the attention decoupling features of the target area, reducing the scope of original image modification. Then, based on the combined features of the style cues and the attention decoupling features, the original image is denoised to obtain an adjusted image, forming a comprehensive dimension. Finally, based on the combined features, the original image is denoised again to obtain the adjusted image, ensuring that the target area of ​​the adjusted image matches the style cues. Therefore, this method can be used for image adjustment regardless of significant differences between the non-target areas of the original and reference images, ensuring processing efficiency. This efficiency includes sample processing efficiency during model training and sample selection efficiency during image adjustment. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is an application environment diagram of the image adjustment method in one embodiment;

[0066] Figure 2 This is a schematic diagram of traditional paired data;

[0067] Figure 3 This is a flowchart illustrating an image adjustment method in one embodiment;

[0068] Figure 4 This is a schematic diagram of unpaired data in one embodiment;

[0069] Figure 5 This is a schematic diagram of the process for obtaining attention decoupling features in one embodiment;

[0070] Figure 6 This is a flowchart illustrating the process of obtaining combined features in one embodiment;

[0071] Figure 7 This is a schematic diagram of the model framework of an image adjustment method in one embodiment;

[0072] Figure 8 This is a schematic diagram of the model framework of a target style extractor in one embodiment;

[0073] Figure 9 This is a flowchart illustrating the process of obtaining the adjusted image in one embodiment;

[0074] Figure 10 This is a structural block diagram of an image adjustment device in one embodiment;

[0075] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0076] To make the objectives, technical solutions, and beneficial effects of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0077] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0078] In some related solutions, an in-context-LoRA (LoR) approach is adopted. A typical implementation requires collecting 30-50 sets of domain-specific image pairing data (source image - target image), horizontally stitching the paired images into a composite image, and using edit text prompts for supervised learning to efficiently fine-tune the parameters of a pre-trained T2IDiT model (LoR). However, in practical applications, obtaining high-quality paired data faces many challenges: First, collecting professional-grade paired data requires significant manpower and resources, resulting in high costs; second, specific editing tasks (such as precisely modifying a user's eyebrows to a specific style of thin eyebrows or completely removing eyebrows) impose strict requirements on data quality, meaning the target image and the original image must differ only in the edited area, while other areas must remain highly consistent. This is what is called high-quality paired data. Figure 2 As shown, Figure 2 The original image 202 and reference image 204 are paired data, and they are highly identical. This strict pairing requirement exponentially increases the difficulty of data acquisition. Therefore, existing methods perform poorly in specific applications where paired data is scarce, and cannot meet the diverse needs of practical scenarios such as medical aesthetics and fashion design.

[0079] The image adjustment method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0080] Server 104 acquires a reference image, an original image, and style cues for the target region; the target region in the reference image is matched with the style cues; features are extracted from the target region in the reference image using a cross-attention mechanism to obtain attention decoupling features; based on the combined features of the style cues and the attention decoupling features, the original image is denoised to obtain an adjusted image; wherein the target region in the adjusted image matches the style cues.

[0081] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0082] In one exemplary embodiment, such as Figure 3 As shown, an image adjustment method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and also to a system including a terminal and a server, and implemented through the interaction between the terminal and the server, including the following steps 302 to 306. Wherein:

[0083] Step 302: Obtain style hints for the reference image, the original image, and the target area; match the target area in the reference image with the style hints.

[0084] The target region in the reference image matches the style cue, thus providing reference information for the style cue and providing image information for the target region. If the target region in the original image does not match the style cue, adjustment is required. The reference image and the original image can be unpaired sample images. Unpaired sample images refer to images where the target regions in the reference and original images belong to the same category, but the information represented by the non-target regions in the reference and original images differs. For example, the reference and original images might be two faces with different identities, both of which have eyebrows. In this case, the target region could be the eyebrows, adjusting the target region in the original image according to the target region in the reference image. Unpaired data includes... Figure 4 As shown, Figure 4 This includes the original image 402 and the reference image 404, which are in the form of unpaired data and may differ from each other.

[0085] The target area is a semantically defined area, which can be a part of the face, clothing, or artwork; a part of clothing can also refer to the style of clothing or the style of artwork. The face area includes, but is not limited to, at least one of facial structure, hairstyle changes, and accessory editing. Facial structure includes dimensions such as eyebrow shape adjustment, eye enlargement, and nose shape modification. Individual facial structure can be further refined; for example, eyebrow shape adjustment includes adjusting thin eyebrows, thick eyebrows, and no eyebrows. Hairstyle changes can include changing hair color, hairstyle shifting, and hair volume adjustment; accessory editing includes, but is not limited to, wearing / removing glasses and changing earring styles.

[0086] Style hints are the adjustment requirements for target areas of the original image, indicating the desired style adjustment result for those areas. Style hints can be descriptive labels for the target area (hint text) or contain related content about both the target and non-target areas. For example, enlarging the eyes in the original image to match the eyes in the reference image, or adjusting the eyebrow shape in the original image to match the eyebrow shape in the reference image. For instance, if the target area is the eyebrows, and the non-target areas could be facial features, expressions, and hairstyles, the style hint could be to edit the person's eyebrows to be thinner while preserving their facial features, expressions, and hairstyle.

[0087] In some embodiments, obtaining a reference image, an original image, and a style cue for the target region includes: obtaining a style cue for the original image and the target region; and selecting a reference image from multiple candidate images based on the style cue. Therefore, determining the appropriate reference image based on the style cue enables more efficient reference image selection.

[0088] In some embodiments, obtaining style cues for a reference image, an original image, and a target region includes: obtaining the input reference image, the original image, and the style cues for the target region, respectively. This allows for the acquisition of corresponding information using different paths.

[0089] Step 304: According to the cross-attention mechanism, feature extraction is performed on the target part in the reference image to obtain attention decoupling features.

[0090] In the process of extracting features of the target region according to the cross-attention mechanism, the information used to determine the target region is one information source, and the information of the reference image itself is another information source, thus forming cross-attention feature extraction.

[0091] Attention decoupling features are target region features in the reference image. Due to the use of a cross-attention mechanism, the attention decoupling features are decoupled from the non-target region features of the reference image; therefore, the cross-attention mechanism is part of the target style extractor in this embodiment. For example, the attention decoupling features can be separated style features without facial identity. These style features are applied only to the original image, resulting in the original image showing a person with style features, such as the person's eyebrows being edited to a thinner eyebrow style, thus overcoming the limitations of paired data.

[0092] The cross-attention mechanism, part of the target style extractor in this embodiment, can separate attention-decoupling features without non-target areas, using these as new style features. This overcomes the limitations of matching image data and lowers the stringent requirements for data quality. Even if the adjusted image differs from the original image not only in the editing area of ​​the target area but also in other areas, high consistency is not required. This relaxed matching requirement reduces the difficulty of data acquisition. Therefore, it can be applied even in specific fields where matching data is scarce, meeting the diverse needs of practical scenarios such as medical aesthetics and fashion design.

[0093] In some embodiments, feature extraction is performed on the target region in the reference image according to the cross-attention mechanism to obtain attention decoupling features. This includes: obtaining a target region mask; determining decoupling query features based on the target region mask; and weighting the features corresponding to the reference image according to the decoupling query features to obtain attention decoupling features. Thus, by using the target region mask and the image information of the reference image to form a cross-attention mechanism, attention decoupling features are accurately extracted.

[0094] In some embodiments, feature extraction is performed on the target area in the reference image according to the cross-attention mechanism to obtain attention decoupling features, including: feature extraction of eyebrows in the reference image according to the cross-attention mechanism to obtain attention decoupling features. This provides reference information, allowing the eyebrows of the person in the original image to overcome the limitations of paired data.

[0095] Step 306: Based on the combined features of style cue features and attention decoupling features, the original image is denoised to obtain the adjusted image.

[0096] The features of style hints are obtained by encoding the acquired style hints and then extracting features from the encoded style hints. These features more accurately represent the semantics of the style hints and can be combined with other features. Optionally, the style hints can be double-encoded using the CLIP text encoder to obtain their features.

[0097] Combined features are features obtained by fusing the features of style cues and the features of attention decoupling. Combined features have both the guiding function of style cues and the information indicated by style cues, and can also represent the image information of attention decoupling features.

[0098] In this process, the target area of ​​the adjusted image is matched with the style cues. The adjusted image is the result of denoising the original image with the goal of bringing the target area closer to the combined features. During denoising, the target area of ​​the original image gradually becomes closer to the combined features until the corresponding adjusted image features are obtained. Decoding the adjusted image features yields the adjusted image. The adjusted image is then matched with the non-target areas of the original image to perform feature adjustments on the local target areas, resulting in the adjusted image.

[0099] In some embodiments, the original image is denoised based on the combined features of the style cue features and the attention decoupling features to obtain the adjusted image, including: projecting the attention decoupling features into the latent space aligned with the style cue features through multiple linear layers to obtain aligned attention decoupling features; and combining the style cue features with the aligned attention decoupling features to obtain combined features.

[0100] In some embodiments, the original image is denoised based on a combination of style cue features and attention decoupling features to obtain an adjusted image, including: denoising the original image based on a combination of style cue features and attention decoupling features to obtain an adjusted image; and denoising the original image based on the combined features to obtain an adjusted image.

[0101] In some embodiments, the original image is denoised based on a combination of features from style cues and attention decoupling features to obtain an adjusted image. This includes: performing feature selection on the attention decoupling features according to the style cues to obtain combined features. Then, the information contained in the attention decoupling features is queried using the style cues' features, so that the decoupling attention features are closer to the style cues' features, resulting in a higher matching degree between the combined features and the style cues.

[0102] In some embodiments, the original image is denoised based on the combined features of the style cue features and the attention decoupling features to obtain the adjusted image, including: projecting the attention decoupling features into the latent space aligned with the style cue features through multiple linear layers to obtain aligned attention decoupling features; and combining the style cue features with the aligned attention decoupling features to obtain the target style features.

[0103] In some embodiments, the original image is denoised based on the combined features to obtain an adjusted image, including: fusing and denoising the combined features and the attention features of the original image based on the diffusion and attention model (DiffusionTransformer, T2IDiT) to obtain target fusion features; and generating the adjusted image based on the target fusion features.

[0104] In some embodiments, denoising the original image based on combined features to obtain an adjusted image includes: selecting the original image at the first time step based on the combined features at the first time step to obtain the adjusted image at the first time step; using the adjusted image at the first time step as the original image at the second time step, and guiding the selection of the original image at the second time step based on the combined features at the second time step to obtain the adjusted image at the second time step. The first and second time steps are adjacent time steps, with the first time step preceding the second time step. Thus, each time step performs denoising based on the image processing result of the previous time step, using combined features to provide proximity style cues for target areas in the original image to ensure effective denoising.

[0105] Steps 302-306 involve designing an efficient feature decoupling mechanism at the algorithm level to ensure the accuracy and robustness of the editing process. For example, in this embodiment, when fine-tuning the pre-trained Text-to-Image DiffusionTransformer large model, a dual-input mechanism is used—one input is the original image containing the person's background and identity information, and a text prompt describing the style of the target area; the other input is the target style map, i.e., the reference image. Step 304 forms a target style extractor for processing the target style map. With the assistance of the CLIP image encoder, this module accurately extracts the relevant style features of the target region through an attention masking mechanism and uses feature space projection technology to decouple the style features, obtaining attention-decoupled features, i.e., the extracted style features. Subsequently, in step 306, the extracted style features are fused with the text features and injected into the DiT denoising process, achieving accurate style transfer while perfectly preserving the original image's identity information. This completely breaks through the limitations of paired data and significantly lowers the application threshold; professional-grade editing effects can be achieved with only a single target style image; moreover, the editing process is highly controllable and interpretable, providing a reliable technical solution for practical application scenarios such as medical aesthetics, film and television special effects, and fashion design.

[0106] In the aforementioned image adjustment method, since the target area in the reference image matches the style cues, style cues from the reference image, the original image, and the target area are obtained. This allows the reference image and style cues to form two adjustment dimensions for the original image. The reference image provides image-dimensional information for the target area, while the style cues provide text-dimensional information for the original image. Furthermore, following a cross-attention mechanism, features are extracted from the target area in the reference image to obtain attention decoupling features. This decouples the target area from other information in the reference image, reducing the scope of original image modification. Then, a combined feature based on the style cues and attention decoupling features is formed, creating a comprehensive dimension. The original image is then denoised based on this combined feature to obtain the adjusted image, ensuring that the target area in the adjusted image matches the style cues. Therefore, this method can be used for image adjustment regardless of significant differences between the non-target areas of the original and reference images, ensuring processing efficiency. This efficiency includes sample processing efficiency during model training and sample selection efficiency during image adjustment.

[0107] In some embodiments, the feature extraction process for at least one layer of a cross-attention network is described, as follows: Figure 5 As shown, according to the cross-attention mechanism, feature extraction is performed on the target region in the reference image to obtain attention decoupling features, including steps 502 to 506; wherein:

[0108] Step 502: Determine the decoupling query features based on the target part mask of the reference image.

[0109] The target region mask is at least a target region mask image; the mask image can characterize the location and structure of the target region in the reference image, so as to distinguish the target region from the non-target region.

[0110] The decoupling query feature is a query feature used to determine the attention decoupling features in the cross-attention mechanism for a reference image. Since the target region mask in the reference image is different from the image itself, two information sources can be formed. Moreover, the location and structure determined by using the target region mask are more accurate. Therefore, the cross-attention mechanism adopts this decoupling query feature.

[0111] In some embodiments, determining decoupling query features based on a target region mask of a reference image includes: identifying a target region in the reference image to obtain the region where the target region is located; binarizing the region where the target region is located to obtain a target region mask; encoding the target region mask into a mask feature; and using the mask feature as a query feature in a cross-attention mechanism; wherein the query feature is a decoupling query feature. Thus, by using the target region mask of the reference image as a query feature, the cross-attention mechanism is based on data processing of the reference image under different processing conditions, thereby increasing the matching degree between the decoupling query feature and the target region in the reference image.

[0112] Step 504: Determine the decoupling key features and decoupling value features based on the features of the reference image.

[0113] In the cross-attention mechanism, the decoupling key feature is the key feature used to determine the attention decoupling feature; the decoupling value feature is the value feature used to determine the attention decoupling feature. Since the decoupling query feature is determined based on the target region mask, while both the decoupling key feature and the decoupling value feature are determined based on features of the reference image, the decoupling key feature and the decoupling value feature have a higher similarity, which helps ensure the accuracy of information dissemination.

[0114] In some embodiments, determining decoupling key features and decoupling value features based on features of a reference image includes: dividing the reference image into multiple image blocks; encoding each image block according to the number of image blocks and the feature dimensions of each image block to obtain features of the reference image; the features of the reference image include each feature dimension of each image block; using the features of the reference image as decoupling key features and decoupling value features. Thus, both the decoupling key features and the decoupling value features are features of the reference image, making the attention decoupling features closer to the features of the reference image at the target location, thereby ensuring the accuracy of information extraction.

[0115] Step 506: Based on the decoupling query features, attention features are extracted from the decoupling key features and decoupling value features to obtain attention decoupling features.

[0116] In one embodiment, attention features are extracted from the decoupling key features and decoupling value features based on the decoupling query features to obtain attention decoupling features, including: adjusting the decoupling value features based on the product of the decoupling query features and the decoupling key features to obtain attention decoupling features.

[0117] In this embodiment, the decoupling query features are determined based on the target part mask, which can accurately describe the location and shape of the target part in the reference image, so as to select the target part of the reference image for attention calculation; and the decoupling key features and decoupling value features are determined based on the features of the reference image, so as to focus on the extraction of target part features in the reference image, thereby obtaining attention decoupling features more accurately.

[0118] In one embodiment, attention features are extracted from the decoupling key features and decoupling value features based on the decoupling query features to obtain attention decoupling features. This includes: performing a dot product between the decoupling query features and the decoupling key features to obtain a decoupling dot product value; scaling the decoupling dot product value according to the number of feature dimensions contained in the decoupling key features to obtain attention decoupling weight coefficients; and weighting the decoupling value features according to the attention decoupling weight coefficients to obtain attention decoupling features.

[0119] The decoupling dot product value indicates the degree of matching between the decoupling query feature and the decoupling key feature, enabling targeted feature extraction. The decoupling dot product value is the result of performing a dot product between the decoupling query feature and the decoupling key feature. When the feature dimension distributions of the decoupling query feature and the decoupling key feature do not match, the decoupling key feature can be transposed to match their feature dimension distributions without changing the number of feature dimensions, thus enabling the dot product.

[0120] The number of feature dimensions refers to the number of channels representing information for each feature. Since dot product requires an equal number of feature dimensions, the number of feature dimensions for the decoupling query feature and the decoupling key feature must be equal. Optionally, the number of feature dimensions may include, but is not limited to, 64 or 128 feature dimensions. Scaling the number of feature dimensions allows for a moderate size of the attention decoupling weight coefficients, resulting in relatively moderate processing efficiency and ensuring efficient information extraction.

[0121] In one embodiment, the attention decoupling features are obtained by weighting the decoupling value features according to the attention decoupling weight coefficients, including: normalizing the attention decoupling weight coefficients to obtain normalized attention decoupling weight coefficients; and determining the attention decoupling features based on the product of the decoupling value features and the normalized attention decoupling weight coefficients.

[0122] In this embodiment, decoupling query features and decoupling key features are weighted and scaled to adjust the decoupling value features more efficiently, so as to determine the attention decoupling features more accurately.

[0123] In some embodiments, such as Figure 6As shown, the cross-attention mechanism includes an N-layer cross-attention network; according to the cross-attention mechanism, feature extraction is performed on the target area in the reference image to obtain attention decoupling features, including steps 602-606; correspondingly, the combined features based on style cues and attention decoupling features are used to denoise the original image to obtain the adjusted image, including step 608; wherein:

[0124] Step 602: Based on the i-th layer of the cross-attention network, feature extraction is performed on the target part in the reference image to obtain the i-th attention decoupling feature to be adjusted.

[0125] Cross-attention networks are basic units derived from feature extraction based on the cross-attention mechanism. There is a one-to-one correspondence between the cross-attention network and the attention decoupling features; the attention decoupling feature to be adjusted at layer i is obtained from feature extraction by the cross-attention network at layer i.

[0126] In some embodiments, based on the i-th layer cross-attention network, feature extraction is performed on the target region in the reference image to obtain the i-th attention decoupling feature to be adjusted, including: when the decoupling query feature is determined based on the target region mask of the reference image, the i-th decoupling key feature and the i-th decoupling value feature are determined based on the (i-1)-th feature of the reference image; based on the decoupling query feature, attention feature extraction is performed on the i-th decoupling key feature and the i-th decoupling value feature to obtain the i-th attention decoupling feature to be adjusted.

[0127] In some embodiments, attention features are extracted from the i-th decoupling key feature and the i-th decoupling value feature based on the decoupling query feature to obtain the i-th attention decoupling feature to be adjusted. This includes: performing a dot product between the decoupling query feature and the i-th decoupling key feature to obtain the i-th decoupling dot product value; scaling the decoupling dot product value according to the number of feature dimensions contained in the i-th decoupling key feature to obtain the i-th attention decoupling weight coefficient; and weighting the decoupling value feature according to the i-th attention decoupling weight coefficient to obtain the i-th attention decoupling feature to be adjusted.

[0128] In this process, when i is 2, the decoupling query feature can be determined, and this determination process is omitted in subsequent steps. The (i-1)th feature of the reference image is the feature obtained by encoding the reference image, i.e., the feature obtained by image extraction. When i is greater than 2, the (i-1)th feature of the reference image is the (i-1)th attention decoupling feature of the reference image, i.e., the feature of the latent space. Here, i and i-1 are the update rounds of the attention decoupling feature, and the i-th layer cross-attention network is the cross-attention network that needs to update the i-th attention decoupling feature.

[0129] Step 604: Perform a linear transformation on the i-th attention decoupling feature to be adjusted to obtain the i-th linearly transformed feature.

[0130] The i-th linearly transformed feature is used to avoid gradient vanishing. Before combining the layers, iterative processing is performed using linear transformation to reduce the gradient vanishing problem and ensure that the cross-attention network layers of the cross-attention mechanism are implemented sequentially.

[0131] In some embodiments, linearly transforming the i-th attention decoupling feature to be adjusted to obtain the i-th linearly transformed feature includes: performing fully connected processing on the i-th attention decoupling feature to be adjusted to obtain the i-th linearly transformed feature.

[0132] Step 606: Based on the i-th linearly transformed feature, adjust the (i-1)-th attention decoupling feature to obtain the i-th attention decoupling feature.

[0133] Where i is an integer greater than or equal to 2, and i is less than or equal to N; when i equals N, the i-th attention decoupling feature is the N-th attention decoupling feature. The (i-1)-th attention decoupling feature is the attention decoupling feature determined by the (i-1)-th layer cross-attention network. Specifically, when i equals 2, the (i-1)-th attention decoupling feature is obtained by extracting the target region of the reference image based on the first cross-attention network. When i is greater than 2, the (i-1)-th attention decoupling feature is the attention decoupling feature output by the (i-1)-th cross-attention network. The N-th attention decoupling feature is obtained by adjusting the (N-1)-th attention decoupling feature based on the Nth linearly transformed feature.

[0134] In some embodiments, attention decoupling features are extracted based on the i-th attention decoupling feature to be adjusted and the Nth to i-th layers of the cross-attention network until the N-th attention decoupling feature is obtained. Thus, the output of each layer of the attention network is processed using a linear layer, so that the cross-attention mechanism includes both cross-attention layers and linear transformation processing methods.

[0135] Step 608: Based on the combined features of the style cue features and the Nth attention decoupling features, the original image is denoised to obtain the adjusted image.

[0136] In some embodiments, the original image is denoised based on a combination of the features of the style hints and the Nth attention decoupling features to obtain an adjusted image, including: performing feature selection on the Nth attention decoupling features according to the features of the style hints to obtain a combination feature; and denoising the original image based on the combination feature to obtain the adjusted image.

[0137] In some embodiments, the original image is denoised based on the combined features of the style cue features and the Nth attention decoupling features to obtain an adjusted image, including: mapping the Nth attention decoupling features to a latent space aligned with the style cue features through multiple linear layers to obtain aligned Nth attention decoupling features; combining the style cue features with the aligned Nth attention decoupling features to obtain target style features; and denoising the original image based on the target style features to obtain the adjusted image.

[0138] In this embodiment, the i-th linearly transformed feature of the cross-attention network is combined with the attention decoupling feature of the previous layer of the cross-attention network through a linear transformation method to prevent the gradient vanishing problem in deep networks, thereby ensuring that the multi-layer attention network of the cross-attention mechanism can be trained and optimized more efficiently.

[0139] In some embodiments, the original image is denoised based on a combination of style-cue features and attention-decoupled features to obtain an adjusted image. This includes: determining a (k-1)th fusion query feature based on a (k-1)th combination feature of style-cue features and attention-decoupled features; using the (k-1)th feature of the original image as a (k-1)th fusion key feature and a (k-1)th fusion value feature; adjusting the (k-1)th feature of the original image based on its self-attention feature to obtain a (k)th feature of the original image; extracting attention features from the (k-1)th fusion key feature and the (k-1)th fusion value feature based on the (k-1)th fusion query feature, and adjusting the (k-1)th combination feature according to the extracted (k)th fusion attention feature to obtain a (k)th combination feature; where k is an integer greater than or equal to 2 and less than or equal to M; fusing the (M)th combination feature with the (M)th feature of the original image to obtain a fused feature; denoising the fused feature according to the (M)th combination feature to obtain a denoised feature; and obtaining the adjusted image based on the denoised feature.

[0140] The (k-1)th combined feature is obtained by combining the style cue feature and the attention decoupling feature in the (k-1)th round. When k is 2, the (k-1)th combined feature can be obtained by concatenating the style cue feature and the attention decoupling feature; when k is greater than 2, the (k-1)th combined feature is obtained by extracting and adjusting attention features based on the (k-2)th combined feature, the (k-2)th fusion key feature, and the (k-2)th fusion value feature. The feature obtained by updating the (k-1)th combined feature is the kth combined feature, where k-1 and k are the rounds in which the combined feature is updated.

[0141] The fusion query feature is a query feature used in the cross-attention mechanism for fusion, which is used to select and fuse the original images. Since the fusion query feature combines the features of style hints and attention decoupling features, the fusion query feature and the (k-1)th feature of the original image belong to features from different information sources, thus constituting the cross-attention mechanism used for fusion.

[0142] Here, the k-th fusion attention feature is the result of attention feature extraction from the (k-1)-th fusion key feature and the (k-1)-th fusion value feature. The k-th combined feature can be the normalized result of the sum of the k-th fusion attention feature and the (k-1)-th combined feature, or it can be the result of finely adjusting the (k-1)-th combined feature using the adjustment parameters corresponding to the k-th fusion attention feature. The k-th combined feature is the result of multiple attention feature extractions and adjustments, and can more accurately represent the corresponding text features.

[0143] The (k-1)th feature of the original image is obtained by performing at least one round of self-attention feature extraction and adjustment on the original image. The (k-1)th feature of the original image can reflect the information of the image from a holistic perspective, thus coordinating global information and preventing changes to non-target areas of the original image. Specifically, the (k)th feature of the original image can refer to: the sum of the (k-1)th feature of the original image and its self-attention features, after normalization; or it can refer to: the result of adjusting the (k-1)th feature of the original image using the adjustment parameters corresponding to its self-attention features. The (k)th feature of the original image is the result of meticulous adjustment, achieved through multiple rounds of self-attention feature extraction and adjustment, enabling a more accurate representation of the corresponding image features.

[0144] Since the original image is processed in blocks in the attention mechanism, the relationship between the blocks does not necessarily depend on the order, because distant objects may also be related. Therefore, self-attention needs to be able to directly capture the interaction of all regions within the image to obtain global dependencies.

[0145] The (k-1)th fusion key feature is the key feature used to interact with the (k-1)th combined feature in the cross-attention mechanism used for fusion; the (k-1)th fusion value feature is the value feature used to interact with the (k-1)th combined feature in the cross-attention mechanism used for fusion. The fused feature is obtained by fusing features based on the last round of updates; this fusion method can be concatenation. The denoised feature is obtained by performing an affine transformation on the fused feature using the condition vector corresponding to the Mth combined feature. For example, it can be obtained by processing the Mth combined feature with a lightweight MLP to obtain the condition vector; then, the condition vector and the fused feature are substituted into the spatial adaptive normalization layer AdaIN for further processing.

[0146] In some embodiments, attention features are extracted from the (k-1)th fusion key feature and the (k-1)th fusion value feature based on the (k-1)th fusion query feature to obtain the k-th combined feature, including: performing a dot product between the (k-1)th fusion query feature and the (k-1)th fusion key feature to obtain the (k-1)th fusion dot product value; scaling the (k-1)th fusion dot product value according to the number of feature dimensions contained in the (k-1)th fusion key feature to obtain the attention (k-1)th fusion weight coefficient; and weighting the (k-1)th fusion value feature according to the attention (k-1)th fusion weight coefficient to obtain the attention (k-1)th fusion feature.

[0147] In one embodiment, the (k-1)th feature of the original image can be used as the (k-1)th fusion bond feature and the (k-1)th fusion value feature, respectively.

[0148] In this embodiment, the features of the original image are selected based on the fusion query features determined by the combined features, resulting in corresponding fusion key features and fusion value features. This forms a cross-attention mechanism for fusion, leading to the update of the combined features. Since the information of the combined features is not used to determine the correlation between image patches, while the self-attention mechanism can capture the information of all image patches within the image and realize the interaction between image patches, the information complementarity of the combined features is formed based on the self-attention features of the original image, forming a dual-input information processing method. Furthermore, the combined features and the self-attention features of the original image are fused to better preserve the information in the original image through the denoised features, ensuring that the similarity between the adjusted image and the original image is appropriate.

[0149] In some embodiments, obtaining an adjusted image based on denoised features includes: adjusting the denoised features based on a first weight parameter to obtain gating weights corresponding to the denoised features; adjusting the denoised features based on a second weight parameter to obtain gating query features, gating key features, and gating value features corresponding to the denoised features; extracting attention features from the gating key features and gating value features corresponding to the denoised features based on the gating query features to obtain attention features to be adjusted corresponding to the denoised features; adjusting the attention features to be adjusted corresponding to the denoised features based on the gating weights to obtain gating adjusted features; adjusting the denoised features based on the gating adjusted features to obtain target fusion features; and generating an adjusted image based on the target fusion features.

[0150] The first weight parameter is the weight parameter under the gating mechanism, and the second weight parameter is the weight parameter under the attention mechanism used for noise reduction. Adjusting the first and second weight parameters separately allows the parameters involved in the gating and attention mechanisms to be combined to form a corresponding gated attention mechanism, thereby more accurately adjusting the noise reduction intensity. The gating weight can adjust the noise of the attention mechanism to avoid excessive noise reduction intensity; excessive noise reduction intensity can cause the non-target areas in the adjusted image to mismatch with the non-target areas in the original image.

[0151] The gated query features, gated key features, and gated value features corresponding to the denoised features are obtained by sequentially adjusting the gated query weight parameters, gated key weight parameters, and gated value weight parameters in the second weight parameters. The gated query weight parameters, gated key weight parameters, and gated value weight parameters are weight parameters obtained through model training.

[0152] The attention feature to be adjusted is obtained by weighting the gating value feature corresponding to the denoised feature based on the gating query feature corresponding to the denoised feature and the dot product of the gating key feature corresponding to the denoised feature.

[0153] Gated features are used to adjust noise intensity. Since the attention features to be adjusted are features under the attention mechanism, and the attention mechanism may over-focus on information of the target area and lose information of the original image, that is, over-denoising occurs.

[0154] The target fusion feature is the result of applying noise adjustment to the denoised feature based on the gated feature. Optionally, the sum of the gated feature and the denoised feature can be linearized to determine the target fusion feature.

[0155] In one embodiment, adjusting the denoised features based on a first weight parameter to obtain the gating weights corresponding to the denoised features includes: obtaining the gating weights corresponding to the denoised features based on the product of the first weight parameter and the denoised features. For example, the product of the first weight parameter and the denoised features can be normalized to obtain the gating weights corresponding to the denoised features.

[0156] In one embodiment, the attention feature to be adjusted corresponding to the denoised feature is adjusted according to the gating weight corresponding to the denoised feature to obtain the gating adjusted feature, including: performing a dot product on the gating weight and the attention feature to be adjusted corresponding to the denoised feature to obtain the gating dot product value; the gating dot product value is the gating adjusted feature.

[0157] In one embodiment, generating an adjusted image based on target fusion features includes: decoding the target fusion features to obtain the adjusted image.

[0158] In this embodiment, the intensity of feature transfer of the denoised features is limited by the gating attention mechanism to precisely control the intensity of style transfer, thereby removing information of non-target parts in the denoised features and making the adjusted image closer to the original image.

[0159] In some embodiments, generating an adjusted image based on target fusion features includes: determining new denoised features based on target fusion features; returning to the step of obtaining gating weights based on the new denoised features to obtain new target fusion features; and generating an adjusted image based on the new target fusion features that satisfy the stop adjustment condition.

[0160] The new denoised features are obtained by updating the initial target fusion features at least once; the initial target fusion features are determined based on the initial denoised features, which are obtained by denoising the fused features according to the Mth combined features.

[0161] The new target fusion feature is obtained by performing the target fusion feature determination step again based on the new denoised feature.

[0162] In an exemplary embodiment, the step of obtaining gating weights based on the new denoised features to obtain new target fusion features includes: adjusting the new denoised features based on a first weight parameter to obtain gating weights corresponding to the new denoised features; adjusting the new denoised features based on a second weight parameter to obtain gating query features, gating key features, and gating value features corresponding to the new denoised features; extracting attention features from the gating key features and gating value features corresponding to the new denoised features based on the gating query features to obtain attention features to be adjusted corresponding to the new denoised features; adjusting the attention features to be adjusted corresponding to the new denoised features according to the gating weights to obtain gating adjusted features; and adjusting the new denoised features based on the gating adjusted features to obtain new target fusion features. Thus, at least one more round of target fusion feature update process is executed to gradually control noise intensity.

[0163] The new target fusion feature that meets the condition for stopping adjustment refers to the new target fusion feature having reached a preset value in terms of the number of updates or the matching degree with the original image. For example, the new target fusion feature that meets the condition for stopping adjustment refers to the target fusion feature obtained from the last update when the number of target fusion feature updates reaches a threshold.

[0164] In this embodiment, the gated attention mechanism can change the update process of the denoised features in multiple rounds to control the noise intensity more finely.

[0165] In one exemplary embodiment, such as Figure 7 As shown, the input architecture in step 302 is a dual-path heterogeneous input architecture, which includes an identity preservation path and a style reference path.

[0166] Identity Preservation Path: The input consists of an original image containing background and identity information of the person, along with text prompts describing the target style. These text prompts are style hints. The original image is processed by a VQ-VAE image encoder for multi-scale feature extraction before being fed into the DiT backbone network. The text prompts undergo dual semantic encoding using both the CLIP text encoder and the T5 encoder. The CLIP text encoder is used for guidance, while the T5 encoder is used for feature concatenation decoupled from the attention mechanism.

[0167] Style reference path: Input a single target style image, i.e., a reference image. After feature decoupling by the target style extractor, the style features are injected into the DiT generation process.

[0168] Correspondingly, step 306 includes: the two features interact in the spatiotemporal dimension through an adaptive fusion module, and finally output a high-quality target image through a cascaded upsampling decoder.

[0169] In one exemplary embodiment, such as Figure 8 As shown, step 302 is performed using the target style extractor, which includes:

[0170] This module adopts a cascaded feature decoupling architecture:

[0171] 1) Basic Feature Extraction Layer: Given the target pattern image The pre-trained CLIP-ViT image encoder is employed. CLIP utilizes contrastive learning to align relevant image and text embeddings, which contain latent features of image appearance. Therefore, it leverages its powerful cross-modal representation capabilities to capture the global semantic features of the target style map. .

[0172]

[0173] in L is the number of image blocks, p is the size of the image blocks, and d is the feature dimension.

[0174] The input image has a width of W and a height of H. After the input image encoder, the input image undergoes image block processing, being divided into L image blocks, each of size p*p. After flattening each image, a d-dimensional feature vector is obtained through linear projection. Finally, the fc of L d-dimensional vectors is obtained.

[0175] Attention decoupling module: Consists of N stacked cross-attention layers and linear layers. For example, N equals 5, 6, or 7. In the cross-attention layers, values ​​(K) and (V) are used as keys. Therefore, the challenge lies in designing a suitable query vector (Q) to guide the extraction of the target style. This takes into account the masking of relevant regions. This method, which can directly describe the target region and target shape, designs a Transformer-based encoder to obtain the embedding vector, which is then used as the query vector for the cross-attention layer.

[0176]

[0177] Cross-attention mechanism (i-th layer):

[0178]

[0179] Where both K and V must be target style features. .

[0180] Meanwhile, skip connections are used to maintain gradient flow:

[0181]

[0182] in This represents the feature vector output by the current layer. This represents the feature vector output by the previous layer, where d is the feature dimension. In this formula, Attention_i represents the attention value of the current layer. After undergoing dimensional transformation through a linear layer, the i-th linearly changing feature is obtained; the i-th linearly changing feature and the attention feature obtained by undergoing the same dimensional transformation through a linear layer as the previous layer are then processed. The features from the previous layer are added together. This is done to prevent the vanishing gradient problem in deep networks. Preserving the original information in the multi-layered computation output allows for more efficient training and optimization of the network.

[0183] Finally, using a feature projection head, attention-decoupled features are mapped to a latent space aligned with text features through multiple linear layers to obtain target style features, i.e., decoupled features.

[0184] In one exemplary embodiment, such as Figure 9As shown, the features extracted by the target style extractor are incorporated into the DiT structure, guiding the model to generate an adjusted image containing the target style. A progressive feature fusion strategy is used: the primary fusion input features include the first combined feature obtained by concatenating text features and sample features, as well as the image features of the original image. This is used to establish a semantic mapping between the text prompt and the target style, for example, associating the text prompt "thin eyebrows" with the eyebrow texture features of the style image. The advanced fusion input features include concatenating the combined features output from the primary fusion with the k-th image feature of the original image. This is used to control the style transfer intensity and maintain geometric consistency; for example, ensuring that the eyebrow texture only applies to the correct area of ​​the face and does not spread to irrelevant areas such as the hairline.

[0185] 1) Initial Integration Stage: In the prompt text The text feature T encoded by the T5 encoder is concatenated with the target style feature along the channel dimension:

[0186]

[0187] Image features after encoding of the original image The data is fed into multiple dual-stream Transformer modules. In these modules, the combined features are correlated with the visual style of the original image to obtain text features with the target style. Image features containing original image structural information The formula for the k-th layer dual-stream Transformer module is:

[0188]

[0189] Where LN represents the normalization layer:

[0190]

[0191] γ and β are learnable parameters, and μ and σ are the mean and variance of the feature dimension;

[0192] Where CrossAttn is the cross-attention layer calculation to obtain the k-th combined feature. The expression for the k-th combination feature is as follows:

[0193]

[0194] Where d is the feature dimension, and K and V are both the (k-1)th features of the original image, i.e., image features. Q is the (k-1)th fused query feature, i.e., the text feature. .

[0195] SelfAttn is used for self-attention calculation:

[0196]

[0197] Where X is d is the feature dimension. , and These are all learnable parameter matrices.

[0198] 3) Advanced Fusion Stage: The two features of the Mth pair are concatenated and fed into multiple single-stream Transformer modules for deep feature integration to obtain the Mth combined feature, with the formula:

[0199]

[0200] The spatially adaptive normalization layer AdaIN in the single-stream Transformer module can maintain the geometric consistency of the image:

[0201]

[0202] Where h is a text style feature The conditional vector is generated by a lightweight MLP, where γ and β are parameters of the affine transformation, and z is the denoised feature; the initial denoised feature is... The subsequent denoised features are .

[0203] Its cross-modal attention layer can precisely control the intensity of style transfer; the cross-modal attention layer of the l-th layer is used to obtain... Its expression is as follows:

[0204]

[0205] Among them, the gating attention mechanism:

[0206]

[0207] Wherein, Sigmoid(WgX) is the gating weight, and the first weight parameter includes The second weighting parameter includes , and The first and second weight parameters are both learnable parameter matrices, and the sigmoid is the activation function, ensuring that the output value is in the (0, 1) interval.

[0208] 3) Decoding output: A decoder with a multi-scale feature pyramid structure is used to gradually upsample and generate the final result, ensuring the unity of local editing and global coordination.

[0209] Therefore, this embodiment provides a non-paired sample image editing framework based on feature decoupling. This framework, through an innovative dual-path feature decoupling mechanism, achieves a technological breakthrough by enabling high-quality image editing with only a single target style image. Compared to existing technologies, this embodiment demonstrates significant advantages in data efficiency, editing accuracy, and application breadth, and can be widely applied in multiple fields such as digital content creation, medical aesthetics, and virtual makeup try-on. Compared to existing methods, this embodiment has the following significant features:

[0210] Improved data efficiency: Reduces the 30-50 sets of professional-grade paired data required by traditional methods to a single sample; reduces data acquisition costs by more than 98%; supports training on unpaired data, breaking through the bottleneck of data acquisition.

[0211] Multi-scenario applicability: Facial feature editing: supports eyebrow shape adjustment (thin eyebrows / thick eyebrows / no eyebrows), eye enlargement, nose shape modification, etc.; Hairstyle transformation: enables hair color change, hairstyle transfer, hair volume adjustment, etc.; Accessory editing: glasses wearing / removal, earring style replacement, etc.; Cross-domain application: has been successfully expanded to clothing style transfer, art style conversion, and other fields.

[0212] Modular and scalable architecture: The target style extractor adopts a plug-in design. The current version integrates the CLIP-ViT encoder and supports replacement with advanced vision encoders such as DINOv2.

[0213] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0214] Based on the same inventive concept, this application also provides an image adjustment apparatus for implementing the image adjustment method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more image adjustment apparatus embodiments provided below can be found in the limitations of the image adjustment method described above, and will not be repeated here.

[0215] In one exemplary embodiment, such as Figure 10As shown, an image adjustment device is provided, comprising:

[0216] The acquisition module 1002 is used to acquire a reference image, an original image, and style hints for the target area; the target area in the reference image matches the style hints.

[0217] Extraction module 1004 is used to extract features from the target part in the reference image according to the cross-attention mechanism to obtain attention decoupling features;

[0218] The denoising module 1006 is used to denoise the original image based on the combined features of the style cue features and the attention decoupling features to obtain an adjusted image; wherein the target part of the adjusted image matches the style cue.

[0219] In one embodiment, the extraction module 1004 is used for:

[0220] Decoupling query features are determined based on the target region mask of the reference image;

[0221] Decoupling key features and decoupling value features are determined based on the features of the reference image;

[0222] Based on the decoupling query features, attention features are extracted from the decoupling key features and the decoupling value features to obtain attention decoupling features.

[0223] In one embodiment, the extraction module 1004 is used for:

[0224] The decoupling dot product value is obtained by performing a dot product between the decoupling query feature and the decoupling key feature.

[0225] Based on the number of feature dimensions contained in the decoupling key features, the decoupling dot product value is scaled to obtain the attention decoupling weight coefficient;

[0226] Based on the attention decoupling weight coefficients, the decoupling value features are weighted to obtain the attention decoupling features.

[0227] In one embodiment, the cross-attention mechanism includes an N-layer cross-attention network;

[0228] The extraction module 1004 is used for:

[0229] Based on the i-th layer cross-attention network, feature extraction is performed on the target part in the reference image to obtain the i-th attention decoupling feature to be adjusted;

[0230] The i-th attention decoupling feature to be adjusted is linearly transformed to obtain the i-th linearly transformed feature.

[0231] Based on the characteristics after the i-th linear transformation, the (i-1)-th attention decoupling feature is adjusted to obtain the i-th attention decoupling feature; where i is an integer greater than or equal to 2, and i is less than or equal to N; when i equals N, the i-th attention decoupling feature is the N-th attention decoupling feature;

[0232] The noise reduction module 1006 is used for:

[0233] Based on the combined features of the style cue features and the Nth attention decoupling features, the original image is denoised to obtain the adjusted image.

[0234] In one embodiment, the noise reduction module 1006 is used for:

[0235] Based on the (k-1)th combined feature of the style hint feature and the attention decoupling feature, determine the (k-1)th fusion query feature;

[0236] The (k-1)th feature of the original image is used as the (k-1)th fusion key feature and the (k-1)th fusion value feature;

[0237] Based on the self-attention feature of the (k-1)th feature of the original image, the (k-1)th feature of the original image is adjusted to obtain the kth feature of the original image;

[0238] Based on the (k-1)th fusion query feature, attention features are extracted from the (k-1)th fusion key feature and the (k-1)th fusion value feature. Then, based on the extracted k-th fusion attention feature, the (k-1)th combined feature is adjusted to obtain the k-th combined feature; k is an integer greater than or equal to 2 and less than or equal to M.

[0239] The Mth combined feature is fused with the Mth feature of the original image to obtain the fused feature;

[0240] The fused features are denoised based on the Mth combined features to obtain the denoised features.

[0241] Based on the denoised features, the adjusted image is obtained.

[0242] In one embodiment, the noise reduction module 1006 is used for:

[0243] Based on the first weight parameter, the denoised features are adjusted to obtain the gating weights corresponding to the denoised features;

[0244] Based on the second weight parameter, the denoised features are adjusted to obtain the gating query features, the gating key features, and the gating value features corresponding to the denoised features.

[0245] Based on the gating query feature corresponding to the denoised feature, attention feature extraction is performed on the gating key feature and the gating value feature corresponding to the denoised feature to obtain the attention feature to be adjusted corresponding to the denoised feature.

[0246] Based on the gating weights corresponding to the denoised features, the attention features to be adjusted corresponding to the denoised features are adjusted to obtain the gating adjusted features.

[0247] The gating-adjusted features are then adjusted, and the denoised features are further adjusted to obtain the target fusion features;

[0248] An adjusted image is generated based on the target fusion features.

[0249] In one embodiment, the noise reduction module 1006 is used for:

[0250] Based on the target fusion features, determine the new denoised features;

[0251] Based on the new denoised features, the step of obtaining the gating weights is returned to obtain the new target fusion features;

[0252] An adjusted image is generated based on the new target fusion features that satisfy the condition for stopping adjustment.

[0253] Each module in the aforementioned image adjustment device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0254] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 11As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an image adjustment method.

[0255] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0256] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0257] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0258] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0259] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0260] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0261] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0262] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image adjustment method, characterized in that, The method includes: Obtain a reference image, an original image, and style hints for the target region; the target region in the reference image matches the style hints. Based on the cross-attention mechanism, feature extraction is performed on the target area in the reference image to obtain attention decoupling features; Based on the combined features of the style cue features and the attention decoupling features, the original image is denoised to obtain the adjusted image; The target area of ​​the adjusted image matches the style prompt.

2. The method according to claim 1, characterized in that, The step of extracting features from the target region in the reference image according to the cross-attention mechanism to obtain attention decoupling features includes: Decoupling query features are determined based on the target region mask of the reference image; Decoupling key features and decoupling value features are determined based on the features of the reference image; Based on the decoupling query features, attention features are extracted from the decoupling key features and the decoupling value features to obtain attention decoupling features.

3. The method according to claim 2, characterized in that, The step of extracting attention features from the decoupling key features and the decoupling value features based on the decoupling query features to obtain attention decoupling features includes: The decoupling dot product value is obtained by performing a dot product between the decoupling query feature and the decoupling key feature. Based on the number of feature dimensions contained in the decoupling key features, the decoupling dot product value is scaled to obtain the attention decoupling weight coefficient; Based on the attention decoupling weight coefficients, the decoupling value features are weighted to obtain the attention decoupling features.

4. The method according to claim 1, characterized in that, The cross-attention mechanism includes an N-layer cross-attention network; The step of extracting features from the target region in the reference image according to the cross-attention mechanism to obtain attention decoupling features includes: Based on the i-th layer cross-attention network, feature extraction is performed on the target part in the reference image to obtain the i-th attention decoupling feature to be adjusted; The i-th attention decoupling feature to be adjusted is linearly transformed to obtain the i-th linearly transformed feature. Based on the characteristics after the i-th linear transformation, the (i-1)-th attention decoupling feature is adjusted to obtain the i-th attention decoupling feature; where i is an integer greater than or equal to 2, and i is less than or equal to N; when i equals N, the i-th attention decoupling feature is the N-th attention decoupling feature; The combined features based on the style cues and the attention decoupling features are used to denoise the original image to obtain the adjusted image, including: Based on the combined features of the style cue features and the Nth attention decoupling features, the original image is denoised to obtain the adjusted image.

5. The method according to claim 1, characterized in that, The combined features based on the style cues and the attention decoupling features are used to denoise the original image to obtain the adjusted image, including: Based on the (k-1)th combined feature of the style hint feature and the attention decoupling feature, determine the (k-1)th fusion query feature; The (k-1)th feature of the original image is used as the (k-1)th fusion key feature and the (k-1)th fusion value feature; Based on the self-attention feature of the (k-1)th feature of the original image, the (k-1)th feature of the original image is adjusted to obtain the kth feature of the original image; Based on the (k-1)th fusion query feature, attention features are extracted from the (k-1)th fusion key feature and the (k-1)th fusion value feature. Then, based on the extracted k-th fusion attention feature, the (k-1)th combined feature is adjusted to obtain the k-th combined feature; k is an integer greater than or equal to 2 and less than or equal to M. The Mth combined feature is fused with the Mth feature of the original image to obtain the fused feature; The fused features are denoised based on the Mth combined features to obtain the denoised features. Based on the denoised features, the adjusted image is obtained.

6. The method according to claim 5, characterized in that, The process of obtaining the adjusted image based on the denoised features includes: Based on the first weight parameter, the denoised features are adjusted to obtain the gating weights corresponding to the denoised features; Based on the second weight parameter, the denoised features are adjusted to obtain the gating query features, the gating key features, and the gating value features corresponding to the denoised features. Based on the gating query feature corresponding to the denoised feature, attention feature extraction is performed on the gating key feature and the gating value feature corresponding to the denoised feature to obtain the attention feature to be adjusted corresponding to the denoised feature. Based on the gating weights corresponding to the denoised features, the attention features to be adjusted corresponding to the denoised features are adjusted to obtain the gating adjusted features. The gating-adjusted features are then adjusted, and the denoised features are further adjusted to obtain the target fusion features; An adjusted image is generated based on the target fusion features.

7. The method according to claim 6, characterized in that, The step of generating the adjusted image based on the target fusion features includes: Based on the target fusion features, determine the new denoised features; Based on the new denoised features, the step of obtaining the gating weights is returned to obtain the new target fusion features; An adjusted image is generated based on the new target fusion features that satisfy the condition for stopping adjustment.

8. An image adjustment device, characterized in that, The device includes: The acquisition module is used to acquire a reference image, an original image, and style hints for the target area; the target area in the reference image matches the style hints. The extraction module is used to extract features from the target area in the reference image according to the cross-attention mechanism to obtain attention decoupling features; A denoising module is used to denoise the original image based on the combined features of the style cue features and the attention decoupling features to obtain an adjusted image; wherein the target part of the adjusted image matches the style cue.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images