Customized figure image generation method and device based on face adaptive replacement

Abstract

The invention discloses a customized figure image generation method and device based on face adaptive replacement. The method comprises the steps that ID features of a customized object reference image are extracted through a portrait encoder, and a text encoder encodes target scene cues to obtain scene features; dimensionality reduction is carried out on the ID features through an identity dimensionality reduction embedding module to obtain compact ID features, and the compact ID features are embedded into scene features to obtain ID embedded scene features; and respectively guiding a portrait and a scene generator by using the ID features and the ID embedded scene features, and outputting human face and scene noise through a corresponding classifier-free guiding function. The same processing strategy is adopted for the two types of noise to generate attention tensors, a maximum value index is taken to obtain an adaptive mask, noise prediction output is calculated in combination with a diffusion sampling time step and a set threshold value, and a customized figure image is generated through diffusion sampling iteration. The problems that in the prior art, under complex semantics, identity and scenes are difficult to cooperate, human face details are prone to distortion, manual intervention is relied on, and the generalization ability is weak are solved.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image generation technology, specifically to a method and apparatus for generating customized human images based on adaptive face replacement. Background Technology

[0002] Customized Personal Image Generation (CPIG) technology, as an important application branch of generative AI in the field of computer vision, focuses on capturing the core biometric features of a target face and integrating textual semantic descriptions to achieve image synthesis that combines identity recognition with personalized expression. It has already demonstrated broad application potential in scenarios such as film and television character customization, artistic portrait creation, and virtual scene interaction. With the maturity of technologies such as diffusion models and Transformers, CPIG technology has formed a diverse range of technical approaches: DreamBooth achieves object-level customization through fine-tuning the diffusion model, laying the foundation for subsequent technologies; Textual Inversion optimizes text embedding vectors to reduce deployment costs; and solutions such as PhotoMaker and InstantID further enhance identity fidelity, adopting no-fine-tuning or slightly-fine-tuning modes to improve practicality and drive the technology closer to industrial applications.

[0003] However, existing CPIG technology still suffers from core shortcomings in practical applications: First, it lacks consistency in identity under complex semantic guidance. When text prompts include multi-dimensional requirements such as character actions and complex scenes, scene and action information can easily squeeze the identity feature representation space, leading to problems such as distorted facial details and reduced identity recognition in the generated images. Second, it lacks complete scene narrative. Technical solutions generally have a bias of "face region priority," overemphasizing local facial features while neglecting the complete depiction of full-body posture and body movements. Furthermore, spatial logic conflicts and stylistic disjointness easily occur between characters and background scenes, making it impossible to achieve "character-scene collaborative narrative." Third, it relies on manual intervention and has weak generalization ability. Traditional methods require manual annotation of facial region masks, which is cumbersome and prone to edge fragmentation and artifacts. At the same time, it is difficult to adapt to diverse scenes such as side faces, occlusions, and dynamic postures, which seriously restricts the industrialization of the technology.

[0004] Therefore, there is an urgent need for a customized human image generation method based on adaptive face replacement to solve the problems of existing technologies, such as difficulty in coordinating identity and scene under complex semantics, easy distortion of facial details, reliance on manual intervention and weak generalization ability. Summary of the Invention

[0005] To address these issues, this invention provides a customized character image generation method and apparatus based on adaptive face replacement, which solves the problems of existing customized character image generation technologies, such as difficulty in coordinating identity and scene under complex semantics, easy distortion of facial details, reliance on manual intervention and weak generalization ability, and achieves identity fidelity and scene-coordinated narrative without manual annotation.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a customized person image generation method based on adaptive face replacement, characterized in that it includes: The ID features are obtained by extracting features from the reference image of the customized object through a human face encoder; the scene features are obtained by encoding the target scene prompts through a text encoder; based on the scene features, the ID features are reduced in dimensionality through an identity dimensionality reduction and embedding module to obtain compact ID features; the compact ID features are embedded into the scene features to obtain ID embedded scene features. The image generator is guided by the ID features and outputs face noise through the image classifier-free guidance function; the scene generator is guided by the ID embedded scene features and outputs scene noise through the scene classifier-free guidance function. Based on the facial noise, a facial attention tensor is generated by processing it according to a set processing strategy; based on the scene noise, a scene attention tensor is generated by processing it according to the set processing strategy. An adaptive mask is generated by taking the maximum index of the face attention tensor and the scene attention tensor; The noise prediction output is calculated based on the relationship between the diffusion sampling time step and the set threshold. Based on the noise prediction output, diffusion sampling iteration is performed to generate a customized portrait image.

[0007] As a preferred embodiment of the customized person image generation method based on adaptive face replacement, the formula for calculating the compact ID feature is as follows:

[0008] In the formula, For compact ID features; ID features; To initialize the query vector; SM It is the softmax function; Dimensional permutation; The formula for calculating the ID embedding scene features is as follows:

[0009] In the formula, Embed scene features into the ID; For scene features; CATThis is a tensor concatenation function; N The length of the original token sequence for scene features; K The length of the token sequence for the compact ID feature.

[0010] As a preferred embodiment of the customized portrait image generation method based on adaptive face replacement, the expression of the portrait classifier-free guided function is:

[0011] In the formula, Facial noise; For diffusion process t Noisy images at any given time; CFG is the core logic of the classifier-less bootstrap function; Parameters for the portrait generator; The expression for the classifier-free bootstrapping function in the scenario is:

[0012] In the formula, This is scene noise; These are parameters for the scene generator; For text encoders; These are prompts for the target scene.

[0013] As a preferred embodiment of the customized character image generation method based on adaptive face replacement, in the process of generating the face attention tensor and the scene attention tensor through the set processing strategy, the set processing strategy includes element-level absolute value calculation, dimension reshaping, spatial scaling and normalization processing. The expression for the face attention tensor is:

[0014] In the formula, Face attention tensor; R For dimension reshaping functions; SM It is the softmax function; For the target dimension; It is a learnable significant factor; This is the initial attention tensor; The expression for the scene attention tensor is:

[0015] In the formula, For scene attention tensor; These are learnable significant factors.

[0016] As a preferred embodiment of the customized human image generation method based on adaptive face replacement, in the process of calculating the noise prediction output based on the relationship between the diffusion sampling time step and the set threshold, if the diffusion sampling time step is greater than the set threshold, the noise prediction output is calculated by setting a calculation formula; if the diffusion sampling time step is not greater than the set threshold, the scene noise is used as the noise prediction output. The set calculation formula is:

[0017] In the formula, For noise prediction output; M For adaptive masking.

[0018] The present invention also provides a customized person image generation device based on adaptive face replacement, which employs the above-mentioned customized person image generation method based on adaptive face replacement, including: The ID feature and ID embedding scene feature acquisition unit is used to extract features from the reference image of the customized object through a human face encoder to obtain ID features; to encode the target scene prompt words through a text encoder to obtain scene features; based on the scene features, to reduce the dimensionality of the ID features through an identity dimensionality reduction and embedding module to obtain compact ID features; and to embed the compact ID features into the scene features to obtain ID embedding scene features. The face noise and scene noise acquisition unit is used to guide the portrait generator based on the ID features and output face noise through the portrait classifier-free guidance function; and to guide the scene generator based on the ID embedded scene features and output scene noise through the scene classifier-free guidance function. A face / scene attention tensor acquisition unit is used to process the face noise according to a set processing strategy to generate a face attention tensor; and to process the scene noise according to the set processing strategy to generate a scene attention tensor. An adaptive mask generation unit is used to generate an adaptive mask by taking the maximum index of the face attention tensor and the scene attention tensor; The noise prediction output calculation unit is used to calculate the noise prediction output based on the relationship between the diffusion sampling time step and a set threshold. A customized portrait image generation unit is used to perform diffusion sampling iteration based on the noise prediction output to generate a customized portrait image.

[0019] As a preferred embodiment of a customized person image generation device based on adaptive face replacement, the calculation formula for the compact ID feature in the ID feature and ID embedding scene feature acquisition unit is as follows:

[0020] In the formula, For compact ID features; ID features; To initialize the query vector; SM It is the softmax function; Dimensional permutation; The formula for calculating the ID embedding scene features is as follows:

[0021] In the formula, Embed scene features into the ID; For scene features; CAT This is a tensor concatenation function; N The length of the original token sequence for scene features; K The length of the token sequence for the compact ID feature.

[0022] As a preferred embodiment of a customized portrait image generation device based on adaptive face replacement, the expression of the portrait classifier-free guided function in the face noise and scene noise acquisition unit is as follows:

[0023] In the formula, Facial noise; For diffusion process t Noisy images at any given time; CFG is the core logic of the classifier-less bootstrap function; Parameters for the human portrait generator; The expression for the classifier-free bootstrapping function in the scenario is:

[0024] In the formula, This is scene noise; These are parameters for the scene generator; For text encoders; These are prompts for the target scene.

[0025] As a preferred embodiment of a customized human image generation device based on adaptive face replacement, in the face / scene attention tensor acquisition unit, during the process of generating the face attention tensor and the scene attention tensor through the set processing strategy, the set processing strategy includes performing element-level absolute value calculation, dimension reshaping, spatial scaling and normalization processing. The expression for the face attention tensor is:

[0026] In the formula, Face attention tensor; R For dimension reshaping functions; SM It is the softmax function; For the target dimension; It is a learnable significant factor; This is the initial attention tensor; The expression for the scene attention tensor is:

[0027] In the formula, For scene attention tensor; These are learnable significant factors.

[0028] As a preferred embodiment of a customized human image generation device based on adaptive face replacement, in the noise prediction output calculation unit, during the process of calculating the noise prediction output based on the relationship between the diffusion sampling time step and the set threshold, if the diffusion sampling time step is greater than the set threshold, then the noise prediction output is calculated by setting a calculation formula; if the diffusion sampling time step is not greater than the set threshold, then the scene noise is used as the noise prediction output. The set calculation formula is:

[0029] In the formula, For noise prediction output; M For adaptive masking.

[0030] The present invention has the following advantages: First, it balances identity preservation with scene collaboration: through deep collaboration between the FAR strategy and the IdDRE module, it accurately preserves the facial identity features of the target under the guidance of complex semantics, while fully depicting the character's full-body posture and complex scenes, thus achieving "character-scene collaborative narrative".

[0031] Second, it eliminates the need for manual intervention and improves efficiency: Adaptive masks replace manual annotation of face regions, avoiding defects such as edge fragmentation and artifacts, simplifying the operation process, and greatly improving generation efficiency and industrial adaptability.

[0032] Third, it has strong generalization ability: it can adapt to diverse scenarios such as side profile, occlusion, and dynamic pose, and the fine-tuning-free framework can be deployed based on any CPIG method, is compatible with multiple encoders and generative models, and has a wide range of application scenarios.

[0033] Fourth, the technical architecture is flexible and efficient: the lightweight architecture does not require fine-tuning of the core model, balances identity fidelity, scene integrity and generation speed, and reduces the threshold for technology implementation and deployment costs. Attached Figure Description

[0034] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0035] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0036] Figure 1 This is a flowchart illustrating the customized human image generation method based on adaptive face replacement provided in Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of the specific implementation process of the customized human image generation method based on adaptive face replacement provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram illustrating the qualitative evaluation results of the existing CPIG method and the present invention in one possible embodiment provided in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the architecture of the customized human image generation device based on adaptive face replacement provided in Embodiment 2 of the present invention. Detailed Implementation

[0037] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Example 1

[0039] See Figure 1 and Figure 2Embodiment 1 of the present invention provides a customized person image generation method based on adaptive face replacement, including the following steps: S1. Extract features from the reference image of the customized object using a human face encoder to obtain ID features; encode the target scene prompts using a text encoder to obtain scene features; based on the scene features, reduce the dimensionality of the ID features using an identity dimensionality reduction and embedding module to obtain compact ID features; embed the compact ID features into the scene features to obtain ID-embedded scene features. S2. Based on the ID features, guide the portrait generator and output face noise through the portrait classifier-free guidance function; based on the ID embedded scene features, guide the scene generator and output scene noise through the scene classifier-free guidance function. S3. Based on the facial noise, a facial attention tensor is generated by processing it according to a set processing strategy; based on the scene noise, a scene attention tensor is generated by processing it according to the set processing strategy. S4. Generate an adaptive mask by taking the maximum index of the face attention tensor and the scene attention tensor; S5. Calculate the noise prediction output based on the relationship between the diffusion sampling time step and the set threshold; S6. Based on the noise prediction output, perform diffusion sampling iteration to generate a customized portrait image.

[0040] In this embodiment, in step S1, the ID feature is obtained by extracting features from the reference image of the customized object through a human face encoder; the scene feature is obtained by encoding the target scene prompt words through a text encoder; based on the scene feature, the ID feature is reduced in dimensionality through an identity dimensionality reduction and embedding module to obtain a compact ID feature; the compact ID feature is embedded into the scene feature to obtain an ID-embedded scene feature.

[0041] Specifically, firstly, the facial encoder uses a model adapted to the arbitrary customized human image generation (CPIG) method. It takes 3-5 clear reference images of a customized object as input and captures core biometric features such as facial contours, facial structure, and skin texture through a feature extraction network. The output is an ID feature with dimensions B×N×D (B is the batch size, N is the token sequence length, and D is the feature dimension), ensuring the complete preservation of identity information. Secondly, the text encoder uses a model with strong semantic understanding capabilities to encode target scene prompts containing human actions and environmental details, converting natural language into machine-recognizable scene features. However, these scene features may implicitly contain virtual human identity information, which can easily interfere with the fidelity of the target identity.

[0042] Subsequently, in the Face Adaptive Replacement (FAR) strategy, to ensure that the generated scene image contains people, the corresponding scene cues typically include descriptions of people such as "a person" or "a man." In cases where there is no explicit reference person, the model generates a virtual character to demonstrate the completeness of the generated image. However, the introduction of a virtual character interferes with the identity of the target person in the CPIG task, resulting in a loss of fidelity in the target person's identity. Therefore, this invention, based on the FAR strategy, further proposes an Identity Dimensionality Reduction Embedding (IdDRE) module. This module aims to cover the virtual character information contained within the scene with the target ID features before using scene features to guide the scene generator in denoising, thereby improving the fidelity of the identity in the final generated image.

[0043] Specifically, for the custom object ID features provided by the CPIG method, the identity information is scattered across a 77-byte token sequence. Directly embedding this into the scene features can easily lead to confusion between the ID features and scene features, thus affecting the expression of ID and scene semantics in the generated image. Therefore, this invention proposes the IdDRE module, which first reduces the dimensionality of the scattered identity information in the ID features into a compact sequence, and then embeds it into the scene features in a concatenated form, achieving coverage of the native virtual character IDs in the scene features. For identity dimensionality reduction, the method takes the target ID feature as input and uses an attention mechanism to reduce the dimensionality to a length of... Compact ID features.

[0044] The formula for calculating the compact ID feature is as follows:

[0045] In the formula, For compact ID features; ID features; To initialize the query vector; SM It is the softmax function; Dimensional permutation.

[0046] After obtaining the compact features, they are concatenated and embedded into the starting position of the scene features to obtain the final ID embedded in the scene features.

[0047] The formula for calculating the ID embedding scene features is as follows:

[0048] In the formula, Embed scene features into the ID; For scene features; CAT This is a tensor concatenation function; NThe length of the original token sequence for scene features; K The length of the token sequence for the compact ID feature.

[0049] For the portion exceeding the specified length after splicing The first feature is directly discarded, and the invalid padding mechanism of the text encoder ensures that the scheme does not cause the loss of key features. Subsequently, the ID-embedded scene features will replace the original scene features to guide the scene generator in noise prediction. In summary, IdDRE uses the target ID features to cover the virtual character identity information implicit in the scene features, suppressing irrelevant identity interference from the source. This solves the problem of scene semantic interference caused by scattered ID features, and ensures a natural fit between the target ID and the scene expression through concatenation embedding, providing a "identity-scene collaboration" feature foundation for subsequent noise prediction.

[0050] In this embodiment, in step S2, the portrait generator is guided based on the ID feature, and the face noise is output through the portrait classifier-free guidance function; the scene generator is guided based on the ID embedded scene feature, and the scene noise is output through the scene classifier-free guidance function.

[0051] Specifically, for face noise generation, the face generator needs to select a model compatible with the CPIG method used to extract ID features in step S1. Using the ID features as a guiding signal, noise prediction is performed through a face-classifier-free guided function (CFG) with parameters specific to the generator. This function optimizes the noise generation direction by fusing ID features with model parameters, ensuring that the output face noise accurately corresponds to the identity features of the customized object, thus providing a foundation for subsequent face region generation.

[0052] For scene noise generation, the scene generator adopts a model with the same architecture as the portrait generator. Guided by the ID embedded scene features obtained in step S1, it replaces the original scene features to avoid interference from the virtual character's identity. Noise prediction is performed through a scene-free classifier-guided function that associates with the generator's proprietary parameters. This function fully analyzes the semantic information such as actions and environment in the scene features to generate scene noise that can completely present the target scene and the character's actions, achieving a preliminary separation and accurate characterization of identity requirements and scene requirements at the noise level.

[0053] The expression for the human face classifier-free bootstrapping function is as follows:

[0054] In the formula, Facial noise; For diffusion process t Noisy images at any given time; CFG is the core logic of the classifier-less bootstrap function; Parameters for the human portrait generator; The expression for the classifier-free bootstrapping function in the scenario is:

[0055] In the formula, This is scene noise; These are parameters for the scene generator; For text encoders; These are prompts for the target scene.

[0056] In this embodiment, in step S3, the face noise is processed by a set processing strategy to generate a face attention tensor; the scene noise is processed by the set processing strategy to generate a scene attention tensor.

[0057] Specifically, for facial noise, element-wise absolute value calculation is first performed to highlight salient regions within the noise. Then, a dimension reshaping function is used to adjust the noise to a specified dimension, followed by continuous spatial scaling to align the attention features with the spatial dimension of the noise prediction, resulting in an initial attention tensor. Finally, spatial normalization is applied to the initial attention tensor, and a learnable saliency factor is introduced to further enhance the saliency of the facial region, ultimately generating the facial attention tensor. Scene noise processing follows the same logic, extracting a scene attention tensor from the scene noise through the above four steps, highlighting key semantic regions in the scene, and ensuring that both types of attention tensors have consistent dimensions and formats, laying the foundation for subsequent tensor comparison and mask generation.

[0058] In this embodiment, taking facial noise as an example, element-level absolute value calculation, dimension reshaping, and spatial scaling are performed sequentially to obtain the initial attention tensor aligned with noise prediction as follows:

[0059] In the formula, This is the initial attention tensor; This indicates the calculation of element-level absolute values; The dimension reshaping function reshapes the input to... Specified dimension; This is a scaling operation for a continuous space, used to align the spatial dimensions of attention features with those of noise prediction; This is a compound function symbol that indicates that functions are executed sequentially from right to left.

[0060] To ensure the rationality of the attention tensor calculation, the initial attention tensor is further spatially normalized, and a learnable saliency factor is introduced. To enhance regional saliency, the formula is as follows: The expression for the face attention tensor is:

[0061] In the formula, Face attention tensor; R For dimension reshaping functions; SM It is the softmax function; For the target dimension; It is a learnable significant factor; This is the initial attention tensor.

[0062] In this embodiment, the expression for the scene attention tensor is:

[0063] In the formula, For scene attention tensor; These are learnable significant factors.

[0064] In this embodiment, in step S4, an adaptive mask is generated by taking the maximum index of the face attention tensor and the scene attention tensor.

[0065] Specifically, since the face attention tensor highlights salient regions related to the face, and the scene attention tensor highlights salient regions related to the scene, by taking the maximum index of the two tensors in the spatial dimension, it is possible to automatically determine whether each pixel in the image is more likely to belong to a face region or a scene region. Specifically, for each pixel in the image, its values ​​in the face attention tensor and scene attention tensor are compared. If the face attention tensor value is larger, the pixel is marked as belonging to a face region; if the scene attention tensor value is larger, it is marked as belonging to a scene region. The resulting binary matrix is ​​the adaptive mask. This mask eliminates the need for manual annotation of face regions and can adaptively adapt to diverse scenes such as side profiles, occlusions, and dynamic poses, effectively avoiding defects such as edge fragmentation and artifacts caused by manual annotation, and providing accurate region segmentation basis for subsequent noise fusion.

[0066] The formula for calculating the adaptive mask is as follows:

[0067] In the formula, M For adaptive masking.

[0068] In this embodiment, in step S5, the noise prediction output is calculated based on the relationship between the diffusion sampling time step and the set threshold.

[0069] Specifically, during the diffusion sampling process, the threshold value is set to a range of 10-20, which can be adaptively adjusted according to the scene complexity and identity fidelity requirements. When the diffusion sampling time step is no greater than the set threshold, the diffusion process is still in the critical stage of building the overall scene structure. At this time, scene noise is directly used as the noise prediction output to ensure that the core information such as the spatial logic and style tone of the scene can be fully presented, and to avoid prematurely introducing facial noise to interfere with the scene construction.

[0070] When the diffusion sampling time step is greater than the set threshold, the overall structure of the scene has basically taken shape. Then, the noise prediction output is calculated by setting a calculation formula. If the diffusion sampling time step is not greater than the set threshold, the scene noise is used as the noise prediction output. The set calculation formula is:

[0071] In the formula, For noise prediction output; M For adaptive masking.

[0072] The masked area uses facial noise to ensure accurate restoration of the target's identity features, while the masked area uses scene noise to maintain the integrity of the scene semantics and achieve a natural fusion of the facial area and the scene area.

[0073] In this embodiment, in step S6, based on the noise prediction output, diffusion sampling iteration is performed to generate a customized portrait image.

[0074] Specifically, a mainstream diffusion sampler is adopted, with a reasonable number of sampling steps set. The noise prediction result output in step S5 is used as the denoising guidance signal for each iteration. During the iteration process, the model gradually removes noise components from the image based on noise prediction, while continuously optimizing image details by combining identity information from ID features and semantic information from scene features. On the one hand, it accurately restores the facial features of customized objects to ensure identity recognition; on the other hand, it fully presents the spatial adaptation relationship between the person's full-body posture, body movements, and the target scene, solving problems such as style disjointedness and awkward transitions. After multiple rounds of diffusion sampling iterations, the final output is a customized character image that combines identity fidelity and scene-based narrative, meeting the personalized needs of film and television production, artistic creation, and other scenarios.

[0075] In one possible implementation, a verification example is provided as follows: The generation effects of this invention were compared with those of four other related works in four test cases (reference image + text prompt). Figure 3 As shown.

[0076] pass Figure 3 It can be seen that: First, existing CPIG methods only generate close-up facial images, resulting in background inconsistencies, such as mismatch between the scene and text prompts, incomplete character forms, such as lack of full-body poses, and inconsistent semantic expression.

[0077] Secondly, this invention breaks through the limitations of close-up facial images and can generate images containing complete human postures, actions, and complex scenes, fully realizing "human-scene collaborative narrative". The visual narrative integrity and richness are significantly better than existing methods.

[0078]

[0079] Table 1. Quantitative evaluation results of existing CPIG methods and the present invention.

[0080] Table 1 shows the quantitative evaluation results of existing CPIG methods and this invention on five commonly used evaluation metrics: Face sim, DINO, CLIP-I, CLIP-T, and FID. Face sim evaluates the facial similarity between the generated image and the reference image; DINO comprehensively evaluates image similarity and image-text alignment; CLIP-I evaluates pixel-level image similarity (reference image and generated image); CLIP-T evaluates image-text alignment; and FID evaluates the similarity to a dataset with complete scene representation and character imagery (lower values ​​are better). Given the core problem this invention aims to solve—overcoming the limitations of close-up facial images and achieving customized character image generation guided by complex semantics, with identity fidelity and collaborative generation of actions and scenes—the FID metric needs to be emphasized. This metric evaluates the generation effect of character imagery, actions, and scene completeness from the overall layout level of the generated image. Experimental results show that this invention achieves optimal or near-optimal results on the metrics evaluating facial similarity and image-text alignment, verifying the effectiveness of this method in balancing identity fidelity and complex scene generation. In terms of FID (Focus on Identification and Data Interpretation) metrics, this invention outperforms other existing works by several times, which fully demonstrates the advantages of this invention in "character posture integrity and scene coordination" and solves the problem of lack of scene narrative integrity.

[0081]

[0082] Table 2 User survey results of existing CPIG methods and the present invention

[0083] As shown in Table 2, the results of user surveys on four evaluation dimensions (based on a survey of 100 users, with a rating range of 0-100) are presented. The results show that the present invention is more popular with users in four aspects: identity similarity, text-image alignment (semantic consistency), image quality and visual aesthetics, and scene and character completeness. It is significantly better than existing methods, which fully demonstrates that the present invention meets users' core needs for "character-scene collaborative narrative".

[0084] The application scenarios of this invention are as follows: In the creation of customized scenes for film and television characters, this invention can accurately preserve the core facial features of actors and integrate them into various fictional scenes and action settings such as ancient costumes and science fiction, efficiently generating customized character images that meet the needs of the plot and reducing the cost of scene construction and shooting.

[0085] In the context of artistic portrait creation, this invention can deeply integrate a specific character image with an abstract art style and a personalized scene based on the artist's textual creative prompts, generating customized works that combine identity recognition with artistic expression, thus expanding the boundaries of creation.

[0086] In virtual scene interaction scenarios, this invention can embed user facial features into virtual environments such as games and metaverses to generate personalized virtual images that adapt to the action logic and visual style of virtual scenes, thereby enhancing user immersion and interactive experience.

[0087] In advertising and marketing scenarios, this invention can quickly integrate the facial features of brand ambassadors or target users into images corresponding to different product usage scenarios and promotional copy, generating customized advertising materials that fit the brand's tone and improving marketing efficiency.

[0088] In the creative compositing of wedding photography scenes, this invention can accurately integrate the facial features of the couple with scenic spots and unique scenes from around the world according to the needs of the newlyweds, generating personalized wedding images that do not require on-site shooting, thus enriching shooting options and creative expression.

[0089] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0090] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0091] Example 2

[0092] See Figure 4 Embodiment 2 of the present invention also provides a customized person image generation device based on adaptive face replacement, comprising: ID feature and ID embedding scene feature acquisition unit 001 is used to extract features from the reference image of the customized object through a human face encoder to obtain ID features; encode the target scene prompt words through a text encoder to obtain scene features; based on the scene features, reduce the dimensionality of the ID features through an identity dimensionality reduction and embedding module to obtain compact ID features; and embed the compact ID features into the scene features to obtain ID embedding scene features. The face noise and scene noise acquisition unit 002 is used to guide the portrait generator based on the ID features and output face noise through the portrait classifier-free guidance function; and to guide the scene generator based on the ID embedded scene features and output scene noise through the scene classifier-free guidance function. The face / scene attention tensor acquisition unit 003 is used to generate a face attention tensor by processing the face noise according to a set processing strategy; and to generate a scene attention tensor by processing the scene noise according to the set processing strategy. The adaptive mask generation unit 004 is used to generate an adaptive mask by taking the maximum index of the face attention tensor and the scene attention tensor. The noise prediction output calculation unit 005 is used to calculate the noise prediction output based on the relationship between the diffusion sampling time step and the set threshold. The customized portrait image generation unit 006 is used to perform diffusion sampling iteration based on the noise prediction output to generate a customized portrait image.

[0093] In this embodiment, in the ID feature and ID embedding scene feature acquisition unit 001, the calculation formula for the compact ID feature is as follows:

[0094] In the formula, For compact ID features; ID features; To initialize the query vector; SM It is the softmax function; Dimensional permutation; The formula for calculating the ID embedding scene features is as follows:

[0095] In the formula, Embed scene features into the ID; For scene features; CAT This is a tensor concatenation function; N The length of the original token sequence for scene features; K The length of the token sequence for the compact ID feature.

[0096] In this embodiment, the expression of the portrait classifier-free guided function in the face noise and scene noise acquisition unit 002 is as follows:

[0097] In the formula, Facial noise; For diffusion process t Noisy images at any given time; CFG is the core logic of the classifier-less bootstrap function; Parameters for the human portrait generator; The expression for the classifier-free bootstrapping function in the scenario is:

[0098] In the formula, This is scene noise; These are parameters for the scene generator; For text encoders; These are prompts for the target scene.

[0099] In this embodiment, in the face / scene attention tensor acquisition unit 003, during the process of generating the face attention tensor and the scene attention tensor by processing through the set processing strategy, the set processing strategy includes performing element-level absolute value calculation, dimension reshaping, spatial scaling and normalization processing. The expression for the face attention tensor is:

[0100] In the formula, Face attention tensor; R For dimension reshaping functions; SM It is the softmax function; For the target dimension; It is a learnable significant factor; This is the initial attention tensor; The expression for the scene attention tensor is:

[0101] In the formula, For scene attention tensor; These are learnable significant factors.

[0102] In this embodiment, in the noise prediction output calculation unit 005, during the process of calculating the noise prediction output based on the relationship between the diffusion sampling time step and the set threshold, if the diffusion sampling time step is greater than the set threshold, the noise prediction output is calculated by setting a calculation formula; if the diffusion sampling time step is not greater than the set threshold, the scene noise is used as the noise prediction output. The set calculation formula is:

[0103] In the formula, For noise prediction output; M For adaptive masking.

[0104] It should be noted that the information interaction and execution process between the modules of the above system are based on the same concept as the method embodiment in Embodiment 1 of this application, and the resulting technical effects are the same as those in the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, and it will not be repeated here.

[0105] Example 3

[0106] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium storing program code for a customized person image generation method based on adaptive face replacement. The program code includes instructions for executing the customized person image generation method based on adaptive face replacement of Embodiment 1 or any possible implementation thereof.

[0107] Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0108] Example 4

[0109] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor; The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor can call the program instructions to execute the customized human image generation method based on adaptive face replacement according to Embodiment 1 or any possible implementation thereof.

[0110] Specifically, a processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0111] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0112] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing systems. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Optionally, they can be implemented using program code executable by a computing system, thereby storing them in a storage system for execution by the computing system. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0113] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A customized person image generation method based on adaptive face replacement, characterized in that, include: The ID features are obtained by extracting features from the reference image of the customized object using a human face encoder. The target scene prompts are encoded using a text encoder to obtain scene features; Based on the scene features, the ID features are reduced in dimensionality using the identity dimensionality reduction and embedding module to obtain compact ID features; the compact ID features are then embedded into the scene features to obtain ID-embedded scene features. Based on the ID features, the portrait generator is guided to output face noise through the portrait classifier-free guidance function. The scene generator is guided by the embedded scene features of the ID, and the scene noise is output through the scene classifier-free guidance function. Based on the aforementioned facial noise, a facial attention tensor is generated by processing it according to a set processing strategy. Based on the scene noise, the scene attention tensor is generated by processing it according to the set processing strategy. An adaptive mask is generated by taking the maximum index of the face attention tensor and the scene attention tensor; The noise prediction output is calculated based on the relationship between the diffusion sampling time step and the set threshold. Based on the noise prediction output, diffusion sampling iteration is performed to generate a customized portrait image.

2. The customized character image generation method based on adaptive face replacement according to claim 1, characterized in that, The formula for calculating the compact ID feature is as follows: In the formula, For compact ID features; ID features; To initialize the query vector; SM It is the softmax function; Dimensional permutation; The formula for calculating the ID embedding scene features is as follows: In the formula, Embed scene features into the ID; For scene features; CAT This is a tensor concatenation function; N The length of the original token sequence for scene features; K The length of the token sequence for the compact ID feature.

3. The customized character image generation method based on adaptive face replacement according to claim 2, characterized in that, The expression for the image classifier-free bootstrapping function is: In the formula, Facial noise; For diffusion process t Noisy images at any given time; CFG is the core logic of the classifier-less bootstrap function; Parameters for the human portrait generator; The expression for the classifier-free bootstrapping function in the scenario is: In the formula, This is scene noise; These are parameters for the scene generator; For text encoders; These are prompts for the target scene.

4. The customized character image generation method based on adaptive face replacement according to claim 3, characterized in that, In the process of generating the face attention tensor and the scene attention tensor through the set processing strategy, the set processing strategy includes element-level absolute value calculation, dimension reshaping, spatial scaling and normalization processing. The expression for the face attention tensor is: In the formula, Face attention tensor; R For dimension reshaping functions; SM It is the softmax function; For the target dimension; It is a learnable significant factor; This is the initial attention tensor; The expression for the scene attention tensor is: In the formula, For scene attention tensor; These are learnable significant factors.

5. The customized character image generation method based on adaptive face replacement according to claim 4, characterized in that, In the process of calculating the noise prediction output based on the relationship between the diffusion sampling time step and the set threshold, if the diffusion sampling time step is greater than the set threshold, the noise prediction output is calculated by the set calculation formula; if the diffusion sampling time step is not greater than the set threshold, the scene noise is used as the noise prediction output. The set calculation formula is: In the formula, For noise prediction output; M For adaptive masking.

6. A customized person image generation device based on adaptive face replacement, employing the customized person image generation method based on adaptive face replacement as described in any one of claims 1-5, characterized in that, include: The ID feature and ID embedding scene feature acquisition unit is used to extract features from the reference image of the customized object through the portrait encoder to obtain ID features; The target scene prompts are encoded using a text encoder to obtain scene features; Based on the scene features, the ID features are reduced in dimensionality using the identity dimensionality reduction and embedding module to obtain compact ID features; the compact ID features are then embedded into the scene features to obtain ID-embedded scene features. The face noise and scene noise acquisition unit is used to guide the portrait generator based on the ID features and output face noise through the portrait classifier-free guidance function; and to guide the scene generator based on the ID embedded scene features and output scene noise through the scene classifier-free guidance function. A face / scene attention tensor acquisition unit is used to process the face noise based on the set processing strategy to generate a face attention tensor. Based on the scene noise, the scene attention tensor is generated by processing it according to the set processing strategy. An adaptive mask generation unit is used to generate an adaptive mask by taking the maximum index of the face attention tensor and the scene attention tensor; The noise prediction output calculation unit is used to calculate the noise prediction output based on the relationship between the diffusion sampling time step and a set threshold. A customized portrait image generation unit is used to perform diffusion sampling iteration based on the noise prediction output to generate a customized portrait image.

7. The customized person image generation device based on adaptive face replacement according to claim 6, characterized in that, In the ID feature and ID embedding scene feature acquisition unit, the calculation formula for the compact ID feature is as follows: In the formula, For compact ID features; ID features; To initialize the query vector; SM It is the softmax function; Dimensional permutation; The formula for calculating the ID embedding scene features is as follows: In the formula, Embed scene features into the ID; For scene features; CAT This is a tensor concatenation function; N The length of the original token sequence for scene features; K The length of the token sequence for the compact ID feature.

8. The customized person image generation device based on adaptive face replacement according to claim 7, characterized in that, In the face noise and scene noise acquisition unit, the expression of the image classifier-free guided function is: In the formula, Facial noise; For diffusion process t Noisy images at any given time; CFG is the core logic of the classifier-less bootstrap function; Parameters for the human portrait generator; The expression for the classifier-free bootstrapping function in the scenario is: In the formula, This is scene noise; These are parameters for the scene generator; For text encoders; These are prompts for the target scene.

9. The customized person image generation device based on adaptive face replacement according to claim 8, characterized in that, In the face / scene attention tensor acquisition unit, during the process of generating the face attention tensor and the scene attention tensor through the set processing strategy, the set processing strategy includes element-level absolute value calculation, dimension reshaping, spatial scaling and normalization processing. The expression for the face attention tensor is: In the formula, Face attention tensor; R For dimension reshaping functions; SM It is the softmax function; For the target dimension; It is a learnable significant factor; This is the initial attention tensor; The expression for the scene attention tensor is: In the formula, For scene attention tensor; These are learnable significant factors.

10. The customized person image generation device based on adaptive face replacement according to claim 9, characterized in that, In the noise prediction output calculation unit, during the process of calculating the noise prediction output based on the relationship between the diffusion sampling time step and the set threshold, if the diffusion sampling time step is greater than the set threshold, the noise prediction output is calculated by the set calculation formula; if the diffusion sampling time step is not greater than the set threshold, the scene noise is used as the noise prediction output. The set calculation formula is: In the formula, For noise prediction output; M For adaptive masking.

Images