Artificial intelligence-based virtual avatar generation method and system

Abstract

The application relates to the technical field of artificial intelligence and computer vision, and provides a virtual image generation method and system based on artificial intelligence, which comprises the following steps: obtaining text prompt data and element structure prior information and visual identity representation information of a target role; taking the element structure prior information as a spatial hard constraint to inject a pre-trained diffusion model, and taking the visual identity representation information as a semantic soft constraint to inject the diffusion model; inputting the text prompt data into the diffusion model for iterative denoising, outputting an initial image, and performing feature alignment correction on a current latent variable based on reference feature representation of a target element during the iterative denoising process; performing feature deviation detection and positioning on the initial image to obtain a deviation region; performing local redrawing repair in the deviation region based on a standard element image until a preset convergence condition is met, and obtaining a virtual image. The application realizes accurate pixel-level maintenance and full-automatic closed-loop correction of a virtual image landmark local feature.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and computer vision technology, specifically to a method and system for generating virtual avatars based on artificial intelligence. Background Technology

[0002] Currently, in the field of virtual avatar generation, text-based image generation techniques, represented by diffusion models, typically use text prompts to drive image generation. However, when generating images in batches for IP characters with strict brand specifications, issues such as color changes, proportional imbalances, or structural loss may still occur in iconic local visual elements (e.g., specific hair ornaments, emblems, etc.). Existing technologies struggle to apply pixel-level constraints to minute local elements, structural control methods rely on manually prepared line art and cannot constrain internal textures, and image prompting methods are prone to blurring at small details. This results in generated content failing to meet the application requirements of IP consistency. Therefore, there is an urgent need for a virtual avatar generation solution that can automatically maintain precise consistency in the core visual features of the character. Summary of the Invention

[0003] To address the problem of poor IP consistency caused by the drift of iconic local elements during the virtual character generation process in existing technologies, this application proposes an artificial intelligence-based virtual character generation method and system to achieve automated and accurate preservation and closed-loop correction of the visual features of the target character.

[0004] To achieve the above objectives, this application adopts the following technical solution: A method for generating virtual avatars based on artificial intelligence, comprising: Obtain text prompt data and prior information on the element structure and visual identity representation of the target character; The prior information of the element structure is injected into the pre-trained diffusion model as a hard spatial constraint, and the visual identity representation information is injected into the diffusion model as a soft semantic constraint. The text prompt data is input into the diffusion model for iterative denoising, and an initial image is output; wherein, during the iterative denoising process, feature alignment correction is performed on the current latent variables based on the reference feature representation of the target element; the target element is the iconic local element of the target character; The initial image is subjected to feature deviation detection and localization to obtain the deviation region; Within the deviation region, the initial image is locally redrawn and repaired based on the standard element image of the target element until the preset convergence condition is met, thus obtaining the virtual image image.

[0005] The above scheme decouples the global identity semantics from the local structural space and introduces a real-time alignment correction mechanism based on reference features during the denoising process. Combined with subsequent automated deviation detection and local redrawing repair, it achieves end-to-end feature consistency control from generation to review and correction, effectively eliminating feature drift.

[0006] As one implementation, the prior information of the element structure includes the pixel-level mask of the target element, and the spatial positional relationship and size ratio of the target element relative to the facial key points of the target character; The visual identity representation information is obtained by pooling and fusing the identity features of multiple reference images of the target character that have been pre-acquired; The step of injecting the prior information of the element structure as a hard spatial constraint into the pre-trained diffusion model and injecting the visual identity representation information as a soft semantic constraint into the diffusion model includes: The prior information of the element structure is encoded and then sent to the structure control network of the diffusion model to apply spatial hard constraints to the generation process of the diffusion model. The visual identity representation information is injected through the cross-attention layer of the diffusion model to apply semantic soft constraints.

[0007] This implementation constructs a structural prior that includes relative spatial relationships and size proportions, and utilizes pooled identity features to enable the model to maintain the geometric stability and identity robustness of elements under different poses and perspectives, thus avoiding the deformation problem caused by a single absolute coordinate constraint.

[0008] As one implementation method, the reference feature representation based on the target element performs feature alignment correction on the current latent variable, including: In the iterative denoising loop, multiple key time steps are selected for anchoring operations; For each key time step, extract the local features corresponding to the target element from the intermediate layer feature map obtained by decoding the current latent variable through the diffusion model; Calculate the perceptual loss between the local features and the reference feature representation of the target element; wherein the reference feature representation is a deep feature of the target element pre-extracted from a reference image of the target character; The current latent variable is updated using gradients based on the perceptual loss; wherein the update step size of the gradient update is dynamically adjusted according to the current time step and the feature attributes of the target element.

[0009] This implementation introduces perceptual loss gradient backpropagation at key nodes of the denoising loop, directly correcting feature shifts at the latent space level. Compared to post-processing only at the output, it can ensure accurate restoration of details from the generation source.

[0010] As one implementation, the anchoring operation employs a layered progressive anchoring strategy; wherein, the layered progressive anchoring strategy includes: In the deep feature map of the UNet decoder of the diffusion model, global identity features obtained by mapping the visual identity representation information are used for anchoring; In the mid-layer feature map of the UNet decoder, the element category features determined based on the prior information of the element structure are used for anchoring; wherein, the element category features correspond to the component category to which the target element belongs; In the shallow feature map of the UNet decoder, the target element is anchored using pixel-level texture features extracted based on the reference feature representation.

[0011] This implementation leverages the characteristic that different levels of the UNet decoder correspond to different semantic granularities, enabling precise layered control from overall style and component structure to micro-texture, thus avoiding generation artifacts caused by cross-scale feature interference.

[0012] As one implementation, the update step size of the gradient update is dynamically adjusted based on the current time step and the feature attributes of the target element, including: The update step size is adjusted based on the degree of deviation between the current time step and the perceptual loss representation, using an annealing strategy. Specifically, when the current time step belongs to the first half of the total denoising steps, a preset first step size is used for rapid correction; when the current time step belongs to the second half of the total denoising steps, a preset second step size smaller than the first step size is used for fine correction. The first half and the second half are obtained by dividing the total denoising steps based on a preset ratio coefficient. The degree of deviation is one of the feature attributes.

[0013] This implementation uses an annealing strategy to dynamically adjust the optimization intensity, which ensures that the generation can quickly converge to the correct feature manifold in the early stage, and prevents the already formed fine texture structure from being destroyed due to excessive step size in the later stage, thus improving the stability of generation.

[0014] As one implementation method, the feature deviation detection and localization of the initial image includes: The initial image is input into a pre-trained multimodal model. The multimodal model uses a hierarchical questioning strategy to examine each preset element region in the initial image to check whether the color, shape, and details conform to the standard element image of the target element. When feature deviations are detected, the model outputs deviation descriptions and location information. The standard element image is a local image that is pre-cropped and extracted from the reference image of the target character and contains only the target element. The positioning information is semantically segmented and remapped to obtain a binary mask of the deviation region; The local redrawing and repair of the initial image based on the standard element image of the target element includes: The diffusion model is reused, and denoising is performed on the region of the binary mask based on the deviation description. During the denoising process, the standard element image of the target element is injected as a local image condition.

[0015] This implementation constructs an automated visual review closed loop based on a multimodal large model. By converting natural language localization into a precise binary mask and combining it with standard element images as strong conditions for local redrawing, it achieves pixel-level self-repair of defects without human intervention.

[0016] As one implementation, before inputting the initial image into the pre-trained multimodal model, the method further includes: A coarse- and fine-grained cascaded review network is adopted: first, a lightweight image classification network screens the initial image for defects to determine whether there are any suspicious defects; only when the lightweight image classification network confirms the existence of a suspicious defect is the initial image input into the multimodal model for fine attribution and localization. The binary mask of the deviation region is further expanded at the edges through morphological dilation.

[0017] This implementation balances detection accuracy and computational efficiency through a cascaded review mechanism. At the same time, it performs morphological dilation processing on the mask to ensure that the redrawn area can cover the edge transition zone and avoid obvious splicing marks after repair.

[0018] As one implementation, after performing denoising processing on the region of the binary mask based on the deviation description, the method further includes: The region boundaries of the binary mask are subjected to Poisson blending or edge feathering; wherein, the pixel width of the edge feathering is adaptively determined according to the size of the binary mask.

[0019] This implementation method further eliminates the differences in lighting and tone between the locally redrawn area and the original background through adaptive edge blending processing, ensuring the visual naturalness and overall harmony of the final output image.

[0020] In one implementation, the diffusion model is a model fine-tuned by identity adaptation; wherein, during the identity adaptation fine-tuning process, the network layers related to geometric deformation in the diffusion model are frozen, and only the normalization layer and attention mapping layer related to texture and color are updated.

[0021] This implementation selectively freezes geometry-related layers, which, while giving the model specific IP textures and color schemes, preserves the original pose generalization ability and compositional diversity of the base model, thus solving the overfitting and pattern collapse problems caused by traditional full-scale fine-tuning.

[0022] Furthermore, this application also provides an artificial intelligence-based virtual avatar generation system for implementing the artificial intelligence-based virtual avatar generation method described above, including: The data acquisition module is used to acquire text prompt data and prior information on the element structure and visual identity representation of the target character; The decoupled injection module is used to inject the prior information of the element structure as a hard spatial constraint into the pre-trained diffusion model, and to inject the visual identity representation information as a soft semantic constraint into the diffusion model. The feature anchoring module is used to input the text prompt data into the diffusion model for iterative denoising and output an initial image; wherein, during the iterative denoising process, feature alignment correction is performed on the current latent variables based on the reference feature representation of the target element; the target element is the iconic local element of the target character; The deviation localization module is used to detect and locate feature deviations in the initial image to obtain the deviation region; The local redrawing module is used to locally redraw and repair the initial image within the deviation area based on the standard element image of the target element until a preset convergence condition is met, thereby obtaining a virtual image image.

[0023] The system achieves automated execution of the entire virtual character generation process through a modular architecture. The collaborative work of each module ensures high-fidelity reproduction of IP character characteristics and improves production efficiency.

[0024] Beneficial effects: 1. This application solves the feature drift problem from the generation mechanism by decoupling the element structure prior and visual identity representation, combined with feature alignment correction in the iterative denoising process. This makes the shape, color and details of the iconic local elements pixel-level accurate and consistent, meeting the stringent quality requirements of IP derivative content.

[0025] 2. This application constructs a closed-loop visual review and local redrawing repair mechanism based on a multimodal model, which can automatically detect and iteratively correct residual defects, replacing the traditional manual image selection and retouching process, and significantly improving production efficiency and yield in batch generation scenarios.

[0026] 3. This application adopts a layered progressive anchoring strategy and an annealing gradient update step size, which takes into account both global style consistency and local texture accuracy. At the same time, it retains geometric generalization ability through identity adaptation fine-tuning strategy, so that new characters can be quickly integrated with only a few reference images, which has strong generalization and low usage threshold. Attached Figure Description

[0027] Figure 1 A flowchart illustrating an artificial intelligence-based virtual avatar generation method provided in this application embodiment; Figure 2 A schematic flowchart illustrating the feature alignment correction method provided in this application embodiment; Figure 3 This is a system architecture diagram of an artificial intelligence-based virtual avatar generation system provided for embodiments of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0030] Example 1 like Figure 1 As shown, this application provides a method for generating virtual avatars based on artificial intelligence. This method aims to solve the feature drift problem that easily occurs in existing text-based image generation technologies when generating specific IP characters by constructing a full-link processing flow from data acquisition, dual-constraint injection, latent space feature correction to closed-loop detection and repair. The method includes the following steps: Step S100: Obtain text prompt data and prior information on the element structure and visual identity representation of the target character.

[0031] Specifically, text prompt data is used to describe high-level semantic content such as the desired generated scene, action, or style. Prior element structural information refers to structured data extracted in advance from reference materials of the target character to characterize the geometric attributes of its distinctive local elements. This may include pixel-level masks, keypoint coordinates, relative positional relationships, or size proportions, providing precise spatial topological constraints. Visual identity representation information is an abstract semantic vector obtained through feature extraction and fusion of multiple reference images, used to express the character's overall style, color scheme, and identity traits. Separating structural and identity information lays the data foundation for subsequent decoupled control.

[0032] Step S200: Inject the prior information of element structure as a hard spatial constraint into the pre-trained diffusion model, and inject the visual identity representation information as a soft semantic constraint into the diffusion model.

[0033] Specifically, hard spatial constraints impose strict geometric boundary restrictions on the latent space of the generated image, forcing the generated content to conform to preset shape and position specifications within a specific region, preventing structural deformation. Soft semantic constraints, on the other hand, guide the model to focus on specific identity features through attention mechanisms and other methods, allowing for natural variations in texture and lighting within a certain range. In one implementation, hard spatial constraints can be implemented using a structural control network (such as a ControlNet-like architecture), while soft semantic constraints can be injected through cross-attention layers. The advantage of this dual-constraint collaborative mechanism is that hard constraints are responsible for building a stable "skeleton," ensuring that iconic elements do not shift or deform; soft constraints are responsible for filling in rich "flesh and blood," maintaining the character's spirit and texture. The two interact orthogonally within the model, avoiding the rigidity of generation caused by overly strong constraints or the loss of features caused by overly weak constraints, thus preserving the diversity of generation while ensuring IP consistency.

[0034] Step S300: Input the text prompt data into the diffusion model for iterative denoising and output the initial image; wherein, during the iterative denoising process, feature alignment correction is performed on the current latent variables based on the reference feature representation of the target element; the target element is the iconic local element of the target character.

[0035] Specifically, iterative denoising is the core process of image generation using the diffusion model. The feature alignment correction introduced in this process is essentially a real-time optimization operation performed at the latent space level, rather than post-processing at the final pixel level. The latent space contains a high-dimensional semantic compressed representation of the image. Within this space, the perceptual loss is calculated using the reference feature representations of target elements (such as specific hair ornaments, badges, tie knots, etc.) and the gradient is backpropagated. This directly corrects the direction of the generated manifold, bringing it closer to standard features. Compared to pixel-level repair, latent space correction offers higher semantic consistency and computational efficiency, suppressing feature drift from the generation source and avoiding problems such as "similar in form but not in spirit" or obvious edge stitching marks. The specific type of target element can be flexibly configured according to IP operation needs; any key visual symbols that constitute character recognition can be included in the correction scope.

[0036] Step S400: Perform feature deviation detection and localization on the initial image to obtain the deviation region.

[0037] Specifically, due to the randomness of the generative model, even after the aforementioned corrections, the initial image may still contain subtle color deviations, texture blurring, or localized defects. The feature deviation detection and localization step aims to automatically identify these defects. The detection subject can be a pre-trained multimodal large model, a dedicated image classifier, or a traditional computer vision algorithm, or a combination thereof. The localization result is usually represented by a mask or bounding box indicating the location of the deviation. This step constitutes a key link in automated quality control, transforming the subjective judgment that originally relied on human visual review into objective, quantifiable machine signals, providing a clear target area for subsequent precise repair.

[0038] In step S500, within the deviation region, the initial image is locally redrawn and repaired based on the standard element image of the target element until the preset convergence condition is met, thereby obtaining the virtual image image.

[0039] Specifically, local redrawing and restoration refers to re-generating only the deviated areas of the image while keeping other areas unchanged. The standard element image of the target element is injected as a strong condition to ensure that the restored content strictly conforms to IP specifications. The standard element image is typically a "gold standard" material that has undergone manual verification or high-precision cropping. Preset convergence conditions are the basis for determining whether the restoration process should terminate, and these conditions can take various forms. For example, a preset convergence condition could be reaching a preset maximum iteration threshold to prevent infinite loops from consuming computational power; it could also be that the confidence score or similarity score returned by the deviation detection module is higher than a preset quality threshold, indicating that the restoration result has met the standard; or it could be that no new deviations are found in several consecutive detections. Through this closed-loop iterative mechanism, the system can adaptively eliminate residual defects until a fully compliant virtual avatar image is output, thus achieving a technological leap from "probabilistic generation" to "deterministic delivery."

[0040] Example 2 Building upon Example 1, this example further elaborates on the construction details of prior information on element structure and visual identity representation information, as well as their specific injection methods into the model. As one implementation method, the prior information on element structure includes a pixel-level mask of the target element, and the spatial positional relationship and size proportion of the target element relative to the facial key points of the target character. Specifically, the pixel-level mask provides the precise binary contour boundary of the target element, serving as the foundational data for subsequent spatial constraints. Compared to traditional absolute coordinate positioning, this application uses the spatial positional relationship relative to facial key points (such as the center of the eyes, the tip of the nose, or the chin point) to describe the element's position. This relative coordinate system design effectively resists absolute positional shifts caused by rotation, pitch, or perspective transformation of the character's head during generation, ensuring that accessories such as hair ornaments and glasses always follow the movement of the main facial structure, avoiding the visual error of "head turning but ornaments not turning." Simultaneously, the size proportion parameter is introduced to maintain the harmony between the target element and the overall facial proportions, preventing the generation of distorted elements that are too large or too small under different resolutions or compositions.

[0041] As one implementation method, visual identity representation information is obtained by pooling and fusing the identity features of multiple reference images of a pre-acquired target character. Specifically, a single reference image is often limited by specific lighting conditions, shooting angle, or background interference, and the directly extracted feature vectors may contain a large amount of noise unrelated to IP identity. Therefore, this application adopts a multi-source data fusion strategy. First, a pre-trained feature extraction network (e.g., a CLIP image encoder fine-tuned for anime style) is used to extract high-dimensional feature vectors from multiple reference images, and then pooling operations are performed on these vectors. The pooling fusion method can be a simple arithmetic average, a weighted average based on image quality scores, or a statistical aggregation of features extracted after data enhancement processing such as random cropping, rotation, and color dithering of the reference images. The visual identity representation information obtained through this fusion mechanism is essentially a high-level abstraction and purification of the core visual characteristics of the target character (such as specific color schemes, line styles, and facial features), stripping away accidental scene factors, thereby providing robust and generalizable identity semantic guidance for the generation process.

[0042] Having clarified the construction methods of the two types of information mentioned above, this embodiment further elaborates on their differentiated injection paths in the diffusion model. As one implementation method, the element structure prior information is encoded and fed into the structure control network of the diffusion model to impose hard spatial constraints on the generation process of the diffusion model; visual identity representation information is injected through the cross-attention layer of the diffusion model to impose soft semantic constraints. Specifically, the structure control network typically adopts a bypass branch architecture (such as a ControlNet-like network) that runs in parallel or in series with the main UNet. After receiving the encoded structure prior information, it superimposes the spatial constraint signals onto the corresponding layers of the main network through zero-convolution or similar learnable adaptation layers. This injection method directly imposes geometric boundary constraints on the feature map of the latent space, forcing the generated content to conform to preset shape and position specifications within a specific region, exhibiting strong determinism and resistance to being covered by text prompts. In contrast, the visual identity representation information is mapped into key-value pairs and injected into the cross-attention layer of the UNet decoder, participating in attention calculation together with the text conditional embedding. This approach is a probabilistic semantic guidance method, which allows the model to flexibly adjust local texture details and lighting effects based on text prompts while maintaining overall identity consistency.

[0043] This embodiment achieves a dynamic balance between spatial geometric stability and identity semantic robustness through the aforementioned decoupled injection mechanism. The structural control network is responsible for building an insurmountable "skeleton," ensuring that iconic elements do not undergo structural displacement or deformation; the cross-attention layer is responsible for filling in the flexible "flesh and blood," giving the character vivid texture and style. The two interact orthogonally within the model, avoiding the stiffness and texture loss caused by relying solely on structural constraints, and overcoming the detail drift and feature loss caused by relying solely on semantic constraints. Thus, while ensuring strict IP compliance, it maximizes the preservation of artistic diversity and creative freedom in AI generation.

[0044] Example 3 Building upon Example 1, this example further details the feature alignment correction mechanism in the iterative denoising process. As one implementation method, feature alignment correction is performed on the current latent variable based on the reference feature representation of the target element, such as... Figure 2 As shown, the process includes: Step S301, selecting multiple key time steps for anchoring operations in the iterative denoising loop; Step S302, for each key time step, extracting local features corresponding to the target element from the intermediate layer feature map obtained by decoding the current latent variable through the diffusion model; Step S303, calculating the perceptual loss between the local features and the reference feature representation of the target element; wherein, the reference feature representation is a deep feature of the target element pre-extracted from the reference image of the target character; Step S304, updating the gradient of the current latent variable based on the perceptual loss; wherein, the update step size of the gradient update is dynamically adjusted according to the current time step and the feature attributes of the target element. Specifically, the anchoring operation is not performed in every denoising step, but rather at several key time steps (e.g., 20%, 50%, and 80% of the total steps) to balance computational overhead and correction effect. When extracting local features, spatial alignment algorithms such as ROI Align are employed. Based on the mask coordinates provided by prior knowledge of the element structure, feature vectors of the target element region are accurately extracted from the multi-scale feature map output by the UNet decoder. This operation effectively resists feature misalignment caused by pose changes during generation. The perceptual loss is preferably calculated using LPIPS (Learned Perceptual Image Patch Similarity) or cosine similarity. Compared to the traditional mean squared error (MSE), these metrics better align with the human visual system's perception of structure and texture, guiding latent variables to converge towards a standard feature manifold that conforms to human visual cognition. Gradient update involves differentiating the perceptual loss with respect to the current latent variable and directly correcting the latent space representation along the negative gradient direction. This real-time correction at the generation source offers higher semantic consistency and edge naturalness compared to post-processing restoration.

[0045] As one implementation method, the anchoring operation employs a layered progressive anchoring strategy. This strategy includes: anchoring in the deep feature maps of the UNet decoder of the diffusion model using global identity features mapped from visual identity representation information; anchoring in the mid-level feature maps of the UNet decoder using element category features determined based on prior information about element structure; where the element category features correspond to the component category to which the target element belongs; and anchoring the target element in the shallow feature maps of the UNet decoder using pixel-level texture features extracted based on reference feature representations. Specifically, different levels of the UNet decoder correspond to different semantic granularities and spatial resolutions. Deep feature maps have lower resolution but higher semantic abstraction; using global identity features for anchoring at this layer ensures that the overall style, tone, and character essence of the generated image remain consistent, laying the correct foundation for subsequent detail generation. Mid-level feature maps carry component-level structural information; using element category features (such as general features of categories like "hair ornament" and "badge") for anchoring at this layer constrains the geometric shape and spatial layout of the target element, preventing structural distortions. Shallow feature maps offer high resolution and rich texture details. Anchoring at this layer directly utilizes pixel-level texture features extracted from standard element images, accurately reproducing the microscopic texture of iconic elements, such as metallic sheen and fabric texture. This layered, progressive strategy follows a generation process from the whole to the part, from abstract to concrete, effectively avoiding cross-level feature interference caused by single-scale anchoring and significantly improving the accuracy of reproducing complex IP elements.

[0046] As one implementation method, the update step size of gradient update is dynamically adjusted based on the current time step and the feature attributes of the target element. This includes: adjusting the update step size based on the degree of deviation between the current time step and the perceptual loss representation using an annealing strategy; specifically, when the current time step belongs to the first half of the total denoising steps, a preset first step size is used for rapid correction; when the current time step belongs to the second half of the total denoising steps, a preset second step size smaller than the first step size is used for fine correction; the first and second halves are divided based on a preset proportional coefficient, and the degree of deviation is one of the feature attributes. Specifically, the core idea of ​​the annealing strategy is to simulate a physical cooling process, causing the optimization intensity to gradually decrease as the generation process progresses. The preset proportional coefficient is used to define the boundary between the first and second halves, for example, it can be set to 0.5 or 0.6, meaning that when the total number of steps is 30, the first 15 or 18 steps constitute the first half. In the first half, the generated content is still in a rough shaping stage, and feature deviations are usually large. At this time, using a larger first step size (e.g., 0.5) can quickly bring latent variables closer to the target feature manifold, improving convergence efficiency. In the second half, image details have been basically generated. If a large step size is still used, it is easy to overcorrect, destroying the formed fine texture or introducing high-frequency noise. Therefore, a smaller second step size (e.g., 0.1) is switched to fine-tune, which can eliminate residual deviations and maintain the naturalness and stability of the image. In addition, the step size can also be linked with the perceptual loss value. When the deviation is small, the step size is automatically reduced to avoid unnecessary disturbances. Through this dynamic adjustment mechanism, the system can achieve robust control of the pixel-level accuracy of IP elements while ensuring generation efficiency.

[0047] Example 4 Building upon Example 1, this example further details the automated closed-loop mechanism for feature deviation detection, localization, and local redrawing repair. As one implementation method, feature deviation detection and localization of the initial image includes: inputting the initial image into a pre-trained multimodal model; the multimodal model then uses a hierarchical questioning strategy to examine each preset element region in the initial image to verify whether the color, shape, and details conform to the standard element image of the target element; and outputs deviation descriptions and localization information when feature deviations are detected. The standard element image is a local image pre-cropped and extracted from the reference image of the target character, containing only the target element. Specifically, the hierarchical questioning strategy refers to constructing a sequence of prompts from the whole to the part. For example, first, it asks "Is the overall identity of the character correct?"; after confirming that it is correct, it then asks specific target elements such as hair ornaments and emblems one by one, asking "Is the color of the [element name] in the image [standard color value]?", "Is the shape consistent with the reference image?", and so on. Figure 1This strategy effectively guides the multimodal model to focus on fine-grained features, avoiding illusions or missed detections caused by global semantic interference. The standard element image serves as the benchmark for verification; it contains only the target element itself and no background noise, ensuring the purity and accuracy of the comparison process. It should be noted that the multimodal model can be a Visual Language Model (VLM) or other neural networks with image-text understanding capabilities.

[0048] To improve detection efficiency and reduce computational overhead, one implementation method involves using a coarse-grained cascaded review network before inputting the initial image into the pre-trained multimodal model. First, a lightweight image classification network screens the initial image for defects, determining if any suspicious defects exist. Only when the lightweight image classification network confirms a suspicious defect is the initial image input into the multimodal model for fine-grained attribution and localization. Specifically, lightweight image classification networks (such as ResNet and MobileNet) are characterized by fast inference speed and low resource consumption, making them suitable for quickly filtering out most generated images that meet quality standards. Only when the defect confidence score output by the classifier exceeds a preset threshold is the computationally intensive multimodal model triggered for precise analysis. This cascaded architecture significantly reduces the average review time in batch generation scenarios while ensuring detection accuracy, making real-time pipeline production possible.

[0049] After obtaining the deviation description and location information, it needs to be converted into machine-executable repair instructions. One approach is to perform semantic segmentation and remapping on the location information to obtain a binary mask of the deviation region. Specifically, the location information output by the multimodal model is usually a natural language description (such as "the edge of the bow in the upper left corner") or a rough bounding box coordinate, which is often not accurate enough for direct redrawing. Semantic segmentation and remapping refers to using a pre-trained segmentation model or grounding algorithm to parse the text location information into a pixel-level binary mask, accurately outlining the contour of the area to be repaired. Further, as an implementation method, the binary mask of the deviation region is also expanded at the edges through morphological dilation. Specifically, morphological dilation can expand the mask boundary outward by several pixels, aiming to cover the transition zone between the target element and the background. Since defects in the generated image often exist not only within the elements but may also affect the light and shadow fusion areas at the edges, if only the original mask area is redrawn, it is very easy to leave obvious stitching gaps between the repaired area and the original image. The buffer area reserved through dilation provides the necessary operational space for subsequent edge fusion.

[0050] After determining the precise repair area, local redrawing and repair are performed. As one implementation method, local redrawing and repair of the initial image is performed based on the standard element image of the target element. This includes: reusing the diffusion model, performing denoising processing on the region of the binary mask based on the deviation description, and injecting the standard element image of the target element as a local image condition during the denoising process. Specifically, "reusing" here means that the redrawing stage uses the same or isomorphic diffusion model weights as the generation stage, which ensures the consistency of the repaired content with the original image in terms of style, resolution, and semantic space. Unlike conventional redrawing that relies solely on text prompts, this application encodes the standard element image and injects it as an additional local image condition into the cross-attention layer or control network. This dual conditional injection mechanism (text description + image reference) constitutes a strongly constrained repair mode, forcing the model to generate content highly consistent with the standard material within the mask region, fundamentally eliminating the risk of feature drift during the repair process.

[0051] To eliminate the visual disconnect between the repaired area and the original background, one implementation method, after denoising the binary mask region based on the deviation description, also includes: applying Poisson blending or edge feathering to the region boundaries of the binary mask; wherein, the pixel width of the edge feathering is adaptively determined according to the size of the binary mask. Specifically, Poisson blending smooths gradient differences by solving partial differential equations and is suitable for scenes with complex lighting or textures; edge feathering achieves transparency transitions by applying Gaussian blur to the mask boundaries. The logic for adaptively determining the pixel width is as follows: for large deviation areas (such as an entire garment), a wider feather band (e.g., 10-20 pixels) is usually needed to achieve a natural gradient blend; while for minor detail flaws (such as missing buttons), excessively wide feathering can lead to blurred details, so a narrower feather band (e.g., 2-5 pixels) should be used. This adaptive strategy takes into account the visual naturalness of repair tasks at different scales, ensuring that the final output virtual image not only has pixel-level consistency but also achieves commercial-grade image quality where repair traces are imperceptible to the human eye. Through the complete closed loop of detection, localization, redrawing and fusion described above, the system can automatically iteratively correct generated defects without human intervention until the preset convergence conditions are met.

[0052] Example 5 Based on the inference process described in the foregoing embodiments, this embodiment further explains the training phase of the diffusion model. As one implementation, the diffusion model is a model fine-tuned for identity adaptation; wherein, during identity adaptation fine-tuning, the network layers related to geometric deformation in the diffusion model are frozen, and only the normalization layer and attention mapping layer related to texture and color are updated. Specifically, identity adaptation fine-tuning refers to customized training using a small number of reference images of the target character on the basis of a general text-based image base model, enabling the model to learn the unique visual identifiers of that character. Unlike conventional full-scale fine-tuning, this application employs a selective freezing strategy to resolve the contradiction between overfitting and decreased generalization ability. The network layers related to geometric deformation typically include the standard convolutional layers, downsampling layers, and the parts responsible for spatial feature transformation in the residual blocks of the UNet backbone. These layers encode the general topological structure, perspective relationships, and prior knowledge of human anatomy in the image. Locking their parameters prevents the model from forgetting the original compositional capabilities and pose diversity of the base during training with few samples. In contrast, normalization layers (such as GroupNorm and LayerNorm) mainly adjust the statistical distribution of feature maps, directly affecting the color tone and lighting style of the generated image; attention mapping layers (including Query, Key, and Value projection matrices for self-attention and cross-attention) are responsible for semantic alignment and the synthesis of texture details. These two types of layers are most sensitive to IP identity features, allowing their parameter updates to efficiently inject the target character's color scheme, line texture, and signature patterns into the model weights.

[0053] The micro-mechanism of this partial fine-tuning mechanism lies in achieving decoupled learning of structural priors and style knowledge. If full fine-tuning is used, with only 3 to 10 reference images, the model is prone to developing a memory effect on the training set, causing the generated characters to only reproduce fixed poses from the reference images. Once new action cues are input, limb distortion or structural collapse occurs. By freezing the geometry layer, a stable and universal "skeleton generator" is essentially preserved for the generation process. Regardless of the input pose description, the model can utilize the spatial understanding learned during pre-training to construct a reasonable layout. Simultaneously, the trainable normalization layer and attention layer act as a "skin renderer," focusing on attaching the IP's unique visual symbols to this skeleton. For example, in a fine-tuning experiment for a certain anime character, when the learning rate is set to 1e... -5 up to 5e -5 When the interval and training steps are between 500 and 2000, this strategy enables the model to not only accurately reproduce the metallic luster of the character's hair ornaments and the dark patterns of the clothing using only 5 reference images, but also maintain body proportion coordination when generating complex postures such as running and jumping that did not appear in the training set. This verifies that it effectively preserves the posture generalization ability of the base model while giving the IP specific textures.

[0054] It should be noted that the specific objects of the selective freezing mentioned above are not fixed. In some embodiments, if the identity features of the target character contain special geometric shapes (such as the unique joint structure of a non-humanoid mecha), some shallow convolutional modules can be unfrozen as appropriate to adapt to specific contour requirements, as long as the frozen state of most geometric layers is maintained overall to avoid global overfitting. In addition, regularization techniques (such as Dropout, weight decay) or mixed precision training strategies can be used during fine-tuning to further improve convergence stability. This embodiment limits parameter updates during the training phase to texture and color-sensitive layers, providing an inherent consistency basis for feature alignment correction in the aforementioned inference phase from the model weight level. This ensures that even without strong external constraints during inference, the model itself has a high degree of IP feature restoration, thereby reducing the computational cost of online generation and the frequency of error correction.

[0055] Example 6 This application also provides an artificial intelligence-based virtual avatar generation system 600, used to implement the artificial intelligence-based virtual avatar generation method described in the foregoing embodiments. This system adopts a modular architecture design, decoupling the complex generation and correction processes into independent functional units to improve the system's maintainability and scalability. Figure 3 As shown, the system 600 includes a data acquisition module 601, a decoupling injection module 602, a feature anchoring module 603, a deviation positioning module 604, and a local redrawing module 605.

[0056] The data acquisition module 601 is used to acquire text prompt data and prior information on the element structure and visual identity representation of the target role. Specifically, this module serves as a unified interface for the system to interact with external data sources. In actual deployment, it can be configured to connect to a local high-performance storage array to read a pre-built IP role asset library, or it can receive real-time reference materials uploaded by users in the cloud via an API gateway. This module integrates a data standardization preprocessing unit, which can uniformly convert raw inputs from different sources and in different formats into standard tensors or serialized objects recognizable by downstream modules. For example, for prior information on element structure, this module verifies the validity of the mask resolution and keypoint coordinates; for visual identity representation information, it verifies the dimensional consistency of the feature vectors. This standardized data access mechanism enables the system to seamlessly adapt to diverse business scenarios, whether it's batch offline production or interactive online generation, all can complete resource loading through the same data pipeline.

[0057] The decoupled injection module 602 is used to inject prior information about element structure as a hard spatial constraint into the pre-trained diffusion model, and inject visual identity representation information as a soft semantic constraint into the diffusion model. Specifically, this module undertakes the core functions of signal routing and condition assembly within the model. It does not directly participate in the generation calculation of image pixels, but is responsible for accurately distributing different types of conditional signals to designated layers of the diffusion model before or during inference. For example, this module can load the encoded structural prior tensor into GPU memory for the structural control network to read, and simultaneously map identity basis vectors into key-value pairs and write them into the cache of the cross-attention layer. Through this physically or logically decoupled injection design, the system can flexibly switch between different constraint strategies without modifying the weights of the base model, achieving plug-and-play generation capabilities.

[0058] The feature anchoring module 603 is used to input text prompt data into the diffusion model for iterative denoising and output an initial image. During iterative denoising, feature alignment correction is performed on the current latent variables based on the reference feature representation of the target element; the target element is the distinctive local element of the target character. Specifically, this module is the system's generation engine and real-time optimization controller. It not only schedules the diffusion model to execute standard denoising sampling steps but also incorporates high-precision temporal control logic to trigger anchoring operations at preset key time steps. This module dynamically calls the ROI Align operator to extract features and calculate perceptual loss by listening to the state signal of the denoising loop, thereby driving gradient backpropagation to update latent variables. Unlike the aforementioned method embodiments that focus on algorithm principles, this module focuses on scheduling efficiency and memory management in engineering implementation. For example, it uses asynchronous pipeline technology to overlap feature extraction and denoising calculation to avoid the correction operation becoming a bottleneck for generation speed. This integrated engine design ensures that the multi-scale feature anchoring strategy can operate stably with limited computing resources, guaranteeing the generation quality and output rate of the initial image.

[0059] The deviation localization module 604 is used to detect and locate feature deviations in the initial image to obtain the deviation region. Specifically, this module constitutes the system's automated quality inspection unit. It receives the initial image output by the feature anchoring module and performs defect screening according to a preset review strategy. In a preferred embodiment, this module encapsulates a coarse-grained cascaded review network. It first calls a lightweight classifier for fast filtering, and only activates a computationally intensive multimodal large model for fine-grained attribution when a suspicious defect is detected. This module also includes a semantic segmentation and remapping subunit, which can automatically convert the deviation localization information described in natural language into an accurate binary mask and perform morphological dilation operations as needed. This intelligent detection architecture enables the system to significantly reduce the average processing time per image while ensuring review accuracy, meeting the stringent throughput requirements of industrial production lines.

[0060] The local redrawing module 605 is used to locally redraw and repair the initial image within the deviation area based on the standard element image of the target element until a preset convergence condition is met, resulting in a virtual avatar image. Specifically, this module is the system's closed-loop correction execution unit. It reuses the diffusion model to perform conditional redrawing on specific areas based on the mask and description information provided by the deviation localization module. This module not only generates the repaired content but also integrates edge blending processing, adaptively selecting Poisson blending or feathering algorithms based on the mask size to eliminate stitching artifacts. More importantly, this module possesses iterative control logic, enabling it to feed the repaired image back to the deviation localization module for secondary verification, forming an automated error correction closed loop until the image quality meets the standard or the maximum number of iterations is reached. Through this end-to-end automated repair mechanism, the system completely eliminates reliance on manual post-processing, significantly improving the yield rate and delivery efficiency of virtual avatar generation.

[0061] It should be noted that the above module division is only a schematic representation of logical function and not a limitation on the physical implementation. In practical applications, multiple modules can be combined and deployed on the same processor, or they can be distributed across different computing nodes and communicate via a network. Furthermore, the modular design of the system gives it strong scalability. For example, when a more advanced visual language model or diffusion model backbone emerges, only the corresponding module components need to be replaced and the interface protocol adapted to achieve an overall performance upgrade without reconstructing the entire system architecture. This flexibility ensures that the technical solution of this application can continuously adapt to the rapid iteration of artificial intelligence technology and extend the product's lifecycle.

[0062] Example 7 To more clearly illustrate the implementation effect and commercial value of the technical solution of this application, the following example of generating a poster of a classic anime character A wearing Christmas attire will be used to verify the entire process of the artificial intelligence-based virtual image generation method described in the aforementioned embodiments.

[0063] In the offline preparation phase, the system first executes a data acquisition and prior construction process. Specifically, five high-resolution reference images of character A are acquired, covering different perspectives and expressions, including a frontal smile, a slight profile, a close-up of the hair ornament, and a close-up of the badge. A pre-trained semantic segmentation network is used to parse each reference image, accurately extracting pixel-level masks of the two iconic elements: the bow hair ornament and the golden dragon emblem. The normalized coordinates and size proportions of these elements relative to the midpoint of the line connecting the eyes are recorded, serializing them into prior information about the element structure. Simultaneously, an image encoder finely tuned for anime style is used to extract identity feature vectors from the five reference images, which are then fused using weighted average pooling to obtain 512-dimensional visual identity representation information. This offline construction process provides a robust geometric constraint benchmark and robust identity semantic guidance for subsequent generation, ensuring that even in complex poses, character A's core visual symbols still possess a structured expression that can be accurately invoked.

[0064] During the online generation phase, users input text prompts containing trigger words, such as "[V] Character A, wearing a red Christmas outfit with white fur trim, holding a gift, with a snowflake background, maintaining the original art style." The system automatically loads pre-built visual identity representation information and injects it into the diffusion model through a cross-attention layer. Simultaneously, it encodes prior information about element structure and sends it to the structure control network to apply hard spatial constraints. During iterative denoising, the system strictly implements a multi-scale feature alignment correction mechanism. Specifically, when denoising reaches step 20 out of a total of 30 steps, the first anchoring operation is activated. Local features of the bow region are extracted from the shallow feature map decoded from the current latent variables. Perceptual loss is calculated by comparing this with pre-stored standard bow features, and the latent variables are updated along the gradient direction with a first-step length of 0.5 to achieve rapid correction. When denoising reaches step 10, the second anchoring operation is performed, at which point the step size automatically decreases to a second-step length of 0.1 for fine-tuning. This dynamic step size adjustment based on the annealing strategy ensures that the color and shape of the bow can be quickly brought back to the correct manifold in the early stages of generation, while avoiding damage to the already formed clothing folds and lighting details due to excessive correction in the later stages of generation, and finally decoding and outputting the initial image.

[0065] During the closed-loop review and remediation phase, the system inputs the initial image into a multimodal model for automated compliance detection. Specifically, a hierarchical questioning strategy is adopted. After confirming the overall character identity is correct, questions are asked one by one regarding the bow area: "Is the left half of the bow in the image blue, the right half white, and is the middle connection a gold diamond clasp?" Assuming the multimodal model returns a deviation description of "the right half is purplish and the diamond clasp is missing," the system then performs semantic segmentation and remapping on this location information to obtain an accurate binary mask, and expands the edges outward through morphological dilation to cover the transition zone. Subsequently, the system performs local redrawing and remediation within the deviation area, reusing the diffusion model and converting the deviation description into text conditions, while injecting a standard close-up image of the bow as a local image condition, forcing the model to generate a gold diamond clasp and correct color scheme that conforms to the IP specifications within the mask area. After redrawing, a 5-pixel-wide edge feathering is applied to the boundary of the remediation area to eliminate splicing marks. The system sends the repaired image back into the review process until no deviations are detected or the preset convergence conditions are met, and finally outputs a fully compliant virtual avatar image.

[0066] The above-described full-process verification demonstrates that the technical solution of this application successfully eliminates the feature drift problem of Character A. Throughout the entire generation and correction process, the precise color matching of the bow, the metallic texture of the diamond-shaped buttons, and their natural integration with the Christmas attire all meet commercially viable standards for direct delivery. More importantly, the entire process requires no manual image selection or retouching, achieving end-to-end automated production from text prompts to compliant images, fully validating the technological advancement and practical value of this application in the mass design scenario of IP derivatives.

[0067] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating virtual avatars based on artificial intelligence, characterized in that, include: Obtain text prompt data and prior information on the element structure and visual identity representation of the target character; The prior information of the element structure is injected into the pre-trained diffusion model as a hard spatial constraint, and the visual identity representation information is injected into the diffusion model as a soft semantic constraint. The text prompt data is input into the diffusion model for iterative denoising, and an initial image is output; wherein, during the iterative denoising process, feature alignment correction is performed on the current latent variables based on the reference feature representation of the target element; the target element is the iconic local element of the target character; The initial image is subjected to feature deviation detection and localization to obtain the deviation region; Within the deviation region, the initial image is locally redrawn and repaired based on the standard element image of the target element until the preset convergence condition is met, thus obtaining the virtual image image.

2. The method for generating virtual avatars based on artificial intelligence according to claim 1, characterized in that, The prior information of the element structure includes the pixel-level mask of the target element, and the spatial positional relationship and size ratio of the target element relative to the facial key points of the target character; The visual identity representation information is obtained by pooling and fusing the identity features of multiple reference images of the target character that have been pre-acquired; The step of injecting the prior information of the element structure as a hard spatial constraint into the pre-trained diffusion model and injecting the visual identity representation information as a soft semantic constraint into the diffusion model includes: The prior information of the element structure is encoded and then sent to the structure control network of the diffusion model to apply spatial hard constraints to the generation process of the diffusion model. The visual identity representation information is injected through the cross-attention layer of the diffusion model to apply semantic soft constraints.

3. The method for generating virtual avatars based on artificial intelligence according to claim 1, characterized in that, The reference feature representation based on the target element performs feature alignment correction on the current latent variable, including: In the iterative denoising loop, multiple key time steps are selected for anchoring operations; For each key time step, extract the local features corresponding to the target element from the intermediate layer feature map obtained by decoding the current latent variable through the diffusion model; Calculate the perceptual loss between the local features and the reference feature representation of the target element; wherein the reference feature representation is a deep feature of the target element pre-extracted from a reference image of the target character; The current latent variable is updated using gradients based on the perceptual loss; wherein the update step size of the gradient update is dynamically adjusted according to the current time step and the feature attributes of the target element.

4. The method for generating virtual avatars based on artificial intelligence according to claim 3, characterized in that, The anchoring operation employs a layered, progressive anchoring strategy; wherein, the layered, progressive anchoring strategy includes: In the deep feature map of the UNet decoder of the diffusion model, global identity features obtained by mapping the visual identity representation information are used for anchoring; In the mid-layer feature map of the UNet decoder, the element category features determined based on the prior information of the element structure are used for anchoring; wherein, the element category features correspond to the component category to which the target element belongs; In the shallow feature map of the UNet decoder, the target element is anchored using pixel-level texture features extracted based on the reference feature representation.

5. The method for generating virtual avatars based on artificial intelligence according to claim 3, characterized in that, The update step size of the gradient update is dynamically adjusted based on the current time step and the feature attributes of the target element, including: The update step size is adjusted based on the degree of deviation between the current time step and the perceptual loss representation, using an annealing strategy. Specifically, when the current time step belongs to the first half of the total denoising steps, a preset first step size is used for rapid correction; when the current time step belongs to the second half of the total denoising steps, a preset second step size smaller than the first step size is used for fine correction. The first half and the second half are obtained by dividing the total denoising steps based on a preset ratio coefficient. The degree of deviation is one of the feature attributes.

6. The method for generating virtual avatars based on artificial intelligence according to claim 1, characterized in that, The feature deviation detection and localization of the initial image includes: The initial image is input into a pre-trained multimodal model. The multimodal model uses a hierarchical questioning strategy to examine each preset element region in the initial image to check whether the color, shape, and details conform to the standard element image of the target element. When feature deviations are detected, the model outputs deviation descriptions and location information. The standard element image is a local image that is pre-cropped and extracted from the reference image of the target character and contains only the target element. The positioning information is semantically segmented and remapped to obtain a binary mask of the deviation region; The local redrawing and repair of the initial image based on the standard element image of the target element includes: The diffusion model is reused, and denoising is performed on the region of the binary mask based on the deviation description. During the denoising process, the standard element image of the target element is injected as a local image condition.

7. The method for generating virtual avatars based on artificial intelligence according to claim 6, characterized in that, Before inputting the initial image into the pre-trained multimodal model, the method further includes: A coarse- and fine-grained cascaded review network is adopted: first, a lightweight image classification network screens the initial image for defects to determine whether there are any suspicious defects; only when the lightweight image classification network confirms the existence of a suspicious defect is the initial image input into the multimodal model for fine attribution and localization. The binary mask of the deviation region is further expanded at the edges through morphological dilation.

8. The method for generating virtual avatars based on artificial intelligence according to claim 6, characterized in that, After performing denoising processing on the region of the binary mask based on the deviation description, the method further includes: The region boundaries of the binary mask are subjected to Poisson blending or edge feathering; wherein, the pixel width of the edge feathering is adaptively determined according to the size of the binary mask.

9. The method for generating virtual avatars based on artificial intelligence according to claim 1, characterized in that, The diffusion model is a model fine-tuned by identity adaptation; wherein, during the identity adaptation fine-tuning process, the network layers related to geometric deformation in the diffusion model are frozen, and only the normalization layer and attention mapping layer related to texture and color are updated.

10. A virtual avatar generation system based on artificial intelligence, used to implement the virtual avatar generation method based on artificial intelligence as described in any one of claims 1 to 9, characterized in that, include: The data acquisition module is used to acquire text prompt data and prior information on the element structure and visual identity representation of the target character; The decoupled injection module is used to inject the prior information of the element structure as a hard spatial constraint into the pre-trained diffusion model, and to inject the visual identity representation information as a soft semantic constraint into the diffusion model. The feature anchoring module is used to input the text prompt data into the diffusion model for iterative denoising and output an initial image; wherein, during the iterative denoising process, feature alignment correction is performed on the current latent variables based on the reference feature representation of the target element; the target element is the iconic local element of the target character; The deviation localization module is used to detect and locate feature deviations in the initial image to obtain the deviation region; The local redrawing module is used to locally redraw and repair the initial image within the deviation area based on the standard element image of the target element until a preset convergence condition is met, thereby obtaining a virtual image image.

Images