Role generation method and system based on lora fast training

Abstract

The application provides a role generation method and system based on lora fast training, and relates to the technical field of data processing.The method comprises the following steps: injecting a semantic feature vector in a role feature file into a condition control path of a pre-training diffusion model during training, and inserting a LoRA low-rank decomposition matrix into an attention layer at a corresponding network position according to an architecture type of the pre-training diffusion model to obtain a role LoRA weight file through iterative updating; and injecting the semantic feature vector in the role feature file into the condition control path of the pre-training diffusion model according to the pre-training diffusion model, the role LoRA weight file, the role feature file and a text prompt containing a trigger word, and generating a target role image through the collaborative guidance of the semantic feature vector and the LoRA weight.The application can effectively handle the case that a role subject is divided into multiple unconnected areas, and avoid the loss of subject information caused by excessive segmentation.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for character generation based on LoRa fast training. Background Technology

[0002] Diffusion models have become the mainstream technical paradigm in the field of image generation. Text-to-image generation techniques, represented by the StableDiffusion series, are widely used in digital content creation, game character design, digital human customization, and film and television concept art generation. To ensure that the AI-generated image content conforms to the appearance characteristics of a specific character, it is usually necessary to perform customized fine-tuning on the pre-trained base diffusion model.

[0003] Currently, the mainstream methods for generating customized characters mainly include the following categories: (1) DreamBooth-like full fine-tuning method. DreamBooth injects the concept of a specific subject into the model weights by fine-tuning all parameters of the diffusion model UNet. This method can achieve a high degree of character restoration, but it has obvious drawbacks: the training speed is extremely slow, and it usually takes hundreds to thousands of iterations to fully fine-tune a character, which can take tens of minutes to several hours on a single GPU; the sample requirement is large, usually requiring 3-5 high-quality training samples; the memory requirement is extremely high, usually requiring more than 24GB of memory, and the deployment environment requirements are harsh; the size of the output fully fine-tuned model is in the GB level, and the storage and switching costs are extremely high in multi-character scenarios.

[0004] (2) IP-Adapter-type image cue adapter method. IP-Adapter introduces an image cue adapter to inject the semantic features of the reference image into the cross-attention layer to guide the generation of results. This method does not require additional training and has a fast inference speed, but the consistency of the characters is poor. It can only retain the overall style and coarse-grained appearance of the characters and cannot accurately reproduce the personalized features such as facial details and clothing textures. This is because the image features have not been specially aligned and optimized with the specific concepts of the characters, resulting in insufficient feature guidance accuracy.

[0005] (3) Efficient fine-tuning method for general LoRA parameters. In order to reduce the number of fine-tuning parameters and improve training speed, those skilled in the art usually use efficient fine-tuning methods such as LoRA to fine-tune only a small number of low-rank matrix parameters. However, the existing general LoRA schemes have the following shortcomings: they lack an automated data processing pipeline, and cannot systematically assess the quality of the input original images, crop the subject, and expand the diversity, resulting in unstable training data quality; they lack explicit extraction and persistent reuse mechanism for role features, and under conditions of very few samples (such as only 1 image), the model has difficulty learning a stable role appearance representation, and the learning effect of role consistency is not ideal; in terms of data preprocessing, existing schemes usually rely on manual annotation and manual cropping, which is cumbersome and inefficient, and it is difficult to support the rapid customization needs of large-scale multi-role models.

[0006] Furthermore, as diffusion model architectures evolve from traditional convolutional UNet to Transformer-based DiT (DiffusionTransformer) (such as Stable Diffusion 3, Flux, PixArt-α, etc.), most existing role customization solutions are only designed for a single architecture and lack cross-architecture general adaptability, making it difficult to adapt to the ever-evolving basic model ecosystem. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a character generation method and system based on LoRa fast training, which can effectively handle the situation where the main body of the character is segmented into multiple disconnected regions, avoid the loss of main body information due to over-segmentation, and remove irrelevant background regions to improve the purity of training data.

[0008] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: Firstly, a character generation method based on LoRa fast training, the method comprising: The original character image is segmented into main character instances. Convex polygons are generated based on the contour point set of the segmented character instances. The shortest distance between any two convex polygons is calculated based on the separating axis theorem. When the shortest distance is less than the preset merging distance threshold, the corresponding convex polygons are merged. The merged character instance region is cropped and normalized to obtain the cleaned image. Based on the cleaned images, an enhanced image set is obtained through data augmentation processing; Based on the enhanced image set, automatic image descriptions are generated for each enhanced image, and trigger words corresponding to the characters are inserted into the description text to form an image-text pair training dataset; Based on the cleaned images, semantic feature vectors are extracted using a pre-trained image encoder, and the semantic feature vectors are serialized into character feature files. Based on the training dataset and character feature files, the pre-trained diffusion model is fine-tuned using low-rank adaptation LoRA. During training, the semantic feature vectors from the character feature files are injected into the conditional control pathway of the pre-trained diffusion model, and the low-rank LoRA decomposition matrix is ​​inserted into the attention layer at the corresponding network position according to the architecture type of the pre-trained diffusion model. The character LoRA weight file is obtained through iterative updates. Based on the pre-trained diffusion model, the character LoRA weight file, the character feature file, and text prompts containing trigger words, the semantic feature vectors from the character feature file are injected into the conditional control path of the pre-trained diffusion model. Through the synergistic guidance of the semantic feature vectors and LoRA weights, the target character image is generated.

[0009] Secondly, a character generation system based on LoRa for rapid training includes: The data cleaning module is used to perform image quality assessment, duplicate detection and deduplication, instance segmentation, convex polygon construction, shortest distance calculation and merging of convex polygons, cropping and size normalization processing based on the original character images, and output the cleaned images. The data augmentation module is used to perform geometric transformation, color perturbation, background replacement and multi-resolution sampling processing based on the cleaned image, and output an enhanced image set; The data labeling module is used to automatically describe the enhanced images and insert role trigger words based on the enhanced image set, and output the image-text pair training dataset; The feature extraction module is used to extract semantic feature vectors from the cleaned image using a pre-trained image encoder and serialize them into a character feature file. The training module is used to load the pre-trained diffusion model and perform LoRA fine-tuning training based on the image and text training dataset and the character feature file. During training, the semantic feature vector is injected into the model conditional control path, and the LoRA low-rank decomposition matrix is ​​automatically inserted into the corresponding attention layer according to the model architecture, and the character LoRA weight file is output. The inference module is used to receive text prompts containing trigger words based on the pre-trained diffusion model, the character LoRA weight file, and the character feature file, inject semantic feature vectors into the model's conditional control path, and generate a target character image.

[0010] The above-described solution of the present invention has at least the following beneficial effects: High-quality data cleaning is achieved through character subject instance segmentation and intelligent cropping mechanism based on convex polygon merging. By using instance segmentation technology to accurately extract the outline of the character subject and adaptively merging the shortest distance between convex polygons based on the separating axis theorem, the system can effectively handle the situation where the character subject is segmented into multiple disconnected regions, avoid the loss of subject information due to over-segmentation, and remove irrelevant background regions to improve the purity of training data.

[0011] By using an automated pipeline for data augmentation and automatic labeling, the data threshold for role customization is lowered. By performing multi-dimensional enhancement processing on cleaned images, such as geometric transformation, color perturbation, and background replacement, a single image is expanded into a diverse training sample set, effectively mitigating the risk of overfitting under conditions of very few samples. Combined with automatic image description generation and trigger word injection, no manual annotation is required throughout the process. Even in extreme scenarios with only one input image, an effective image-text pair training dataset can be constructed, solving the problem of existing technologies' dependence on a large number of training samples and manual annotation.

[0012] By explicitly extracting character semantic feature vectors and reusing persistent files, stable character consistency is achieved in both training and inference phases. A pre-trained image encoder extracts high-dimensional semantic feature vectors from cleaned images and serializes them into feature files. These feature files serve as additional conditional inputs to guide LoRA learning during training and are directly loaded into the conditional control pathway of the diffusion model during inference, establishing a stable feature transfer bridge between training and inference. Even with only one original input image, inference can accurately reconstruct facial features, clothing textures, and other appearance details by loading the feature file. This solves the problems of insufficient inference consistency and stability caused by implicit character feature encoding in existing IP-Adapter-type solutions, as well as the unsatisfactory learning performance of general LoRA solutions due to the lack of explicit feature guidance under very few sample conditions.

[0013] By employing a cross-architecture LoRA low-rank matrix adaptive insertion strategy, universal compatibility with different diffusion model architectures is achieved. Through a collaborative guidance mechanism of semantic feature vectors and LoRA weights, highly consistent character image generation is achieved, reducing storage and switching costs in multi-character scenarios. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the character generation method based on LoRa fast training provided in an embodiment of the present invention.

[0015] Figure 2 This is a schematic diagram of a character generation system based on LoRa fast training provided in an embodiment of the present invention. Detailed Implementation

[0016] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0017] Example 1: A character generation method based on LoRA fast training (single input image scene) This embodiment uses a user providing only one original character image as input, with the target base model being StableDiffusion XL (UNet architecture), to detail the complete implementation process of the method of this invention. In this embodiment, the character is a fictional game character, and the character trigger words are pre-set, consisting of letters, numbers, and underscores, each corresponding to a specific character.

[0018] Step S1: Data cleaning. Data cleaning refers to a series of automated quality processing operations performed on the user-input raw character image, including image quality assessment and filtering, duplicate image detection and deduplication, character subject instance segmentation, convex polygon construction, convex polygon shortest distance calculation and merging, cropping, and size normalization. The final output is a cleaned image with a uniform format, prominent subject, and clean background, providing high-quality input for subsequent data augmentation and feature extraction. Specifically, this may include: Step S1.1, Image Quality Assessment and Filtering. Image quality assessment refers to calculating a quality score for each input image using a sharpness assessment algorithm, determining whether the image is acceptable based on a preset quality threshold, and filtering out low-quality images such as blurry, overexposed, or underexposed images. The system receives the original character image input by the user. The system first performs image quality assessment on this image. The specific implementation process is as follows: Laplacian convolution constructs a Laplacian convolution kernel, which is a fixed-size two-dimensional matrix. The center element of the kernel has a positive value, its neighboring elements have negative values, and the remaining elements have zero values. This kernel is then slid across the image pixel by pixel. For each pixel, the grayscale value of each pixel within the kernel's coverage area is multiplied by the corresponding value of the kernel. The sum of all products yields the Laplacian response value for that pixel. The purpose of Laplacian convolution is to extract edge regions in an image where grayscale values ​​change drastically; a larger response value indicates a sharper and clearer edge.

[0019] Variance calculation involves iterating through the response values ​​of all pixels in the Laplacian response map. First, the average of all response values ​​is calculated by dividing the sum of all response values ​​by the total number of pixels. For each response value, the square of its difference from the average is calculated. Finally, all squared values ​​are summed and divided by the total number of pixels to obtain the variance value. The variance value is the image sharpness quality score. The purpose of variance calculation is to measure the statistical distribution of edge sharpness across the entire image; a larger variance indicates more sharp edges in the image and better overall image quality.

[0020] The threshold determination involves comparing the calculated quality score with a preset quality threshold. The preset quality threshold is determined by collecting a batch of manually labeled qualified and unqualified character image samples, calculating the Laplacian gradient variance for each sample, and selecting the boundary value that best distinguishes between qualified and unqualified samples as the quality threshold. If the quality score is greater than or equal to the threshold, the image quality is considered qualified and retained; if the quality score is less than the threshold, the image quality is considered unqualified, automatically filtered, and removed from the input image set, while simultaneously returning a filtering prompt message to the user. In this embodiment, the input image is considered qualified and retained because its calculated quality score is greater than the threshold.

[0021] This embodiment automatically evaluates image sharpness through Laplacian gradient variance, which can automatically filter out low-quality images caused by shooting shake, abnormal exposure, etc., ensuring the quality of training data from the source, avoiding the negative impact of blurry or overexposed / underexposed images on subsequent model training, and improving the consistency and sharpness of the final character generation.

[0022] Step S1.2, duplicate image detection and deduplication, refers to the operation of detecting and removing highly repetitive samples in the input image set through perceptual hashing or feature vector similarity calculation methods. The purpose is to retain effective images with appearance diversity and avoid overfitting of the model due to redundant training data. Since the original character image in this embodiment is only 1 image, the system automatically skips the duplicate image detection and deduplication step and directly enters the character main instance segmentation stage. The skip logic is as follows: at the beginning of the data cleaning process, the system first checks the number of elements in the input image set. If the number is equal to the preset minimum number threshold (which is set to 1 image), the skip deduplication flag is set to true, and subsequent deduplication steps read this flag and skip directly.

[0023] When there are multiple input images, the system performs duplicate image detection and deduplication. Specifically, it employs a perceptual hashing algorithm, scaling each image to a preset hash size and converting it to grayscale. After calculating the discrete cosine transform, the low-frequency components of the transform coefficients are selected. Each coefficient is compared to the median of all coefficients; if greater than the median, it is set to 1; otherwise, it is set to 0, generating a fixed-bit binary hash value. Then, the Hamming distance between any two image hash values ​​is calculated, which is the number of different corresponding bit values ​​in the two binary strings. If the Hamming distance is less than a preset Hamming distance threshold, it is determined to be a duplicate image pair, one of which is retained, and the remaining duplicate images are discarded. The Hamming distance threshold is determined by collecting a batch of image pairs with known duplication relationships, calculating the Hamming distance of each image pair, and selecting a boundary value that accurately distinguishes between duplicate and non-duplicate image pairs.

[0024] This embodiment automatically detects and removes duplicate images using perceptual hashing and Hamming distance, avoiding a large number of highly similar redundant samples in the training dataset. This prevents the model from overfitting to specific compositions or poses, while reducing unnecessary training computation overhead and improving training efficiency. For single-image scenes, automatic skipping logic avoids unnecessary calculations and improves the flexibility of the processing flow.

[0025] Step S1.3, Character Subject Instance Segmentation, where character subject instance segmentation refers to using a pre-trained instance segmentation model to perform pixel-level classification of the character subject in the image, outputting the contour point set of the character instance, and accurately separating the character subject region from the background region. Character subject instance segmentation is performed on images that meet the required quality standards. This embodiment uses an instance segmentation model based on Mask R-CNN. The specific implementation process is as follows: Load the network structure and pre-trained weights of the Mask R-CNN instance segmentation model. This model contains three core components: a backbone network, which adopts the ResNet architecture to extract multi-scale convolutional features of the image. ResNet is a residual network that solves the gradient vanishing problem in deep networks through residual connections; a Region Proposal Network (RPN), which is used to generate candidate target regions by sliding on the feature map and outputs the target existence score and bounding box regression offset for each candidate region; and a head network, which includes a classification branch and a mask branch. The classification branch outputs the target class probability distribution for each candidate region, and the mask branch outputs a pixel-level binary segmentation mask for each candidate region.

[0026] A qualified image is input into the Mask R-CNN model. The backbone network performs convolution and pooling operations layer by layer to extract multi-scale feature maps. Pooling involves dividing the feature map into regions of a fixed window size and taking the maximum or average value of the feature values ​​within each region (max pooling) or average pooling, thereby reducing the spatial size of the feature map and expanding the receptive field. The region proposal network generates anchor boxes of various scales and aspect ratios on the feature map, centered at each spatial location. Each anchor box is classified (to determine whether it contains a target) and regressed (to adjust the position and size of the anchor box), and the top-scoring candidate regions are selected. The head network performs Region of Interest Alignment (ROI Align) on each candidate region. A fixed number of points are uniformly sampled within the candidate region, and the feature value of each sampled point is obtained using bilinear interpolation. This maps candidate regions of any size to feature maps of a fixed size. The classification branch outputs the probability that the region belongs to the role category, and the masking branch outputs a binary segmentation mask for the region.

[0027] For the binary segmentation mask output by the mask branch, pixels with a value of 1 represent the main character area, and pixels with a value of 0 represent the background area. A contour tracking algorithm is used to extract the contour point set of the character instance. The specific implementation of the contour tracking algorithm is as follows: starting from the top left corner of the mask, scan pixel by pixel to find the first unvisited pixel with a value of 1 as the contour starting point; check the neighboring pixels of this pixel in a clockwise direction to find the next neighboring pixel with a value of 1 as the contour point; repeat this process until returning to the starting point to obtain the closed contour point sequence of the connected region. The contour point set is a set of ordered two-dimensional coordinate points, each point consisting of an x-coordinate and a y-coordinate, representing the boundary position between the main character area and the background area.

[0028] In this embodiment, because the color of the character's clothing is similar to the color of some elements in the background, the instance segmentation model divides the main body of the character into two disconnected regions: a first region and a second region, with a certain pixel gap between the two regions. This gap is caused by the close color distance between the clothing color and the background color in the RGB color space, resulting in a segmentation break at the boundary. By using the Mask R-CNN instance segmentation model to achieve pixel-level extraction of the main body of the character, it is possible to accurately separate the character from the background, providing accurate boundary information for subsequent convex polygon construction and intelligent cropping. At the same time, the contour tracking algorithm can extract a complete closed contour point sequence. Even if the main body of the character is segmented into multiple disconnected regions, it can be repaired in subsequent steps through the convex polygon merging mechanism, effectively avoiding the loss of main body information due to over-segmentation.

[0029] Step S1.4, Convex Polygon Construction: Convex polygon construction refers to calculating the convex hull of the contour point set for each character instance. This convex hull is the smallest convex polygon containing all points in the contour point set, transforming the irregular point set boundary into a regular polygon representation for subsequent geometric distance calculation and merging operations. The convex hull is generated for the contour point set of each character instance. This embodiment uses the Graham scan algorithm. The specific implementation process is as follows: Iterate through all points in the contour point set and find the point with the smallest y-coordinate as the reference point. The comparison method is as follows: if two points have different y-coordinates, take the point with the smaller y-coordinate; if two points have the same y-coordinate, take the point with the smaller x-coordinate. The purpose of selecting the reference point is to determine the starting reference point for the convex hull construction, ensuring the stability and determinism of the algorithm.

[0030] Using the reference point as the pole, calculate the polar angles of all other points in the contour point set relative to the reference point. The polar angle is calculated as follows: subtract the reference point's ordinate from the current point's ordinate to obtain the ordinate difference; subtract the reference point's abscissa from the current point's abscissa to obtain the abscissa difference; divide the ordinate difference by the arctangent of the abscissa difference to obtain the polar angle of that point relative to the reference point. Sort the points according to their polar angles from smallest to largest. If two points have the same polar angle, calculate the Euclidean distance between these two points and the reference point. The Euclidean distance is calculated as the square root of the sum of the squares of the abscissa differences and the squares of the ordinate differences. Keep the points with greater distances and delete the points with smaller distances. The purpose of polar angle sorting is to arrange the points counterclockwise, providing an ordered input sequence for subsequent convex hull determination.

[0031] To construct a stack-based convex hull, the system maintains a stack data structure to store candidate vertices of the convex hull, with the top of the stack always pointing to the most recently added candidate vertex. First, the first and second sorted vertices (the pivot point) are pushed onto the stack. Then, starting from the third sorted vertex, each vertex is processed sequentially: before each processing step, the top two points of the stack (the top vertex and the second-to-last vertex) are taken, and the vectors from the second-to-last vertex to the top vertex and from the top vertex to the current vertex are calculated. The cross product of these two vectors is calculated as follows: the horizontal component of the vector from the second-to-last vertex to the top vertex multiplied by the vertical component of the vector from the top vertex to the current vertex, minus the horizontal component of the vector from the second-to-last vertex to the top vertex multiplied by the horizontal component of the vector from the top vertex to the current vertex. If the cross product is greater than 0, it indicates that the three points form a counter-clockwise rotation (left turn), and the current vertex is pushed onto the stack. If the cross product is less than or equal to 0, it indicates that the three points form a clockwise rotation (right turn) or are collinear, and the top vertex is popped from the stack. Then, the cross product of the new top two points and the current vertex is rechecked, and this process is repeated until the left turn condition is met, at which point the current vertex is pushed onto the stack. Cross product determination is the core of the convex hull algorithm. Its function is to ensure that the vertices that constitute the boundary of the convex polygon are always retained in the stack, and to remove points that are concave inside.

[0032] The convex polygon is output by traversing all points. The sequence of points retained in the stack after traversing all points is the vertex set of the convex polygon, arranged in order from the bottom to the top of the stack. The convex polygon of the first region contains the first number of vertices, and the convex polygon of the second region contains the second number of vertices.

[0033] In this embodiment, the Graham scanning algorithm is used to transform the irregular set of contour points into a regular convex polygon representation, which simplifies the complexity of subsequent geometric calculations. The cross product determination mechanism can efficiently filter out the vertices that constitute the convex hull boundary and remove redundant points that are recessed inside, so that the convex polygon can compactly wrap the main body area of ​​the character, providing an accurate geometric basis for subsequent separation axis theorem distance calculation and intelligent merging determination.

[0034] Step S1.5 uses the Separating Axis Theorem (SAT) to accurately calculate the shortest Euclidean distance between two convex polygons. This is used to determine whether the two convex polygons belong to the same subject and whether they should be merged. The Separating Axis Theorem states that if two convex polygons do not intersect, there must exist a separating axis, i.e., a projection axis in a certain direction, such that the projection intervals of the two convex polygons on this axis do not overlap. The gap between the two projection intervals is the separating distance in that direction. The minimum value among all possible separating axis directions is the shortest distance between the two convex polygons.

[0035] Step S1.5.1: Construct a candidate normal vector set, where the candidate normal vector set refers to the set of all possible projection directions used as the basis for determining the separation axis, which is formed by merging the unit outward normal vectors of all edges of the two convex polygons.

[0036] Calculate the unit outward normal vector of each side of the first convex polygon. For each side of the first convex polygon, the side is defined by two adjacent vertices in the polygon's closing order. Calculate the direction vector of the side: subtract the x-coordinate of the previous vertex from the x-coordinate of the next vertex to obtain the x-component of the direction vector; subtract the y-coordinate of the previous vertex from the y-coordinate of the next vertex to obtain the y-component of the direction vector. The direction vector represents the directed line segment of the side from its starting point to its ending point.

[0037] To calculate the outer normal vector of this edge, rotate the direction vector 90 degrees counterclockwise. The operation of rotating 90 degrees counterclockwise is as follows: take the opposite of the vertical component of the direction vector as the horizontal component of the outer normal vector, and take the horizontal component of the direction vector as the vertical component of the outer normal vector; the outer normal vector is perpendicular to this edge and points out of the polygon.

[0038] Calculate the unit outward normal vector: First, calculate the magnitude of the outward normal vector. The magnitude is calculated by adding the square of the horizontal component to the square of the vertical component, and then taking the square root. Then, divide the horizontal component of the outward normal vector by the magnitude, and divide the vertical component by the magnitude to obtain the unit outward normal vector. The unit outward normal vector retains the direction of the outward normal vector while normalizing its length to 1 for easier subsequent projection calculations. Perform the above calculation on each edge of the first convex polygon to obtain the first set of normal vectors. The number of normal vectors in the set is equal to the number of edges of the first convex polygon.

[0039] Calculate the unit outward normal vector of each side of the second convex polygon using the same calculation method as in the first step. Calculate the unit outward normal vector for each side of the second convex polygon one by one to obtain the second set of normal vectors. The number of normal vectors in the set is equal to the number of sides of the second convex polygon.

[0040] Merge the normal vector sets by combining all normal vectors in the first and second normal vector sets into a single candidate normal vector set. The merging operation directly takes the union of the two sets and retains all normal vectors. Each normal vector in the candidate normal vector set represents a possible separating axis direction, and the total number of normal vectors is the sum of the number of sides of the two convex polygons.

[0041] Step S1.5.2: For each normal vector in the candidate normal vector set, project all vertices of the two convex polygons onto the normal vector to obtain two projection intervals. Then, determine whether the two convex polygons overlap or separate along the normal vector direction based on the positional relationship between the two projection intervals, and calculate the separation distance. For each normal vector in the candidate normal vector set, perform the following operations in sequence: The projection of the first convex polygon's vertices involves traversing all vertices of the first convex polygon and calculating the projection value of each vertex onto the current normal vector. The projection value is calculated as follows: the vertex's x-coordinate multiplied by the x-component of the normal vector, plus the vertex's y-coordinate multiplied by the y-component of the normal vector. The projection value represents the vertex's scalar position along the normal vector direction. After traversal, the minimum value is found among all projection values, which is taken as the minimum value of the first projection interval; the maximum value is found, which is taken as the maximum value of the first projection interval. The first projection interval is defined by these minimum and maximum values, representing the scalar range occupied by the first convex polygon along the current normal vector direction.

[0042] The projection of the vertices of the second convex polygon is performed using the same method as in the first step. All vertices of the second convex polygon are traversed, and the projection value of each vertex on the current normal vector is calculated. The minimum value is taken as the minimum value of the second projection interval, and the maximum value is taken as the maximum value of the second projection interval. The second projection interval is defined by the minimum and maximum values.

[0043] To determine the relationship between projection intervals, calculate the minimum value of the first projection interval minus the maximum value of the second projection interval to obtain the first difference. Calculate the minimum value of the second projection interval minus the maximum value of the first projection interval to obtain the second difference. Take the larger of the first difference and the second difference as the overlap length under the candidate normal vector.

[0044] An overlap length greater than 0 indicates that there is an overlapping area between the two projection intervals, and the overlap length is the width of the overlapping area; an overlap length equal to 0 indicates that the boundaries of the two projection intervals are exactly connected; an overlap length less than 0 indicates that there is a gap between the two projection intervals, and the gap width is the absolute value of the overlap length.

[0045] Separation distance calculation: The separation distance under the given normal vector is calculated based on the sign of the overlap length. If the overlap length is less than 0, it means that the two projection intervals are separated, and the separation distance is equal to the absolute value of the overlap length. The specific calculation logic is as follows: if the first difference is greater than the second difference, the separation distance is equal to the absolute value of the first difference; if the second difference is greater than or equal to the first difference, the separation distance is equal to the absolute value of the second difference; if the overlap length is greater than or equal to 0, it means that the two projection intervals overlap or connect, and the two convex polygons intersect or are tangent in the direction of the normal vector, and the separation distance is recorded as 0; the separation distance represents the shortest gap distance between the two convex polygon projection intervals in the direction of the normal vector, and a separation distance of 0 means that the direction of the normal vector is not the separation axis.

[0046] Traverse and collect: Traverse all normal vectors in the candidate normal vector set, perform the first to fourth steps above for each normal vector, collect the separation distance value corresponding to each normal vector, and obtain the separation distance set. In the separation distance set, each element corresponds to the separation distance in the direction of a normal vector.

[0047] Step S1.5.3, determining the shortest distance, refers to the operation of taking the minimum value from the set of separation distances as the shortest distance between two convex polygons.

[0048] The minimum value is retrieved from the set of separation distances by iterating through and comparing them. The iteration and comparison are implemented as follows: the minimum value is initialized as a maximum value, and then each element in the set of separation distances is iterated through sequentially. If the current element value is less than the current minimum value, the minimum value is updated with the current element value. After the iteration is complete, the minimum value is the shortest distance between the two convex polygons.

[0049] This embodiment uses the separating axis theorem to accurately calculate the shortest distance between two convex polygons, which can accurately determine whether the main body region of a character that has been over-segmented by the instance segmentation model belongs to the same character, providing a reliable quantitative basis for subsequent merging judgment. Compared with simple bounding box overlap detection, the separating axis theorem can handle the fine distance calculation between any convex polygons, effectively solving the problem that the main body of a character is mistakenly segmented into multiple disconnected regions due to similar clothing colors to the background, and avoiding the loss of main body information.

[0050] Step S1.6, Convex Polygon Merging and Cropping, refers to determining whether two convex polygons should be merged based on the comparison result between the shortest distance and the preset merging distance threshold, and then cropping and normalizing the size of the merged character instance region to finally output the cleaned image.

[0051] The merging determination involves comparing the shortest distance calculated in step S1.5.3 with a preset merging distance threshold. The preset merging distance threshold is determined as follows: A batch of character instance segmentation results are collected, showing samples that belong to the same character entity but are segmented into multiple disconnected regions. The distribution of the shortest distances between disconnected regions in each sample is statistically analyzed, and the distance value that covers most of the segmentation breaks of the same entity is taken as the merging distance threshold. If the shortest distance is less than the merging distance threshold, the character instance regions corresponding to the two convex polygons are determined to belong to the same character entity, and a merging operation is performed. If the shortest distance is greater than or equal to the merging distance threshold, the two convex polygons correspond to different character instance regions, remain independent, and no merging is performed.

[0052] In this embodiment, if the shortest distance is less than the merging distance threshold, it is determined that the two convex polygons belong to the same character body, and merging is performed.

[0053] Convex polygon merging involves combining all vertices of the first and second convex polygons into a union vertex set. The Graham scan algorithm (using the same algorithm flow as in step S1.4) is then re-executed on this union vertex set to calculate its convex hull, resulting in the merged convex polygon. The merged convex polygon completely covers the entire area of ​​the main character, resolving the issue of lost main character information caused by over-segmentation during instance segmentation.

[0054] The bounding rectangle is calculated for the merged convex polygon. The bounding rectangle is an axis-aligned rectangle, meaning its sides are parallel to the coordinate axes. It is determined as follows: traverse all vertices of the merged convex polygon, record the minimum x-coordinate of all vertices as the left boundary of the rectangle, and the maximum x-coordinate as the right boundary; record the minimum y-coordinate of all vertices as the bottom boundary, and the maximum y-coordinate as the top boundary. The purpose of the bounding rectangle is to provide a regular clipping region, encompassing the irregular main body area of ​​the character.

[0055] The original image is cropped by its bounding rectangle, retaining only the pixels within the rectangle and removing those outside. This cropping operation makes the main subject the absolute core of the image, eliminating irrelevant background information that could interfere with subsequent training.

[0056] The cropped image is scaled to a preset target size. The scaling method uses bilinear interpolation: for each pixel location in the target image, its corresponding floating-point coordinates in the source image are calculated based on the scaling ratio. The pixel value of the nearest integer pixels around these floating-point coordinates is then taken. First, two linear interpolations are performed horizontally (using a distance-weighted average), and then one linear interpolation is performed vertically to obtain the target pixel value. Bilinear interpolation maintains image smoothness during scaling, reducing jagged edges and blur. Size normalization ensures that all training data have a uniform input size, meeting the requirement of a fixed input size for the diffusion model.

[0057] After the above steps, the cleaned image is obtained; the main character in the cleaned image is intact, the background is clean, and the size is uniform, providing high-quality basic data for subsequent data augmentation and feature extraction.

[0058] This embodiment intelligently determines whether separated character regions belong to the same subject and automatically merges them through adaptive comparison of the shortest distance and merging threshold, effectively correcting the over-segmentation problem in instance segmentation. The convex polygon bounding rectangle cropping after merging can accurately remove irrelevant background regions, making the character subject the absolute core of the training data. Size normalization ensures that all training data have a uniform format, meeting the requirements of the diffusion model for a fixed input size. At the same time, bilinear interpolation maintains image quality, providing clean and standardized basic data for subsequent data augmentation and feature extraction.

[0059] Step S2, data augmentation, refers to performing a series of automated image transformation operations on the cleaned images, including geometric transformation, color perturbation, background replacement, and multi-resolution sampling, to expand a small number of original images into a diverse set of augmented images, thereby increasing the number and diversity of training samples and reducing the risk of overfitting the model under conditions of very few samples.

[0060] Step S2.1, geometric transformation enhancement, refers to performing spatial geometric transformation operations such as random horizontal flipping and random rotation within a set angle range on the image to generate variant images with different viewpoints and compositions, enabling the model to learn the consistency of the character's appearance in different spatial directions; the following geometric transformation operations are performed on the cleaned image: For random horizontal flipping, the system generates a uniformly distributed random number between 0 and 1. The uniformly distributed random number is generated by calling a pseudo-random number generator, using the current system timestamp as the random seed, and outputting a floating-point number uniformly distributed between 0 and 1. If this random number is less than a preset flipping probability threshold, a horizontal flip operation is performed: the pixel order of each row of the image matrix is ​​reversed, that is, the pixel value of the i-th row and j-th column in the original image is assigned to the pixel position of the i-th row and (image width minus 1 minus j)-th column in the flipped image. If the random number is greater than or equal to the flipping probability threshold, the original image remains unchanged. The purpose of horizontal flipping is to enable the model to learn the left-right symmetry of the character's appearance, avoiding overfitting to the character's orientation. The preset flipping probability threshold is 0.5, meaning that the probability of flipping and not flipping are equal.

[0061] A rotation angle is randomly sampled within a preset rotation angle range. The sampling method is as follows: a uniformly distributed random number between 0 and 1 is generated, multiplied by the angle range width, and then added to the lower limit of the angle range to obtain the rotation angle. The lower limit of the preset rotation angle range is a negative angle value, and the upper limit is a positive angle value. Equal absolute values ​​indicate that the allowed clockwise and counterclockwise rotation amplitudes are the same. The rotation angle is obtained by rotating the image around its center. The specific implementation of the rotation operation is as follows: a 2x2 rotation matrix is ​​constructed. The element in the first row and first column of the matrix is ​​the cosine of the rotation angle, the element in the first row and second column is the negative of the sine of the rotation angle, the element in the second row and first column is the sine of the rotation angle, and the element in the second row and second column is the cosine of the rotation angle. For each pixel position in the target image, its coordinates are subtracted from the coordinates of the image center to obtain the relative coordinates. The relative coordinates are then multiplied by the inverse of the rotation matrix (i.e., the transpose of the rotation matrix, since the rotation matrix is ​​an orthogonal matrix) to obtain the corresponding floating-point coordinates in the source image. Then, bilinear interpolation is used to obtain the source pixel value. The blank areas generated after rotation are filled with preset fill pixel values. The purpose of the rotation operation is to generate variants with different tilt angles, thereby enhancing the robustness of the model to changes in the character's pose angle.

[0062] For the cleaned image, the above geometric transformation combination is repeated several times to generate multiple geometrically enhanced images. Each time, the horizontal flipping and rotation angle are independently and randomly sampled. The multiple geometrically enhanced images differ from each other in flipping state and rotation angle, forming a set of variations of the character in different spatial orientations. The number of repetitions is determined based on the number of input images and the desired enhancement ratio.

[0063] This embodiment generates diverse spatial geometric variants through random horizontal flipping and random rotation, allowing the model to be exposed to character appearances with different orientations and angles during training. This enables the model to learn the spatial invariance of character appearance features, avoids overfitting to specific orientations, and improves the model's generalization ability to different poses and composition requirements during the inference phase.

[0064] Step S2.2, color enhancement, refers to randomly adjusting the brightness, contrast, and saturation of the image, as well as applying hue shifts, to simulate changes in the character's appearance under different lighting conditions and shooting parameters. This allows the model to learn the invariance of the character's appearance features under color changes. For each geometrically enhanced image, the following color perturbation operation is performed: A brightness adjustment factor is randomly sampled within a preset brightness factor range. The sampling method is as follows: a uniformly distributed random number between 0 and 1 is generated, multiplied by the width of the brightness factor range, and then the lower limit of the brightness factor range is added to obtain the brightness factor. The lower limit of the brightness factor range is less than 1, and the upper limit is greater than 1, allowing the brightness to be either decreased or increased. The red, green, and blue channel values ​​of each pixel in the image are multiplied by the brightness factor. If the brightness factor is greater than 1, the image becomes brighter overall, simulating strong lighting conditions; if the brightness factor is less than 1, the image becomes darker overall, simulating weak lighting conditions. The purpose of brightness adjustment is to enable the model to learn the color consistency of the character's appearance under different lighting intensities.

[0065] A contrast adjustment factor is randomly sampled within a preset contrast factor range. The grayscale mean of all pixels in the image is calculated: for each pixel, the red channel value is multiplied by the red weight coefficient, the green channel value by the green weight coefficient, and the blue channel value by the blue weight coefficient. The sum of these three is the grayscale value of that pixel. The sum of each weight coefficient is 1, and the green weight coefficient is greater than the red and blue weight coefficients (reflecting the human eye's greater sensitivity to green). The grayscale mean is obtained by summing the grayscale values ​​of all pixels and dividing by the total number of pixels. Then, for each color channel of each pixel, the grayscale mean is subtracted from the channel value to obtain the deviation. The deviation is multiplied by the contrast factor and then added to the grayscale mean to obtain the adjusted channel value. If the contrast factor is greater than 1, the bright areas of the image are brighter and the dark areas are darker, resulting in enhanced contrast; if the contrast factor is less than 1, the bright and dark areas of the image tend to be similar, resulting in reduced contrast. The purpose of contrast adjustment is to simulate the contrast differences under different shooting devices and environments.

[0066] Randomly sample a saturation adjustment factor within the preset saturation factor range. Convert the image from the RGB color space to the HSV color space. The RGB to HSV conversion method is as follows: take the maximum and minimum values ​​of the red, green, and blue channels; the difference between the maximum and minimum values ​​is the color difference. The hue value (H) is calculated based on the channel where the maximum value is located: if the maximum value is in the red channel, the hue value equals the difference between the green and blue channels divided by the color difference, multiplied by 60 degrees (if the color difference is 0, the hue value is 0); if the maximum value is in the green channel, the hue value equals the difference between the blue and red channels divided by the color difference, plus 2, multiplied by 60 degrees; if the maximum value is in the blue channel, the hue value equals the difference between the red and green channels divided by the color difference, plus 4, multiplied by 60 degrees. The saturation value (S) is calculated as follows: if the maximum value is 0, the saturation is 0; otherwise, the saturation equals the color difference divided by the maximum value. The lightness value (V) equals the maximum value. Then, each value in the saturation channel is multiplied by a saturation factor. If the result exceeds the upper limit of saturation, it is truncated to the upper limit value; if the result is less than the lower limit of saturation, it is truncated to the lower limit value. Finally, the HSV color space is converted back to the RGB color space. The purpose of saturation adjustment is to simulate the appearance of a character under different levels of color vibrancy, so that the model is not affected by differences in image saturation.

[0067] A hue offset is randomly sampled within a preset hue offset range. The image is converted from RGB to HSV color space (conversion method as in step 3). The hue offset is added to each value of the hue channel. If the result exceeds the upper limit of the hue period, one period is subtracted; if the result is less than the lower limit of the hue period, one period is added (cyclic modulo). The purpose of the hue offset is to simulate the changes in character appearance under different white balance settings and different light source color temperatures, allowing the model to learn the invariance of character appearance features to hue changes. For each geometrically enhanced image, the above color enhancement combination is performed several times. Each time, each adjustment factor is independently and randomly sampled, and multiple color variants are generated for each geometric variant. The number of executions is determined based on the number of input images and the desired enhancement ratio.

[0068] This embodiment simulates diverse lighting environments, shooting conditions, and display device differences in the real world by using multi-dimensional random perturbations of brightness, contrast, saturation, and hue. This enables the model to learn the invariance of character appearance features under color changes, avoids overfitting the model to specific lighting conditions, and improves the model's ability to respond to different lighting descriptions and color styles during the inference stage.

[0069] Step S2.3, background replacement enhancement, refers to using an image segmentation model to segment the main character region from the color-enhanced image, and then overlaying the main character region with a solid color background or a stylized background to generate an enhanced image with background replacement. This effectively reduces the model's overfit to the original fixed background and allows the model to focus on the appearance features of the character itself rather than background information.

[0070] An image segmentation model is loaded, which outputs a segmentation mask for the target region based on the input image and cue information. The image segmentation model used in this embodiment consists of three parts: an image encoder, a cue encoder, and a mask decoder. The image encoder employs a Vision Transformer architecture, encoding the input image into high-dimensional image feature embeddings; the cue encoder receives cue information such as points, boxes, or text, and encodes the cue into cue feature embeddings; the mask decoder performs cross-attention fusion of the image feature embeddings and the cue feature embeddings, outputting a segmentation mask.

[0071] The system constructs segmentation prompts: It takes the center point coordinates of the circumscribed rectangle of the merged convex polygon obtained during the data cleaning stage. The center point coordinates are calculated as follows: the x-coordinate of the left boundary of the circumscribed rectangle plus the x-coordinate of the right boundary, divided by 2, gives the x-coordinate of the center point; the y-coordinate of the lower boundary of the circumscribed rectangle plus the y-coordinate of the upper boundary, divided by 2, gives the y-coordinate of the center point. This center point is used as the input for the image segmentation model, with the prompt type being a foreground point, indicating that the point is located within the target region to be segmented. After receiving the image and the point prompt, the image segmentation model outputs a segmentation mask. The segmentation mask is a binary image with the same size as the input image, where the pixel value of the main character region is 1, and the pixel value of the background region is 0.

[0072] The second step involves the system randomly selecting a background type: A uniformly distributed random number between 0 and 1 is generated. If the random number is less than the preset background type selection threshold, a solid color background is selected for overlay; otherwise, a stylized background overlay is selected. A solid color is randomly selected from a preset color library, which is a list containing multiple RGB color values, covering solid colors with different brightness and hues. The random selection method is to generate a random integer within the index range of the color library and use it as an index to retrieve the corresponding color. A solid color background image with the same size as the color-enhanced image is created, and all three channel values ​​of all pixels are set to the RGB values ​​of the selected color. Then, overlay compositing is performed: Each pixel position is traversed. If the segmentation mask value at that position is 1, the composite image at that position takes the pixel value of the foreground image (color-enhanced image); if the segmentation mask value at that position is 0, the composite image at that position takes the pixel value of the solid color background image. The purpose of overlay compositing is to seamlessly blend the subject with the background while maintaining complete control over the background.

[0073] A background image is randomly selected from a pre-defined stylized background image library. This library contains images of various scene styles, including indoor scenes, natural landscapes, cityscapes, and abstract textures. The random selection process involves reading the path list of all image files in the library, generating a random integer within the path list's index range as an index, and loading the corresponding background image. The background image is then scaled to the same size as the color-enhanced image. It is then overlaid using the same masking method as for a solid-color background. For each color-enhanced image, background replacement is performed several times, generating multiple background-replaced enhanced images. The number of iterations depends on the number of input images and the desired enhancement ratio.

[0074] This embodiment replaces the background by overlaying the main character with a variety of solid-color backgrounds or stylized scene backgrounds, effectively severing the false association between the character's appearance and the original fixed background. This prevents the model from relying on background information to identify the character during training, thereby forcing the model to focus on the character's own visual features (such as face, clothing, body shape, etc.) and significantly improving the model's ability to maintain character consistency under different scene backgrounds during the inference stage.

[0075] Step S2.4, multi-resolution sampling, refers to the operation of generating multiple versions of the background replacement and enhancement image at different resolutions, so that the training data covers different image sizes, improves the generalization ability of the LoRA model under multi-scale inputs, and ensures that the model can stably generate highly consistent character images regardless of the output resolution specified by the user during inference.

[0076] For each background replacement and enhancement image, multiple versions at different resolutions are generated. The number and specific sizes of the resolution versions are preset based on the resolution range supported by the base model and training requirements. The generation method involves scaling the images to the preset resolution sizes. The scaling method uses bilinear interpolation: first, the horizontal scaling ratio (target width divided by source image width) and the vertical scaling ratio (target height divided by source image height) are calculated. Then, for each pixel position in the target image, its corresponding floating-point coordinates in the source image are calculated based on the scaling ratio. The pixel values ​​of the nearest integer pixels around these floating-point coordinates are taken and weighted by distance to obtain the target pixel value.

[0077] The purpose of multi-resolution sampling is to provide data support for the multi-scale training of the diffusion model. During training, the diffusion model adds noise and removes noise from images at different resolutions. Multi-resolution sampling enables the LoRA weights to adapt to feature representations at different scales.

[0078] After the four sub-steps of geometric transformation enhancement, color enhancement, background replacement enhancement, and multi-resolution sampling, the original cleaned image is expanded into an enhanced image set. The number of images in the enhanced image set is determined by the product of the number of times each sub-step is executed. Under conditions of very few samples (e.g., only one input image), the number of images in the enhanced image set should be sufficient to support the effective convergence of LoRA fine-tuning training.

[0079] This embodiment generates training samples covering different sizes through multi-resolution sampling, allowing the LoRA weights to be exposed to character feature representations at different scales during training. As a result, regardless of the output resolution specified by the user during the inference phase, the model can stably generate high-quality, highly consistent character images, avoiding the problem of reduced generation quality caused by mismatch between training and inference resolutions.

[0080] Step S3, automatic image description and trigger word injection, specifically includes: The process involves calling a multimodal visual language model (VLM) to perform forward inference on each enhanced image, automatically generating natural language descriptions of the character's appearance attributes, clothing attributes, expressions, and postures. This achieves fully automated annotation of training data without human intervention.

[0081] The system invokes a multimodal visual language model, which consists of three core components: a visual encoder, employing a Vision Transformer architecture, which segments the image into fixed-size image patches and encodes them into an embedding sequence; a query transformer, or Q-Former (Querying Transformer), which acts as a bridge between the visual encoder and the language decoder, extracting language-related visual representations from visual features through a set of learnable query vectors; and a language decoder, employing an autoregressive language model architecture, which generates subsequent text word by word based on the prefix text and visual representations. The specific implementation process is as follows: Image input and visual encoding involve scaling the enhanced image to the standard input size of the visual encoder using bilinear interpolation. The scaled image is then segmented into image patches of a preset size. The pixel values ​​of each image patch are flattened into a vector, which is then mapped to an embedding vector using a linear projection matrix. The number of rows in the linear projection matrix corresponds to the dimension of the flattened image patch vector, and the number of columns corresponds to the hidden dimension of the visual encoder. A corresponding positional encoding is added to each embedding vector. This positional encoding is a set of learnable vectors, with each position corresponding to a unique encoded vector representing the spatial location of the image patch within the image. The embedding vector sequence is then input into a multi-layer Vision Transformer for forward propagation. Each Transformer layer contains a multi-head self-attention layer and a feedforward network layer, outputting a feature vector sequence with the same dimension as the input. The feature vector sequence output from the last layer is then used as the visual features.

[0082] The visual-language bridging mechanism, the query transformer, receives visual features and a fixed set of learnable query vectors as input. These query vectors interact with the visual features through a cross-attention layer, where the query originates from the query vector, and the keys and values ​​are derived from the visual features, extracting a compact visual representation relevant to the linguistic description. The output of the query transformer is a set of visually augmented query vectors, which serve as the visual conditional input to the language decoder.

[0083] The language description generation process begins with a language decoder using a predefined prefix phrase as the starting text. This prefix text is converted into a sequence of tokens, each token mapped to a word embedding vector via a word embedding matrix. Positional encoding is then added before inputting this vector into the language decoder's Transformer layer. Within each Transformer layer, the language decoder's self-attention layer handles dependencies within the text sequence, while the cross-attention layer uses text features as queries and the visual representation output by the query transformer as keys and values, integrating visual information into the text generation process. The language decoder then autoregressively predicts the probability distribution of the next token word by word. Autoregression means generating a token, appending it to the end of an existing sequence, and then using this sequence to predict the next token until an end marker is generated. The token with the highest probability is taken as the output, ultimately generating a complete natural language description text. This description text covers character appearance attributes (such as hairstyle, hair color, eye color, skin tone, and facial features), clothing attributes (such as style, color, texture, and accessories), facial expressions, and posture.

[0084] This embodiment automatically generates descriptive text through a multimodal visual language model, achieving full automation of training data annotation without human intervention, significantly reducing the data preparation threshold and labor costs for character customization; at the same time, the model can automatically capture fine-grained information such as character appearance attributes, clothing attributes, expressions and postures, providing rich and accurate text conditions for subsequent training, enabling LoRA fine-tuning to learn the precise correspondence between character appearance and text description.

[0085] Trigger word injection refers to the process of inserting a trigger word uniquely corresponding to a role at a specified position in the automatically generated descriptive text, resulting in descriptive text with the trigger word. The purpose of the trigger word is to bind the role concept to specific text tags during the training phase. During the inference phase, the user only needs to include the trigger word in the prompts to activate the corresponding role's LoRA weights and features. The specific implementation process is as follows: The system reads a preset trigger word string from the character configuration. Each trigger word is a string consisting of letters, numbers, and underscores, corresponding one-to-one with a character. The trigger word is then inserted into the automatically generated description text at a specified position. This specified position can be at the beginning, end, or a specific location within the text. The insertion method is string concatenation: the trigger word string, separators (such as commas and spaces), and the original description text are concatenated in sequence to obtain the description text with the trigger word.

[0086] This embodiment establishes a stable binding relationship between character concepts and specific text tags by inserting a unique trigger word into the description text, enabling the model to learn to associate the trigger word with the character's appearance features during training. In the inference phase, users only need to include the trigger word in the prompt to accurately activate the corresponding character LoRA weights and feature files, without the need for complex parameter configuration or model switching operations, thus achieving convenient character generation control.

[0087] Tag quality verification refers to verifying the text length and content matching of descriptive text with trigger words, filtering out low-quality text that fails the verification, and ensuring the quality of the annotation operation for training images and text.

[0088] Text length validation uses a preset text length threshold to count the total number of characters (including letters, spaces, and punctuation) in each descriptive text containing trigger words. If the total number of characters is greater than or equal to the threshold, the validation passes; if the total number of characters is less than the threshold, the descriptive text is deemed too brief and is filtered out, and the corresponding augmented image is also removed from the training dataset. The purpose of text length validation is to filter out low-quality descriptions caused by failures in visual language model generation or overly simplistic image content.

[0089] Content matching verification utilizes an image-text matching model to calculate the semantic similarity between the enhanced image and the descriptive text with trigger words. The image-text matching model includes an image encoder and a text encoder, which are jointly trained on large-scale image-text pair data to ensure that the matched image-text pairs have a high cosine similarity in the embedding space.

[0090] The enhanced image is scaled to the standard input size of the image encoder and input into the image encoder. The vector corresponding to the classification label output from the last layer is taken as the image embedding vector. The text with trigger words is converted into a word sequence by a text segmenter, and start and end tags are added. This sequence is input into the text encoder, and the output vector corresponding to the last word is mapped by a linear projection layer to obtain the text embedding vector. The image embedding vector and the text embedding vector are multiplied element-wise and then summed to obtain the vector dot product. The magnitude of the image embedding vector (the square root of the sum of the squares of each element) and the magnitude of the text embedding vector are calculated separately. The vector dot product is divided by the product of the magnitudes of the image embedding vector and the text embedding vector to obtain the cosine similarity. The cosine similarity ranges from -1 to +1, with a larger value indicating a better semantic match between the image and the text.

[0091] A preset similarity threshold is set. If the cosine similarity is greater than or equal to the threshold, the image-text pair passes the verification and is retained. If the cosine similarity is less than the threshold, it is determined that the descriptive text and image content do not match and are filtered out. The purpose of content matching verification is to filter out descriptive errors or mismatches between images and text that may be generated by the visual language model.

[0092] The training dataset consists of image-text pairs formed by pairing trigger-word description texts with corresponding enhanced images that pass text length and content matching checks. The dataset is stored in a disk file in a structured text format: each line in the file is a structured data object containing an image file path field and a description text field. The structured text format facilitates line-by-line reading and streaming processing, making it suitable for loading large-scale training data.

[0093] This embodiment automatically filters out low-quality and mismatched labeled samples by double-checking text length and content matching, ensuring the labeling quality of the training dataset; high-quality image-text pairs in the training data enable LoRA fine-tuning to learn a more accurate correspondence between character appearance and text description, thereby improving the consistency and accuracy of character generation in the reasoning stage.

[0094] Step S4, semantic feature vector extraction and role feature file generation, specifically includes: Semantic feature vector extraction refers to the operation of using a pre-trained image encoder to perform forward reasoning on the cleaned image to extract high-dimensional semantic feature vectors. These feature vectors encode the semantic information of a character's appearance, including facial features, clothing features, and body proportions. Here, a cleaned image is used instead of an enhanced image to ensure the purity and stability of the feature extraction and avoid noise interference from enhancement. This embodiment uses a pre-trained visual encoder as the image encoder. This encoder adopts the Vision Transformer architecture and is pre-trained on large-scale image-text pairs, enabling it to extract high-dimensional feature vectors containing rich semantic information. The specific implementation process is as follows: The cleaned image is scaled to the standard input size of the visual encoder using bilinear interpolation. The image pixel values ​​are normalized from the original integer range to a floating-point range of 0 to 1 (each pixel value is divided by 255). Then, each color channel is standardized separately: the pixel value of each channel is subtracted from the mean value of that channel on the pre-training dataset, and then divided by the standard deviation of that channel on the pre-training dataset. Standardization ensures that the data distribution of the input image matches the data distribution during the training of the pre-trained encoder, guaranteeing the accuracy of feature extraction.

[0095] The preprocessed image is segmented into image patches of a preset size. The pixel values ​​of each image patch are flattened into a vector, which is then mapped to an embedding vector through a linear projection matrix. The parameters of the linear projection matrix are learned during the pre-training phase and remain frozen during feature extraction. Simultaneously, a learnable location encoding vector is added to each image patch embedding vector. This location encoding vector provides the Transformer with spatial location information for the image patch, as the Transformer's self-attention mechanism itself lacks location awareness. Furthermore, a learnable classification label embedding vector is added before the image patch sequence; this classification label is used to aggregate global image information.

[0096] The embedding vector sequence (one classification label plus several image patches) is input into a multi-layer Vision Transformer. Each Transformer layer contains the following sub-layers: Multi-head self-attention layer: Each vector in the input sequence is mapped to a query vector, key vector, and value vector through three different linear projection matrices (query projection matrix, key projection matrix, and value projection matrix), respectively. The query vector, key vector, and value vector are then divided into multiple subspaces according to a preset number of attention heads. In each subspace, the dot product of the query vector and the key vector is calculated, divided by the square root of the key vector's dimension (a scaling factor to prevent the dot product from becoming too large and causing the softmax gradient to vanish), and then normalized using the softmax function (softmax function: taking the exponent of each element and then dividing by the sum of all element exponents, so that the output value is between 0 and 1 and the sum is 1), resulting in the attention weight matrix. The attention weight matrix is ​​multiplied by the value vector to obtain the attention output for that subspace. The attention outputs of all subspaces are concatenated and then mapped through the output projection matrix to obtain the final output of the multi-head self-attention layer.

[0097] Feedforward network layer: It contains two linear projection matrices and a non-linear activation function. The input is first mapped to a higher dimension (usually several times the hidden dimension) through the first linear projection matrix, then through the non-linear activation function, and finally mapped back to the original dimension through the second linear projection matrix.

[0098] Each sublayer is followed by a residual connection (which directly adds the sublayer's input to its output) and layer normalization (which calculates the mean and variance of each sample's feature dimensions, then normalizes and scales / translates them). The residual connection allows gradients to propagate directly to shallower layers, mitigating the vanishing gradient problem in deeper networks; the layer normalization stabilizes the feature distribution during training.

[0099] The vector corresponding to the classification label output by the last Transformer layer is taken. This vector is aggregated with global information from multiple Transformers and contains the semantic information of the entire image. This vector is then mapped to the multimodal embedding space through a linear projection layer to obtain the final semantic feature vector. This semantic feature vector encodes the appearance semantic information of the character, including features such as hairstyle and hair color, eye color and skin color, clothing style, color and texture, and geometric proportions of facial features.

[0100] In this embodiment, a high-dimensional semantic feature vector is extracted from the cleaned image by a pre-trained visual encoder, compressing the character's appearance information into a compact vector representation. This vector encodes the character's facial features, clothing features, body proportions, and other core semantic information. Using the cleaned image instead of the augmented image for feature extraction ensures the purity and stability of the feature vector, preventing interference from data augmentation noise. The semantic extraction capability learned by the pre-trained encoder on large-scale data enables the extraction of sufficiently rich character appearance feature representations even with only one input image.

[0101] Multi-image feature aggregation refers to the operation of fusing the high-dimensional semantic feature vectors of multiple cleaned images through mean pooling or attention-weighted fusion to aggregate them into a single stable character feature representation when there are multiple cleaned images. The purpose of aggregation is to extract the most representative and stable character semantic features from multiple character images from different angles and poses. In this embodiment, there is only one cleaned image, so the semantic feature vector extracted in step S4.1 is directly used as the final character semantic feature vector, skipping the aggregation step. If there are multiple images, the system provides two optional aggregation strategies: Mean pooling strategy: The feature vectors of all images are summed element-wise, and then each element of the sum vector is divided by the number of images to obtain the aggregated average feature vector. Mean pooling is computationally simple and efficient, and is suitable for scenes where the image quality is similar and the viewpoint coverage is uniform. It assumes that each image contributes equally to the representation of the character's features.

[0102] Attention-weighted fusion strategy: A learnable query vector with the same dimension as the feature vectors is used. For each feature vector, the dot product of the query vector and the feature vector (element-wise multiplication and summation) is calculated to obtain the attention score of that feature vector. The attention scores of all feature vectors are normalized using a softmax function: the natural exponent of each score is taken, and then divided by the sum of the natural exponents of all scores to obtain the attention weight of each feature vector. The weight values ​​are between 0 and 1, and the sum of all weights is 1. Each feature vector is multiplied by its corresponding attention weight, and then all weighted feature vectors are summed element-wise to obtain the aggregated weighted feature vector. The purpose of attention-weighted fusion is to automatically assign higher weights to images with more information and more complete character appearances, suppressing the influence of low-quality or severely occluded images.

[0103] In this embodiment, for multi-image scenes, two aggregation strategies are provided: mean pooling and attention-weighted fusion. These strategies can extract the most representative feature representations from character images from multiple angles and poses. The attention-weighted fusion strategy automatically evaluates the information content of each image through a learnable query vector, assigns higher weights to high-quality images, and suppresses the interference of low-quality samples. Compared with simple mean pooling, it can generate more stable and discriminative character semantic features.

[0104] Feature file serialization refers to the operation of organizing semantic feature vectors and related metadata into a standardized data structure and persistently storing them in binary format to a disk file. The purpose of serialization is to achieve persistent storage and cross-stage reuse of role features; feature files extracted during the training phase can be directly loaded and used during the inference phase without needing to be extracted again. The specific implementation process is as follows: The first step is metadata organization. The system builds a metadata dictionary containing the following fields: Role ID field: String type, stores the unique identifier of the role, used to uniquely distinguish different roles in a multi-role feature library.

[0105] Trigger word string field: String type, stores the trigger word bound to this role, used to activate the corresponding role during the reasoning phase.

[0106] Feature vector dimension field: Integer type, stores the number of dimensions of the semantic feature vector, used to verify whether the vector dimensions are correct when loading the feature file.

[0107] Encoder type field: String type, stores the name and version of the pre-trained image encoder used for feature extraction, used to select the correct encoder or confirm compatibility when loading the feature file.

[0108] Extract timestamp field: string type, stores the time of feature extraction, using standard time format, for version management and traceability.

[0109] The source image hash value field is a string type that stores the hash value calculated for the cleaned image. The hash value is calculated by inputting the binary data of the image file into a hash algorithm. The algorithm processes the data in blocks and outputs a fixed-length hash value, represented in hexadecimal characters. The hash value is used to verify the correspondence between the feature file and the original image, preventing the feature file from being misused or tampered with.

[0110] The second step is serialization and storage. The semantic feature vector (a multi-dimensional floating-point tensor) and the metadata dictionary are combined into a single data structure and saved to a disk file using a binary serialization format. The serialization process involves calling a serialization function to convert the data structure into a binary byte stream and writing it to the target file path. This feature file is extremely lightweight, facilitating storage, network transmission, and multi-role feature library management. Furthermore, the feature file is independent of the specific base model and can be flexibly used with different versions of the diffusion model.

[0111] This embodiment achieves persistent storage and cross-stage reuse of role features by serializing semantic feature vectors and metadata into independent feature files. Feature files extracted during the training phase can be directly loaded during the inference phase without re-extraction, improving process efficiency. The feature file size is extremely lightweight (usually only a few KB), much smaller than LoRA weight files and the complete model, resulting in extremely low storage costs in multi-role scenarios. The feature file is independent of the specific base model, has good architectural compatibility, and can be flexibly used with different versions of diffusion models. The hash value verification mechanism in the metadata ensures the correspondence between the feature file and the original image, preventing misuse.

[0112] Step S5: LoRA fine-tuning training. LoRA fine-tuning training refers to the operation of fine-tuning the pre-trained diffusion model by combining the training dataset and character feature files with the image and text. During training, only a small number of low-rank factorization matrix parameters are updated, and all parameters of the original model are frozen, which greatly reduces the number of trainable parameters and training time. At the same time, the semantic feature vectors in the character feature files are injected into the conditional control path of the model to enhance the learning of character features.

[0113] Model loading and architecture identification refers to loading a pre-trained diffusion model and automatically identifying its architecture type (UNet architecture or DiT architecture) so that the LoRA low-rank decomposition matrix can be inserted into the corresponding network position according to the architecture type.

[0114] Load the pre-trained diffusion model, which contains the following core components: A Variational Autoencoder (VAE) consists of an encoder and a decoder. The encoder compresses the pixel-space image into the latent space, with the compression ratio determined by the number of downsampling layers in the VAE. The decoder restores the latent space representation back to the pixel-space image. The VAE's role is to significantly reduce computation by performing diffusion and denoising in the low-dimensional latent space.

[0115] Text encoder: Converts text prompts into a sequence of text feature vectors, which serve as conditional inputs to the diffusion model. Different versions of the diffusion model may use text encoders with different architectures and scales.

[0116] Denoising Network: The denoising network is the core of the diffusion model, responsible for predicting noise based on the current noisy latent representation, diffusion time step, and conditional information. The architecture of the denoising network is divided into two main categories: UNet architecture and DiT (Diffusion Transformer) architecture.

[0117] The architecture type field in the loaded model configuration dictionary is read. The architecture type field is a string that is written by the model developer during model training and saving. If the value of this field contains a UNet-related identifier, it is determined to be a UNet architecture; if the value of this field contains a DiT or Transformer-related identifier, it is determined to be a DiT architecture. In this embodiment, the loaded model is a UNet architecture.

[0118] The denoising network structure of the UNet architecture, from input to output, is as follows: input convolutional layer (performs preliminary feature extraction of latent representation and additional conditions), downsampling encoder (contains multiple downsampling stages, each stage containing multiple residual blocks, each residual block followed by a downsampling layer, halving the feature map space size and doubling the number of channels), intermediate layer (contains residual blocks, located at the lowest spatial resolution layer of UNet), upsampling decoder (contains multiple upsampling stages, each stage containing multiple residual blocks, each residual block preceded by an upsampling layer, doubling the feature map space size and halving the number of channels, while concatenating the feature maps from the corresponding stages of the encoder through skip connections), and output convolutional layer (maps the feature map back to the latent space dimension). Each residual block contains: a temporal embedding projection layer (encodes the diffusion time step into a conditional vector), a textual conditional cross-attention layer (using image features as queries and text features as keys and values), a self-attention layer (using the image features themselves as queries, keys, and values), and a feedforward network layer.

[0119] In this embodiment, by automatically identifying the model architecture type, the system can adaptively select the correct LoRA insertion strategy and feature injection location without requiring the user to manually specify the model architecture, thus achieving universal compatibility with different diffusion models such as UNet and DiT. The architecture identification mechanism enables the system to adapt to the ever-evolving ecosystem of basic models, and when a new diffusion model architecture appears, it can be supported simply by extending the architecture identification rules.

[0120] LoRA low-rank factorization matrix insertion refers to the operation of inserting LoRA low-rank factorization matrices (matrix A and matrix B) into the attention layer of the corresponding network location based on the identified model architecture type. The core idea of ​​LoRA is that the weight matrix of the pre-trained model has low-rank characteristics in terms of parameter update when adapting to new tasks. Therefore, the parameter update can be approximated by the product of two low-rank matrices, which greatly reduces the number of trainable parameters.

[0121] Based on the identified UNet architecture type, the LoRA low-rank decomposition matrix is ​​inserted into the query projection matrix, key projection matrix, value projection matrix, output projection matrix, and linear projection matrix of the feedforward network layer in each residual block attention layer of the UNet.

[0122] The insertion method is illustrated using the query projection matrix as an example. The original query projection matrix is ​​a two-dimensional weight matrix, with the number of rows representing the input feature dimension and the number of columns representing the output feature dimension. In the standard forward computation, the input feature matrix is ​​multiplied by the original query projection matrix to obtain the query matrix. During this computation, the parameters of the original query projection matrix remain frozen, i.e., the `requires_grad` attribute of the parameters is set to `False`, and they do not participate in gradient backpropagation or parameter updates.

[0123] After inserting LoRA, the system creates two additional low-rank decomposition matrices: Matrix A has the dimension of the input feature dimension multiplied by the rank r of its columns, where r is the rank, and r is much smaller than both the input and output feature dimensions. Matrix A is initialized by randomly sampling each element from a Gaussian distribution (normal distribution) with a mean of 0 and a standard deviation of 1 divided by the square root of r. The Gaussian random sampling is implemented by first generating a uniformly distributed random number between 0 and 1, then converting it to a standard normally distributed random number using a Box-Muller transformation, and finally multiplying it by the target standard deviation to obtain the initial value. The purpose of using Gaussian random initialization for matrix A is to break the symmetry, allowing different dimensions to learn different features.

[0124] Matrix B: Dimensions are rank r (r rows) multiplied by the output feature dimension (columns). Matrix B is initialized with all elements set to 0. This zero initialization ensures that at the start of training, the product of matrix A and matrix B is zero, the LoRA bias term is zero, and the output of the original model remains unchanged, guaranteeing that training starts from the performance level of the original model.

[0125] The forward computation process after inserting LoRA is as follows: First, the input feature matrix is ​​multiplied by the original query projection matrix (frozen, not updated) to obtain the original query output; Second, the input feature matrix is ​​multiplied by matrix A to obtain an intermediate matrix (the dimension of which is batch size multiplied by sequence length multiplied by r); Third, the intermediate matrix is ​​multiplied by matrix B to obtain the LoRA bias output (the dimension of which is batch size multiplied by sequence length multiplied by output feature dimension); Fourth, the corresponding elements of the original query output and the LoRA bias output are added to obtain the final query matrix. During backpropagation, the gradients of the loss function with respect to matrices A and B are calculated and the parameters of these two matrices are updated respectively; the original query projection matrix does not participate in the gradient calculation.

[0126] The rank r is set within a preset range. The lower limit of this range ensures minimum expressive power, while the upper limit is constrained by the balance between the number of trainable parameters and training resources. A larger rank r results in more trainable parameters and stronger expressive power, but also increases training time and memory usage. A smaller rank r results in fewer parameters and faster training, but may be insufficient to capture specific appearance features of a character. The principle for selecting rank r is to achieve a balance between parameter efficiency and expressive power.

[0127] Similarly, insert the corresponding LoRA matrix pairs for the key projection matrix, value projection matrix, output projection matrix, and linear projection matrix of the feedforward network layer.

[0128] This embodiment, through LoRA low-rank factorization matrix insertion, requires training only a very small number of parameters (typically one-tenth to one-hundredth of the original model parameters) to achieve customized adaptation for the character, significantly reducing training memory requirements and training time, making minute-level fast training on consumer-grade GPUs possible; Gaussian random initialization of matrix A and all-zero initialization of matrix B ensure that training starts at the performance level of the original model, avoiding instability in the early stages of training; the flexibly configurable rank r parameter allows users to freely balance training efficiency and generation quality based on character complexity and available resources.

[0129] Character feature file loading and injection refers to loading semantic feature vectors from the character feature file generated by the feature extraction module during the training process, and injecting them as additional conditional inputs into the conditional control path of the pre-trained diffusion model. Together with textual conditions, these vectors guide the noise prediction network to learn character-specific appearance information.

[0130] Feature file loading: The system reads the character feature file from the disk, uses a deserialization function to convert the binary byte stream into a data structure, extracts semantic feature vectors (multidimensional floating-point tensors) and metadata dictionary from it, verifies whether the trigger word string in the metadata is consistent with the trigger word of the currently trained character, and whether the feature vector dimension is consistent with the expected dimension.

[0131] Feature projection involves introducing a learnable linear projection layer, which is a two-dimensional matrix. The number of rows corresponds to the input dimension of the semantic feature vector, and the number of columns corresponds to the conditional feature dimension of the cross-attention layer. The parameters of the projection matrix are randomly initialized before training using a Gaussian distribution with a mean of 0 and a standard deviation of 1 divided by the square root of the input dimension. During training, these parameters participate in gradient updates. The projection operation is calculated by multiplying the semantic feature vector (considered as a matrix with 1 row multiplied by the input dimension's columns) by the projection matrix (input dimension rows multiplied by the conditional feature dimension columns), resulting in the mapped role conditional feature vector. The projection layer's role is to map the semantic feature vector from the image encoder's embedding space to the conditional embedding space of the diffusion model's cross-attention layer, aligning the feature spaces of the two different models.

[0132] Feature injection occurs in the cross-attention layer of the denoising network. The original conditional input comes from the text feature vector sequence output by the text encoder. This text feature vector sequence is mapped to a text key vector sequence via a key projection matrix and to a text value vector sequence via a value projection matrix. In the cross-attention calculation, the query is derived from the query vector sequence projected from the image features of the current layer of the denoising network, where the key is the text key vector sequence and the value is the text value vector sequence.

[0133] The feature injection method of this invention is as follows: the mapped character conditional feature vector is treated as an additional conditional vector, and it is concatenated with the text feature vector sequence in the sequence dimension to form an extended key sequence and an extended value sequence. Specifically, the character conditional feature vector is processed through the key projection matrix and the value projection matrix (sharing these two projection matrices with the text features) to obtain a character key vector and a character value vector (each a single vector with a sequence length of 1). The character key vector is concatenated to the end of the text key vector sequence to obtain the extended key sequence; the character value vector is concatenated to the end of the text value vector sequence to obtain the extended value sequence. During cross-attention calculation, the query performs a dot product with all vectors in the extended key sequence, and after softmax normalization, the attention weights are obtained. The attention weights are then weighted and summed with all vectors in the extended value sequence to obtain the cross-attention output. Through this injection method, the character semantic feature vector participates in cross-attention calculation at every denoising time step, jointly guiding the noise prediction process with the text conditions.

[0134] This embodiment establishes a stable feature transfer bridge between training and inference by explicitly injecting the character semantic feature vector into the conditional control path of the diffusion model. This allows the model to perceive the character's appearance semantic information at each denoising time step, thus enabling it to learn a stable character appearance representation even under conditions with very few samples (such as only one image). The learnable parameters of the projection layer enable adaptive alignment between the embedding space of the image encoder and the conditional space of the diffusion model, improving the effectiveness of feature injection. The synergistic effect of the feature vector and the text conditions allows the model to maintain character consistency while flexibly responding to diverse descriptions such as pose, scene, and lighting in the text prompts.

[0135] Training hyperparameter configuration refers to setting various hyperparameters for LoRA fine-tuning training, including LoRA rank, number of training steps, learning rate scheduling strategy, warm-up steps, and gradient clipping threshold, in order to control the training process and optimize the results.

[0136] Set the rank r of the LoRA low-rank decomposition matrix to a preset value. The selection principle for rank r is: within the preset rank range, choose an appropriate value based on the number of input images, character complexity, and training resources. The fewer the number of input images or the more complex the character appearance, the larger the selected rank r should be to provide sufficient expressive power; the more input images or the simpler the character appearance, the smaller the selected rank r can be to improve training efficiency.

[0137] The number of training steps is adaptively calculated based on the effective sample size of the augmented image set, the batch size, and the preset number of repetition rounds. The total number of training steps is calculated as: effective sample size multiplied by the number of repetition rounds divided by the batch size. If the calculated number of steps exceeds the preset recommended number of steps, the system automatically adjusts the number of repetition rounds to bring the number of steps back to the recommended range. The lower limit of the recommended number of steps ensures that the model has enough iterations to learn the role features, while the upper limit prevents overtraining and overfitting.

[0138] A cosine annealing strategy is used to schedule the learning rate. The learning rate curve of the cosine annealing strategy decreases slowly in the early stage of training (maintaining a large learning rate for rapid convergence), decreases rapidly in the middle stage of training, and flattens out at the end of training (approaching the minimum learning rate for fine-tuning). The specific calculation method is as follows: Training progress ratio calculation: The current number of training steps is divided by the total number of training steps to obtain the training progress ratio, which ranges from 0 to 1.

[0139] Cosine decay coefficient calculation: Multiply the training progress ratio by π to obtain an angle value, which ranges from 0 to π; take the cosine value of this angle value, which ranges from 1 to -1; add 1 to the cosine value and divide by 2 to obtain the decay coefficient, which ranges from 0 to 1, is 1 at the beginning of training and 0 at the end of training.

[0140] Current learning rate calculation: Multiply the decay factor by the difference between the initial learning rate and the minimum learning rate, and add the minimum learning rate to obtain the learning rate for the current step. The initial learning rate and minimum learning rate are preset based on parameters such as the base model type, the amount of training data, and the LoRA rank.

[0141] The number of warm-up steps is set to a preset value, much smaller than the total number of training steps. During the warm-up phase, the learning rate linearly increases from the minimum learning rate to the initial learning rate. The linear increase is calculated as follows: the current number of warm-up steps is divided by the total number of warm-up steps to obtain the warm-up progress ratio, which ranges from 0 to 1; the warm-up progress ratio is multiplied by the difference between the initial learning rate and the minimum learning rate, and then added to the minimum learning rate to obtain the current learning rate during the warm-up phase. The purpose of the warm-up mechanism is to avoid gradient oscillations and training instability caused by an excessively large learning rate in the early stages of training, allowing the model parameters to adjust slowly in the early stages of training and gradually enter a stable training state.

[0142] The gradient norm threshold is set to a preset value. After each backpropagation gradient calculation, a gradient clipping operation is performed: the global L2 norm of the gradients of all trainable parameters (i.e., the parameters of all LoRA matrices A and B, as well as the projection layer parameters) is calculated. The L2 norm is calculated by squaring each gradient value, summing all the squared values, and then taking the square root of the sum. If the L2 norm is greater than the threshold, each gradient value is multiplied by the threshold and divided by the L2 norm (i.e., the gradient is scaled proportionally so that the scaled L2 norm is exactly equal to the threshold). If the L2 norm is less than or equal to the threshold, the gradient remains unchanged. The purpose of gradient clipping is to prevent excessive parameter updates caused by anomalous single batch data or gradient explosion, and to ensure the stability and convergence of the training process.

[0143] In this embodiment, adaptive training step calculation dynamically adjusts the number of training steps based on the actual amount of data, avoiding underfitting or overfitting caused by a fixed number of steps; cosine annealing learning rate scheduling enables rapid convergence in the early stages of training and fine-tuning in the later stages, balancing training speed and model quality; the warm-up mechanism effectively avoids gradient oscillations in the early stages of training, improving training stability; gradient clipping prevents parameter mutations caused by abnormal samples, ensuring a smooth and controllable training process.

[0144] Loss function and optimization refer to defining the objective function (loss function) for training optimization and calculating the loss value, backpropagating the gradient, and updating the LoRA parameters in each training iteration.

[0145] This embodiment uses the standard diffusion denoising loss, i.e., the mean squared error noise prediction loss. For each pair of image and text data in the training dataset, the training process for each iteration is as follows: The enhanced image is input into the VAE encoder. The VAE encoder performs layer-by-layer convolution, downsampling, and normalization operations on the image, compressing it from the pixel space to the latent space. The spatial size of the latent space is the original image size divided by the VAE compression ratio, and the number of channels is the number of latent channels in the VAE. The role of VAE encoding is to map the high-dimensional pixel space image to the low-dimensional latent space, enabling the diffusion model to perform noise addition and denoising in the latent space, significantly reducing the computational cost.

[0146] A diffusion time step is uniformly and randomly sampled from an integer range of 1 to the total number of diffusion steps. The uniform random sampling method is as follows: generate a uniformly distributed random number between 0 and 1, multiply it by the total number of diffusion steps, round down, and then add 1; the diffusion time step represents the degree of noise addition, the larger the time step, the more noise is added, and the more severely the image information is damaged.

[0147] A random noise tensor with the same shape as the latent representation is sampled from a standard normal distribution (a Gaussian distribution with a mean of 0 and a standard deviation of 1). The standard normal distribution sampling is performed by generating a uniformly distributed random number between 0 and 1 for each element position of the tensor, and then converting it to a standard normal distribution random number using a Box-Muller transform. The noise tensor serves to simulate random noise during the diffusion process during training, acting as the prediction target for the denoising network.

[0148] The noisy latent representation is calculated based on the noise scheduling parameters. The noise scheduling parameter is the cumulative signal rate, whose value varies with the diffusion time step and is calculated by the noise scheduler of the diffusion model according to a preset noise scheduling strategy. The noisy calculation process is as follows: First, take the square root of the cumulative signal rate and multiply it by the latent representation (element-wise multiplication); second, take 1 and subtract the square root of the cumulative signal rate, and multiply it by random noise; third, add the results of the first and second steps element-wise to obtain the noisy latent representation. The purpose of the noisy operation is to construct training sample pairs: the input is the noisy latent representation and the time step, and the output target is the added random noise.

[0149] The noisy latent representation, the diffusion time step, the text feature vector sequence encoded by the text encoder, and the character semantic feature vector mapped by the projection layer are all input into the denoising network. The denoising network processes the input: the time step is converted into a temporal embedding vector through sinusoidal positional encoding or learnable embeddings and injected into each residual block; the text feature vector sequence and character feature vector interact with image features in the cross-attention layer; the image features undergo layer-by-layer processing through downsampling, intermediate layers, and upsampling. The denoising network finally outputs a predicted noise tensor, whose shape is the same as the random noise tensor sampled in the third step.

[0150] Calculate the mean squared error loss: Subtract the noise tensor predicted by the denoising network from the random noise tensor sampled in the third step to obtain the noise prediction error tensor; square each element in the noise prediction error tensor, and then average all elements (sum divided by the total number of elements) to obtain the loss value for the current iteration. The mean squared error loss measures the difference between the predicted noise and the actual noise; the smaller the loss value, the more accurate the noise prediction of the denoising network.

[0151] Starting with the loss value, the partial derivatives (gradients) of the loss value with respect to each trainable parameter (parameters of all LoRA matrices A and B, as well as parameters of the projection layer) are calculated using an automatic differentiation mechanism. The automatic differentiation mechanism is based on backpropagation of the computation graph: starting with the loss value, the gradients of each parameter are calculated layer by layer using the chain rule, following the reverse order of the forward computation. After calculation, gradient clipping is performed on the gradients (in the manner described in step S5.4). Then, the parameters are updated using an adaptive optimizer. The optimizer updates by maintaining a first-order momentum (exponential moving average of the gradient) and a second-order momentum (exponential moving average of the squared gradient) for each parameter. The first-order momentum is updated by multiplying the previous first-order momentum by the first decay factor, plus the current gradient multiplied by 1 minus the first decay factor. The second-order momentum is updated by multiplying the previous second-order momentum by the second decay factor, plus the current gradient squared by 1 minus the second decay factor. Bias correction is applied to the first and second-order momentum (divided by 1 minus the decay factor raised to the power of the training steps). The parameter update is calculated as: learning rate multiplied by the corrected first-order momentum, divided by the square root of the corrected second-order momentum, plus a small constant (to prevent division by zero). The optimizer's role is to add weight decay regularization to the adaptive learning rate, effectively accelerating convergence and preventing overfitting. All parameters of the original denoising network remain frozen and do not participate in gradient calculation or parameter updates.

[0152] This embodiment employs standard diffusion denoising mean square error loss, fully leveraging the powerful denoising capabilities of the pre-trained diffusion model. LoRA fine-tuning only requires lightweight adaptation on this basis. Training in the VAE latent space significantly reduces computational cost, enabling rapid training on consumer-grade GPUs. The adaptive optimizer, combining first-order and second-order momentum, achieves adaptive learning rate adjustment of parameters, accelerating convergence and improving training stability. Freezing the original model parameters ensures that pre-training capabilities are not compromised, and role-specific adaptation is performed only through LoRA bias.

[0153] Training convergence and output refer to the process of extracting the parameters of the LoRA low-rank decomposition matrix and saving them as a character LoRA weight file after training is completed.

[0154] During training, the loss value is recorded for each training step. Training is considered converged and stopped when the decrease in loss value over a consecutive number of steps is less than a preset convergence threshold, or when the preset total number of training steps is reached. The convergence threshold is preset based on the base model type and the amount of training data.

[0155] Extract all parameters (excluding frozen parameters from the original model) from the trained model's LoRA matrices A and B. Organize the extracted parameters into a dictionary structure by layer name, with the key being the layer name (including the layer's path in the denoising network and the LoRA matrix identifier) ​​and the value being the corresponding parameter tensor. Save the LoRA weight file using a secure tensor serialization format. This serialization format's file structure includes a header (storing metadata such as tensor name, shape, and data type) and a binary data body (storing the raw byte data of each tensor), offering advantages such as fast loading speed, readability without code execution, and no security risks.

[0156] The LoRA weight file and the character feature file together constitute a complete representation of the character. The combined size of the two is much smaller than the complete model of the DreamBooth method, and it has significant advantages in storage and switching efficiency in multi-character scenarios.

[0157] In this embodiment, the convergence determination mechanism ensures that training stops in a timely manner after achieving satisfactory results, avoiding overfitting caused by overtraining; the LoRA weight file contains only low-rank matrix parameters, and its size is much smaller than the complete model (usually several MB to tens of MB), resulting in extremely low storage costs in multi-role scenarios; it is saved using a secure tensor serialization format, which ensures fast loading speed and no risk of code execution; the LoRA weight file is used in conjunction with the role feature file to achieve separation of training and inference, and the same feature file can be used with LoRA weights of different architectures, providing good flexibility and reusability.

[0158] Step S6, inference generation, refers to the operation of injecting the semantic feature vector from the character feature file into the conditional control path of the pre-trained diffusion model based on the pre-trained diffusion model, the character LoRA weight file, the character feature file, and the text prompt containing trigger words. The target character image is generated through the synergistic guidance of the semantic feature vector and the LoRA weight.

[0159] During the inference phase, the system loads the following three resources: first, a pre-trained base diffusion model; second, the LoRA weight file for the character output during the training phase; and third, the character feature file pre-generated by the feature extraction module. The user inputs text prompts containing trigger words, describing the desired character pose, scene, action, and lighting, and the system executes the following inference process: The system loads the pre-trained model's VAE, text encoder, and denoising network. Semantic feature vectors are loaded from the character feature file. Parameters for all LoRA matrices A and B are loaded from the character LoRA weight file. Each LoRA matrix pair is merged with its corresponding original weight matrix using matrix multiplication: the original weight matrix is ​​added to the product of matrix A and matrix B, resulting in a bias-adjusted weight matrix. Matrix multiplication is performed by multiplying the corresponding elements of the i-th row of matrix A and the j-th column of matrix B, then summing the results to obtain the element in the i-th row and j-th column of the product matrix. The merging operation is completed once before inference; no additional computation is required during inference, resulting in inference speed almost identical to the original model.

[0160] The user-input text prompts are encoded into a text feature vector sequence using a text encoder. The text encoding process is similar to the Transformer forward propagation described in step S4.1, and the final output is a text feature vector sequence.

[0161] The semantic feature vectors are mapped to the conditional feature dimension of the cross-attention layer through a projection layer (sharing projection layer parameters with the training phase), and then injected into the cross-attention layer of the denoising network. The injection method is exactly the same as in the training phase: the mapped character feature vectors are projected through key and value respectively to obtain character key vectors and character value vectors, which are then concatenated to the end of the text feature vector sequence to form extended keys and values.

[0162] Starting with pure random noise (sampling a random noise tensor following a standard normal distribution in the latent space as the initial noisy latent representation), the denoising process is progressively reduced from the maximum diffusion time step to the minimum diffusion time step, guided by the noise scheduler of the diffusion model. In each denoising step: the denoising network takes the current noisy latent representation, the current time step, the text feature vector sequence, and the character feature vector as inputs to predict the noise for the current step; then, according to the denoising formula of the noise scheduler, it subtracts the weighted component of the predicted noise from the current noisy latent representation to obtain a clearer latent representation, which serves as the input for the next step. The denoising process can utilize an accelerated sampler to compress the number of sampling steps to be much smaller than the total number of diffusion steps. The principle of the accelerated sampler is: in the Markov chain of the diffusion process, by introducing non-Markov deterministic skip steps, the number of sampling steps is significantly reduced while maintaining the generation quality.

[0163] The final denoised latent representation is input into the VAE decoder. The VAE decoder performs layer-by-layer convolution, upsampling, and normalization operations on the latent representation, restoring the latent space representation to an RGB image in pixel space. The output image size is the preset target size.

[0164] In the generated target character image, the facial features and clothing features of the character are highly consistent with the original input image, while accurately responding to the posture, scene, action and illumination descriptions in the text prompt. The collaborative guidance mechanism of the semantic feature vector and LoRA weights ensures that the character appearance features learned in the training stage are accurately restored in the inference stage, realizing highly consistent character image generation.

[0165] In this embodiment, through the collaborative guidance of the semantic feature vector and LoRA weights, highly consistent restoration of character appearance features and flexible response to text prompts are realized; the semantic feature vector provides stable character semantic constraints, ensuring that core features such as the character's face and clothing in the generated image are consistent with the original input; the LoRA weight bias enables the model to accurately reproduce the detailed texture and style features of the character; the synergistic effect of the two enables the inference stage to not only maintain character consistency, but also flexibly respond to different posture, scene, action and illumination descriptions; the LoRA weights are merged into the original weights in one go before inference, so the inference speed is almost the same as that of the original model, with no additional computational overhead; the use of an accelerated sampler greatly reduces the number of inference steps and time while maintaining generation quality, improving user experience.

[0166] Embodiment 2: Character generation method based on LoRA rapid training (multiple input images, DiT architecture) In this embodiment, multiple original character images provided by the user are used as input, and the target base model is StableDiffusion 3 (DiT architecture, namely diffusion transformer architecture) as an example, to illustrate the implementation process of the method of the present invention under the multi-image scenario and DiT architecture. The character in this embodiment is a fictional game character, and the character trigger word is preset.

[0167] The user inputs multiple original character images, including character photos of different angles and different shot scales. For each image, the same method as in Embodiment 1 is used to calculate the Laplacian gradient variance quality score. The preset quality threshold is the same as that in Embodiment 1. Among the images, those with a quality score greater than or equal to the threshold are judged as qualified and retained; those with a quality score less than the threshold are judged as unqualified, automatically filtered and removed from the input image set.

[0168] In this embodiment, low-quality images are filtered through automatic quality assessment, ensuring that all multiple input images have sufficient clarity, providing a high-quality data basis for subsequent multi-angle feature fusion, and avoiding the impact of individual blurred images on the accuracy of overall character feature extraction.

[0169] Duplicate detection is performed on qualified images. A perceptual hashing algorithm is used, implemented in the same way as the multi-image scenario described in Example 1: each image is scaled to a preset hash size and converted to grayscale. After calculating the discrete cosine transform, the low-frequency components of the transform coefficients are taken. Each coefficient is compared with the median of all coefficients; if greater than the median, it is set to 1; otherwise, it is set to 0, generating a fixed-bit binary hash value. Then, the Hamming distance between the hash values ​​of any two images is calculated. A preset Hamming distance threshold is set; if the Hamming distance is less than the threshold, it is determined to be a duplicate image pair, one of which is retained, and the remaining duplicate images are discarded.

[0170] In this embodiment, in a multi-image scenario, perceptual hashing is used to avoid highly similar redundant images from entering the training set, ensuring that the training data has sufficient perspective and pose diversity, enabling the model to learn character appearance features from multiple angles and improve the generalization ability of character consistency.

[0171] Character main body instance segmentation is performed on the deduplicated images; the same Mask R-CNN model as in Example 1 is used, and the specific implementation process is consistent with Example 1; in this example, the main body of the character in each image is segmented into a single connected region, and there are no cases of multiple disconnected regions.

[0172] For each image, the outline point set of the character instance is extracted, and the Graham scan algorithm is used to generate a convex polygon. The specific implementation is the same as in Example 1. The convex polygon generated for each image contains a different number of vertices depending on the size of the character's main body and the complexity of the outline.

[0173] Since each image contains only one convex polygon (the main character is a single connected region), there is no need to calculate the shortest distance between convex polygons. Instead, the images are directly cropped and normalized (uniformly scaled to the preset target size) based on the bounding rectangle of each convex polygon, resulting in multiple cleaned images.

[0174] In this embodiment, multiple images provide complementary character appearance information from different angles. Instance segmentation and convex polygon construction ensure accurate extraction of the main character in each image. Cropping and normalization unify the format of all images, providing standardized input for subsequent multi-image feature fusion.

[0175] In this embodiment, the main characters in each image are all single connected regions, and the convex polygon merging step is not actually executed; the system detects that each image contains only one convex polygon during the process and automatically skips the merging step. However, the interface and logic of the merging step are retained in the system to ensure the system's general processing capability for cases containing multiple disconnected regions.

[0176] Data augmentation is performed on multiple cleaned images separately. In this embodiment, since there are already several input images containing multi-angle information, the augmentation magnitude of each cleaned image can be appropriately reduced compared to Embodiment 1. Each cleaned image is expanded into several augmented images, and the total number of augmented images is the product of the number of cleaned images and the number of augmented images per image.

[0177] The enhancement operations are the same as in Example 1, and are performed sequentially: geometric transformation (random horizontal flipping plus random rotation within a preset angle range, generating several geometric variants for each image), color enhancement (brightness factor, contrast factor, saturation factor, and hue offset are randomly sampled within their respective preset ranges, generating several color variants for each geometric variant), background replacement (generating several background replacement variants for each color variant, randomly selecting a solid color or stylized background), and multi-resolution sampling (generating multiple versions with different resolutions). The specific implementation process of each enhancement operation is completely consistent with steps S2.1 to S2.4 in Example 1.

[0178] In this embodiment, multiple input images already provide multi-angle information. Appropriately reducing the enhancement level can avoid data redundancy, while still further enhancing the diversity of training data through geometric, color, and background enhancements. The enhanced dataset has richer variations on the basis of covering multiple angles, enabling LoRA fine-tuning to learn more robust character appearance feature representations.

[0179] For each enhanced image in the enhanced image set, a multimodal visual language model is invoked to generate a natural language description, with the specific implementation process being consistent with Example 1. The generated description covers information such as the character's appearance attributes, clothing attributes, and posture.

[0180] Trigger words are inserted at specified positions in each descriptive text, using the same insertion method as in Example 1. Text length and content matching are validated using the same validation method as in Example 1. The validated text-image pairs form the text-image pair training dataset.

[0181] In this embodiment, automatic annotation in a multi-image scene can generate differentiated descriptive text for each enhanced image, covering character appearance information from different angles and poses, enriching the semantic diversity of the image and text in the training data, and helping the model learn a more comprehensive correspondence between character appearance and text description; For multiple cleaned images, semantic feature vectors are extracted using a pre-trained visual encoder, in the same way as in Example 1, resulting in multiple feature vectors.

[0182] Since multiple images exist, attention-weighted fusion is used for aggregation. The specific implementation is consistent with the attention-weighted fusion strategy described in Example 1, introducing a learnable query vector with the same dimension as the feature vector. For each feature vector, the dot product of the query vector and the feature vector is calculated to obtain an attention score. All attention scores are normalized using a softmax function to obtain attention weights. Each feature vector is multiplied by its corresponding attention weight and then summed element-wise to obtain the aggregated character semantic feature vector. Attention-weighted fusion automatically assigns higher weights to images with more information (such as frontal photos containing a complete full-body appearance) and lower weights to images with less information (such as close-ups containing only the face), generating more representative character feature representations. The aggregated semantic feature vector and metadata (character ID, trigger words, dimension, encoder type, extraction timestamp, etc.) are serialized into a character feature file, with the serialization method consistent with step S4.3 in Example 1.

[0183] In this embodiment, the attention-weighted fusion strategy can automatically evaluate the information content of each image and assign corresponding weights. Compared with simple mean pooling, it can generate more representative and stable character semantic features. The feature fusion of multi-angle images enables the character feature vector to contain more comprehensive appearance information (such as front, side, and back views), and can more accurately restore the appearance consistency of the character from various angles during the inference stage.

[0184] This embodiment uses Stable Diffusion 3 (SD3) as the basic model, and its denoising network adopts the DiT (Diffusion Transformer) architecture, which is fundamentally different from the UNet architecture in Embodiment 1.

[0185] The architecture identification method is the same as in Example 1. The system reads the architecture type field in the model configuration and determines it to be a DiT architecture.

[0186] The core structure of the DiT architecture is as follows: The input image is first segmented into fixed-size image patches. Each image patch is transformed into an embedding vector through linear projection and then positionally encoded. Subsequently, the embedding vector sequence is input into a multi-layer Transformer block. Each Transformer block contains: a multi-head self-attention layer (which maps the input sequence through query, key, and value projection matrices and performs self-attention calculation, consistent with the Transformer self-attention layer described in step S4.1), and a feedforward network layer (containing two linear projection matrices and a non-linear activation function, consistent with the feedforward network layer described in step S4.1). Each sub-layer is followed by residual connections and layer normalization. The output of the Transformer block is mapped back to the image patch space through the final linear projection layer and reconstructed into the latent image representation. Furthermore, the diffusion time step and textual conditions in the DiT architecture are injected into each Transformer block through adaptive layer normalization (AdaLN): after the time step is encoded by a multi-layer perceptron, the scaling and offset parameters of the output layer normalization, as well as the scaling parameters before and after the self-attention and feedforward networks, are used to conditionally modulate the features.

[0187] Based on the DiT architecture type, the LoRA low-rank decomposition matrix is ​​inserted into the attention projection matrices of each Transformer layer: query projection matrix, key projection matrix, value projection matrix, and output projection matrix. The insertion method is consistent with the description in Example 1: the original weight matrix remains frozen, and new matrices A (dimension is the input feature dimension multiplied by the rank r columns) and B (dimension is the rank r rows multiplied by the output feature dimension columns) are added. Matrix A is randomly initialized with a Gaussian distribution, and matrix B is initialized with all zeros. During forward computation, the original weight output plus the output of matrix A multiplied by matrix B is used as the bias term.

[0188] In this embodiment, the value of rank r can be appropriately reduced compared to the UNet architecture. Under the DiT architecture, since the Transformer has strong feature representation capabilities and rich conditional modulation mechanisms through adaptive layer normalization, a lower rank is usually sufficient to capture the character-specific appearance features, while further reducing the number of trainable parameters.

[0189] In this embodiment, the system automatically identifies the DiT architecture and adaptively adjusts the LoRA insertion strategy, demonstrating the universal compatibility of the present invention with different diffusion model architectures. The strong feature representation capability of the Transformer under the DiT architecture enables effective role adaptation with a lower rank, further reducing the number of trainable parameters and training time. The synergistic effect of adaptive layer normalization and LoRA bias provides an efficient conditional modulation mechanism for role customization under the DiT architecture.

[0190] Training hyperparameter configuration: The total number of training steps is adaptively calculated based on the effective sample size, batch size, and preset repetition rounds of the augmented image set; the learning rate adopts a cosine annealing strategy, and the initial and minimum learning rates are preset based on the DiT architecture and the amount of training data; the number of warm-up steps is preset; and the gradient clipping threshold is preset. The calculation and configuration methods of each hyperparameter are consistent with those in Example 1.

[0191] Role feature vector injection method: Similar to Example 1, the semantic feature vector is mapped to the conditional feature dimension of the DiT cross-attention layer through a learnable projection layer, and then concatenated with the text feature vector sequence to serve as the extended cross-attention key and value. The injection method is consistent with the description in Example 1, the difference being that the position and dimension of the cross-attention layer in the DiT architecture are different from those in UNet.

[0192] Loss function and optimization: The standard diffusion denoising mean squared error loss is adopted, and the calculation and optimization methods are consistent with those in Example 1. After training, the LoRA weight file for the roles is exported and saved using the secure tensor serialization format.

[0193] In this embodiment, the LoRA fine-tuning under the DiT architecture maintains the same training process and optimization strategy as the UNet architecture, reducing the learning cost for users; in scenarios with multiple input images, the amount of training data is more sufficient, the model can learn more comprehensive character appearance features, training converges faster, and the generation quality is higher.

[0194] The reasoning process is similar to that in Example 1, loading the DiT architecture basic model, the role LoRA weight file, and the role feature file, and the user inputs a text prompt containing trigger words.

[0195] The system injects character feature vectors into the cross-attention layer of the DiT architecture, and adjusts the Transformer attention projection matrix by adjusting the LoRA weight bias. The two work together to guide the denoising process and generate highly consistent target character images. The appearance features of the characters in the generated images are highly consistent with the original input images, and at the same time, they accurately respond to the pose, scene and lighting descriptions in the prompts.

[0196] In this embodiment, the inference generation under the DiT architecture fully utilizes the global attention mechanism of Transformer, which can generate richer scene details and more natural pose changes while maintaining character consistency. The feature vectors trained from multiple images with LoRA weights and attention weighted fusion make the appearance consistency of the character better in the inference stage under multiple angles.

[0197] Example 3: Cross-Architecture Inference Compatibility Verification This embodiment verifies the universal compatibility of the system of the present invention with different diffusion model architectures.

[0198] Scenario 1: Inference under the UNet architecture Load the LoRA weight file and character feature file trained on the UNet architecture, and perform inference on the UNet architecture model. The system automatically recognizes the current model as a UNet architecture, inserts the LoRA matrix into the attention layers (query, key, value, output projection matrix) and feedforward network layers of each residual block of UNet, and injects the character feature vector into the UNet cross-attention layer. Inference executes normally, and the appearance features of the character in the generated image are highly consistent with the original input image.

[0199] The beneficial effect of this scenario is that it verifies the consistency between training and inference of the system of the present invention on the UNet architecture, and the character appearance features learned in the training phase are accurately restored in the inference phase.

[0200] Scenario 2: Inference under the DiT Architecture Load the LoRA weight file and character feature file trained on the DiT architecture, and perform inference on the DiT architecture model. The system automatically recognizes the current model as a DiT architecture, inserts the LoRA matrix into the attention projection matrices of each Transformer layer, and injects the character feature vectors into the DiT cross-attention layer. Inference executes normally, and the appearance features of the character in the generated image are highly consistent with the original input image.

[0201] The beneficial effects of this scenario are: it verifies the consistency of training and inference of the system of the present invention on the DiT architecture, and the system can adaptively handle LoRA insertion and feature injection of the DiT architecture.

[0202] Example 4: Multi-role feature library management and quick switching This embodiment illustrates the advantages of the present invention in storage and switching in multi-role scenarios.

[0203] Suppose a user needs to manage multiple different roles. Each role corresponds to storing the following two files: a role LoRA weight file and a role feature file. The size of the role LoRA weight file depends on the underlying model architecture, LoRA rank, and number of insertion layers, typically ranging from several megabytes to tens of megabytes. The size of the role feature file depends on the feature vector dimension and metadata size, typically ranging from several kilobytes to tens of kilobytes.

[0204] The total storage footprint for multiple roles is the sum of the LoRA weight file sizes for each role plus the sum of the feature file sizes for each role. Since the LoRA weight file is much smaller than the full fine-tuning model (typically in the GB range), and the feature file is extremely lightweight, the total storage footprint for multiple roles is still much smaller than the storage footprint of a single DreamBooth full fine-tuning model. The storage efficiency improvement is approximately the size of the full model divided by the sum of the LoRA weight file and feature file sizes for a single role.

[0205] During inference, the system automatically matches the corresponding role files based on the trigger words in the user's input prompts. The matching method is as follows: During the initialization phase, the system scans all role feature files in the specified directory, reads the trigger word string field from the metadata of each feature file, and constructs a mapping table (a hash table data structure, with the trigger word as the key and the role file path and LoRA weight file path as the value) from trigger words to role file paths. The purpose of this mapping table is to achieve efficient lookup of trigger words to role files, with a constant time complexity, unaffected by the number of roles.

[0206] When a user inputs a suggestion word, the system segments the suggestion word and checks each word against a trigger word in the mapping table. If a match is found, the system retrieves the corresponding LoRA weight file path and feature file path for the role based on the mapping table and loads both files. The loading process only requires reading two small files and takes very little time. When switching roles, the system unloads the LoRA weights of the current role (restoring the bias-adjusted weight matrix to its original state) and loads the LoRA weights and feature file for the new role. This process also takes very little time, and the user experiences no noticeable delay.

[0207] The full-scale fine-tuning method requires unloading the current complete model and loading a new model when switching roles, involving the transfer of a large amount of data between GPU memory and system memory, which is time-consuming. The solution of this invention achieves extremely low storage costs and extremely fast role switching speeds in multi-role scenarios, significantly improving the practicality of multi-role management and user experience.

[0208] This embodiment achieves constant-time-complexity character lookup through a mapping table from trigger words to character files, and the switching speed is not affected by the number of characters. The lightweight size of the LoRA weight file and feature file makes the storage cost of multiple characters extremely low, improving storage efficiency by hundreds of times compared to the full fine-tuning method. The millisecond-level character switching speed allows users to switch seamlessly in multi-character creation scenarios, greatly improving workflow efficiency. The separate design of feature files and LoRA weight files allows the same character to share the same feature file while training LoRA weights for different architecture models, further reducing storage and management costs in multi-model scenarios.

[0209] A character generation system based on LoRa for fast training includes: The data cleaning module is used to perform image quality assessment, duplicate detection and deduplication, instance segmentation, convex polygon construction, shortest distance calculation and merging of convex polygons, cropping and size normalization processing based on the original character images, and output the cleaned images. The data augmentation module is used to perform geometric transformation, color perturbation, background replacement and multi-resolution sampling processing based on the cleaned image, and output an enhanced image set; The data labeling module is used to automatically describe the enhanced images and insert role trigger words based on the enhanced image set, and output the image-text pair training dataset; The feature extraction module is used to extract semantic feature vectors from the cleaned image using a pre-trained image encoder and serialize them into a character feature file. The training module is used to load the pre-trained diffusion model and perform LoRA fine-tuning training based on the image and text training dataset and the character feature file. During training, the semantic feature vector is injected into the model conditional control path, and the LoRA low-rank decomposition matrix is ​​automatically inserted into the corresponding attention layer according to the model architecture, and the character LoRA weight file is output. The inference module is used to receive text prompts containing trigger words based on the pre-trained diffusion model, the character LoRA weight file, and the character feature file, inject semantic feature vectors into the model's conditional control path, and generate a target character image.

[0210] The above embodiments detail the specific implementation methods and systems of the LoRA-based fast training method and system of the present invention in different scenarios such as single image, multiple images, UNet architecture, DiT architecture, cross-architecture use, and multi-role management. The parameter values ​​involved in each embodiment (such as threshold, factor range, number of steps, learning rate, etc.) are preset values ​​that can be flexibly adjusted according to factors such as the characteristics of the base model, the amount of training data, and the complexity of the role, and are not fixed limitations. It should be noted that the above embodiments are only preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art can make several improvements and modifications without departing from the principles and scope of protection of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A character generation method based on LoRa fast training, characterized in that, The method includes: The original character image is segmented into main character instances. Convex polygons are generated based on the contour point set of the segmented character instances. The shortest distance between any two convex polygons is calculated based on the separating axis theorem. When the shortest distance is less than the preset merging distance threshold, the corresponding convex polygons are merged. The merged character instance region is cropped and normalized to obtain the cleaned image. Based on the cleaned images, an enhanced image set is obtained through data augmentation processing; Based on the enhanced image set, automatic image descriptions are generated for each enhanced image, and trigger words corresponding to the characters are inserted into the description text to form an image-text pair training dataset; Based on the cleaned images, semantic feature vectors are extracted using a pre-trained image encoder, and the semantic feature vectors are serialized into character feature files. Based on the training dataset and character feature files, the pre-trained diffusion model is fine-tuned using low-rank adaptation LoRA. During training, the semantic feature vectors from the character feature files are injected into the conditional control pathway of the pre-trained diffusion model, and the low-rank LoRA decomposition matrix is ​​inserted into the attention layer at the corresponding network position according to the architecture type of the pre-trained diffusion model. The character LoRA weight file is obtained through iterative updates. Based on the pre-trained diffusion model, the character LoRA weight file, the character feature file, and text prompts containing trigger words, the semantic feature vectors from the character feature file are injected into the conditional control path of the pre-trained diffusion model. Through the synergistic guidance of the semantic feature vectors and LoRA weights, the target character image is generated.

2. The method according to claim 1, characterized in that, Before performing character subject instance segmentation on the original character image, the following steps are also included: The original character images are evaluated for image quality, and blurry, overexposed or underexposed images are filtered out to obtain images of acceptable quality. Perform duplicate image detection and deduplication on qualified images to obtain deduplicated images, and then perform the character subject instance segmentation on the deduplicated images; When there is only one original character image, skip duplicate image detection and deduplication, and perform character main instance segmentation on images with acceptable quality.

3. The method according to claim 2, characterized in that, The calculation of the shortest distance between any two convex polygons based on the separating axis theorem includes: Obtain the first convex polygon and the second convex polygon; calculate the unit outward normal vector of each edge of the first convex polygon to obtain the first set of normal vectors; calculate the unit outward normal vector of each edge of the second convex polygon to obtain the second set of normal vectors; merge the first set of normal vectors and the second set of normal vectors to obtain the candidate set of normal vectors. Based on the candidate normal vectors, all vertices of the first convex polygon are projected to obtain the first projection interval, which is defined by the minimum and maximum values ​​of the projection values. Based on the candidate normal vectors, all vertices of the second convex polygon are projected to obtain the second projection interval, which is defined by the minimum and maximum values ​​of the projection values. Calculate the first difference based on the minimum value of the first projection interval and the maximum value of the second projection interval; Calculate the second difference based on the minimum value of the second projection interval and the maximum value of the first projection interval; Based on the first difference and the second difference, the overlap length under the candidate normal vector is obtained; When the overlap length is less than zero, calculate the separation distance under the candidate normal vector based on the endpoint values ​​of the first projection interval and the second projection interval; when the overlap length is not less than zero, record the separation distance as zero; traverse all normal vectors in the candidate normal vector set and collect the separation distance under each candidate normal vector. The minimum value among all collected separation distances is used to obtain the shortest distance between the first convex polygon and the second convex polygon.

4. The method according to claim 3, characterized in that, The data augmentation process includes: The cleaned image is randomly horizontally flipped and randomly rotated within a set angle range to obtain a geometrically enhanced image; The brightness, contrast, and saturation of the geometrically enhanced image are randomly adjusted, and the hue is shifted to obtain a color-enhanced image. The main character region is segmented from the color-enhanced image using an image segmentation model. The main character region is then overlaid with a solid color background or a stylized background to obtain a background replacement and enhancement image. Multiple versions of the background replacement enhancement image are generated at different resolutions to obtain an enhanced image set.

5. The method according to claim 4, characterized in that, The training dataset comprising the image-text pairs includes: The multimodal visual language model is invoked to perform forward inference on the enhanced image, generating natural language description text that describes the character's appearance attributes, clothing attributes, expressions, and postures; Insert a trigger word that uniquely corresponds to the role at a specified position in the natural language description text to obtain description text with the trigger word; The text descriptions with trigger words are validated for text length and content matching. Texts that fail the validation are filtered out, and the text descriptions with trigger words that pass the validation and their corresponding enhanced images are retained to form the image-text pair training dataset.

6. The method according to claim 5, characterized in that, The process of extracting semantic feature vectors and serializing them into a role feature file includes: High-dimensional semantic feature vectors are extracted from the cleaned images using CLIP or DINOv2 visual encoders. When there are multiple cleaned images, the high-dimensional semantic feature vectors of each image are aggregated into a semantic feature vector by mean pooling or attention weighted fusion. The semantic feature vector, along with the role ID and trigger word string, is used as metadata, serialized into a role feature file, and persistently stored.

7. The method according to claim 6, characterized in that, The LoRA fine-tuning training includes: Set the rank of the LoRA low-rank decomposition matrix to be in the range of 4 to 32; The number of training steps is adaptively set within the range of 100 to 500 steps, depending on the effective sample size of the augmented image set. A cosine annealing strategy is used to schedule the learning rate, and the number of warm-up steps and gradient clipping are set to enable training to converge quickly.

8. The method according to claim 7, characterized in that, The step of inserting the LoRA low-rank decomposition matrix into the attention layer at the corresponding network location according to the architecture type of the pre-trained diffusion model includes: Identify the architecture of the pre-trained diffusion model; When the architecture is identified as UNet, the LoRA low-rank decomposition matrix is ​​inserted into the query projection matrix, key projection matrix, value projection matrix and feedforward network layer in the attention layer of each residual block of UNet. When the DiT architecture is identified as a diffusion transformer, the low-rank LoRA decomposition matrix is ​​inserted into the attention projection matrix of each layer of the Transformer. When the number of original character images is 1, the data augmentation process expands the single cleaned image into 10 to 50 images containing diverse appearance changes to form an augmented image set, and combines it with the character feature file to complete LoRA fine-tuning training.

9. A character generation system based on LoRA fast training, characterized in that, The system comprising the method of any one of claims 1 to 8, wherein the method is: The data cleaning module is used to perform image quality assessment, duplicate detection and deduplication, instance segmentation, convex polygon construction, shortest distance calculation and merging of convex polygons, cropping and size normalization processing based on the original character images, and output the cleaned images. The data augmentation module is used to perform geometric transformation, color perturbation, background replacement and multi-resolution sampling processing based on the cleaned image, and output an enhanced image set; The data labeling module is used to automatically describe the enhanced images and insert role trigger words based on the enhanced image set, and output the image-text pair training dataset; The feature extraction module is used to extract semantic feature vectors from the cleaned image using a pre-trained image encoder and serialize them into a character feature file. The training module is used to load the pre-trained diffusion model and perform LoRA fine-tuning training based on the image and text training dataset and the character feature file. During training, the semantic feature vector is injected into the model conditional control path, and the LoRA low-rank decomposition matrix is ​​automatically inserted into the corresponding attention layer according to the model architecture, and the character LoRA weight file is output. The inference module is used to receive text prompts containing trigger words based on the pre-trained diffusion model, the character LoRA weight file, and the character feature file, inject semantic feature vectors into the model's conditional control path, and generate a target character image.

Images