Cultural feature digital human generation method fusing LoRA fine tuning and cross-modal interaction

Abstract

The invention discloses a cultural feature digital human generation method fusing LoRA fine tuning and cross-modal interaction, and belongs to the technical field, and the method comprises the following steps: S1, data collection and quantization parameter extraction, S2, high-precision 3D digital human model training, S3, cultural feature adaptation model fine tuning, S4, synchronous output of actions and voices, and S5, emotional interaction response driving. Through the combination of a score distillation optimization method of a skeleton condition and a high-precision 3D scanning technology, high-precision modeling of a historical figure digital human is realized, the face scanning precision reaches 0.1 mm, the body modeling error is controlled within 0.5 cm, the posture characteristics and the elegant temperament of the historical figure are precisely restored, and the method has the advantages of high modeling precision and high modeling precision. The problems of image distortion and posture deformity in traditional modeling are effectively solved, and a high-fidelity image basis is provided for cultural celebrity digital people.

Description

Technical Field

[0001] This invention relates to the field of digital human generation technology, and more specifically, to a culturally distinctive digital human generation method that integrates LoRA fine-tuning and cross-modal interaction. Background Technology

[0002] Culturally distinctive digital humans are virtual images created using digital technology with traditional cultural elements at their core, and are a new type of digital technology product that combines cultural dissemination and interactive functions.

[0003] The prior art patent document with authorization announcement number CN117392292A discloses "a 3D digital human generation method and system", which includes: acquiring an image of a target object, obtaining a two-dimensional face image based on the image of the target object; obtaining a complete 3D digital human model and exporting the edited bone weights; acquiring a set of facial expression coefficients to be used; acquiring the three-dimensional coordinates of each key point in the natural state corresponding to each face image frame and the face rotation parameters; acquiring the position of each joint point in the 3D digital human model, and performing fusion driving on the 3D digital human model.

[0004] The patent document with authorization announcement number CN118521685B discloses "A method for generating 3D digital humans based on AIGC", which includes data collection, data preprocessing, feature extraction, model generation, texture mapping, animation generation, model rendering, data transmission, data adjustment, and data generation. It establishes coordinates for the generated 3D digital human, compares the trajectory curves of individual joints in the coordinates with corresponding preset trajectory curves, determines whether the operation of a single joint meets preset standards based on the obtained overlap, and, if it is initially determined that the operation of a single joint does not meet preset standards, determines whether to adjust the preset rendering ratio based on the distance between the endpoint of the obtained trajectory curve of the single joint and the endpoint of the preset trajectory curve.

[0005] While existing technologies can fit 3D digital human models that are closer to real human faces, preventing the low accuracy problem caused by relying solely on image or voice expression coefficient sets, and making digital human emotional expression more vivid, while effectively improving the accuracy of 3D digital human operation and improving generation efficiency and running smoothness by adjusting parameters such as rendering ratio, existing technologies still have problems such as low cultural feature restoration, insufficient modeling accuracy, poor dynamic adaptation of movements and clothing, and lack of emotional response in interaction. Summary of the Invention

[0006] This invention mainly provides a method for generating culturally distinctive digital humans that integrates LoRA fine-tuning and cross-modal interaction, which can solve the problems mentioned in the background technology.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a culturally distinctive digital human generation method integrating LoRA fine-tuning and cross-modal interaction, comprising:

[0008] S1. High-precision 3D scanners are used to collect physical data of historical figures' portraits and sculptures to reconstruct basic geometric models. At the same time, descriptions of human figures' physical characteristics are extracted from historical documents and converted into quantitative constraint parameters.

[0009] S2. The score distillation optimization algorithm that embeds the physiological structure constraints of the human skeleton and the historical posture quantitative constraint parameters obtained in S1 into the skeleton conditions is used to train and generate a high-precision 3D digital human model that meets the constraints.

[0010] S3. Construct a dataset containing static samples of culturally distinctive clothing and makeup, as well as samples of dynamic clothing deformation. Based on the ControlNet framework, perform LoRA lightweight fine-tuning on the model to enable digital human generation to achieve clothing and dynamic adaptation effects that conform to cultural characteristics.

[0011] S4. Collect real-person motion and facial expression data through motion capture equipment to train motion transfer models, and integrate speech recognition and text-to-speech technology to achieve synchronous output of digital human motion and speech;

[0012] S5. Perform emotion recognition on the user's voice and text input, and convert the recognition results into the corresponding facial expressions, actions and voice parameters of the digital human to drive the digital human to complete emotional interactive responses.

[0013] Furthermore, in S1, the high-precision 3D scanner has a scanning accuracy of 0.05 mm, the historical documents include historical books and local chronicles related to the target person, and the quantitative constraint parameters include body feature data such as the ratio of shoulder width to height and the threshold of the standing posture tilt angle.

[0014] Furthermore, in S2, the physiological structural constraints of the human skeleton include the range of motion of joints and the distribution area of ​​muscles. The score distillation optimization algorithm of the skeleton conditions realizes the gradient backpropagation adjustment of model parameters by transforming the dual constraints into a penalty term of the loss function.

[0015] Furthermore, in S3, the static samples contain cultural elements such as the wide robes and sleeves of Song Dynasty scholar-officials, the headdresses and hats, and the phoenix costumes and silver ornaments of She ethnic women. The dynamic deformation samples are labeled with the mapping relationship between motion parameters and clothing folds and draping shapes. The LoRA lightweight fine-tuning only adjusts 5% of the parameters of the ControlNet model.

[0016] Furthermore, in S4, the motion capture device collects real-person motion and facial expression data, including body language and facial micro-expressions. The motion transfer model achieves accurate mapping of real-person motion to digital humans through training. The linkage between speech recognition and text-to-speech technology is synchronized through a timestamp alignment mechanism.

[0017] Furthermore, in S5, emotion recognition includes voice emotion analysis and text semantic analysis. Voice emotion analysis extracts features such as tone, speech rate, and volume, while text semantic analysis parses the emotional tendency of the content. The parameters converted include facial expression amplitude, body movement intensity, and voice tone change values.

[0018] Furthermore, the high-precision 3D digital human model has a facial scanning accuracy of 0.1 mm, a body modeling error of <0.5 cm, and facial contours of the brow bone and cheekbone that highly match historical portraits.

[0019] Furthermore, the synchronization delay between the digital human's movements and speech is <300 milliseconds, and the synchronization accuracy is >95%.

[0020] The beneficial effects of this invention's culturally distinctive digital human generation method, which integrates LoRA fine-tuning and cross-modal interaction, are as follows:

[0021] By combining the score distillation optimization method based on skeletal conditions with high-precision 3D scanning technology, high-precision modeling of digital human figures of historical figures has been achieved. The facial scanning accuracy reaches 0.1 mm, and the body modeling error is controlled within 0.5 cm. The model accurately restores the physical characteristics and refined temperament of historical figures, effectively solving the problems of image distortion and posture deformity in traditional modeling, and providing a high-fidelity image foundation for digital human figures of cultural celebrities.

[0022] By integrating ControlNet technology with LoRA fine-tuning and cross-modal interaction technology, the cultural recognizability of digital human clothing and makeup has been improved, accurately presenting the characteristics of Luling culture. Furthermore, through efficient synchronization of actions and voice and emotional perception response, a natural and smooth immersive cultural dialogue has been achieved, allowing users to deeply experience the scene of communicating with historical figures and promoting innovation in cultural dissemination methods. Attached Figure Description

[0023] The present invention will now be described in further detail with reference to the accompanying drawings and specific implementation methods.

[0024] Figure 1 This is a schematic diagram of the process for generating a culturally distinctive digital human that integrates LoRA fine-tuning and cross-modal interaction, as described in this invention. Detailed Implementation

[0025] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Example 1

[0027] like Figure 1 As shown, a technical solution is provided: a culturally distinctive digital human generation method that integrates LoRA fine-tuning and cross-modal interaction, including:

[0028] Step 1: Data Acquisition and Quantitative Parameter Extraction

[0029] High-precision 3D scanners are used to collect physical data of historical figures' portraits and sculptures to reconstruct basic geometric models. At the same time, descriptions of human body shapes are extracted from historical documents and converted into quantitative constraint parameters.

[0030] Specifically, the high-precision 3D scanner has a scanning accuracy of 0.05 mm, historical documents include historical books and local chronicles related to the target person, and quantitative constraint parameters include body feature data such as the ratio of shoulder width to height and the threshold of the standing posture tilt angle.

[0031] Among them, the 3D scanner needs to scan the portraits of historical figures (such as Wang Yangming, Wen Tianxiang, etc.), sculptures and related cultural relics in all directions, generate point cloud data, and then reconstruct the basic geometric model of the target figure based on the point cloud data.

[0032] In addition, for historical figures from Luling such as Wang Yangming and Wen Tianxiang, physical descriptions such as "imposing and robust" and "dignified and imposing" were extracted from historical books and local chronicles such as the History of Song and the Biographies of Luling Figures. These descriptions were then transformed into quantifiable constraint parameters using semantic parsing tools.

[0033] The quantitative constraint parameters also include detailed features such as the angular relationship between the head and the body, ensuring that the extracted parameters can fully reflect the physical characteristics of historical figures.

[0034] Step 2: High-precision 3D digital human model training

[0035] The score distillation optimization algorithm (SDS) is used to train and generate a high-precision 3D digital human model that conforms to the constraints by embedding the physiological structure constraints of the human skeleton and the quantitative constraint parameters of historical postures (such as "imposing appearance" and "groomy demeanor") obtained in S1 into the skeleton conditions.

[0036] Among them, the physiological structure constraints of the human skeleton include the range of motion of joints (e.g., the maximum bending angle of the elbow joint is 145°) and the distribution area of ​​muscles (e.g., the degree of prominence of the pectoral and back muscles). The score distillation optimization algorithm of the skeleton conditions transforms the physiological structure constraints of the human skeleton and the historical posture quantification constraint parameters into the penalty term of the loss function. During model training, if the generated posture or contour deviates from the constraint range, the loss value will increase significantly, thereby forcing the model to adjust the parameters through gradient backpropagation, and finally outputting a high-precision 3D digital human model that meets the dual constraints.

[0037] In addition, during the model training process, the generated 3D digital human model needs to be continuously compared with the detailed features of historical portraits and sculptures, such as the contours of the brow bone and cheekbone, and the facial scanning accuracy needs to be ensured to reach 0.1 mm through iterative optimization.

[0038] For body modeling, it is necessary to strictly control the error according to the quantitative constraint parameters so that the modeling error of each part of the body is less than 0.5 cm. In particular, it is necessary to ensure that key postures such as standing posture are consistent with historical descriptions to avoid deformed postures.

[0039] Step 3: Fine-tuning the cultural feature adaptation model

[0040] We constructed a dataset containing static samples of culturally distinctive clothing and makeup, as well as samples of dynamic clothing deformation. Based on the ControlNet framework, we performed LoRA lightweight fine-tuning on the model to enable digital human generation to have clothing and dynamic adaptation effects that conform to cultural characteristics.

[0041] Among them, the static samples contain cultural elements such as the wide robes and sleeves of Song Dynasty scholar-officials, the headdress and hat, and the phoenix dress and silver ornaments of She ethnic women. The dynamic deformation samples are labeled with the mapping relationship between motion parameters and clothing folds and draping shapes. The LoRA lightweight fine-tuning only adjusts 5% of the parameters of the ControlNet model.

[0042] In addition, static samples must cover 2,000+ Luling cultural features (such as the width of the sleeves of Song Dynasty clothing, the ratio of clothing length to height, and the complexity of the patterns of She ethnic silver ornaments). Each sample is labeled with corresponding cultural attribute tags such as era and identity.

[0043] Dynamic deformation samples require the use of motion capture equipment to collect motion data of real people wearing cultural costumes, such as bowing and walking, to establish a mapping relationship between "motion angle and degree of clothing deformation". This ensures that the model can automatically generate a suitable clothing shape based on the digital human's movements after fine-tuning, avoiding clipping or distortion.

[0044] Step 4: Simultaneous output of actions and voice.

[0045] By collecting real-person movement and facial expression data using motion capture equipment to train a motion transfer model, and integrating speech recognition and text-to-speech technologies, the digital human's movements and speech can be output synchronously.

[0046] Specifically, motion capture devices collect real-person motion and facial expression data, including body language and facial micro-expressions. Motion transfer models achieve accurate mapping of real-person motion to digital humans through training. The linkage between speech recognition and text-to-speech technology achieves synchronous control through a timestamp alignment mechanism.

[0047] The synchronization control achieved through the timestamp alignment mechanism specifically involves assigning the same timestamp to each segment of speech signal and the corresponding action data. When the digital human outputs, the system accurately calls the corresponding action frame and speech segment based on the timestamp, ensuring that the two are perfectly matched on the timeline.

[0048] In addition, the motion transfer model needs to undergo multiple rounds of training and optimization to improve the smoothness of digital human movements by 40% compared to traditional models and reduce joint movement stuttering.

[0049] The overall latency of speech recognition and text-to-speech technology needs to be controlled within 300 milliseconds, and the synchronization accuracy needs to be maintained above 95% to meet the requirements of natural interaction.

[0050] Step 5: Driven by Emotional Interaction Response

[0051] Emotion recognition is performed on the user's voice and text input, and the recognition results are converted into the corresponding facial expressions, actions and voice parameters of the digital human, driving the digital human to complete emotional interactive responses.

[0052] Specifically, emotion recognition includes speech emotion analysis and text semantic analysis. Speech emotion analysis extracts features such as pitch, speech rate, and volume, while text semantic analysis analyzes the emotional tendencies of the content. The parameters involved in the conversion include facial expression amplitude, body movement intensity, and speech tone variation.

[0053] Among them, voice emotion analysis requires the use of algorithmic models to quantify the emotional features in the user's voice, such as "tragic" emotion corresponding to a lower pitch and slower speech rate, and "excited" emotion corresponding to an increased volume and faster speech rate;

[0054] In addition, text semantic analysis needs to combine cultural context to analyze the emotional tendency of the dialogue content. For example, when a user mentions "Since ancient times, who has ever escaped death?", it is determined to be a tragic emotion and corresponding digital human response parameters are generated.

[0055] The converted parameters need to match the facial expression points and skeletal movement range of the digital human model (for example, increasing the furrowing of the brows by 30% and reducing the gesture range by 50% when expressing a tragic emotion, to ensure that the emotional expression is natural and in line with the temperament of the historical figure).

[0056] In summary, the above five steps achieve the generation of culturally distinctive digital humans that combine high-precision historical accuracy, strong cultural characteristics, and natural interaction capabilities: a foundation is built from dual-source data collection of "physical objects + documents," a high-precision 3D model is generated using the SDS algorithm with skeleton constraints, and the LoRA-tuned ControlNet is used to give cultural features dynamic adaptation capabilities. Combined with motion and voice synchronization technology, smooth interaction is ensured, and finally, immersive cultural dialogue is achieved through emotional responses. This comprehensively solves the problems of insufficient accuracy, feature distortion, and stiff interaction in traditional technologies applied to cultural scenarios.

[0057] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A culturally distinctive digital human generation method that integrates LoRA fine-tuning and cross-modal interaction, characterized by: S1. High-precision 3D scanners are used to collect physical data of historical figures' portraits and sculptures to reconstruct basic geometric models. At the same time, descriptions of human figures' physical characteristics are extracted from historical documents and converted into quantitative constraint parameters. S2. The score distillation optimization algorithm that embeds the physiological structure constraints of the human skeleton and the historical posture quantitative constraint parameters obtained in S1 into the skeleton conditions is used to train and generate a high-precision 3D digital human model that meets the constraints. S3. Construct a dataset containing static samples of culturally distinctive clothing and makeup, as well as samples of dynamic clothing deformation. Based on the ControlNet framework, perform LoRA lightweight fine-tuning on the model to enable digital human generation to achieve clothing and dynamic adaptation effects that conform to cultural characteristics. S4. Collect real-person motion and facial expression data through motion capture equipment to train motion transfer models, and integrate speech recognition and text-to-speech technology to achieve synchronous output of digital human motion and speech; S5. Perform emotion recognition on the user's voice and text input, and convert the recognition results into the corresponding facial expressions, actions and voice parameters of the digital human to drive the digital human to complete emotional interactive responses.

2. The method for generating culturally distinctive digital humans by integrating LoRA fine-tuning and cross-modal interaction as described in claim 1, characterized in that: In S1, the high-precision 3D scanner has a scanning accuracy of 0.05 mm, the historical documents include historical books and local chronicles related to the target person, and the quantitative constraint parameters include body feature data such as the ratio of shoulder width to height and the threshold of the standing posture tilt angle.

3. The method for generating culturally distinctive digital humans by integrating LoRA fine-tuning and cross-modal interaction as described in claim 1, characterized in that: In S2, the physiological structural constraints of the human skeleton include the range of motion of joints and the distribution area of ​​muscles. The score distillation optimization algorithm of the skeleton conditions realizes the gradient backpropagation adjustment of model parameters by transforming the dual constraints into a penalty term of the loss function.

4. The method for generating culturally distinctive digital humans by integrating LoRA fine-tuning and cross-modal interaction as described in claim 1, characterized in that: In S3, the static samples contain cultural elements such as the wide robes and sleeves of Song Dynasty scholar-officials, the headdresses and hats, and the phoenix costumes and silver ornaments of She ethnic women. The dynamic deformation samples are labeled with the mapping relationship between motion parameters and clothing folds and draping shapes. The LoRA lightweight fine-tuning only adjusts 5% of the parameters of the ControlNet model.

5. The method for generating culturally distinctive digital humans by integrating LoRA fine-tuning and cross-modal interaction as described in claim 1, characterized in that: In S4, the motion capture device collects real-person motion and facial expression data, including body language and facial micro-expressions. The motion transfer model achieves accurate mapping of real-person motion to digital humans through training. The linkage between speech recognition and text-to-speech technology is synchronized through a timestamp alignment mechanism.

6. The method for generating culturally distinctive digital humans by integrating LoRA fine-tuning and cross-modal interaction as described in claim 1, characterized in that: In S5, emotion recognition includes voice emotion analysis and text semantic analysis. Voice emotion analysis extracts features of tone, speech rate, and volume, while text semantic analysis parses the emotional tendency of the content. The parameters for conversion cover facial expression amplitude, body movement intensity, and voice tone change values.

7. The method for generating culturally distinctive digital humans by integrating LoRA fine-tuning and cross-modal interaction as described in claim 2, characterized in that: The high-precision 3D digital human model has a facial scanning accuracy of 0.1 mm, a body modeling error of <0.5 cm, and facial contours of the brow bone and cheekbone that highly match historical portraits.

8. The method for generating culturally distinctive digital humans by integrating LoRA fine-tuning and cross-modal interaction as described in claim 1, characterized in that: The synchronization delay between the digital human's movements and speech is <300 milliseconds, and the synchronization accuracy is >95%.

Images