Digital human personalized shaping method and device based on user psychology and medium

Abstract

The invention discloses a digital human personalized shaping method and device based on user psychology and a medium. The method comprises the following steps: acquiring multi-modal data when a user interacts with a digital human; based on the multi-modal data, according to the style preference of the user, establishing a psychological feature model through an adaptive learning mode; dialogue voice of the digital person is generated according to the psychological feature model, and matched facial expressions and limb actions of the digital person are generated according to the dialogue voice and the psychological features of the user. According to the digital human personalized shaping method and device based on the psychology of the user and the medium, the psychological feature model is established based on the style preference of the user, and the interaction content matched with the conversation voice, the facial expression and the body movement is generated, so that the digital human can show more real and natural personalized behaviors, and the user experience is improved. The problem that it is difficult to carry out personalized behavior shaping on the digital human according to psychological characteristics and interaction preferences of the user can be solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and medium for personalized shaping of digital humans based on user psychology. Background Art

[0002] Digital Human (Digital Human / Meta Human) is a digital human image close to human image created by digital technology; with the continuous development of computer graphics, artificial intelligence, motion capture and drive, image rendering and other technologies, the production and operation of digital humans have become more efficient and convenient, and digital humans have been widely used in many fields such as entertainment, media, finance and education. At present, special equipment and software are mainly used to record the movements of real actors, and then these movement data are applied to virtual character models; drive technology controls the movements of virtual characters through programming or action libraries to make their movements more natural. At the same time, it is also necessary to capture and drive the facial expressions of digital humans to achieve rich emotional expressions.

[0003] However, existing digital human systems usually rely on predefined dialogue templates and fixed voice outputs, lacking a deep understanding of users' personalized needs and the ability to dynamically adjust them. Therefore, existing systems are unable to fully perform adaptive behavior tuning based on users' psychological characteristics and interaction preferences, resulting in a stiff and unnatural interactive experience. Summary of the invention

[0004] The present invention provides a method, device and medium for personalized shaping of digital humans based on user psychology, so as to solve the problem that it is difficult to shape the personalized behavior of digital humans according to the user's psychological characteristics and interaction preferences.

[0005] Acquire multimodal data when users interact with digital humans;

[0006] Based on the multimodal data, and in accordance with the user's style preference, a psychological feature model is established through adaptive learning;

[0007] The conversational speech of the digital human is generated according to the psychological characteristic model, and the facial expressions and body movements of the digital human matching the conversational speech and the psychological characteristics of the user are generated according to the conversational speech.

[0008] The present invention can fully reflect the user's interactive behavior and style preferences by acquiring multimodal data when the user interacts with the digital human. Based on multimodal data, a psychological characteristic model is established through adaptive learning, which can accurately capture the user's personalized needs and fully consider the dynamic characteristics of the user's emotions. The digital human's conversational voice is generated according to the psychological characteristic model, which can ensure the personalized, emotional and natural fluency of the voice, making the digital human's voice more in line with the user's expectations and preferences, and improving the intimacy and credibility of the interaction. The facial expressions and body movements of the digital human that match the conversational voice and the user's psychological characteristics are generated, which can further enhance the realism and immersion of the interaction, so that the personalized shaping of the digital human can be achieved in terms of voice and movement.

[0009] Compared with the existing technology, the present invention establishes a psychological characteristic model based on the user's style preferences, and generates interactive content that matches the conversational voice, facial expressions and body movements, which can allow the digital human to exhibit more realistic and natural personalized behavior. Therefore, it can solve the problem of difficulty in shaping the personalized behavior of the digital human according to the user's psychological characteristics and interaction preferences.

[0010] As a preferred solution, based on the multimodal data and according to the user's style preference, a psychological feature model is established through adaptive learning, specifically:

[0011] Preprocessing the multimodal data to obtain basic data;

[0012] Extracting the user's emotion vector from the basic data through a preset neural network or emotion recognition algorithm to obtain the user's psychological characteristics;

[0013] Based on the user psychological characteristics, an initial psychological characteristic model is established through a collaborative module set; wherein the collaborative module set includes a user psychological characteristic analysis module, a dialogue generation and personalized tuning module, a speech synthesis and adjustment module, a digital human animation and rendering module, and a feedback and adaptive learning module;

[0014] The user's feedback data on the initial psychological characteristic model is obtained, and according to the user preference value in the feedback data, the adaptive learning direction of the initial psychological characteristic model is directionally adjusted in a weighted manner to obtain the psychological characteristic model.

[0015] In this preferred solution, the design of the collaborative module set enables each module to work independently and at the same time cooperate to jointly build the initial psychological feature model. This modular design improves the flexibility and scalability of the system and facilitates subsequent maintenance and upgrades. By obtaining user feedback data on the initial psychological feature model and making weighted adjustments based on user preference values, the system can continuously learn and optimize the psychological feature model. This adaptive learning capability enables the system to continuously track user changes and provide more accurate services.

[0016] As a preferred solution, the multimodal data is preprocessed to obtain basic data, specifically:

[0017] Extracting acoustic features, communication features, and behavioral features from the speech data, text data, and visual data of the multimodal data, respectively;

[0018] Performing multimodal data fusion processing on the acoustic features, the communication features and the behavioral features to obtain comprehensive feature data;

[0019] The comprehensive feature data is subjected to denoising and standardization processing to obtain the basic data.

[0020] This preferred solution extracts features from data in three different modalities: voice, text, and vision, and can more comprehensively capture the user's behavior, emotions, and intentions. Multimodal data fusion processing can integrate information from different modalities to form a richer and more complete feature set. This fusion processing not only improves the efficiency of data processing, but also reduces data redundancy and noise, thereby improving the effect of subsequent analysis.

[0021] As a preferred solution, the conversational speech of the digital human is generated according to the psychological characteristic model, specifically:

[0022] Based on the psychological feature model, initial dialogue content is obtained by controlling the randomness and diversity of dialogue generation;

[0023] Performing preference recognition and extraction on the multimodal data in the dimensions of voice, action and emotion to obtain user preference data;

[0024] The expression and context of the initial conversation content are dynamically adjusted according to the preference data, and based on speech synthesis technology, the tone and speed of the initial conversation content are adjusted according to the preset emotion analysis results to generate the conversation voice of the digital human.

[0025] This preferred solution can more accurately grasp the user's emotional state by identifying the user's preferences in voice, action and emotion dimensions, and adjust the conversation method accordingly, so that the conversation is more in line with the natural communication mode of humans and improves the realism of the conversation. Combined with speech synthesis technology, the system can generate high-quality voice output, further enhancing the immersiveness of the conversation, so that users can experience a more realistic interactive experience when talking to digital people.

[0026] As a preferred solution, the sentiment analysis result is specifically:

[0027] Performing stop word removal and lemmatization processing on the multimodal data to obtain sentiment words and sentences;

[0028] By matching the emotional words and sentences with the emotional dictionary, the emotional expressions and potential emotional tendencies in the emotional words and sentences are identified to obtain the emotional analysis results including emotional classification; wherein the emotional dictionary is a vocabulary library including emotional words of different categories.

[0029] The removal of stop words and word form restoration in this preferred solution can simplify text data, remove meaningless words and word form changes, and make emotional words and sentences focus more on core emotional expressions. The emotional dictionary is a database containing a large number of emotional words and their emotional tendencies. By matching emotional words and sentences with the emotional dictionary, the emotional expressions in the text can be accurately identified. This matching method avoids the errors that may be caused by emotional analysis based on rules or statistical methods, and improves the accuracy of emotional analysis.

[0030] As a preferred solution, facial expressions and body movements of a digital human matching the conversational speech and the psychological characteristics of the user are generated, specifically:

[0031] Performing word segmentation and part-of-speech tagging on the conversation speech to obtain key sentence components;

[0032] Capturing context information across sentences in the conversation speech to obtain contextual relationships;

[0033] Identify emotional words and phrases in the conversation speech to obtain semantic understanding results;

[0034] According to the key sentence components, the contextual relationship and the semantic understanding result, the emotion classification and emotion intensity calculation are performed on the conversation speech to obtain a multimodal emotion analysis result;

[0035] Based on the multimodal emotion analysis results and the psychological characteristics of the user, the facial expressions and body movements that match the user are generated on the digital human through 3D modeling technology.

[0036] This preferred solution can accurately identify keywords and sentence components in a conversation through word segmentation and part-of-speech tagging, thereby more accurately understanding the semantics of the conversation; capturing cross-sentence context information can grasp the overall flow and logical relationship of the conversation, avoiding understanding a single sentence in isolation; and by identifying emotional words and phrases, an in-depth understanding of the emotional color and intensity expressed in the conversation can be achieved.

[0037] As a preferred solution, based on the multimodal emotion analysis results and the psychological characteristics of the user, the facial expressions and body movements that match the user are generated on the digital human through 3D modeling technology, specifically:

[0038] Based on the multimodal emotion analysis results and the psychological characteristics of the user, the expression data corresponding to different emotions are bound to the facial control points of the digital human through 3D modeling technology, and the emotion data in the multimodal emotion analysis results are converted into the facial expressions of the digital human;

[0039] Generate corresponding gestures and body postures for different emotions and semantics according to the emotion intensity and conversation content in the multimodal emotion analysis results;

[0040] The gestures and body postures are bound to the skeletal structure of the digital human to generate the body movements of the digital human.

[0041] This preferred solution generates gestures and body postures based on the intensity of emotions and the content of the conversation, making the digital human's body movements more natural and smooth; these movements are coordinated with facial expressions, together forming a complete and real emotional expression system. In addition, by binding the expression data and body movements to the digital human's facial control points and bone structure through 3D modeling technology, highly realistic emotional expressions and body movements can be achieved.

[0042] The present application also provides a device for personalizing a digital human based on user psychology, including a data module, a model module, and an adjustment module;

[0043] Wherein, the data module is used to obtain multimodal data when the user interacts with the digital human;

[0044] The model module is used to establish a psychological feature model through adaptive learning based on the multimodal data and according to the style preference of the user;

[0045] The adjustment module is used to generate the conversational speech of the digital human according to the psychological characteristic model, and to generate the facial expressions and body movements of the digital human matching the conversational speech and the psychological characteristics of the user.

[0046] As a preferred solution, the model module includes a data unit, a psychological unit, a model unit and a feedback unit;

[0047] Wherein, the data unit is used to pre-process the multimodal data to obtain basic data;

[0048] The psychological unit is used to extract the user's emotion vector from the basic data through a preset neural network or emotion recognition algorithm to obtain the user's psychological characteristics;

[0049] The model unit is used to establish an initial psychological feature model based on the user psychological features through a collaborative module set; wherein the collaborative module set includes a user psychological feature analysis module, a dialogue generation and personalized tuning module, a speech synthesis and adjustment module, a digital human animation and rendering module, and a feedback and adaptive learning module;

[0050] The feedback unit is used to obtain the user's feedback data on the initial psychological characteristic model, and according to the user preference value in the feedback data, directionally adjust the adaptive learning direction of the initial psychological characteristic model in a weighted manner to obtain the psychological characteristic model.

[0051] As a preferred solution, the data unit includes a feature subunit, a fusion subunit and a denoising subunit;

[0052] Wherein, the feature subunit is used to extract acoustic features, communication features and behavioral features from the voice data, text data and visual data of the multimodal data respectively;

[0053] The fusion subunit is used to perform multimodal data fusion processing on the acoustic features, the communication features and the behavioral features to obtain comprehensive feature data;

[0054] The denoising subunit is used to perform denoising and standardization processing on the comprehensive feature data to obtain the basic data.

[0055] As a preferred solution, the adjustment module includes a dialogue unit, a preference unit and a speech unit;

[0056] The dialogue unit is used to obtain the initial dialogue content by controlling the randomness and diversity of dialogue generation based on the psychological feature model;

[0057] The preference unit is used to perform preference identification and extraction on the multimodal data in the dimensions of voice, action and emotion to obtain user preference data;

[0058] The speech unit is used to dynamically adjust the expression and context of the initial dialogue content according to the preference data, and based on speech synthesis technology, adjust the tone and speed of the initial dialogue content according to the preset emotion analysis results to generate the dialogue voice of the digital human.

[0059] As a preferred solution, the sentiment analysis result is specifically:

[0060] Performing stop word removal and lemmatization processing on the multimodal data to obtain sentiment words and sentences;

[0061] By matching the emotional words and sentences with the emotional dictionary, the emotional expressions and potential emotional tendencies in the emotional words and sentences are identified to obtain the emotional analysis results including emotional classification; wherein the emotional dictionary is a vocabulary library including emotional words of different categories.

[0062] As a preferred solution, the adjustment module includes a component unit, a relationship unit, a recognition unit, an analysis unit and an action unit;

[0063] The component unit is used to perform word segmentation and part-of-speech tagging on the conversation speech to obtain key sentence components;

[0064] The relation unit is used to capture context information across sentences in the conversation speech to obtain contextual relations;

[0065] The recognition unit is used to recognize emotional words and phrases in the conversation speech to obtain semantic understanding results;

[0066] The analysis unit is used to perform emotion classification and emotion intensity calculation on the conversation speech according to the key sentence components, the contextual relationship and the semantic understanding result to obtain a multimodal emotion analysis result;

[0067] The action unit is used to generate the facial expressions and body movements that match the multimodal emotion analysis results and the psychological characteristics of the user on the digital human through 3D modeling technology.

[0068] As a preferred solution, the action unit includes an expression subunit, a posture subunit and an action subunit;

[0069] The expression subunit is used to bind the expression data corresponding to different emotions to the facial control points of the digital human through 3D modeling technology based on the multimodal emotion analysis results and the psychological characteristics of the user, and convert the emotion data in the multimodal emotion analysis results into the facial expressions of the digital human;

[0070] The gesture subunit is used to generate corresponding gestures and body postures for different emotions and semantics according to the emotion intensity and conversation content in the multimodal emotion analysis result;

[0071] The action subunit is used to bind the gestures and body postures to the skeletal structure of the digital human to generate the limb movements of the digital human.

[0072] The present application also provides a storage medium on which a computer program is stored. The computer program is called and executed by a computer to implement the above-mentioned method for personalized shaping of digital humans based on user psychology. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a flow chart of a method for personalizing a digital human based on user psychology provided in an embodiment of the present application;

[0074] Figure 2 is a flowchart of user psychological characteristics analysis provided by an embodiment of the present application;

[0075] Figure 3 is a flow chart of dialogue generation and speech synthesis provided by an embodiment of the present application;

[0076] Figure 4 is a flow chart of digital human behavior generation provided by an embodiment of the present application;

[0077] Figure 5 It is a system architecture diagram provided by an embodiment of the present application;

[0078] Figure 6 It is a structural schematic diagram of a digital human personalized shaping device based on user psychology provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0080] In the description of the present application, unless otherwise specified, “several” means two or more.

[0081] It should be noted that in the embodiments of the present application, TTS technology (Text To Speech) refers to text-to-speech technology, NLP technology (Natural Language Processing) refers to natural language processing technology, Blendshape is a weight-based vertex deformation technology, and Morph Target technology is a technology for defining deformation targets in a 3D model.

[0082] The embodiment of the present application provides a method for personalizing a digital human based on user psychology, which is mainly used in situations where it is necessary to adaptively tune the behavior of a digital human according to the user's psychological characteristics and interaction preferences in order to achieve a natural interaction effect.

[0083] Embodiment 1:

[0084] See also Figure 1 The embodiment of the present application provides a method for personalizing a digital human based on user psychology, including S1 to S3, and the specific implementation steps are as follows:

[0085] S1. Obtain multimodal data when users interact with digital humans.

[0086] Step S1 of the embodiment of the present application is specifically as follows:

[0087] Acquire multimodal data when the user interacts with the digital human; wherein the multimodal data includes conversation text, voice signals and facial expressions;

[0088] Preprocess the multimodal data, such as denoising and normalization; input the preprocessed multimodal data into the user psychological feature analysis module.

[0089] S2. Based on multimodal data and in accordance with the user’s style preferences, a psychological feature model is established through adaptive learning.

[0090] Step S2 of the embodiment of the present application includes S2.1 to S2.2, wherein S2.1 is a process of converting multimodal data into basic data, and S2.2 is a process of establishing a psychological characteristic model based on the basic data, specifically:

[0091] S2.1. Based on the user psychological feature analysis module, acoustic features (such as speech speed, timbre and emotional intensity), communication features (such as conversation style, word habits and emotional state) and behavioral features (such as facial expressions and body movements) are extracted from the voice data, text data and visual data of multimodal data respectively;

[0092] Perform multimodal data fusion processing on acoustic features, communication features and behavioral features to obtain comprehensive feature data;

[0093] The comprehensive feature data is denoised and standardized to obtain basic data.

[0094] In this embodiment S2.1, features are extracted from data of three different modalities, namely speech, text and vision, which can more comprehensively capture the user's behavior, emotions and intentions. Multimodal data fusion processing can integrate information from different modalities to form a richer and more complete feature set. This fusion processing not only improves the efficiency of data processing, but also reduces data redundancy and noise, thereby improving the effect of subsequent analysis;

[0095] At the same time, when the signal noise of a certain mode is large, the system can use other modes to compensate and improve the overall robustness and reliability.

[0096] S2.2, extracting the user's emotion vector from the basic data through a convolutional neural network (CNN) or an emotion recognition algorithm (such as a support vector machine SVM) to obtain the user's psychological characteristics;

[0097] Based on the user's psychological characteristics, an initial psychological characteristic model is established through a collaborative module set; wherein the collaborative module set includes a user psychological characteristic analysis module, a dialogue generation and personalized tuning module, a speech synthesis and adjustment module, a digital human animation and rendering module, and a feedback and adaptive learning module; and the user psychological characteristic analysis module is used to collect user multimodal data and extract user psychological characteristics; the dialogue generation and personalized tuning module is used to generate personalized dialogue content according to user psychological characteristics and adjust the interaction style; the speech synthesis and adjustment module is used to synthesize speech output that meets personalized needs according to emotion-driven parameters; the digital human animation and rendering module is used to generate digital human behavior, including expression drive, body movement rendering and adaptive tuning; the feedback and adaptive learning module is used to optimize the psychological characteristic model according to user feedback data.

[0098] Obtaining user feedback data on the initial psychological characteristic model; wherein the feedback data includes user conversation data and non-verbal signals;

[0099] Use NLP and emotion recognition algorithms to extract emotional states from feedback data; and based on the user preference update formula, adjust the adaptive learning direction of the initial psychological feature model in a weighted manner according to the emotional state and user preference values ​​in the feedback data to obtain a psychological feature model; where user preference values ​​include conversation style preferences, emotion and sentiment preferences, speech speed and tone preferences, and non-verbal interaction preferences;

[0100] Update the user's personalized preferences in the psychological characteristic model, such as preference for humor or direct communication.

[0101] Among them, the user preference update formula is:

[0102] Pnew=α×Pold+(1-α)×Pfeedback

[0103] Among them, Pnew is the updated preference, Pold is the original preference, Pfeedback is the preference in the feedback data, and α is the weight coefficient; this formula is used to help the model gradually optimize the user's personalized preferences in adaptive learning.

[0104] To apply the embodiments of this application, please refer to Figure 2 , Figure 2 This is a user psychological characteristic analysis flow chart provided in an embodiment of the present application, which shows the general process of building a psychological characteristic model.

[0105] The method for obtaining each preference value in the user preference value is as follows:

[0106] ① Conversation style preference:

[0107] Based on the feedback data, extract user characteristics such as language style, vocabulary selection, sentence length and conversation rhythm during the interaction process;

[0108] Use NLP technology to perform semantic and style analysis on the language patterns of user characteristics, identify whether the user prefers conversation styles such as humorous, formal, and concise, and obtain conversation style preferences including feature labels or preference levels of user conversation styles (such as preference for humorous style, preference for concise conversation).

[0109] ②Emotional and emotional preferences:

[0110] Using emotion recognition algorithms (such as convolutional neural networks (CNN), support vector machines (SVM), etc.), we extract the user's emotional characteristics (including positive, negative, neutral, etc. emotional tendencies) from the voice, text, and expressions of the feedback data, and obtain the user's emotions and emotional preferences including emotional vectors (such as happiness, sadness, anger, etc.).

[0111] ③Speech speed and tone preference:

[0112] Through speech signal processing technology, the user's preferences for the speaking speed, timbre and volume of the interactive object in the feedback data are identified, and the parameters of the speech output are adjusted in combination with emotion analysis to generate specific speaking speed and tone preferences (such as preference for steady tone and slower speaking speed).

[0113] ④ Non-verbal interaction preferences:

[0114] Based on the user's feedback and reaction data in the feedback data (such as facial expressions, eye tracking, etc.), we can identify their acceptance of body language and reaction preferences, such as whether they prefer more body movements, whether they like more facial expressions, and obtain their non-verbal interaction preferences.

[0115] Among them, the method of adjusting the adaptive learning direction of the initial psychological feature model according to the user preference value in the feedback data is:

[0116] ① Adjust speech output based on speech speed and intonation preferences:

[0117] According to the user's emotional state (such as positive, calm, anxious, etc.) and preference for speech speed, the speech speed and intonation parameters are dynamically adjusted using emotion-driven TTS technology, so that the digital human based on the psychological feature model can express different emotions. For example, in a pleasant situation, the speech speed is faster and the tone is brighter, and in a serious or soothing scene, the speech speed is slower and the tone is lower.

[0118] ② Adjust the words and expressions according to the conversation style preference:

[0119] Use the user's conversation style preferences (such as humor, formality, and conciseness) to adjust the language expression of the digital person corresponding to the psychological feature model. Use NLP technology and the Large Language Model (LLM) to adjust the wording style in combination with the user's preferences to generate personalized conversations for the digital person.

[0120] For example, if the user prefers a humorous style, the system will generate a more relaxed expression with humorous elements; if the user prefers a formal style, it will choose a more formal and rigorous language. During the real-time interaction process, the system will further refine the language style based on the user's immediate response to maintain the consistency and naturalness of the conversation.

[0121] ③ Dynamic adjustment of context:

[0122] During the conversation, the user's emotional changes are dynamically evaluated. If a significant change in the user's emotions is detected (such as from neutral to happy), the speaking speed and wording of the digital person corresponding to the psychological characteristic model will be adjusted accordingly to make it consistent with the user's current emotional and contextual needs.

[0123] In this embodiment S2.2, the design of the collaborative module set enables each module to work independently and at the same time cooperate to jointly construct the initial psychological characteristic model. This modular design improves the flexibility and scalability of the system and facilitates subsequent maintenance and upgrades. By obtaining user feedback data on the initial psychological characteristic model and making weighted adjustments based on user preference values, the system can continuously learn and optimize the psychological characteristic model. This adaptive learning capability enables the system to continuously track user changes and provide more accurate services;

[0124] In addition, the use of adaptive learning technology enables the system to continuously optimize the psychological characteristic model. As the number of interactions increases, the system can gradually adapt to the user's personalized needs and emotional changes. This adaptive mechanism helps to improve the user's long-term satisfaction and the continuity of the digital human interaction experience.

[0125] S3. Generate the digital human's conversational speech based on the psychological characteristic model, and generate the digital human's facial expressions and body movements that match the conversational speech and the user's psychological characteristics.

[0126] Step S3 of the embodiment of the present application includes S3.1 to S3.5, wherein S3.1 is a process of generating dialogue content according to the psychological characteristic model, S3.2 is a process of generating the dialogue voice of the digital human according to the dialogue content, S3.3 is a process of generating multimodal emotion analysis results, S3.4 is a process of generating the gestures and body postures of the digital human according to the multimodal emotion analysis results and the user's psychological characteristics, and S3.5 is a process of continuously optimizing the digital human according to the feedback data, specifically:

[0127] S3.1. The dialogue generation and personalized tuning module based on the psychological feature model uses a large language model to control the randomness and diversity of dialogue generation and generate initial dialogue content that conforms to the user's psychological characteristics. The initial dialogue content is checked for semantic consistency to ensure that the dialogue remains coherent while being personalized.

[0128] Perform preference recognition and extraction on multimodal data in the dimensions of voice, action and emotion to obtain user preference data;

[0129] The expression and context of the initial conversation content are dynamically adjusted according to the preference data to obtain the conversation content; and the style of the conversation content is dynamically adjusted according to the real-time feedback from the user.

[0130] This embodiment S3.1 can more accurately grasp the user's emotional state by identifying the user's preferences in the dimensions of voice, action and emotion, and adjust the conversation method accordingly, so that the conversation is more in line with the natural communication mode of humans and the realism of the conversation is improved.

[0131] S3.2. Based on the speech synthesis and adjustment module, the multimodal data is segmented, stop words are removed, and morphological restoration is performed to obtain emotional words and sentences containing key emotional words and phrases;

[0132] By matching sentiment words and sentences with sentiment dictionaries, we can identify the sentiment expressions and potential sentiment tendencies in sentiment words and sentences, and use machine learning models (such as convolutional neural networks (CNN) or long short-term memory networks (LSTM)) to analyze the context and sentiment patterns of the text, and obtain sentiment analysis results including sentiment classification and sentiment intensity; where the sentiment dictionary is a vocabulary library containing positive and negative sentiment words, and sentiment classification includes categories such as positive, negative, and neutral;

[0133] Combined with real-time user feedback, the sentiment classification results are optimized, and through reinforcement learning, the sentiment classification results are gradually adapted to the user's personalized emotional response;

[0134] Based on speech synthesis technology (such as TTS technology), the tone, speed and emotional expression of the conversation content are adjusted according to the results of sentiment analysis to generate the conversational speech of the digital human; and during speech output, the speed, timbre and emotional intensity of the conversational speech are adjusted through real-time user emotional feedback to make it more natural.

[0135] The specific method of adjusting the conversation content according to the sentiment analysis results is as follows:

[0136] ① Adjust the tone of voice according to the type of emotion:

[0137] According to the emotion classification (such as positive, negative and neutral), different tones are set for voice output. For example, when positive emotions are recognized, the system increases the brightness and liveliness of the tone; when the emotion is negative, the tone becomes low and gentle.

[0138] ② Adjust the speaking speed according to the intensity of emotions:

[0139] When you are in a state of high intensity (such as excitement or anger), increase your speaking speed to convey the urgency of the emotion; when you are in a state of calm or soothing emotion, slow down your speaking speed to create a stable atmosphere.

[0140] ③Dynamically adjust volume and timbre based on emotional expression needs:

[0141] When you are in a happy or excited mood, slightly increase the volume to make the tone brighter; when you are in a calm or negative mood, reduce the volume to make the tone softer.

[0142] ④ Combine multimodal feedback to fine-tune emotional expression in real time:

[0143] Use the user's real-time feedback (such as facial expressions and tone responses) to fine-tune the tone, speed, and emotional intensity of the voice. For example, if the system detects that the user's emotions are fluctuating, the digital human's emotional expression can be strengthened or weakened accordingly.

[0144] ⑤ Emotion-driven TTS technology:

[0145] Based on the results of emotion analysis and preset emotion parameters (such as speaking speed, pitch, and emotion intensity), speech synthesis technology (such as TTS technology) is used to automatically generate emotional speech.

[0146] To apply the embodiments of this application, please refer to Figure 3 , Figure 3 It is a flow chart of dialogue generation and speech synthesis provided in an embodiment of the present application, which shows the general process of text processing and speech output.

[0147] In this embodiment S3.2, combined with speech synthesis technology, the system can generate high-quality speech output, further enhancing the immersiveness of the conversation, so that users can experience a more realistic interactive experience when talking to digital people; and the removal of stop words and word form restoration processing can simplify text data, remove meaningless words and word form changes, and make emotional words and sentences more focused on core emotional expressions. The emotional dictionary is a database containing a large number of emotional words and their emotional tendencies. By matching emotional words and sentences with the emotional dictionary, the emotional expression in the text can be accurately identified. This matching method avoids the errors that may be caused by emotional analysis based on rules or statistical methods, and improves the accuracy of emotional analysis.

[0148] S3.3. Based on the digital human animation and rendering module, the dialogue speech and its content are segmented, POS tagged and parsed to obtain key sentence components; wherein the key sentence components include subject, predicate and object, etc.;

[0149] Capture contextual information across sentences in the conversational speech and its content through contextual understanding models (such as Transformer or RNN architecture) to obtain contextual relationships;

[0150] Based on semantic analysis technology, emotional words and phrases (such as "happy" and "disappointed") in conversation speech and its content are identified through emotional dictionaries and pre-trained emotional recognition models to obtain semantic understanding results;

[0151] According to the key sentence components, contextual relationships and semantic understanding results, an emotion classification model (such as convolutional neural network CNN or support vector machine SVM) is used to classify the emotions and calculate the emotion intensity of the dialogue speech and its content, and obtain multimodal emotion information (such as text, speech and expression) including emotion classification and emotion intensity; by fusing multimodal emotion information, multimodal emotion analysis results are obtained; among them, the multimodal emotion analysis results include positive emotions, negative emotions, neutral emotions and their intensity.

[0152] In this embodiment S3.3, the key words and sentence components in the conversation can be accurately identified through word segmentation and part-of-speech tagging, so as to more accurately understand the semantics of the conversation; capturing cross-sentence context information can grasp the overall process and logical relationship of the conversation, avoiding the understanding of a single sentence in isolation; by identifying emotional words and phrases, the emotional color and intensity expressed in the conversation can be deeply understood;

[0153] In addition, by combining multimodal analysis of user psychological characteristics, this embodiment can accurately extract the user's personalized needs, which helps to automatically adjust the digital human's dialogue, voice style and animation performance in the future, making the interaction more personalized and natural.

[0154] S3.4. Based on the results of multimodal emotion analysis and the psychological characteristics of the user, the expression data corresponding to different emotions (such as smile, frown and surprise, etc.) are bound to the facial control points of the digital human through 3D modeling technology, and the emotion data in the multimodal emotion analysis results are converted into the facial expressions of the digital human in real time through Blendshape or Morph Target technology. For example, the emotion of "happiness" is mapped to the facial posture of smiling and relaxing eyes; while the emotion of "anger" is mapped to expressions such as closed lips and frowning.

[0155] Based on the emotion intensity and conversation content in the multimodal emotion analysis results, animation-driven technology is used to generate corresponding gestures and body postures for different emotions and semantics;

[0156] The gestures and body postures are bound to the digital human's skeletal structure through bone binding technology, so that each bone node (such as shoulders, elbows and knees) is associated with a control weight, allowing each part to move in a coordinated manner, thereby generating the digital human's limb movements and adjusting the digital human's limb movements based on user preference dynamics and preset action libraries.

[0157] The overall performance of the digital human is generated synchronously by combining voice, text and expression data to ensure that the facial, gesture and body posture are consistent in emotional expression; and a gradual transition is used when the emotion changes, so that the changes in expression, gesture and posture are smooth and continuous, avoiding abruptness. For example, when expressing a happy emotion, the voice tone, facial smile and open gestures will change in coordination to enhance the emotional authenticity and expressiveness of the digital human; when changing from a smile to a serious expression, the facial and body postures are gradually adjusted to present a smoother visual effect.

[0158] In this embodiment S3.4, gestures and body postures are generated according to the intensity of emotions and the content of the dialogue, making the body movements of the digital human more natural and smooth; these movements are coordinated with the facial expressions, together forming a complete and real emotional expression system. In addition, by binding the expression data and body movements to the facial control points and bone structure of the digital human through 3D modeling technology, highly realistic emotional expressions and body movements can be achieved;

[0159] In addition, the use of dynamic emotion-driven methods can make the digital human's emotional expression richer and more natural. The system can adjust the tone, expression and posture according to the user's psychological state in real time, so that the digital human can show emotional reactions that are consistent with the content and context of the conversation. This dynamic adjustment mechanism significantly improves the authenticity of the interaction and the user's immersion.

[0160] S3.5. Based on the digital human animation and rendering module, collect the user's interactive feedback data on the digital human (such as emotional response, dialogue satisfaction score, etc.); and after each interaction, update the digital human's body, expression and voice corresponding to the psychological feature model according to the interactive feedback data to optimize the expression form for specific users;

[0161] Reinforcement learning technology is used to continuously improve the psychological characteristic model and the adaptability of digital humans to user reactions.

[0162] To apply the embodiments of this application, please refer to Figure 4 , Figure 4 This is a digital human behavior generation flow chart provided in an embodiment of the present application, which shows the entire process from dialogue content to animation rendering.

[0163] To apply the embodiments of this application, please refer to Figure 5 , Figure 5 It is a system architecture diagram provided in an embodiment of the present application, showing the various modules in the psychological characteristic model and their relationships.

[0164] Overall, this embodiment has the following beneficial effects:

[0165] This application can fully reflect the user's interactive behavior and style preferences by acquiring multimodal data when the user interacts with the digital human. Based on multimodal data, a psychological characteristic model is established through adaptive learning, which can accurately capture the user's personalized needs and fully consider the dynamic characteristics of the user's emotions. Generating the digital human's conversational voice based on the psychological characteristic model can ensure the personalization, emotionality and natural fluency of the voice, making the digital human's voice more in line with the user's expectations and preferences, and improving the intimacy and credibility of the interaction. Generating matching digital human facial expressions and body movements based on the conversational voice and the user's psychological characteristics can further enhance the realism and immersion of the interaction. Therefore, the personalized shaping of the digital human can be achieved in terms of voice and movement, and the adaptive behavior of the digital human can be fully optimized according to the user's psychological characteristics and interaction preferences, so that the user's interactive experience is natural and not stiff;

[0166] This application provides a variety of technical innovations such as personalized tuning, multimodal fusion and adaptive learning, which reduces labor costs economically, improves customer stickiness, and promotes the popularization and application of digital human technology and the availability of mental health support services in society.

[0167] Embodiment 2:

[0168] See also Figure 6 , the embodiment of the present application provides a digital human personalized shaping device based on user psychology, including a data module 10, a model module 20 and an adjustment module 30;

[0169] The data module 10 is used to obtain multimodal data when the user interacts with the digital human;

[0170] Model module 20, used to establish a psychological feature model through adaptive learning based on multimodal data and according to the user's style preference;

[0171] The adjustment module 30 is used to generate the digital human's conversational speech according to the psychological characteristic model, and to generate the digital human's facial expressions and body movements that match the conversational speech and the user's psychological characteristics.

[0172] In one embodiment, the data module 10 is specifically:

[0173] Acquire multimodal data when the user interacts with the digital human; wherein the multimodal data includes conversation text, voice signals and facial expressions;

[0174] Preprocess the multimodal data, such as denoising and normalization; input the preprocessed multimodal data into the user psychological feature analysis module.

[0175] In one embodiment, the model module 20 includes a feature subunit, a fusion subunit, a denoising subunit, a psychological unit, a model unit and a feedback unit, wherein the feature subunit, the fusion subunit and the denoising subunit are processes for converting multimodal data into basic data, and the psychological unit, the model unit and the feedback unit are processes for establishing a psychological feature model based on the basic data, specifically:

[0176] Among them, the feature subunit is used to extract acoustic features (such as speech speed, timbre and emotional intensity), communication features (such as dialogue style, word habits and emotional state) and behavioral features (such as facial expressions and body movements) from the voice data, text data and visual data of multimodal data on the basis of the user psychological feature analysis module;

[0177] The fusion subunit is used to perform multimodal data fusion processing on acoustic features, communication features and behavioral features to obtain comprehensive feature data;

[0178] The denoising subunit is used to denoise and standardize the comprehensive feature data to obtain basic data.

[0179] The feature subunit, fusion subunit and denoising subunit of this embodiment extract features from data of three different modalities: speech, text and vision, and can more comprehensively capture the user's behavior, emotions and intentions. Multimodal data fusion processing can integrate information from different modalities to form a richer and more complete feature set. This fusion processing not only improves the efficiency of data processing, but also reduces data redundancy and noise, thereby improving the effect of subsequent analysis;

[0180] At the same time, when the signal noise of a certain mode is large, the system can use other modes to compensate and improve the overall robustness and reliability.

[0181] The psychological unit is used to extract the user's emotion vector from the basic data through a convolutional neural network (CNN) or an emotion recognition algorithm (such as a support vector machine SVM) to obtain the user's psychological characteristics;

[0182] A model unit is used to establish an initial psychological feature model based on user psychological features through a collaborative module set; wherein the collaborative module set includes a user psychological feature analysis module, a dialogue generation and personalized tuning module, a speech synthesis and adjustment module, a digital human animation and rendering module, and a feedback and adaptive learning module; and the user psychological feature analysis module is used to collect user multimodal data and extract user psychological features; the dialogue generation and personalized tuning module is used to generate personalized dialogue content according to user psychological features and adjust the interaction style; the speech synthesis and adjustment module is used to synthesize speech output that meets personalized needs according to emotion-driven parameters; the digital human animation and rendering module is used to generate digital human behavior, including expression drive, body movement rendering and adaptive tuning; the feedback and adaptive learning module is used to optimize the psychological feature model according to user feedback data.

[0183] A feedback unit, used to obtain user feedback data on the initial psychological characteristic model; wherein the feedback data includes user dialogue data and non-verbal signals;

[0184] The feedback unit is further used to extract the emotional state from the feedback data using NLP and emotion recognition algorithms; and based on the user preference update formula, according to the emotional state and the user preference value in the feedback data, the adaptive learning direction of the initial psychological feature model is adjusted in a weighted manner to obtain the psychological feature model; wherein the user preference value includes conversation style preference, emotion and sentiment preference, speech speed and tone preference, and non-verbal interaction preference;

[0185] Update the user's personalized preferences in the psychological characteristic model, such as preference for humor or direct communication.

[0186] Among them, the user preference update formula is:

[0187] Pnew=α×Pold+(1-α)×Pfeedback

[0188] Among them, Pnew is the updated preference, Pold is the original preference, Pfeedback is the preference in the feedback data, and α is the weight coefficient; this formula is used to help the model gradually optimize the user's personalized preferences in adaptive learning.

[0189] To apply the embodiments of this application, please refer to Figure 2 , Figure 2 This is a user psychological characteristic analysis flow chart provided in an embodiment of the present application, which shows the general process of building a psychological characteristic model.

[0190] The method for obtaining each preference value in the user preference value is as follows:

[0191] ① Conversation style preference:

[0192] Based on the feedback data, extract user characteristics such as language style, vocabulary selection, sentence length and conversation rhythm during the interaction process;

[0193] Use NLP technology to perform semantic and style analysis on the language patterns of user characteristics, identify whether the user prefers conversation styles such as humorous, formal, and concise, and obtain conversation style preferences including feature labels or preference levels of user conversation styles (such as preference for humorous style, preference for concise conversation).

[0194] ②Emotional and emotional preferences:

[0195] Using emotion recognition algorithms (such as convolutional neural networks (CNN), support vector machines (SVM), etc.), we extract the user's emotional characteristics (including positive, negative, neutral, etc. emotional tendencies) from the voice, text, and expressions of the feedback data, and obtain the user's emotions and emotional preferences including emotional vectors (such as happiness, sadness, anger, etc.).

[0196] ③Speech speed and tone preference:

[0197] Through speech signal processing technology, the user's preferences for the speaking speed, timbre and volume of the interactive object in the feedback data are identified, and the parameters of the speech output are adjusted in combination with emotion analysis to generate specific speaking speed and tone preferences (such as preference for steady tone and slower speaking speed).

[0198] ④ Non-verbal interaction preferences:

[0199] Based on the user's feedback and reaction data in the feedback data (such as facial expressions, eye tracking, etc.), we can identify their acceptance of body language and reaction preferences, such as whether they prefer more body movements, whether they like more facial expressions, and obtain their non-verbal interaction preferences.

[0200] Among them, the method of adjusting the adaptive learning direction of the initial psychological feature model according to the user preference value in the feedback data is:

[0201] ① Adjust speech output based on speech speed and intonation preferences:

[0202] According to the user's emotional state (such as positive, calm, anxious, etc.) and preference for speech speed, the speech speed and intonation parameters are dynamically adjusted using emotion-driven TTS technology, so that the digital human based on the psychological feature model can express different emotions. For example, in a pleasant situation, the speech speed is faster and the tone is brighter, and in a serious or soothing scene, the speech speed is slower and the tone is lower.

[0203] ② Adjust the words and expressions according to the conversation style preference:

[0204] Use the user's conversation style preferences (such as humor, formality, and conciseness) to adjust the language expression of the digital person corresponding to the psychological feature model. Use NLP technology and the Large Language Model (LLM) to adjust the wording style in combination with the user's preferences to generate personalized conversations for the digital person.

[0205] For example, if the user prefers a humorous style, the system will generate a more relaxed expression with humorous elements; if the user prefers a formal style, it will choose a more formal and rigorous language. During the real-time interaction process, the system will further refine the language style based on the user's immediate response to maintain the consistency and naturalness of the conversation.

[0206] ③ Dynamic adjustment of context:

[0207] During the conversation, the user's emotional changes are dynamically evaluated. If a significant change in the user's emotions is detected (such as from neutral to happy), the speaking speed and wording of the digital person corresponding to the psychological characteristic model will be adjusted accordingly to make it consistent with the user's current emotional and contextual needs.

[0208] In the psychological unit, model unit and feedback unit of this embodiment, the design of the collaborative module set enables each module to work independently and cooperate with each other to jointly build the initial psychological feature model. This modular design improves the flexibility and scalability of the system and facilitates subsequent maintenance and upgrades. By obtaining user feedback data on the initial psychological feature model and making weighted adjustments based on user preference values, the system can continuously learn and optimize the psychological feature model. This adaptive learning capability enables the system to continuously track user changes and provide more accurate services;

[0209] In addition, the use of adaptive learning technology enables the system to continuously optimize the psychological characteristic model. As the number of interactions increases, the system can gradually adapt to the user's personalized needs and emotional changes. This adaptive mechanism helps to improve the user's long-term satisfaction and the continuity of the digital human interaction experience.

[0210] In one embodiment, the adjustment module 30 includes a dialogue unit, a preference unit, a content unit, a voice unit, a component unit, a relationship unit, a recognition unit, an analysis unit, an expression sub-unit, a posture sub-unit, an action sub-unit and a feedback unit; wherein the dialogue unit, the preference unit and the content unit are processes for generating dialogue content according to the psychological characteristic model, the voice unit is a process for generating the dialogue voice of the digital human according to the dialogue content, the component unit, the relationship unit, the recognition unit and the analysis unit are processes for generating multimodal emotion analysis results, the expression sub-unit, the posture sub-unit and the action sub-unit are processes for generating the gestures and body postures of the digital human according to the multimodal emotion analysis results and the user's psychological characteristics, and the feedback unit is a process for continuously optimizing the digital human according to the feedback data, specifically:

[0211] Among them, the dialogue unit is used for dialogue generation and personalized tuning module based on psychological feature model. It uses large language model to control the randomness and diversity of dialogue generation and generate initial dialogue content that conforms to the user's psychological characteristics. It also performs semantic consistency check on the initial dialogue content to ensure that the dialogue remains coherent while being personalized.

[0212] The preference unit is used to perform preference recognition and extraction on multimodal data in the dimensions of voice, action and emotion to obtain the user's preference data;

[0213] The content unit is used to dynamically adjust the expression and context of the initial conversation content according to the preference data to obtain the conversation content; and dynamically adjust the style of the conversation content according to the real-time feedback from the user.

[0214] The dialogue unit, preference unit and content unit of this embodiment can more accurately grasp the user's emotional state by identifying the user's preferences in the dimensions of voice, action and emotion, and adjust the dialogue method accordingly, so that the dialogue is more in line with the natural communication mode of humans and the authenticity of the dialogue is improved.

[0215] The speech unit is used to perform word segmentation, stop word removal and morphological restoration on the multimodal data based on the speech synthesis and adjustment module to obtain emotional words and sentences containing key emotional words and phrases;

[0216] The speech unit is also used to identify the emotional expression and potential emotional tendency in emotional words and sentences by matching emotional words and sentences with the emotional dictionary, and use machine learning models (such as convolutional neural network CNN or long short-term memory network LSTM) to analyze the context and emotional pattern of the text to obtain the emotional analysis results including emotional classification and emotional intensity; among them, the emotional dictionary is a vocabulary library containing positive and negative emotional words, and the emotional classification includes positive, negative and neutral categories;

[0217] The speech unit is also used to optimize the sentiment classification results based on the user's real-time feedback, and gradually adapt the sentiment classification results to the user's personalized emotional response through reinforcement learning;

[0218] The speech unit is also used to adjust the tone, speaking speed and emotional expression of the conversation content according to the results of emotion analysis based on speech synthesis technology (such as TTS technology) to generate the conversational speech of the digital person; and when the speech is output, the speaking speed, timbre and emotional intensity of the conversational speech are adjusted through real-time user emotional feedback to make it more natural.

[0219] The specific method of adjusting the conversation content according to the sentiment analysis results is as follows:

[0220] ① Adjust the tone of voice according to the type of emotion:

[0221] According to the emotion classification (such as positive, negative and neutral), different tones are set for voice output. For example, when positive emotions are recognized, the system increases the brightness and liveliness of the tone; when the emotion is negative, the tone becomes low and gentle.

[0222] ② Adjust the speaking speed according to the intensity of emotions:

[0223] When you are in a state of high intensity (such as excitement or anger), increase your speaking speed to convey the urgency of the emotion; when you are in a state of calm or soothing emotion, slow down your speaking speed to create a stable atmosphere.

[0224] ③Dynamically adjust volume and timbre based on emotional expression needs:

[0225] When you are in a happy or excited mood, slightly increase the volume to make the tone brighter; when you are in a calm or negative mood, reduce the volume to make the tone softer.

[0226] ④ Combine multimodal feedback to fine-tune emotional expression in real time:

[0227] Use the user's real-time feedback (such as facial expressions and tone responses) to fine-tune the tone, speed, and emotional intensity of the voice. For example, if the system detects that the user's emotions are fluctuating, the digital human's emotional expression can be strengthened or weakened accordingly.

[0228] ⑤ Emotion-driven TTS technology:

[0229] Based on the results of emotion analysis and preset emotion parameters (such as speaking speed, pitch, and emotion intensity), speech synthesis technology (such as TTS technology) is used to automatically generate emotional speech.

[0230] To apply the embodiments of this application, please refer to Figure 3 , Figure 3 It is a flow chart of dialogue generation and speech synthesis provided in an embodiment of the present application, which shows the general process of text processing and speech output.

[0231] The speech unit of this embodiment is combined with speech synthesis technology, and the system can generate high-quality speech output, further enhancing the immersiveness of the conversation, so that the user can feel a more realistic interactive experience when talking with the digital person; and the removal of stop words and word form restoration processing can simplify the text data, remove meaningless words and word form changes, and make the emotional words and sentences focus more on the core emotional expression. The emotional dictionary is a database containing a large number of emotional words and their emotional tendencies. By matching emotional words and sentences with the emotional dictionary, the emotional expression in the text can be accurately identified. This matching method avoids the errors that may be caused by emotional analysis based on rules or statistical methods, and improves the accuracy of emotional analysis.

[0232] The component unit is used to perform word segmentation, part-of-speech tagging and syntactic analysis on the dialogue speech and its content based on the digital human animation and rendering module to obtain key sentence components; wherein the key sentence components include subject, predicate and object, etc.;

[0233] Relation units are used to capture contextual information across sentences in the conversational speech and its content through contextual understanding models (such as Transformer or RNN architectures) to obtain contextual relations;

[0234] The recognition unit is used to identify emotional words and phrases (such as "happy" and "disappointed") in the conversation speech and its content based on semantic analysis technology through the emotional dictionary and the pre-trained emotional recognition model to obtain semantic understanding results;

[0235] The analysis unit is used to perform emotion classification and emotion intensity calculation on the dialogue speech and its content according to key sentence components, contextual relationships and semantic understanding results, using an emotion classification model (such as a convolutional neural network (CNN) or a support vector machine (SVM)), to obtain multimodal emotion information (such as text, speech and expression) including emotion classification and emotion intensity; and obtain multimodal emotion analysis results by fusing multimodal emotion information; wherein the multimodal emotion analysis results include positive emotions, negative emotions, neutral emotions and their intensities.

[0236] The component unit, relationship unit, recognition unit and analysis unit of this embodiment can accurately identify the keywords and sentence components in the conversation through word segmentation and part-of-speech tagging, so as to more accurately understand the semantics of the conversation; capturing cross-sentence context information can grasp the overall process and logical relationship of the conversation, avoiding the understanding of a single sentence in isolation; by identifying emotional words and phrases, the emotional color and intensity expressed in the conversation can be deeply understood;

[0237] In addition, by combining multimodal analysis of user psychological characteristics, this embodiment can accurately extract the user's personalized needs, which helps to automatically adjust the digital human's dialogue, voice style and animation performance in the future, making the interaction more personalized and natural.

[0238] The expression subunit is used to bind the expression data corresponding to different emotions (such as smile, frown and surprise, etc.) to the facial control points of the digital human through 3D modeling technology based on the multimodal emotion analysis results and the user's psychological characteristics, and convert the emotion data in the multimodal emotion analysis results into the facial expressions of the digital human in real time through Blendshape or Morph Target technology. For example, the "happy" emotion is mapped to the facial posture of smiling and relaxing eyes; while the "angry" emotion is mapped to expressions such as closed lips and frowning.

[0239] The gesture subunit is used to generate corresponding gestures and body postures for different emotions and semantics using animation-driven technology based on the emotion intensity and conversation content in the multimodal emotion analysis results;

[0240] The action subunit is used to bind gestures and body postures to the digital human's skeletal structure through bone binding technology, so that each bone node (such as shoulders, elbows and knees) is associated with a control weight, so that each part can move in a coordinated manner, thereby generating the digital human's limb movements and adjusting the digital human's limb movements based on user preference dynamics and preset action libraries.

[0241] The overall performance of the digital human is generated synchronously by combining voice, text and expression data to ensure that the facial, gesture and body posture are consistent in emotional expression; and a gradual transition is used when the emotion changes, so that the changes in expression, gesture and posture are smooth and continuous, avoiding abruptness. For example, when expressing a happy emotion, the voice tone, facial smile and open gestures will change in coordination to enhance the emotional authenticity and expressiveness of the digital human; when changing from a smile to a serious expression, the facial and body postures are gradually adjusted to present a smoother visual effect.

[0242] The gestures and body postures generated by the expression subunit, posture subunit, and action subunit of this embodiment according to the intensity of emotion and the content of the conversation make the body movements of the digital human more natural and smooth; these movements are coordinated with the facial expressions, and together constitute a complete and real emotional expression system. In addition, by binding the expression data and body movements to the facial control points and bone structure of the digital human through 3D modeling technology, highly realistic emotional expression and body movements can be achieved;

[0243] In addition, the use of dynamic emotion-driven methods can make the digital human's emotional expression richer and more natural. The system can adjust the tone, expression and posture according to the user's psychological state in real time, so that the digital human can show emotional reactions that are consistent with the content and context of the conversation. This dynamic adjustment mechanism significantly improves the authenticity of the interaction and the user's immersion.

[0244] The feedback unit is used to collect the user's interactive feedback data on the digital human (such as emotional response, dialogue satisfaction score, etc.) based on the digital human animation and rendering module; and after each interaction, the digital human's body, expression and voice corresponding to the psychological feature model are updated according to the interactive feedback data to optimize the expression form for specific users;

[0245] The feedback unit is also used to adopt reinforcement learning technology to continuously improve the psychological characteristic model and the adaptability of the digital human to user responses.

[0246] To apply the embodiments of this application, please refer to Figure 4 , Figure 4 This is a digital human behavior generation flow chart provided in an embodiment of the present application, which shows the entire process from dialogue content to animation rendering.

[0247] To apply the embodiments of this application, please refer to Figure 5 , Figure 5 It is a system architecture diagram provided in an embodiment of the present application, showing the various modules in the psychological characteristic model and their relationships.

[0248] Overall, this embodiment has the following beneficial effects:

[0249] This application can fully reflect the user's interactive behavior and style preferences by acquiring multimodal data when the user interacts with the digital human. Based on multimodal data, a psychological characteristic model is established through adaptive learning, which can accurately capture the user's personalized needs and fully consider the dynamic characteristics of the user's emotions. Generating the digital human's conversational voice based on the psychological characteristic model can ensure the personalization, emotionality and natural fluency of the voice, making the digital human's voice more in line with the user's expectations and preferences, and improving the intimacy and credibility of the interaction. Generating matching digital human facial expressions and body movements based on the conversational voice and the user's psychological characteristics can further enhance the realism and immersion of the interaction. Therefore, the personalized shaping of the digital human can be achieved in terms of voice and movement, and the adaptive behavior of the digital human can be fully optimized according to the user's psychological characteristics and interaction preferences, so that the user's interactive experience is natural and not stiff;

[0250] This application provides a variety of technical innovations such as personalized tuning, multimodal fusion and adaptive learning, which reduces labor costs economically, improves customer stickiness, and promotes the popularization and application of digital human technology and the availability of mental health support services in society.

[0251] Embodiment three:

[0252] The embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for personalized shaping of a digital human based on user psychology;

[0253] Wherein, the method for personalized shaping of digital human based on user psychology, if implemented in the form of a software functional unit and used as an independent product, can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0254] The above are preferred embodiments of the present invention. It should be noted that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for personalized shaping of digital human based on user psychology, characterized in that: include: Acquire multimodal data when users interact with digital humans; Based on the multimodal data, and in accordance with the user's style preference, a psychological feature model is established through adaptive learning; The conversational speech of the digital human is generated according to the psychological characteristic model, and the facial expressions and body movements of the digital human matching the conversational speech and the psychological characteristics of the user are generated according to the conversational speech.

2. A method for personalized shaping of digital human based on user psychology as claimed in claim 1, characterized in that: Based on the multimodal data and in accordance with the user's style preference, a psychological feature model is established through adaptive learning, specifically: Preprocessing the multimodal data to obtain basic data; Extracting the user's emotion vector from the basic data through a preset neural network or emotion recognition algorithm to obtain the user's psychological characteristics; Based on the user psychological characteristics, an initial psychological characteristic model is established through a collaborative module set; wherein the collaborative module set includes a user psychological characteristic analysis module, a dialogue generation and personalized tuning module, a speech synthesis and adjustment module, a digital human animation and rendering module, and a feedback and adaptive learning module; The user's feedback data on the initial psychological characteristic model is obtained, and according to the user preference value in the feedback data, the adaptive learning direction of the initial psychological characteristic model is directionally adjusted in a weighted manner to obtain the psychological characteristic model.

3. A method for personalized shaping of digital human based on user psychology as claimed in claim 2, characterized in that: The multimodal data is preprocessed to obtain basic data, specifically: Extracting acoustic features, communication features, and behavioral features from the speech data, text data, and visual data of the multimodal data, respectively; Performing multimodal data fusion processing on the acoustic features, the communication features and the behavioral features to obtain comprehensive feature data; The comprehensive feature data is subjected to denoising and standardization processing to obtain the basic data.

4. A method for personalized shaping of digital human based on user psychology as claimed in claim 1, characterized in that: Generating the conversational speech of the digital human according to the psychological characteristic model is specifically as follows: Based on the psychological feature model, initial dialogue content is obtained by controlling the randomness and diversity of dialogue generation; Performing preference recognition and extraction on the multimodal data in the dimensions of voice, action and emotion to obtain user preference data; The expression and context of the initial conversation content are dynamically adjusted according to the preference data, and based on speech synthesis technology, the tone and speed of the initial conversation content are adjusted according to the preset emotion analysis results to generate the conversation voice of the digital human.

5. A method for personalized shaping of digital human based on user psychology as claimed in claim 4, characterized in that: The sentiment analysis results are specifically: Performing stop word removal and lemmatization processing on the multimodal data to obtain sentiment words and sentences; By matching the emotional words and sentences with the emotional dictionary, the emotional expressions and potential emotional tendencies in the emotional words and sentences are identified to obtain the emotional analysis results including emotional classification; wherein the emotional dictionary is a vocabulary library including emotional words of different categories.

6. A method for personalized shaping of digital human based on user psychology as claimed in claim 1, characterized in that: And generate matching digital human facial expressions and body movements according to the conversation voice and the psychological characteristics of the user, specifically: Performing word segmentation and part-of-speech tagging on the conversation speech to obtain key sentence components; Capturing context information across sentences in the conversation speech to obtain contextual relationships; Identify emotional words and phrases in the conversation speech to obtain semantic understanding results; According to the key sentence components, the contextual relationship and the semantic understanding result, the emotion classification and emotion intensity calculation are performed on the conversation speech to obtain a multimodal emotion analysis result; Based on the multimodal emotion analysis results and the psychological characteristics of the user, the facial expressions and body movements that match the user are generated on the digital human through 3D modeling technology.

7. A method for personalized shaping of digital human based on user psychology as claimed in claim 6, characterized in that: Based on the multimodal emotion analysis results and the psychological characteristics of the user, the facial expressions and body movements that match the user are generated on the digital human through 3D modeling technology, specifically: Based on the multimodal emotion analysis results and the psychological characteristics of the user, the expression data corresponding to different emotions are bound to the facial control points of the digital human through 3D modeling technology, and the emotion data in the multimodal emotion analysis results are converted into the facial expressions of the digital human; Generate corresponding gestures and body postures for different emotions and semantics according to the emotion intensity and conversation content in the multimodal emotion analysis results; The gestures and body postures are bound to the skeletal structure of the digital human to generate the body movements of the digital human.

8. A device for shaping digital human personality based on user psychology, characterized in that: Includes data module, model module and adjustment module; Wherein, the data module is used to obtain multimodal data when the user interacts with the digital human; The model module is used to establish a psychological feature model through adaptive learning based on the multimodal data and according to the style preference of the user; The adjustment module is used to generate the conversational speech of the digital human according to the psychological characteristic model, and to generate the facial expressions and body movements of the digital human matching the conversational speech and the psychological characteristics of the user.

9. The device for personalized shaping of digital human based on user psychology as claimed in claim 8, characterized in that: The model module includes a data unit, a psychological unit, a model unit and a feedback unit; Wherein, the data unit is used to pre-process the multimodal data to obtain basic data; The psychological unit is used to extract the user's emotion vector from the basic data through a preset neural network or emotion recognition algorithm to obtain the user's psychological characteristics; The model unit is used to establish an initial psychological feature model based on the user psychological features through a collaborative module set; wherein the collaborative module set includes a user psychological feature analysis module, a dialogue generation and personalized tuning module, a speech synthesis and adjustment module, a digital human animation and rendering module, and a feedback and adaptive learning module; The feedback unit is used to obtain the user's feedback data on the initial psychological characteristic model, and according to the user preference value in the feedback data, directionally adjust the adaptive learning direction of the initial psychological characteristic model in a weighted manner to obtain the psychological characteristic model.

10. A storage medium, characterized in that: The storage medium stores a computer program, which is called and executed by a computer to implement a method for personalized shaping of a digital human based on user psychology as described in any one of claims 1 to 7 above.

Images