Intelligent method and system for social interaction and emotional feedback of virtual objects

Abstract

The present invention relates to the field of data processing technology, and more specifically to an intelligent method and system for social interaction and emotional feedback of virtual objects, including: determining the image, sound and personality characteristics of a target object, defining a virtual object, extracting first facial data and first timbre data respectively, selecting a target video from a video library based on the personality characteristics of the target object, synthesizing the first facial data to the corresponding part of the target video, selecting the first personality data from the personality library, obtaining the first speech content from a dialogue template, and generating the voice data of the virtual object, integrating the action basis and the voice data of the virtual object, and generating a video of the virtual object speaking; obtaining the attribute information and the current emotional state of the target user based on the first shooting data, generating the emotional symbol of the current emotional state of the target user, and updating the video of the virtual object speaking. The social interaction ability of the virtual object and the accuracy of the emotional feedback are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically to an intelligent method and system for social interaction and emotional feedback of virtual objects. Background Art

[0002] With the rapid development of information technology, virtual objects play an increasingly important role in social interaction and emotional expression. Similar prior art includes a Chinese patent application with publication number CN118136056A, which discloses a bionic audio system and its control method, including: a sound capture module, an emotion analysis module, a bionic audio processing module, and a sound feedback module, which are wirelessly connected in sequence. The sound capture module is used to capture the user's vocal signal through a microphone; the analysis module is used to analyze the emotional characteristics in the vocal signal to determine the current emotional state; the bionic audio processing module is used to process and optimize the vocal signal according to the emotional state. Similar prior art also includes a Chinese patent application with publication number CN117880566A, which discloses a digital human live broadcast interaction method and system based on artificial intelligence, including the following steps: based on the audience text and voice input in the live broadcast, voice recognition is used for data conversion, and the sentiment analysis tool library is used to classify emotions using the support vector machine algorithm to generate an emotion classification data set.

[0003] However, the above application documents all ignore important non-verbal communication elements such as vision. In real social interactions, emotional expression is often multimodal, and single audio processing may not be sufficient to fully reflect the user's emotional state. Therefore, the present invention provides an intelligent method and system for virtual object social interaction and emotional feedback, aiming to improve the social interaction ability of virtual objects and the accuracy of emotional feedback, and provide users with a richer and more realistic interactive experience. Summary of the invention

[0004] The present invention defines a virtual object by determining the image, sound and personality characteristics of the target object, extracts the first facial data and the first timbre data respectively, selects the target video from the video library based on the personality characteristics of the target object, synthesizes the first facial data to the corresponding part of the target video, selects the first personality data from the personality library, obtains the first speech content from the dialogue template, generates the voice data of the virtual object, integrates the action basis and the voice data of the virtual object, generates the video of the virtual object speaking; obtains the attribute information and the current emotional state of the target user based on the first shooting data, generates the emotional symbol of the current emotional state of the target user, and updates the video of the virtual object speaking. The social interaction ability of the virtual object and the accuracy of the emotional feedback are improved.

[0005] In order to achieve the above object, the present invention provides an intelligent method for social interaction and emotional feedback of virtual objects, which is implemented through the following steps:

[0006] Step S1: Determine the image, sound and personality characteristics of the target object;

[0007] Step S2: defining a virtual object, extracting first facial data from the image of the target object, extracting first timbre data from the sound, selecting video data with the greatest similarity to the personality characteristics of the target object from a video library as a target video based on the personality characteristics of the target object, synthesizing the first facial data to a corresponding part of the target video, and using the target video as an action basis of the virtual object, wherein the video library stores multiple types of video data;

[0008] Step S3: based on the personality characteristics of the target object, first personality data closest to the personality characteristics of the target object is selected from the personality library, based on the first personality data, first speech content is obtained from the dialogue template, and based on the first timbre data, speech data of the virtual object is generated by using speech synthesis technology, and the action basis of the virtual object and the speech data are integrated to generate a video of the virtual object speaking, wherein the speech data includes timbre, intonation, speech speed and emotional expression, and the video of the virtual object speaking is synchronized with the speech data, and the personality library stores personality data of multiple personality characteristics;

[0009] Step S4: acquiring attribute information of the target user based on the first shooting data, identifying the current emotional state of the target user based on the first voice data input by the target user, and recording first emotional response data of the target user based on the attribute information of the target user and the current emotional state, wherein the first shooting data represents image information of the face or body of the target user captured when the virtual object interacts with the target user for the first time, and the first voice data represents voice information acquired through a voice input device when the virtual object interacts with the target user for the first time;

[0010] Step S5: Based on the first emotional response data of the target user, generate the emotional symbol of the current emotional state of the target user, combine the video of the virtual object speaking with the emotional symbol, update the video of the virtual object speaking, and display it on the terminal device of the target user to interact with the target user, and at the same time store the first emotional response data of the target user and the video of the virtual object speaking.

[0011] As a preferred technical solution of the present invention, after the above step S5, the method further includes:

[0012] Acquire second shooting data from the target user, and match the second shooting data with the emotional symbol to generate second emotional response data of the target user; based on the second emotional response data, record the feedback result of the target user on the video of speaking to the virtual object; based on the feedback result, correct the personality characteristics of the virtual object; and also based on the second emotional response data, optimize the emotional symbol.

[0013] As a preferred technical solution of the present invention, the step S3 of selecting the first personality data closest to the personality characteristics of the target object from the personality library includes:

[0014] Based on psychological theory, a plurality of personality dimensions are defined for each personality data, and a parameter is defined for each personality dimension, and a weight is assigned to each personality dimension based on the importance of the personality dimension in the personality data to form the personality library, wherein the parameter represents the numerical representation of the personality dimension;

[0015] A plurality of first personality dimensions and parameters of each of the first personality dimensions are extracted from the personality characteristics of the target object, and first personality data having the highest similarity to the personality characteristics of the target object is searched in the personality database.

[0016] As a preferred technical solution of the present invention, searching the personality database for first personality data having the highest similarity to the personality characteristics of the target object also includes:

[0017] Based on the parameters of the personality dimension and the weight, the personality characteristic value of the target object is calculated, the personality characteristic value is compared with the value of each personality data in the personality library, and the personality data with the smallest difference between the personality data value and the personality characteristic value is obtained as the first personality data.

[0018] As a preferred technical solution of the present invention, based on the attribute information of the target user and the current emotional state, recording the first emotional response data of the target user includes:

[0019] A series of emotion symbols and a plurality of emotion levels corresponding to each emotion symbol are preset, wherein the emotion levels are divided into primary, secondary and tertiary levels, which are used to indicate the intensity of the relevant emotion under the corresponding emotion symbol;

[0020] Acquire attribute information of a plurality of first users, and divide all first users into a plurality of user categories based on the attribute information of the first users, select an emotion symbol corresponding to the attribute information of the first user from the emotion symbols as an alternative item, display the alternative item to each of the first users, photograph the facial expression of the first user, and use the matching result of the facial expression of the first user and the alternative item as the emotion measurement result of the first user, and convert the emotion measurement result into a numerical emotional response;

[0021] Summarizing the emotion measurement results of the first user in each of the user categories, and calculating the standard deviation of the emotion measurement results of each of the emotion symbols, and using the standard deviation as the dispersion index of the emotion symbol, wherein each of the user categories has the same attribute information;

[0022] Based on the dispersion index of the emotion symbol, a subcategory is created for each of the user categories, and when the dispersion index of a certain emotion symbol in the user category with the same attribute is less than a first threshold, the emotion symbol is stored in the subcategory;

[0023] Acquire the attribute information of the target user, and based on the attribute information of the target user, select the subcategory of the user category corresponding to the target user, and display the emotional symbols in the subcategory to the target user, and based on the current emotional state of the target user, record the first emotional response data of the target user, wherein the emotional response data is represented by a corresponding numerical value.

[0024] As a preferred technical solution of the present invention, after storing the emotional symbol in the subcategory, the method further includes:

[0025] When the dispersion index of a certain emotional symbol in the user category with the same attribute is greater than or equal to the first threshold, storing other emotional symbols outside the subcategory in the user category into other subcategories;

[0026] Combined with the stored emotion record of the target user and the emotion symbols in the feedback of the virtual object, the emotion symbols simultaneously in the emotion record of the target user are obtained from the other subcategories and stored in the subcategory.

[0027] As a preferred technical solution of the present invention, the feedback result also includes:

[0028] The feedback result of the target user speaking to the video of the virtual object is called a second speech content, and keywords are extracted from the second speech content, wherein the keywords also include phrases, adjectives, adverbs or specific nouns expressing emotions;

[0029] Based on a preset emotion dictionary and the expression data of the target user, the emotion symbol and the emotion level corresponding to the emotion symbol are allocated to the keyword.

[0030] The present invention also provides an intelligent system for social interaction and emotional feedback of virtual objects as described above, comprising the following modules:

[0031] A determination unit, used to determine the image, sound and personality characteristics of the target object;

[0032] The configuration unit is configured to: define a virtual object, extract first facial data from the image of the target object, extract first timbre data from the sound, select video data with the greatest similarity to the personality characteristics of the target object from a video library as a target video based on the personality characteristics of the target object, synthesize the first facial data into a corresponding part of the target video, and use the target video as an action basis of the virtual object, wherein the video library stores multiple types of video data;

[0033] Based on the personality characteristics of the target object, first personality data closest to the personality characteristics of the target object is selected from a personality library, based on the first personality data, a first speech content is obtained from a dialogue template, and based on the first timbre data, speech data of the virtual object is generated by using speech synthesis technology, and the action basis of the virtual object and the speech data are integrated to generate a video of the virtual object speaking, wherein the speech data includes timbre, intonation, speech speed and emotional expression, and the video of the virtual object speaking is synchronized with the speech data, and the personality library stores personality data of multiple personality characteristics;

[0034] a control unit, configured to obtain attribute information of the target user based on first shooting data, identify a current emotional state of the target user based on first voice data input by the target user, and record first emotional response data of the target user based on the attribute information of the target user and the current emotional state, wherein the first shooting data represents image information of the face or body of the target user captured when the virtual object interacts with the target user for the first time, and the first voice data represents voice information acquired through a voice input device when the virtual object interacts with the target user for the first time;

[0035] An updating unit is used to generate the emotional symbol of the current emotional state of the target user, combine the video of the virtual object speaking with the emotional symbol, update the video of the virtual object speaking, and display it on the terminal device of the target user to interact with the target user, and at the same time store the first emotional response data of the target user and the video of the virtual object speaking.

[0036] Compared with the prior art, the beneficial effects of the present invention are at least as follows:

[0037] 1. The technical solution of the present invention can provide customized services by determining the image, sound and personality characteristics of the target object, which is helpful to create a more realistic and attractive virtual character. Define a virtual object, extract first facial data from the image of the target object, extract first timbre data from the sound, select video data with the greatest similarity to the personality characteristics of the target object from the video library based on the personality characteristics of the target object as the target video, synthesize the first facial data into the corresponding part of the target video, and use the target video as the action basis of the virtual object, select first personality data closest to the personality characteristics of the target object from the personality library based on the personality characteristics of the target object, obtain first speech content from the dialogue template based on the first personality data, and generate voice data of the virtual object based on the first timbre data by using speech synthesis technology, integrate the action basis and voice data of the virtual object, generate a video of the virtual object speaking, and communicate with the user in a realistic way, improve the social interaction ability and accuracy of emotional feedback of the virtual object, and provide users with a richer and more realistic interactive experience.

[0038] 2. The technical solution of the present invention can timely capture the target user's emotions and timely adjust the expression of the virtual object by acquiring the target user's attribute information based on the first shooting data, identifying the target user's current emotional state based on the first voice data input by the target user, and recording the target user's first emotional response data based on the target user's attribute information and current emotional state. Based on the target user's first emotional response data, an emotional symbol of the target user's current emotional state is generated, the video of the virtual object speaking is combined with the emotional symbol, the video of the virtual object speaking is updated, and displayed on the target user's terminal device to interact with the target user, and the first emotional response data of the target user and the video of the virtual object speaking are stored at the same time. By combining the emotional symbol and the video of the virtual object, feedback matching the user's emotional state is provided, and the real-time application of emotional data is realized. Through the coordination between the above steps, the virtual object can not only interact with the user in a realistic way, but also reflect the user's emotional state in real time, improve the social interaction ability of the virtual object and the accuracy of emotional feedback, and provide users with a richer and more realistic interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0040] Figure 1 A flowchart of the steps of the intelligent method for social interaction and emotional feedback of virtual objects in the present invention;

[0041] Figure 2 This is a structural diagram of the composition of the intelligent system used for social interaction and emotional feedback of virtual objects in the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0043] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0044] With the rapid development of information technology, virtual objects play an increasingly important role in social interaction and emotional expression. However, in existing technologies, most technical solutions ignore important non-verbal communication elements such as vision. In real social interactions, emotional expression is often multimodal, and single audio processing may not be sufficient to fully reflect the user's emotional state.

[0045] In view of the above technical problems, the present invention proposes Figure 1 The intelligent method for virtual object social interaction and emotional feedback shown includes the following steps:

[0046] Step S1: Determine the image, sound and personality characteristics of the target object.

[0047] Specifically, in virtual social interactions, users expect to communicate with virtual objects that are as close to real people as possible. At the same time, different users may have different expectations and needs for virtual objects, such as hoping that the virtual object is their idol. The target object represents the virtual object expected by the user, and the user represents the object that directly interacts with the virtual object. By determining the personalized characteristics of the target object, customized services can be provided. Each virtual object can be optimized according to the characteristics of its target user group, which helps to create more realistic and attractive virtual characters. The accurate capture and expression of personality characteristics can help establish an emotional connection between the virtual object and the user, and increase the user's trust and dependence on the virtual object. When the image, sound and personality characteristics of the virtual object match the user's expectations, the virtual object can be more naturally integrated into the user's social environment, providing a smoother and more pleasant interactive experience.

[0048] Step S2: define a virtual object, extract first facial data from the image of the target object, extract first timbre data from the sound, select video data with the greatest similarity to the personality characteristics of the target object from the video library as the target video based on the personality characteristics of the target object, synthesize the first facial data into the corresponding part of the target video, and use the target video as the action basis of the virtual object, wherein the video library stores various types of video data.

[0049] Specifically, to solve the problem of multimodal data fusion, the facial data and timbre data of the target object are extracted, combined with personality characteristics, and multiple sensory data such as vision and hearing are integrated to reflect the personality and behavioral characteristics of the target object. By selecting video data similar to the personality characteristics, the emotional expression of the virtual object is enriched to make it more vivid and appealing. For example, if the target object is determined to be an excellent customer service representative, and a virtual customer service is created using her as an example, the image, voice and personality evaluation data of the customer service representative are first collected, and facial features such as the shape and position of the eyes and mouth are extracted from the image of the customer service representative. Then, the timbre features are extracted from the voice recording of the customer service representative to ensure that the voice of the virtual object is similar to that of the customer service representative. According to the personality evaluation data, it is found that the customer service representative has the personality characteristics of patience and friendliness. Then, in the video library, those videos with stable voice in pitch and speed are identified and selected. These videos show gentle facial expressions and soothing gestures. Then, the extracted facial data is synthesized into the corresponding parts of the selected video to ensure that the facial expressions are synchronized with the video content. The actions in the synthesized video are used as the basis for the actions of the virtual customer service representative, so that when interacting with customers, the virtual object can show natural and appropriate body language. This enables communication with users in a realistic way, improves the social interaction ability and accuracy of emotional feedback of virtual objects, and provides users with a richer and more realistic interactive experience.

[0050] Step S3: Based on the personality characteristics of the target object, select the first personality data closest to the personality characteristics of the target object from the personality library, obtain the first speech content from the dialogue template based on the first personality data, and generate the voice data of the virtual object using speech synthesis technology based on the first timbre data, integrate the action basis and voice data of the virtual object together, and generate a video of the virtual object speaking, wherein the voice data includes timbre, intonation, speaking speed and emotional expression, and the video of the virtual object speaking is synchronized with the voice data, and the personality library stores personality data of multiple personality characteristics.

[0051] Specifically, in order to make the virtual object provide a more accurate expression effect in the interaction, by selecting the personality data closest to the personality characteristics of the target object from the personality library, the virtual object shows personality characteristics similar to the target object. The combination with natural language enables the virtual object to have a more realistic and natural social interaction. At the same time, the voice data generated by speech synthesis technology contains emotional expression, so that the virtual object can establish an emotional connection with the user. The virtual object integrating action basis and voice data provides a highly personalized user experience. For example, to create a virtual teacher to teach children science courses, first select the personality data closest to the personality characteristics of the target teacher in the personality library, such as "enthusiasm" and "patience". Based on the selected personality data, obtain the child-friendly and inspiring teaching language content from the dialogue template. Then use speech synthesis technology to generate the voice of the virtual teacher according to the extracted timbre data, including appropriate intonation, speaking speed and emotional expression. The action basis and voice data are integrated to generate a video of the virtual teacher's lecture, and the lip shape in the video is synchronized with the voice, and the speaking speed and emotional expression match the teaching content. Through the above steps, this example can create a virtual teacher who has both professional knowledge and can teach in a way that children like. The virtual teacher can also interact with students in a realistic way, provide a high-quality learning experience, and stimulate students' curiosity and interest in learning. The social interaction ability and accuracy of emotional feedback of virtual objects are improved, providing users with a richer and more realistic interactive experience.

[0052] Step S4: Acquire attribute information of the target user based on the first shooting data, identify the current emotional state of the target user based on the first voice data input by the target user, and record first emotional response data of the target user based on the attribute information and the current emotional state of the target user, wherein the first shooting data represents image information of the target user's face or body captured when the virtual object interacts with the target user for the first time, and the first voice data represents voice information acquired through a voice input device when the virtual object interacts with the target user for the first time.

[0053] Specifically, since the user's emotional state affects the social interaction effect with the virtual object, the emotional state of the target user is identified through the first voice data, the problem of real-time monitoring of the emotional state is solved, the target user's emotions can be captured in time, and the expression of the virtual object can be adjusted in time. Real-time identification of emotional state enables the virtual object to respond to the user's emotional needs more accurately, improve the naturalness and effectiveness of the interaction. At the same time, in order to better understand and serve the target user, it is necessary to record the emotional response of the target user. The above steps record the first emotional response data based on the user's attributes and emotional state, which helps to accumulate the user's emotional database and provide data support for subsequent emotional analysis and speaking video optimization. For example, when the virtual customer service interacts with the customer for the first time, the customer's attribute information, such as age, gender, etc., is obtained by shooting images; then the customer's voice information is obtained through the voice input device, and the customer's emotional state, such as anxiety, happiness or frustration, is identified using emotion recognition technology. Finally, combined with the customer's attribute information and emotional state, the customer's first emotional response data when interacting with the virtual customer service is recorded, including the tone of voice, speech speed and expressed emotions. Virtual objects can not only interact with users in a realistic way, but also monitor and record users' emotional states in real time, providing users with more accurate and personalized emotional support. Target users, customers, and users all refer to those who interact with virtual objects.

[0054] Step S5: Based on the first emotional response data of the target user, an emotional symbol of the target user's current emotional state is generated, the video of the virtual object speaking is combined with the emotional symbol, the video of the virtual object speaking is updated, and displayed on the target user's terminal device to interact with the target user, and the first emotional response data of the target user and the video of the virtual object speaking are stored at the same time.

[0055] Specifically, in social interactions, users expect to receive timely and personalized feedback. The above steps provide feedback that matches the user's emotional state by combining emotional symbols and videos of virtual objects, thus realizing the real-time application of emotional data. At the same time, the generation and application of emotional symbols enable virtual objects to communicate with users with richer emotions, and the combination of emotional symbol videos provides a more intuitive and personalized interactive experience, making users feel more understood and cared for. Storing emotional response data and videos of virtual objects speaking provides a basis for subsequent analysis of user emotional patterns and optimization of interaction strategies. For example, based on the user's emotional response data, emotional symbols representing the user's current emotional state are generated, such as smiley faces, sad faces or other emoticons, as well as emotional levels corresponding to emotional symbols, such as very weak, weak, medium, strong, very strong, etc. The updated video is displayed on the user's smart device to interact with the user.

[0056] Through the coordination of the above steps, virtual objects can not only interact with users in a realistic way, but also reflect the user's emotional state in real time, improve the social interaction ability of virtual objects and the accuracy of emotional feedback, and provide users with a richer and more realistic interactive experience.

[0057] Furthermore, after the above step S5, the following steps are further included:

[0058] Acquire second shooting data from the target user, and match the second shooting data with the emotional symbol to generate second emotional response data of the target user. Based on the second emotional response data, record the feedback result of the target user on the video of speaking to the virtual object. Based on the feedback result, correct the personality characteristics of the virtual object. Meanwhile, optimize the emotional symbol based on the second emotional response data.

[0059] Specifically, in social interaction, the user's emotional state may change as the interaction proceeds. Therefore, in the subsequent interaction between the user and the virtual object, the user's second shooting data is continued to be obtained by shooting images, and the second shooting data is obtained and matched with the emotional symbol to generate the user's second emotional response data, so as to solve the problem of continuously monitoring the user's emotional state. Through continuous monitoring and adjustment, the virtual object can better adapt to the user's emotional changes and provide a more natural and adaptable interactive experience. In order to make the virtual object more in line with the user's actual feelings and needs, it is necessary to dynamically adjust the character characteristics of the virtual object according to user feedback, and adjust the character characteristics of the virtual object by recording the feedback results of the user's speech video to the virtual object to make it more in line with the user's actual feelings. For example, if the user responds positively to the coach's encouragement, the frequency of use of encouraging language is increased. And by optimizing the emotional symbol, the problem of the mismatch between the emotional symbol and the user's emotional state is solved. The interaction of the virtual object is made more in line with the user's expectations and needs, thereby further improving the social interaction ability of the virtual object and the accuracy of emotional feedback, and providing users with a richer and more realistic interactive experience.

[0060] Furthermore, the step S3 above selects the first personality data closest to the personality characteristics of the target object from the personality library, including:

[0061] Based on psychological theories, multiple personality dimensions are defined for each personality data, and parameters are defined for each personality dimension. Weights are assigned to each personality dimension based on the importance of the personality dimension in the personality data to form a personality library, wherein the parameters represent the numerical representation of the personality dimension.

[0062] A plurality of first personality dimensions and parameters of each first personality dimension are extracted from the personality characteristics of the target object, and the first personality data having the highest similarity to the personality characteristics of the target object is searched in the personality database.

[0063] Specifically, five personality dimensions can be defined, namely, openness, responsibility, extroversion, agreeableness and neuroticism. For example, the parameter of responsibility is represented by a value ranging from 0 at the minimum to 100 at the maximum. A weight is also assigned to each personality dimension based on the scene or purpose of the interaction. For example, openness accounts for 15% of the entire personality characteristics, responsibility accounts for 15% of the entire personality characteristics, extroversion accounts for 10% of the entire personality characteristics, agreeableness accounts for 20% of the entire personality characteristics, and neuroticism accounts for 40% of the entire personality characteristics. By calculating the personality characteristic value of the target object and the value of the personality data in the personality library, the first personality data with the highest similarity to the personality characteristics of the target object can be determined, creating a more real, personalized virtual object that can trigger emotional resonance, thereby providing higher quality and accurate social interaction and emotional feedback.

[0064] Furthermore, the above-mentioned searching the personality database for the first personality data having the highest similarity to the personality characteristics of the target object further includes:

[0065] Based on the parameters and weights of the personality dimensions, the personality characteristic value of the target object is calculated, the personality characteristic value is compared with the value of each personality data in the personality library, and the personality data with the smallest difference between the personality data value and the personality characteristic value is obtained as the first personality data.

[0066] Specifically, the above steps describe a method based on parameterization and quantification to select the first personality data that is most similar to the personality characteristics of the target object. Personality characteristics are usually abstract and difficult to quantify. This step converts personality characteristics into quantifiable values ​​by using parameters and weights based on personality dimensions, which is convenient for comparison and calculation. There may be multiple personality data in the personality library. By calculating the personality characteristic values ​​of the target object and comparing them with each personality data in the library, the problem of how to accurately match the most similar personality data is solved. The user's expectations of the virtual object largely depend on its personality characteristics. By accurately matching the first personality data, the user's interaction experience with the virtual object can be optimized. Quantification and precise matching help to establish an emotional connection between the user and the virtual object, because the user may feel that the virtual object understands and cares about their needs and feelings more, providing a richer and more realistic interaction experience.

[0067] Furthermore, based on the attribute information and current emotional state of the target user, recording the first emotional response data of the target user includes:

[0068] A series of emotion symbols and multiple emotion levels corresponding to each emotion symbol are preset, wherein the emotion levels are divided into primary, secondary and tertiary levels, which are used to indicate the intensity of the related emotion under the corresponding emotion symbol;

[0069] Specifically, for example, the emotion symbols include icons representing happiness, sadness, anger, fear, surprise, etc. The emotion levels of the happiness icon include level one: very happy, level two: happy, and level three: moderately happy, represented by the values ​​3, 2, and 1 respectively.

[0070] Acquire attribute information of a plurality of first users, and based on the attribute information of the first users, classify all the first users into a plurality of user categories, select an emotion symbol corresponding to the attribute information of the first user from the emotion symbols as an alternative item, display the alternative item to each first user, photograph the facial expression of the first user, and use the matching result of the facial expression of the first user and the alternative item as the emotion measurement result of the first user, and convert the emotion measurement result into a numerical emotional response;

[0071] Summarizing the emotion measurement results of the first user in each user category, and calculating the standard deviation of the emotion measurement results of each emotion symbol, and using the standard deviation as the dispersion index of the emotion symbol, wherein each user category has the same attribute information;

[0072] Specifically, for example, if the attribute information of a user category is "male between 20-30 years old", the system recommends three emotional symbols for this user category: happy, sad, and angry. The three levels of happy emotions (3, 2, 1) are displayed to the user as options. The user chooses the level that best matches his or her current emotion based on his or her feelings. The choices of all users are collected and their facial expressions are recorded to confirm their choices. Assuming that most users choose "3" to indicate very happy, this choice is matched with the results of the user's facial expression analysis to confirm its accuracy. Then calculate the standard deviation of the emotion measurement results of all users who choose "3". If the standard deviation is small, it means that the cognition of this emotional symbol in this user group is relatively consistent.

[0073] Based on the dispersion index of the emotion symbol, a subcategory is created for each user category, and when the dispersion index of a certain emotion symbol in the user category with the same attribute is less than a first threshold, the emotion symbol is stored in the subcategory;

[0074] Acquire the attribute information of the target user, and based on the attribute information of the target user, select a subcategory of the user category corresponding to the target user, and display the emotional symbols in the subcategory to the target user, and based on the current emotional state of the target user, record the first emotional response data of the target user, wherein the emotional response data is represented by a corresponding numerical value.

[0075] Specifically, for example, all first users are first classified according to the gender in the attribute information, and then classified according to the age group in the attribute information, and all first users are divided into multiple user categories. Through the above steps, it can be determined which emotional symbols have lower emotion measurement result changes in a specific user group, thereby more accurately reflecting the user's emotional state, which helps to improve the accuracy and reliability of emotion records.

[0076] Furthermore, after storing the emotion symbol in the subcategory, the step further includes:

[0077] When the dispersion index of a certain emotional symbol in the user category with the same attribute is greater than or equal to a first threshold, other emotional symbols outside the subcategory in the user category are stored in other subcategories;

[0078] Combined with the stored emotion records of the target user and the emotion symbols in the feedback of the virtual object, the emotion symbols that are simultaneously in the emotion records of the target user are obtained from other subcategories and stored in the subcategory.

[0079] Specifically, in the process of collecting and applying emotional symbols, there may be some symbols that show high variability in specific user groups. When the dispersion index of emotional symbols in user categories exceeds a specific threshold, special processing or classification is required. In order to provide more personalized services, the classification and storage of emotional symbols can be optimized based on the user's specific emotional records and feedback to more accurately identify and respond to the user's emotional state. Through the above steps, not only can the user's emotional state be better understood and responded to, but also self-optimization and adjustment can be performed according to the user's specific needs and feedback, and virtual objects can more accurately reflect the user's emotional state.

[0080] Furthermore, the above feedback results also include:

[0081] The feedback result of the target user speaking to the video of the virtual object is called the second speech content, and keywords are extracted from the second speech content, wherein the keywords also include phrases, adjectives, adverbs or specific nouns expressing emotions;

[0082] Based on the preset emotional dictionary and the target user's expression data, emotional symbols and the corresponding emotional levels of the emotional symbols are assigned to the keywords.

[0083] Specifically, the content is combined with emotional symbols to more accurately reflect the user's emotional state.

[0084] The present invention also provides a Figure 2 The intelligent system for social interaction and emotional feedback of virtual objects shown includes the following modules:

[0085] A determination unit, used to determine the image, sound and personality characteristics of the target object;

[0086] The configuration unit is configured to: define a virtual object, extract first facial data from an image of a target object, extract first timbre data from a sound, respectively, select video data having the greatest similarity to the personality characteristics of the target object from a video library as a target video based on the personality characteristics of the target object, synthesize the first facial data to a corresponding part of the target video, and use the target video as an action basis of the virtual object, wherein the video library stores multiple types of video data;

[0087] Based on the personality characteristics of the target object, first personality data closest to the personality characteristics of the target object is selected from the personality library, first speech content is obtained from the dialogue template based on the first personality data, speech data of the virtual object is generated by using speech synthesis technology based on the first timbre data, and the action basis and speech data of the virtual object are integrated to generate a video of the virtual object speaking, wherein the speech data includes timbre, intonation, speech speed and emotional expression, and the video of the virtual object speaking is synchronized with the speech data, and the personality library stores personality data of multiple personality characteristics;

[0088] a control unit, configured to obtain attribute information of a target user based on first shooting data, identify a current emotional state of the target user based on first voice data input by the target user, and record first emotional response data of the target user based on the attribute information and the current emotional state of the target user, wherein the first shooting data represents image information of a face or body of the target user captured when the virtual object interacts with the target user for the first time, and the first voice data represents voice information acquired through a voice input device when the virtual object interacts with the target user for the first time;

[0089] An updating unit is used to generate an emotional symbol of the current emotional state of the target user, combine the video of the virtual object speaking with the emotional symbol, update the video of the virtual object speaking, and display it on the terminal device of the target user to interact with the target user, and at the same time store the first emotional response data of the target user and the video of the virtual object speaking.

[0090] In summary, the present invention can provide customized services by determining the image, sound and personality characteristics of the target object, which is helpful to create a more realistic and attractive virtual character. Define a virtual object, extract first facial data from the image of the target object, extract first timbre data from the sound, select video data with the greatest similarity to the personality characteristics of the target object from the video library based on the personality characteristics of the target object as the target video, synthesize the first facial data into the corresponding part of the target video, and use the target video as the action basis of the virtual object, select the first personality data closest to the personality characteristics of the target object from the personality library based on the personality characteristics of the target object, obtain the first speech content from the dialogue template based on the first personality data, and generate the voice data of the virtual object based on the first timbre data by using speech synthesis technology, integrate the action basis and voice data of the virtual object together, and generate a video of the virtual object speaking, which can communicate with the user in a realistic way, improve the social interaction ability and the accuracy of emotional feedback of the virtual object, and provide users with a richer and more realistic interactive experience.

[0091] Since the user's emotional state affects the social interaction effect with the virtual object, by acquiring the target user's attribute information based on the first shooting data, identifying the target user's current emotional state based on the first voice data input by the target user, and recording the target user's first emotional response data based on the target user's attribute information and the current emotional state, the target user's emotions can be captured in time and the expression of the virtual object can be adjusted in time. Based on the target user's first emotional response data, an emotional symbol of the target user's current emotional state is generated, the video of the virtual object speaking is combined with the emotional symbol, the video of the virtual object speaking is updated, and displayed on the target user's terminal device to interact with the target user, and the first emotional response data of the target user and the video of the virtual object speaking are stored at the same time. By combining the emotional symbol and the video of the virtual object, feedback matching the user's emotional state is provided, and the real-time application of emotional data is realized. Through the coordination between the above steps, the virtual object can not only interact with the user in a realistic way, but also reflect the user's emotional state in real time, improve the social interaction ability of the virtual object and the accuracy of emotional feedback, and provide users with a richer and more realistic interactive experience.

[0092] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0093] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the above-mentioned program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0094] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0096] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent method for social interaction and emotional feedback of virtual objects, characterized in that: The method comprises: Step S1: Determine the image, sound and personality characteristics of the target object; Step S2: defining a virtual object, extracting first facial data from the image of the target object, extracting first timbre data from the sound, selecting video data with the greatest similarity to the personality characteristics of the target object from a video library as a target video based on the personality characteristics of the target object, synthesizing the first facial data to a corresponding part of the target video, and using the target video as an action basis of the virtual object, wherein the video library stores multiple types of video data; Step S3: based on the personality characteristics of the target object, first personality data closest to the personality characteristics of the target object is selected from the personality library, based on the first personality data, first speech content is obtained from the dialogue template, and based on the first timbre data, speech data of the virtual object is generated by using speech synthesis technology, and the action basis of the virtual object and the speech data are integrated to generate a video of the virtual object speaking, wherein the speech data includes timbre, intonation, speech speed and emotional expression, and the video of the virtual object speaking is synchronized with the speech data, and the personality library stores personality data of multiple personality characteristics; Step S4: acquiring attribute information of the target user based on the first shooting data, identifying the current emotional state of the target user based on the first voice data input by the target user, and recording first emotional response data of the target user based on the attribute information of the target user and the current emotional state, wherein the first shooting data represents image information of the face or body of the target user captured when the virtual object interacts with the target user for the first time, and the first voice data represents voice information acquired through a voice input device when the virtual object interacts with the target user for the first time; Step S5: Based on the first emotional response data of the target user, an emotional symbol of the current emotional state of the target user is generated, the video of the virtual object speaking is combined with the emotional symbol, the video of the virtual object speaking is updated, and the video is displayed on the terminal device of the target user to interact with the target user, and the first emotional response data of the target user and the video of the virtual object speaking are stored at the same time; The step S3 of selecting the first personality data closest to the personality characteristics of the target object from the personality library includes: Based on psychological theory, a plurality of personality dimensions are defined for each personality data, and a parameter is defined for each personality dimension, and a weight is assigned to each personality dimension based on the importance of the personality dimension in the personality data to form the personality library, wherein the parameter represents the numerical representation of the personality dimension; A plurality of first personality dimensions and parameters of each of the first personality dimensions are extracted from the personality characteristics of the target object, and first personality data having the highest similarity to the personality characteristics of the target object is searched in the personality database.

2. The method according to claim 1, characterized in that After the above step S5, the method further includes: Acquire second shooting data from the target user, and match the second shooting data with the emotional symbol to generate second emotional response data of the target user; based on the second emotional response data, record the feedback result of the target user on the video of speaking to the virtual object; based on the feedback result, correct the personality characteristics of the virtual object; and also based on the second emotional response data, optimize the emotional symbol.

3. The method according to claim 1, characterized in that Searching the personality database for first personality data having the highest similarity to the personality characteristics of the target object also includes: Based on the parameters of the personality dimension and the weight, the personality characteristic value of the target object is calculated, the personality characteristic value is compared with the value of each personality data in the personality library, and the personality data with the smallest difference between the personality data value and the personality characteristic value is obtained as the first personality data.

4. The method according to claim 1, characterized in that: Based on the attribute information of the target user and the current emotional state, recording the first emotional response data of the target user includes: A series of emotion symbols and a plurality of emotion levels corresponding to each emotion symbol are preset, wherein the emotion levels are divided into primary, secondary and tertiary levels, which are used to indicate the intensity of the relevant emotion under the corresponding emotion symbol; Acquire attribute information of a plurality of first users, and divide all first users into a plurality of user categories based on the attribute information of the first users, select an emotion symbol corresponding to the attribute information of the first user from the emotion symbols as an alternative item, display the alternative item to each of the first users, photograph the facial expression of the first user, and use the matching result of the facial expression of the first user and the alternative item as the emotion measurement result of the first user, and convert the emotion measurement result into a numerical emotional response; Summarizing the emotion measurement results of the first user in each of the user categories, and calculating the standard deviation of the emotion measurement results of each of the emotion symbols, and using the standard deviation as the dispersion index of the emotion symbol, wherein each of the user categories has the same attribute information; Based on the dispersion index of the emotion symbol, a subcategory is created for each of the user categories, and when the dispersion index of a certain emotion symbol in the user category with the same attribute is less than a first threshold, the emotion symbol is stored in the subcategory; Acquire the attribute information of the target user, and based on the attribute information of the target user, select the subcategory of the user category corresponding to the target user, and display the emotional symbols in the subcategory to the target user, and based on the current emotional state of the target user, record the first emotional response data of the target user, wherein the emotional response data is represented by a corresponding numerical value.

5. The method according to claim 4, characterized in that After storing the emotional symbol in the subcategory, the following further comprises: When the dispersion index of a certain emotional symbol in the user category with the same attribute is greater than or equal to the first threshold, storing other emotional symbols outside the subcategory in the user category into other subcategories; Combined with the stored emotion record of the target user and the emotion symbols in the feedback of the virtual object, the emotion symbols simultaneously in the emotion record of the target user are obtained from the other subcategories and stored in the subcategory.

6. The method according to claim 2, characterized in that The feedback results also include: The feedback result of the target user speaking to the video of the virtual object is called a second speech content, and keywords are extracted from the second speech content, wherein the keywords also include phrases, adjectives, adverbs or specific nouns expressing emotions; Based on a preset emotion dictionary and the expression data of the target user, the emotion symbol and the emotion level corresponding to the emotion symbol are allocated to the keyword.

7. An intelligent system for virtual object social interaction and emotional feedback, used to implement the method according to any one of claims 1 to 6, characterized in that: The system includes the following modules: A determination unit, used to determine the image, sound and personality characteristics of the target object; The configuration unit is configured to: define a virtual object, extract first facial data from the image of the target object, extract first timbre data from the sound, select video data with the greatest similarity to the personality characteristics of the target object from a video library as a target video based on the personality characteristics of the target object, synthesize the first facial data into a corresponding part of the target video, and use the target video as an action basis of the virtual object, wherein the video library stores multiple types of video data; Based on the personality characteristics of the target object, first personality data closest to the personality characteristics of the target object is selected from a personality library, based on the first personality data, a first speech content is obtained from a dialogue template, and based on the first timbre data, speech data of the virtual object is generated by using speech synthesis technology, and the action basis of the virtual object and the speech data are integrated to generate a video of the virtual object speaking, wherein the speech data includes timbre, intonation, speech speed and emotional expression, and the video of the virtual object speaking is synchronized with the speech data, and the personality library stores personality data of multiple personality characteristics; a control unit, configured to obtain attribute information of the target user based on first shooting data, identify a current emotional state of the target user based on first voice data input by the target user, and record first emotional response data of the target user based on the attribute information of the target user and the current emotional state, wherein the first shooting data represents image information of the face or body of the target user captured when the virtual object interacts with the target user for the first time, and the first voice data represents voice information acquired through a voice input device when the virtual object interacts with the target user for the first time; an updating unit, configured to generate the emotion symbol of the current emotional state of the target user, combine the video of the virtual object speaking with the emotion symbol, update the video of the virtual object speaking, and display it on the terminal device of the target user to interact with the target user, and store the first emotion reaction data of the target user and the video of the virtual object speaking; The configuration unit is also used to define multiple personality dimensions for each personality data based on psychological theories, and define parameters for each personality dimension, and assign weights to each personality dimension based on the importance of the personality dimension in the personality data to form the personality library, wherein the parameters represent the quantification of the personality dimension; extract multiple first personality dimensions and parameters of each of the first personality dimensions from the personality characteristics of the target object, and search the personality library for the first personality data with the highest similarity to the personality characteristics of the target object.

Images