A digital human platform interaction system and method based on artificial intelligence

Abstract

The present invention discloses an artificial intelligence-based digital human platform interaction system and method, and relates to the field of human-computer interaction technology. The present invention utilizes digital twin technology to combine facial features and preferences of users in different emotions to construct a virtual user image; the generated digital human image and voice of each user are used to construct a digital human unique to each user, and the digital human of each user is saved to construct a digital human warehouse; when the user interacts with the digital human, the user's facial data and the virtual user image are collected in real time for matching, and after matching, a digital human unique to the real-time user is selected according to the virtual user image and displayed; the digital human collects answers on the network according to interaction questions and classifies all collected answers; selects answers to interaction questions that satisfy the user; and the digital human modifies the answers to the interaction questions according to the user's emotions and outputs them to the real-time user.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to an artificial intelligence-based digital human platform interaction system and method. Background Art

[0002] Digital humans initially appeared primarily in film and gaming, used to create simple virtual characters or character animations. These early digital humans were relatively crude, primarily achieved through manual modeling and keyframe animation. Their interactivity was also very limited, typically limited to simple movements and dialogue according to pre-set scripts. With the initial development of computer graphics and artificial intelligence technologies, digital humans' appearance and performance improved considerably, and they began to possess some simple interactive features, such as limited interaction with users through mouse clicks or keyboard input. Simultaneously, some virtual assistant-style digital human applications began to emerge, but the technology for speech recognition and natural language understanding was still immature, resulting in a relatively rigid interactive experience. Major breakthroughs in deep learning technology in the field of artificial intelligence have brought about a qualitative leap in digital human interaction. Digital humans are now widely used in various fields, such as intelligent customer service, virtual anchors, and education. As voice interaction technology matures, digital humans can relatively accurately recognize and understand user voice commands and respond with natural and fluent speech. Furthermore, facial expressions and body movements are becoming more natural and lifelike, better matching the spoken content and enhancing the realism and fun of interactions. However, when users interact with digital humans today, they are presented with a single image. Users feel alienated and disconnected during the interaction, and when digital humans generate answers to user questions, they are unable to change the form of the answers based on the user's real-time status. This has a significant impact on the user's experience and senses when interacting with digital humans. Summary of the Invention

[0003] The purpose of the present invention is to provide a digital human platform interaction system and method based on artificial intelligence to solve the problems raised in the prior art.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A digital human platform interaction method based on artificial intelligence, the method comprising the following steps:

[0006] S100. Collect statistics on all users who interact with the digital human, collect the different emotions and corresponding facial data of each user when interacting with the digital human in the historical records, extract the facial features of the users when they have different emotions, collect the feedback data of each user on the interaction results when they interact with the digital human in the historical records, and use the feedback data to determine the user's preferences for the interaction results; use digital twin technology to combine the facial features and preferences of the users when they have different emotions to construct a virtual user image;

[0007] Furthermore, the specific steps for using digital twin technology to combine the user's facial features and preferences when they have different emotions to construct a virtual user image are as follows:

[0008] S101. Count all users who interact with the digital human to obtain a user number of n. Collect the emotion types of each user when interacting with the digital human in the historical records. , It represents the 1st, 2nd, 3rd, ..., nth emotion of each user when interacting with the digital human in the collected historical records, where n is a positive integer; extract the facial data of the user when interacting with the digital human in different emotions, and use the Dlib library to find the key points in the facial data. , Indicates the 1st, 2nd, 3rd, ..., mth key points in the facial data to be searched, where m is a positive integer;

[0009] S102: Establish a coordinate system in the user's facial data and extract the coordinates of each key point. , Represent the coordinates of the 1st, 2nd, 3rd, ..., mth key points in the facial data, and calculate the geometric distance between every two key points. The formula is:

[0010] ;

[0011] In the formula, Represents the geometric distance between the i-th and j-th key points among the m key points, and represents the horizontal and vertical coordinates of the i-th key point, and Represents the horizontal and vertical coordinates of the jth key point; i and j range from 1 to m; the geometric distance of each key point under different user emotions is calculated as follows: , Indicates the 1st, 2nd, 3rd, ..., For the geometric distance of key points, the geometric distance set A of key points of the user with different emotions is used as the facial feature of each emotion;

[0012] S103, collecting the feedback data of each user on the interaction results when interacting with the digital human in the historical records, and extracting the result features of the digital human when the user interacted with the digital human in the historical records; matching the collected user feedback data with the corresponding result features of the digital human to obtain the component elements combined into , represents the 1st, 2nd, 3rd, ..., kth element combination obtained after matching, where F represents the feedback data, J represents the result feature of the digital human; k is a positive integer; the number of times each element combination occurs is collected and the average value is calculated, and the element combination with a number greater than the average value is defined as a frequent group; the confidence of all frequent groups is calculated using the formula:

[0013] ;

[0014] In the formula, represents the confidence of the calculated t-th frequent group, Indicates the number of times the t-th frequent group is collected, Represents the number of feedback data collected in the tth frequent group; calculate the confidence of all frequent groups, and after judgment, select the result characteristics of the digital human in the frequent group with the largest confidence as the user's preference, and repeat the calculation to obtain the preferences of all users; use digital twin technology to combine the user's facial features and preferences when in different emotions to construct a virtual user image.

[0015] By leveraging historical facial data and feedback from user interactions with digital humans, we can derive facial features and user preferences for different emotions and construct a virtual user avatar. By analyzing the avatar, we can gain a deeper understanding of the user's appearance, facial expressions, and movement preferences, among other personalized information. Based on these analytical results, the digital human can provide users with interactive content and service methods that better suit their personal preferences and needs, making the interaction process more natural, smooth, and comfortable, and enhancing the user's sense of participation and immersion.

[0016] Virtual avatars can serve as a form of external expression of users' subconscious needs. Analyzing these avatars helps uncover unspoken latent needs. This allows businesses or service providers to proactively plan and optimize products or services to better meet future user needs and enhance user satisfaction and loyalty.

[0017] By continuously analyzing the avatar, the effectiveness of strategy implementation can be monitored and evaluated in real time. Based on the analysis results, problems and deviations in strategy execution can be promptly identified, and corresponding adjustments and optimizations can be made to ensure that the strategy always progresses towards the intended goal, achieving effective management and continuous optimization of the business.

[0018] S200: For each user's virtual user image, collect the virtual images browsed by each user on the Internet, extract the appearance features of the browsed virtual images, and construct a digital human image unique to each user; collect audio data listened to by each user on the Internet, extract the voice features of the audio data, and construct a digital human voice unique to each user; construct a digital human unique to each user using the generated digital human image and voice, and save each user's digital human to construct a digital human warehouse;

[0019] Furthermore, the specific steps for saving each user's Digital Human and building a Digital Human warehouse are as follows:

[0020] S201. For each user's virtual user image constructed, collect the virtual images browsed by each user on the Internet, extract the facial key points of the collected virtual images using a face detection algorithm, standardize the data value of each facial key point to the same data format, and calculate the performance value of each facial key point. The formula is:

[0021] ;

[0022] In the formula, B represents the performance value of each key point, R represents the total number of differences between every two data values ​​in each key point, and H and H' represent the different data values ​​in each key point. After calculating the performance value of each key point, image segmentation technology is used to segment each key point in the facial data into a separate appearance feature.

[0023] Extract the number of times users browsed each segmented appearance feature in the historical records and construct a probability model, which is expressed as: ; In the model represents the probability of the vth appearance feature, represents the number of views of the v-th appearance feature, Represents the total number of times users have viewed all appearance features; uses a deep learning model to train the probabilities of all appearance features, and selects all appearance features with a probability greater than 1 / 2 as the user's favorite appearance; finally, uses image generation technology to combine all the user's favorite appearance features to generate the user's favorite digital human image;

[0024] S202: Collect audio data listened to by each user on the network, extract timbre features from the collected audio data using an audio processing algorithm, calculate the probability of each timbre feature using the same probability model as in S201, and select the timbre feature with the highest probability as the user's favorite audio feature; repeat the operation to calculate the favorite audio feature for each user;

[0025] S203. After calculating the digital human image and audio features preferred by each user, an adversarial network algorithm is used to construct a digital human unique to each user based on the digital human image and audio features preferred by each user, and each user's digital human is saved to construct a digital human warehouse.

[0026] Different users have different preferences and needs for the appearance and voice of their digital human beings. Some may prefer a youthful, energetic appearance and a crisp, pleasant voice, while others may prefer a mature, stable appearance and a deep, mellow voice. Generating a digital human unique to each user can precisely meet these diverse needs, providing users with a digital companion that suits their personal tastes.

[0027] When users interact with a digital human, a familiar appearance and voice help them more quickly understand the digital human's message. For example, in a voice interaction scenario, the user's own voice style allows them to more instinctively grasp the key points and emotions in the digital human's speech, reducing the comprehension barriers that may arise from unfamiliar voices, thereby making communication smoother and improving the efficiency of information transfer.

[0028] S300: When a user interacts with a digital human, the user's facial data is collected in real time and matched with the virtual user image. After the match, a digital human unique to the real-time user is selected based on the virtual user image and displayed.

[0029] Furthermore, after matching, the specific steps of selecting a digital person unique to the real-time user based on the virtual user image and displaying it are as follows:

[0030] S301. When a user interacts with a digital human, the user's facial data is collected in real time, key points in the real-time user's facial data are extracted, the data value of each key point is collected accordingly, and the data value of each key point in the virtual user's image is extracted; the facial similarity between the real-time user and all virtual user images is calculated using the formula:

[0031] ;

[0032] In the formula, represents the facial similarity between the real-time user and each virtual user image, E represents the number of key points in the extracted real-time user facial data, Represents the data value of the p-th key point of the real-time user, Represents the data value of the pth key point in the virtual user image;

[0033] S302: After calculating the facial similarity between the real-time user and each virtual user image, select the virtual user image with the facial similarity closest to 1 and match it with the real-time user to obtain the mapped virtual image of the real-time user; and select a digital human unique to the real-time user from the digital human warehouse based on the mapped virtual image of the real-time user for display.

[0034] S400: After selecting a real-time user's digital human, voice data from the user's interaction with the digital human is collected, and the user's voice data is analyzed using a natural language processing algorithm to generate interactive questions that the real-time user asks the digital human. The digital human collects answers online based on the interactive questions and classifies all collected answers.

[0035] Furthermore, the specific steps for the digital human to collect answers on the Internet based on the interactive questions and classify all the collected answers are as follows:

[0036] S401: After selecting a real-time user's digital human, collect the user's voice data when interacting with the digital human, analyze the user's voice data using a natural language processing algorithm, and generate interactive questions that the real-time user asks the digital human;

[0037] S402: The digital human collects answers on the Internet according to the interactive questions and classifies all the collected answers into categories according to the interactive result features in the S100 historical records. , Represents the 1st, 2nd, 3rd, ..., bth answers obtained after classification, where b is a positive integer.

[0038] S500: Extract the preferences of the user's virtual image, judge each of the classified answers, and select the answer to the interactive question that satisfies the user;

[0039] Furthermore, the specific steps for selecting the answers to the interactive questions that satisfy the user are as follows:

[0040] S501: Extract the preferences of the user's virtual image to judge each answer after classification, and select the answer with the same result characteristics as the user's preference as the optimal answer.

[0041] S600: After selecting the answer to the interactive question, collect real-time facial data of the user and the digital human when interacting, and make a corresponding judgment on the real-time user's emotions in the user's virtual image. The digital human modifies the answer to the interactive question according to the user's emotions and outputs it to the real-time user.

[0042] Furthermore, the specific steps for the digital human to modify the answer to the interactive question based on the user's emotions and output it to the real-time user are as follows:

[0043] S601: After selecting the answer to the interactive question, collect the real-time facial data of the user and the digital human when they interact, extract the coordinates of each key point in the real-time user facial data, and calculate the geometric distance between each two key points in the real-time user facial data. , Indicates the 1st, 2nd, 3rd, ..., The geometric distance between the key points is compared with the geometric distance A' of the key points of the real-time user and the geometric distance set A of the key points in the mapped virtual image of the real-time user;

[0044] S602: When A=A', extract the user emotions corresponding to the A set as the real-time user's interactive emotions, add real-time user interactive emotion modifiers to the selected best answer using a natural language processing algorithm, and finally output it to the real-time user.

[0045] Analyzing elements like the avatar's expressions and posture can help the digital human better understand the user's emotional state. This allows the digital human to respond emotionally accordingly, establishing a deeper emotional connection with the user, allowing the user to feel more understood and resonate during the interaction, and improving the quality and depth of the interaction.

[0046] An artificial intelligence-based digital human platform interaction system, comprising a data collection module, a virtual user image construction module, a digital human construction module, a real-time interaction module, and an output module;

[0047] The data collection module is used to collect data on interactions with digital humans in user history records;

[0048] The virtual user image construction module is used to extract the user's facial features when they have different emotions based on the facial data of the user during interaction in the collected historical records, use the feedback data to obtain the user's preferences for the interaction results, and use digital twin technology to combine the user's facial features and preferences when they have different emotions to construct a virtual user image;

[0049] The digital human construction module is used to extract the appearance features of the browsing virtual images in the collected historical records to construct a digital human image unique to each user, extract the voice features of the audio data to construct a digital human voice unique to each user, and construct a digital human unique to each user;

[0050] The real-time interaction module is used to collect the real-time user's facial data to search for and map the virtual image, then select the digital human's answer based on the user's preferences, and judge the real-time user's emotions based on the real-time user's facial data;

[0051] The output module is used to add modifiers to the selected best answer according to the real-time user's emotions and then output it to the real-time user.

[0052] The virtual user image building module includes emotion unit and preference unit;

[0053] The emotion unit is used to extract the facial features of the user when they have different emotions based on the facial data of the user when they interact in the collected historical records;

[0054] The preference unit is used to calculate the preference of each user based on the collected user feedback data and result features.

[0055] The real-time interaction module includes a mapping virtual image unit, an answer selection unit, and an emotion judgment unit;

[0056] The mapping virtual image unit is used to collect facial data of the real-time user, calculate the similarity between the real-time user and the virtual user image, and select the mapping virtual image of the real-time user;

[0057] The answer selection unit is used to collect real-time user facial data, search for mapped virtual images, and then select digital human answers based on user preferences;

[0058] The emotion judgment unit is used to judge the real-time user emotion based on the real-time user's facial data.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. This invention utilizes historical facial data and feedback from user interactions with digital humans to derive facial features and user preferences for different emotions, thereby constructing a virtual user avatar. Analysis of the avatar provides a deep understanding of the user's personalized information, including their appearance, facial expressions, and movement preferences. Based on these analysis results, the digital human can provide users with interactive content and service methods that better suit their personal preferences and needs, making the interaction process more natural, smooth, and comfortable, and enhancing the user's sense of participation and immersion.

[0061] 2. After the present invention builds a digital human unique to each user, the familiar appearance and voice of the digital human during interaction helps the user more quickly understand the information conveyed by the digital human. For example, in a voice interaction scenario, the user's own voice style allows them to more instinctively grasp the key points and emotions in the digital human's speech, reducing the comprehension barriers that may arise from unfamiliar voices, thereby making communication smoother and improving the efficiency of information transfer. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a module distribution diagram of an artificial intelligence-based digital human platform interaction system of the present invention;

[0063] Figure 2 This is a schematic diagram of the steps of an artificial intelligence-based digital human platform interaction method of the present invention. DETAILED DESCRIPTION

[0064] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0065] Example: Figure 1-Figure 2 As shown, the present invention provides a technical solution.

[0066] A digital human platform interaction method based on artificial intelligence, the method comprising the following steps:

[0067] S100. Collect statistics on all users who interact with the digital human, collect the different emotions and corresponding facial data of each user when interacting with the digital human in the historical records, extract the facial features of the users when they have different emotions, collect the feedback data of each user on the interaction results when they interact with the digital human in the historical records, and use the feedback data to determine the user's preferences for the interaction results; use digital twin technology to combine the facial features and preferences of the users when they have different emotions to construct a virtual user image;

[0068] The specific steps for using digital twin technology to combine the user's facial features and preferences when they have different emotions to build a virtual user image are as follows:

[0069] S101. Count all users who interact with the digital human to obtain a user number of n. Collect the emotion types of each user when interacting with the digital human in the historical records. , It represents the 1st, 2nd, 3rd, ..., nth emotion of each user when interacting with the digital human in the collected historical records, where n is a positive integer; extract the facial data of the user when interacting with the digital human in different emotions, and use the Dlib library to find the key points in the facial data. , Indicates the 1st, 2nd, 3rd, ..., mth key points in the facial data to be searched, where m is a positive integer;

[0070] S102: Establish a coordinate system in the user's facial data and extract the coordinates of each key point. , Represent the coordinates of the 1st, 2nd, 3rd, ..., mth key points in the facial data, and calculate the geometric distance between every two key points. The formula is:

[0071] ;

[0072] In the formula, Represents the geometric distance between the i-th and j-th key points among the m key points, and represents the horizontal and vertical coordinates of the i-th key point, and Represents the horizontal and vertical coordinates of the jth key point; i and j range from 1 to m; the geometric distance of each key point under different user emotions is calculated as follows: , Indicates the 1st, 2nd, 3rd, ..., For the geometric distance of key points, the geometric distance set A of key points of the user with different emotions is used as the facial feature of each emotion;

[0073] S103, collecting the feedback data of each user on the interaction results when interacting with the digital human in the historical records, and extracting the result features of the digital human when the user interacted with the digital human in the historical records; matching the collected user feedback data with the corresponding result features of the digital human to obtain the component elements combined into , represents the 1st, 2nd, 3rd, ..., kth element combination obtained after matching, where F represents the feedback data, J represents the result feature of the digital human; k is a positive integer; the number of times each element combination occurs is collected and the average value is calculated, and the element combination with a number greater than the average value is defined as a frequent group; the confidence of all frequent groups is calculated using the formula:

[0074] ;

[0075] In the formula, represents the confidence of the calculated t-th frequent group, Indicates the number of times the t-th frequent group is collected, Represents the number of feedback data collected in the tth frequent group; calculate the confidence of all frequent groups, and after judgment, select the result characteristics of the digital human in the frequent group with the largest confidence as the user's preference, and repeat the calculation to obtain the preferences of all users; use digital twin technology to combine the user's facial features and preferences when in different emotions to construct a virtual user image.

[0076] By leveraging historical facial data and feedback from user interactions with digital humans, we can derive facial features and user preferences for different emotions and construct a virtual user avatar. By analyzing the avatar, we can gain a deeper understanding of the user's appearance, facial expressions, and movement preferences, among other personalized information. Based on these analytical results, the digital human can provide users with interactive content and service methods that better suit their personal preferences and needs, making the interaction process more natural, smooth, and comfortable, and enhancing the user's sense of participation and immersion.

[0077] Virtual avatars can serve as a form of external expression of users' subconscious needs. Analyzing these avatars helps uncover unspoken latent needs. This allows businesses or service providers to proactively plan and optimize products or services to better meet future user needs and enhance user satisfaction and loyalty.

[0078] By continuously analyzing the avatar, the effectiveness of strategy implementation can be monitored and evaluated in real time. Based on the analysis results, problems and deviations in strategy execution can be promptly identified, and corresponding adjustments and optimizations can be made to ensure that the strategy always progresses towards the intended goal, achieving effective management and continuous optimization of the business.

[0079] S200: For each user's virtual user image, collect the virtual images browsed by each user on the Internet, extract the appearance features of the browsed virtual images, and construct a digital human image unique to each user; collect audio data listened to by each user on the Internet, extract the voice features of the audio data, and construct a digital human voice unique to each user; construct a digital human unique to each user using the generated digital human image and voice, and save each user's digital human to construct a digital human warehouse;

[0080] The specific steps to save each user's Digital Human and build a Digital Human warehouse are:

[0081] S201. For each user's virtual user image constructed, collect the virtual images browsed by each user on the Internet, extract the facial key points of the collected virtual images using a face detection algorithm, standardize the data value of each facial key point to the same data format, and calculate the performance value of each facial key point. The formula is:

[0082] ;

[0083] In the formula, B represents the performance value of each key point, R represents the total number of differences between every two data values ​​in each key point, and H and H' represent the different data values ​​in each key point. After calculating the performance value of each key point, image segmentation technology is used to segment each key point in the facial data into a separate appearance feature.

[0084] Extract the number of times users browsed each segmented appearance feature in the historical records and construct a probability model, which is expressed as: ; In the model represents the probability of the vth appearance feature, represents the number of views of the v-th appearance feature, Represents the total number of times users have viewed all appearance features; uses a deep learning model to train the probabilities of all appearance features, and selects all appearance features with a probability greater than 1 / 2 as the user's favorite appearance; finally, uses image generation technology to combine all the user's favorite appearance features to generate the user's favorite digital human image;

[0085] S202: Collect audio data listened to by each user on the network, extract timbre features from the collected audio data using an audio processing algorithm, calculate the probability of each timbre feature using the same probability model as in S201, and select the timbre feature with the highest probability as the user's favorite audio feature; repeat the operation to calculate the favorite audio feature for each user;

[0086] S203. After calculating the digital human image and audio features preferred by each user, an adversarial network algorithm is used to construct a digital human unique to each user based on the digital human image and audio features preferred by each user, and each user's digital human is saved to construct a digital human warehouse.

[0087] Different users have different preferences and needs for the appearance and voice of their digital human beings. Some may prefer a youthful, energetic appearance and a crisp, pleasant voice, while others may prefer a mature, stable appearance and a deep, mellow voice. Generating a digital human unique to each user can precisely meet these diverse needs, providing users with a digital companion that suits their personal tastes.

[0088] When users interact with a digital human, a familiar appearance and voice help them more quickly understand the digital human's message. For example, in a voice interaction scenario, the user's own voice style allows them to more instinctively grasp the key points and emotions in the digital human's speech, reducing the comprehension barriers that may arise from unfamiliar voices, thereby making communication smoother and improving the efficiency of information transfer.

[0089] S300: When a user interacts with a digital human, the user's facial data is collected in real time and matched with the virtual user image. After the match, a digital human unique to the real-time user is selected based on the virtual user image and displayed.

[0090] After matching, the specific steps for selecting a digital person unique to the real-time user based on the virtual user image and displaying it are as follows:

[0091] S301. When a user interacts with a digital human, the user's facial data is collected in real time, key points in the real-time user's facial data are extracted, the data value of each key point is collected accordingly, and the data value of each key point in the virtual user's image is extracted; the facial similarity between the real-time user and all virtual user images is calculated using the formula:

[0092] ;

[0093] In the formula, represents the facial similarity between the real-time user and each virtual user image, E represents the number of key points in the extracted real-time user facial data, Represents the data value of the p-th key point of the real-time user, Represents the data value of the pth key point in the virtual user image;

[0094] S302: After calculating the facial similarity between the real-time user and each virtual user image, select the virtual user image with the facial similarity closest to 1 and match it with the real-time user to obtain the mapped virtual image of the real-time user; and select a digital human unique to the real-time user from the digital human warehouse based on the mapped virtual image of the real-time user for display.

[0095] S400: After selecting a real-time user's digital human, voice data from the user's interaction with the digital human is collected, and the user's voice data is analyzed using a natural language processing algorithm to generate interactive questions that the real-time user asks the digital human. The digital human collects answers online based on the interactive questions and classifies all collected answers.

[0096] The specific steps for the digital human to collect answers from the network based on interactive questions and classify all collected answers are as follows:

[0097] S401: After selecting a real-time user's digital human, collect the user's voice data when interacting with the digital human, analyze the user's voice data using a natural language processing algorithm, and generate interactive questions that the real-time user asks the digital human;

[0098] S402: The digital human collects answers on the Internet according to the interactive questions and classifies all the collected answers into categories according to the interactive result features in the S100 historical records. , Represents the 1st, 2nd, 3rd, ..., bth answers obtained after classification, where b is a positive integer.

[0099] S500: Extract the preferences of the user's virtual image, judge each of the classified answers, and select the answer to the interactive question that satisfies the user;

[0100] The specific steps to select the answers to interactive questions that satisfy users are as follows:

[0101] S501: Extract the preferences of the user's virtual image to judge each answer after classification, and select the answer with the same result characteristics as the user's preference as the optimal answer.

[0102] S600: After selecting the answer to the interactive question, collect real-time facial data of the user and the digital human when interacting, and make a corresponding judgment on the real-time user's emotions in the user's virtual image. The digital human modifies the answer to the interactive question according to the user's emotions and outputs it to the real-time user.

[0103] The specific steps for the digital human to modify the answer to the interactive question based on the user's emotions and output it to the real-time user are as follows:

[0104] S601: After selecting the answer to the interactive question, collect the real-time facial data of the user and the digital human when they interact, extract the coordinates of each key point in the real-time user facial data, and calculate the geometric distance between each two key points in the real-time user facial data. , Indicates the 1st, 2nd, 3rd, ..., The geometric distance between the key points is compared with the geometric distance A' of the key points of the real-time user and the geometric distance set A of the key points in the mapped virtual image of the real-time user;

[0105] S602: When A=A', extract the user emotions corresponding to the A set as the real-time user's interactive emotions, add real-time user interactive emotion modifiers to the selected best answer using a natural language processing algorithm, and finally output it to the real-time user.

[0106] Analyzing elements like the avatar's expressions and posture can help the digital human better understand the user's emotional state. This allows the digital human to respond emotionally accordingly, establishing a deeper emotional connection with the user, allowing the user to feel more understood and resonate during the interaction, and improving the quality and depth of the interaction.

[0107] An artificial intelligence-based digital human platform interaction system, comprising a data collection module, a virtual user image construction module, a digital human construction module, a real-time interaction module, and an output module;

[0108] The data collection module is used to collect data on interactions with digital humans in user history records;

[0109] The virtual user image construction module is used to extract the user's facial features when they have different emotions based on the facial data of the user during interaction in the collected historical records, use the feedback data to obtain the user's preferences for the interaction results, and use digital twin technology to combine the user's facial features and preferences when they have different emotions to construct a virtual user image;

[0110] The digital human construction module is used to extract the appearance features of the browsing virtual images in the collected historical records to construct a digital human image unique to each user, extract the voice features of the audio data to construct a digital human voice unique to each user, and construct a digital human unique to each user;

[0111] The real-time interaction module is used to collect the real-time user's facial data to search for and map the virtual image, then select the digital human's answer based on the user's preferences, and judge the real-time user's emotions based on the real-time user's facial data;

[0112] The output module is used to add modifiers to the selected best answer according to the real-time user's emotions and then output it to the real-time user.

[0113] The virtual user image building module includes emotion unit and preference unit;

[0114] The emotion unit is used to extract the facial features of the user when they have different emotions based on the facial data of the user when they interact in the collected historical records;

[0115] The preference unit is used to calculate the preference of each user based on the collected user feedback data and result features.

[0116] The real-time interaction module includes a mapping virtual image unit, an answer selection unit, and an emotion judgment unit;

[0117] The mapping virtual image unit is used to collect facial data of the real-time user, calculate the similarity between the real-time user and the virtual user image, and select the mapping virtual image of the real-time user;

[0118] The answer selection unit is used to collect real-time user facial data, search for mapped virtual images, and then select digital human answers based on user preferences;

[0119] The emotion judgment unit is used to judge the real-time user emotion based on the real-time user's facial data.

[0120] Example: Now a digital human platform is constructed, and the number of users is counted as 3. A virtual user image is constructed based on the historical records of the three users' interactions;

[0121] Assume there are three users, namely user 1, user 2, and user 3; facial features include the corners of the eyes, the corners of the mouth, and the ends of the eyebrows; and emotions include joy and depression.

[0122] The facial features of different emotions in the virtual user images of the three users are obtained as follows: User 1 - joy (10, 5, 13), depression (8, 5, 9); the virtual user images of User 2 and User 3 are obtained in turn;

[0123] Based on the characteristics of the interaction results in the history, it is found that user 1 prefers simplicity, user 2 prefers detail, and user 3 prefers ornateness;

[0124] Based on the browsing image and audio data of each user in history, digital characters unique to three users are constructed: User 1 (child, immature), User 2 (animal, stable), and User 3 (cartoon girl, cheerful);

[0125] The real-time user's facial data is collected and similarity is calculated. The real-time user is determined to be user 2, and the digital person is selected as an animal and has a steady voice. Based on the real-time user's interactive question, "Introduce "Dream of the Red Chamber", the answers are classified into concise, detailed, and ornate.

[0126] Select the detailed introduction category based on the mapped virtual image; collect real-time user facial data, obtain the real-time emotion as joy, and use the natural language processing algorithm to add joy modifiers to the answer, for example: "It's a pleasure to serve you, etc.", and finally output it to the user.

[0127] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A digital human platform interaction method based on artificial intelligence, characterized by: The method comprises the following steps: S100, collect statistics on all users who interact with the digital human, collect different emotions and corresponding facial data of each user when interacting with the digital human in the historical records, extract the facial features of the user when they are in different emotions, collect the feedback data of the user on the interaction results when each user interacts with the digital human in the historical records, and use the feedback data to obtain the user's preference for the interaction results; use the digital twin technology to combine the facial features and preferences of the user when they are in different emotions to construct a virtual user image; S200, for each user's virtual user image constructed, collect the virtual images browsed by each user on the network, extract the appearance features of the browsed virtual images to construct a digital human image unique to each user; collect the audio data listened to by each user on the network, extract the voice features of the audio data, and construct a digital human voice unique to each user; construct a digital human unique to each user with the generated digital human image and voice of each user, and save the digital human of each user to construct a digital human warehouse; S300, when the user interacts with the digital human, the user's facial data is collected in real time and matched with the virtual user image, and after matching, a digital human unique to the real-time user is selected according to the virtual user image and displayed; S400: After selecting the digital human of the real-time user, voice data of the user interacting with the digital human is collected, and the user voice data is analyzed using a natural language processing algorithm to generate interactive questions that the real-time user asks the digital human; the digital human collects answers on the network according to the interactive questions and classifies all the collected answers; S500, extracting preferences from the user's virtual image to judge each of the classified answers, and selecting an answer to the interactive question that satisfies the user; S600: After the answer to the interactive question is selected, the real-time facial data of the user and the digital human when interacting is collected, and the emotions of the real-time user are judged in the user's virtual image. The digital human modifies the answer to the interactive question according to the user's emotions and outputs it to the real-time user.

2. The method for digital human platform interaction based on artificial intelligence according to claim 1, characterized in that: The specific steps of using the digital twin technology in S100 to combine the facial features and preferences of the user in different emotions to construct a virtual user image are: S101. Count all users who interact with the digital human to obtain a user number of n, and collect the emotion types of each user when interacting with the digital human in the historical records. , It represents the 1st, 2nd, 3rd, ..., nth emotions of each user in the collected historical records when interacting with the digital human, where n is a positive integer; extract the facial data of the user when interacting with the digital human when the user has different emotions, and use the Dlib library to find the key points in the facial data: , Indicates the 1st, 2nd, 3rd, ..., mth key points in the facial data to be searched, where m is a positive integer; S102: Establish a coordinate system in the user's facial data and extract the coordinate points of each key point. , Represent the coordinates of the 1st, 2nd, 3rd, ..., mth key points in the facial data, and calculate the geometric distance between every two key points. The formula is: ； In the formula, Represents the geometric distance between the i-th and j-th key points among the m key points, and represents the horizontal and vertical coordinates of the i-th key point, and Represents the horizontal and vertical coordinates of the jth key point; i and j are from 1 to m; the geometric distance of each key point under different user emotions is calculated as follows: , Indicates the 1st, 2nd, 3rd, ..., For the geometric distance of key points, the geometric distance set A of key points of the user with different emotions is used as the facial feature of each emotion; S103, collecting the feedback data of each user on the interaction result when the user interacts with the digital human in the historical records, and correspondingly extracting the result features of the digital human when the user interacts with the digital human in the historical records; The collected user feedback data is matched with the corresponding digital human result features to obtain the constituent elements combined as , represents the 1st, 2nd, 3rd, ..., kth element combination obtained after matching, where F represents the feedback data, J represents the result feature of the digital human; k is a positive integer; the number of times for each element combination is collected and the average value is calculated, and the element combination with a number greater than the average value is defined as a frequent group; the confidence of all frequent groups is calculated, and the formula is: ； In the formula, represents the confidence of the calculated t-th frequent group, represents the number of times the tth frequent group is collected, Indicates the number of feedback data collected in the tth frequent group; The confidence of all frequent groups is calculated, and the result feature of the digital human in the frequent group with the largest confidence is selected as the user's preference after judgment, and the calculation is repeated to obtain the preferences of all users; Digital twin technology is used to combine the user's facial features and preferences when in different emotions to construct a virtual user image.

3. The method for digital human platform interaction based on artificial intelligence according to claim 2, characterized in that: The specific steps of saving each user's digital human and building a digital human warehouse in S200 are: S201, for each user's virtual user image constructed, collect the virtual images browsed by each user on the network, use the face detection algorithm to extract the facial key points of the collected virtual images, standardize the data value of each facial key point, unify them into the same data format, and calculate the performance value of each facial key point, the formula is: ； In the formula, B represents the performance value of each key point, R represents the total number of differences between every two data values ​​in each key point, and H and H' represent the different data values ​​in each key point. After calculating the performance value of each key point, the image segmentation technology is used to segment each key point in the facial data to form a separate appearance feature. The number of views of each segmented appearance feature by the user in the historical records is extracted respectively, and a probability model is constructed, which is expressed as: ; In the model represents the probability of the vth appearance feature, represents the number of views of the vth appearance feature, Indicates the total number of views of all appearance features by users; The deep learning model is used to train the probabilities of all appearance features, and all appearance features with a probability greater than one-half are selected as the user's favorite appearance; finally, the image generation technology is used to combine all the user's favorite appearance features to generate the user's favorite digital human image; S202, collecting audio data listened to by each user on the network, extracting timbre features from the collected audio data using an audio processing algorithm, calculating the probability of each timbre feature using the same probability model as in S201, and selecting the timbre feature with the highest probability as the user's favorite audio feature; repeating the operation to calculate the favorite audio feature of each user; S203. After calculating the digital human image and audio features that each user likes, use an adversarial network algorithm to build a digital human that belongs to each user according to the digital human image and audio features that each user likes, and save the digital human of each user to build a digital human warehouse.

4. The method for digital human platform interaction based on artificial intelligence according to claim 3, characterized in that: The specific steps of selecting and displaying a digital person unique to the real-time user according to the virtual user image after matching in S300 are: S301, when the user interacts with the digital human, collect the user's facial data in real time, extract the key points in the real-time user's facial data, collect the data value of each key point accordingly, and extract the data value of each key point in the virtual user image; calculate the facial similarity of the real-time user and all virtual user images, the formula is: ； In the formula, represents the facial similarity between the real-time user and each virtual user image, E represents the number of key points in the extracted real-time user facial data, Represents the data value of the pth key point of the real-time user, Represents the data value of the pth key point in the virtual user image; S302, after calculating the facial similarity between the real-time user and each virtual user image, select the virtual user image with the facial similarity closest to 1 and match it with the real-time user to obtain the mapped virtual image of the real-time user; and select a digital human unique to the real-time user in the digital human warehouse for display according to the mapped virtual image of the real-time user.

5. The method for digital human platform interaction based on artificial intelligence according to claim 4, characterized in that: The specific steps of the digital human collecting answers on the network according to the interactive questions and classifying all the collected answers in S400 are: S401, after selecting the digital human of the real-time user, collecting the voice data of the user interacting with the digital human, analyzing the user voice data using a natural language processing algorithm, and generating interactive questions that the real-time user asks the digital human; S402: The digital human collects answers on the network according to the interactive questions and then classifies all the collected answers according to the interactive result features in the historical records of S100. , Represents the 1st, 2nd, 3rd, ..., bth answers obtained after classification, where b is a positive integer.

6. The method for digital human platform interaction based on artificial intelligence according to claim 5, characterized in that: The specific steps of selecting the answer to the interactive question that satisfies the user in S500 are: S501, extracting the preferences of the user's virtual image to judge each answer after classification, and selecting the answer with the same result characteristics as the user's preference as the optimal answer.

7. The method for digital human platform interaction based on artificial intelligence according to claim 6, characterized in that: The specific steps of the digital human modifying the answer to the interactive question according to the user's emotions and then outputting it to the real-time user in S600 are: S601, after selecting the answer to the interactive question, collect the real-time facial data of the user and the digital human when interacting, extract the coordinates of each key point in the real-time user facial data, and calculate the geometric distance between every two key points in the real-time user facial data. , Indicates the 1st, 2nd, 3rd, ..., The geometric distance between the key points is compared with the key point geometric distance A' of the real-time user and the key point geometric distance set A in the mapped virtual image of the real-time user; S602, when A=A', extract the user emotions corresponding to the A set as the interactive emotions of the real-time user, add the real-time user interactive emotion modifier to the selected best answer using the natural language processing algorithm, and finally output it to the real-time user.

8. A digital human platform interaction system based on artificial intelligence, characterized by: The digital human platform interaction system includes a data collection module, a virtual user image construction module, a digital human construction module, a real-time interaction module, and an output module; The data collection module is used to collect data on the user's interaction with the digital human in the historical records; The virtual user image construction module is used to extract the facial features of the user when in different emotions based on the facial data of the user during interaction in the collected historical records, obtain the user's preference for the interaction results using the feedback data, and use the digital twin technology to combine the facial features and preferences of the user when in different emotions to construct a virtual user image; The digital human construction module is used to extract the appearance features of the virtual images browsed in the collected historical records to construct a digital human image unique to each user, extract the voice features of the audio data, construct a digital human voice unique to each user, and construct a digital human unique to each user; The real-time interaction module is used to collect the real-time user's facial data to find and map the virtual image, then select the digital human answer according to the user's preference, and judge the real-time user's emotions according to the real-time user's facial data; The output module is used to add modifiers to the selected best answer according to the real-time user's emotions and then output it to the real-time user.

9. The digital human platform interaction system based on artificial intelligence according to claim 8, characterized in that: The virtual user image building module includes an emotion unit and a preference unit; The emotion unit is used to extract the facial features of the user when they are in different emotions based on the facial data of the user when they interact in the collected historical records; The preference unit is used to calculate the preference of each user according to the collected user feedback data and result characteristics.

10. The digital human platform interaction system based on artificial intelligence according to claim 8, characterized in that: The real-time interaction module includes a mapping virtual image unit, an answer selection unit and an emotion judgment unit; The mapping virtual image unit is used to collect facial data of the real-time user, calculate the similarity between the real-time user and the virtual user image, and select the mapping virtual image of the real-time user; The answer selection unit is used to collect real-time user facial data, search for mapped virtual images, and then select digital human answers according to user preferences; The emotion judgment unit is used to judge the real-time user emotion according to the real-time user's facial data.

Images