Digital human interaction method, device and electronic device based on emotion

Abstract

The present invention provides a method, device and electronic device for digital human interaction based on emotion, and relates to the field of artificial intelligence technology. The method comprises: obtaining interactive voice and interactive video; in the process of obtaining interactive voice, determining multiple sub-voices and the voice number of each sub-voice in turn from the interactive voice based on semantic integrity; allocating an idle processing thread to each sub-voice, fusing the sub-voice and sub-video through the processing thread to obtain the sub-emotion corresponding to each sub-voice; determining the interactive emotion and target thread based on the determination time of the sub-emotion and the voice number; controlling the digital human to make an interactive action matching the interactive emotion, and adjusting the state of the target thread to an idle state. In the process of user inputting voice, the present invention can quickly and accurately determine the interactive emotion to respond to the user based on the fusion of sub-voices and sub-videos by multiple processing threads, improve the naturalness of the digital human when interacting with the user, and make the digital human more humanized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an emotion-based digital human interaction method, device and electronic equipment. Background Art

[0002] Digital humans are digital human images created through digital technology, artificial intelligence technology, virtual reality and other technologies. With the rapid development of artificial intelligence technology, digital humans have demonstrated strong interactive communication capabilities and can communicate fluently with users to provide personalized services.

[0003] However, when digital humans communicate with users, they often need to wait for the user to complete the entire voice input before conducting an overall analysis and responding. This method limits the immediacy and naturalness of the conversation, making it difficult for users to obtain instant feedback comparable to real conversations during the conversation, resulting in the digital humans not being natural enough when interacting with users and having a low degree of anthropomorphism. Summary of the invention

[0004] In view of the above problems, the purpose of the present invention is to provide an emotion-based digital human interaction method, device and electronic device, which can improve the naturalness of the interaction between the digital human and the user and make the digital human more humanized.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] On the one hand, an embodiment of the present invention provides an emotion-based digital human interaction method, comprising:

[0007] Acquire interactive voice and interactive video, wherein the interactive voice is the voice input by the interactive object, and the acquisition time of the interactive video and the interactive voice is consistent;

[0008] In the process of acquiring the interactive speech, a plurality of sub-speech and a speech number of each sub-speech are sequentially determined from the interactive speech based on semantic integrity, wherein the speech number represents the determination order of the sub-speech;

[0009] Allocate an idle processing thread to each of the sub-voices, fuse the sub-voices and sub-videos through the processing thread to obtain the sub-emotion corresponding to each sub-voice, and the acquisition time of the sub-voices and the sub-videos is consistent;

[0010] Determine the interactive emotion and the target thread based on the determination time of the sub-emotion and the voice number;

[0011] The digital human is controlled to perform an interactive action matching the interactive emotion, and the state of the target thread is adjusted to an idle state.

[0012] Optionally, in the process of acquiring the interactive speech, determining a plurality of sub-speech and a speech number of each sub-speech from the interactive speech in sequence based on semantic integrity includes:

[0013] In the process of acquiring the interactive speech, if the interactive speech changes, performing semantic integrity detection on the interactive speech;

[0014] If the interactive speech is semantically complete, the interactive speech is used as a sub-speech, and the interactive speech obtained after the sub-speech is used to return to the step of performing semantic integrity detection on the interactive speech if the interactive speech changes, until the interactive speech is inputted;

[0015] Obtaining the maximum number among the existing voice numbers, and determining the voice number of the sub-voice by using the maximum number;

[0016] If the semantics of the interactive speech is incomplete, the process returns to the step of performing semantic integrity detection on the interactive speech if the interactive speech changes, until the interactive speech is completely input.

[0017] Optionally, allocating an idle processing thread to each of the sub-voices, and fusing the sub-voices and sub-videos through the processing threads to obtain the sub-emotion corresponding to each sub-voice, includes:

[0018] Selecting a preset thread in an idle state from a preset thread pool as a processing thread for the sub-voice;

[0019] For each of the processing threads, using the sub-video, converting the sub-speech into a first image;

[0020] Using the sub-speech, converting the sub-video into a first speech;

[0021] The first image, the sub-video, the sub-voice and the first voice are integrated to determine a sub-emotion corresponding to the sub-voice.

[0022] Optionally, the converting the sub-voice into a first image by using the sub-video includes:

[0023] Performing speech recognition and emotion recognition on the sub-speech to obtain speech text and speech emotion corresponding to the sub-speech;

[0024] Performing emotion recognition on the speech text to determine the text emotion corresponding to the speech text;

[0025] The speech emotion and the text emotion are integrated to determine the target speech emotion;

[0026] Acquire an object image of the interactive object from the sub-video;

[0027] The object image is adjusted using the target speech emotion to generate a first image.

[0028] Optionally, the converting the sub-video into a first voice by using the sub-voice includes:

[0029] For each frame of the image in the sub-video, performing feature extraction processing on the image to obtain object features of the interactive object;

[0030] Performing clustering processing on multiple frames of images in the sub-video using the object features to obtain multiple clusters;

[0031] The cluster containing the largest number of images is determined as the target cluster;

[0032] For each frame of the image in the target cluster, a description text corresponding to the image is generated by using an image-to-text model, wherein the description text includes an emotional description of the interactive object;

[0033] The sub-speech is adjusted using the emotion description to generate a first speech.

[0034] Optionally, the fusing the first image, the sub-video, the sub-voice and the first voice to determine the sub-emotion corresponding to the sub-voice includes:

[0035] Calculating the similarity between the first image and each frame image in the sub-video based on the object feature of the interactive object;

[0036] If the number of target images is not less than a preset number, predicting a first probability corresponding to each preset emotion based on the first image, the target image is an image with a similarity greater than a preset similarity;

[0037] If the number of the target images is less than a preset number, predicting a first probability corresponding to each preset emotion based on the sub-video;

[0038] Fusing the speech features of the sub-speech and the first speech to predict a second probability corresponding to each preset emotion;

[0039] A sub-emotion is determined from a plurality of preset emotions using the first probability and the second probability of each preset emotion.

[0040] Optionally, the determining a sub-emotion from a plurality of preset emotions by using the first probability and the second probability of each preset emotion comprises:

[0041] For each preset emotion, weighting the first probability using a first weight to obtain a first target probability;

[0042] Performing weighted processing on the second probability using a second weight to obtain a second target probability;

[0043] Sum the first target probability and the second target probability to obtain a probability of a preset emotion to be used;

[0044] According to the probability of each preset emotion to be used, a sub-emotion is determined from a plurality of preset emotions.

[0045] Optionally, determining the interactive emotion and the target thread based on the determination time of the sub-emotion and the voice number includes:

[0046] The sub-emotion and the voice number corresponding to the sub-voice are combined to generate combined data and write it into the specified queue, wherein the writing time of the combined data is related to the determination time of the sub-emotion;

[0047] If it is detected that the designated queue is not empty, taking the combined data from the head of the designated queue, determining the sub-emotion in the combined data as the interactive emotion, and taking the voice number in the combined data as the designated number;

[0048] Acquire a voice number smaller than the specified number as the target number;

[0049] The combined data including the target number is deleted from the designated queue, and the processing thread for processing the sub-voice corresponding to the target number is determined as the target thread.

[0050] On the other hand, an embodiment of the present invention further provides an emotion-based digital human interaction device, which is used to implement any of the above methods, and the device includes:

[0051] An acquisition module, used to acquire interactive voice and interactive video, wherein the interactive voice is the voice input by the interactive object, and the acquisition time of the interactive video and the interactive voice is consistent;

[0052] A sub-speech determination module is used to determine a plurality of sub-speech and a speech number of each sub-speech from the interactive speech in sequence based on semantic integrity during the process of acquiring the interactive speech, wherein the speech number represents the determination order of the sub-speech;

[0053] A sub-emotion determination module, used for allocating an idle processing thread to each sub-voice, fusing the sub-voice and the sub-video through the processing thread to obtain the sub-emotion corresponding to each sub-voice, wherein the acquisition time of the sub-voice and the sub-video is consistent;

[0054] An interaction determination module, used to determine the interaction emotion and the target thread based on the determination time of the sub-emotion and the voice number;

[0055] The interactive control module is used to control the digital human to make interactive actions matching the interactive emotions and adjust the state of the target thread to an idle state.

[0056] On the other hand, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute the steps in any one of the emotion-based digital human interaction methods provided in the embodiments of the present invention.

[0057] On the other hand, an embodiment of the present invention further provides a computer-readable storage medium, which stores multiple instructions, and the instructions are suitable for a processor to load to execute the steps in any one of the emotion-based digital human interaction methods provided by the embodiments of the present invention.

[0058] On the other hand, an embodiment of the present invention further provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of any one of the emotion-based digital human interaction methods provided in the embodiments of the present invention.

[0059] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0060] The present invention can obtain interactive voice and interactive video, and in the process of obtaining interactive voice, determine the sub-voice and voice number from the interactive voice in turn based on semantic integrity, so as to timely find the semantically complete sub-voice from the longer voice input by the user, and then allocate a processing thread to the sub-voice, and fuse the sub-voice and sub-video through the processing thread to quickly and accurately determine the sub-emotion, and then use the determination time of the sub-emotion and the voice number to determine the interactive emotion and the target thread, control the digital human to make interactive actions matching the interactive emotion, and adjust the target thread to an idle state to ensure timely response during the user input voice, enhance the naturalness of the interaction, and thus make the digital human more humanized and the interactive experience better. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 is a schematic diagram of an application scenario of the emotion-based digital human interaction method provided by an embodiment of the present invention;

[0063] Figure 2is a flow chart of an emotion-based digital human interaction method provided by an embodiment of the present invention;

[0064] Figure 3 It is a schematic diagram of a process for determining a sub-voice and a voice number provided by an embodiment of the present invention;

[0065] Figure 4 is a schematic diagram of a process for determining a sub-emotion provided by an embodiment of the present invention;

[0066] Figure 5 is a schematic diagram of the structure of an emotion-based digital human interaction device provided by an embodiment of the present invention;

[0067] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0069] With the rapid development of artificial intelligence technology, digital humans have gradually penetrated into people's life scenes. As intelligent customer service, digital humans can provide consulting services to users through dialogue. Usually, in addition to providing professional consulting services in specific fields, these digital humans also have basic chatting capabilities and can chat with users.

[0070] However, the inventors found in their research that, when talking to a digital human, the digital human usually listens to the user's voice with a fixed expression and posture, and will only respond after the user's voice input is completed. Although the image of the digital human is very close to that of a real human, the unnatural conversation still makes the user feel like they are talking to a machine, resulting in the interaction effect not being vivid and natural, and the degree of anthropomorphism is low. Therefore, the present invention proposes an emotion-based digital human interaction method, device, and electronic device to improve the naturalness of the interaction between the digital human and the user, making the digital human more anthropomorphic.

[0071] It will be understood that in the specific implementation of the present invention, data related to users, such as interactive voice, interactive video, etc., require user permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0072] See also Figure 1, shows a schematic diagram of an application scenario of the emotion-based digital human interaction method. The application scenario may include a terminal 101 and a server 102, and data can be exchanged between the terminal 101 and the server 102 through a network, and a digital human can be displayed on the terminal 101. The terminal 101 may be a mobile phone, a tablet computer, a smart Bluetooth device, a computer, a large screen, etc.; the server 102 may be a single server or a server cluster consisting of multiple servers.

[0073] Among them, the terminal 101 can be provided with a voice collector and an image collector, and the interactive object can interact with the digital human through the terminal 101, that is, the terminal 101 can obtain the interactive voice input by the interactive object and collect the interactive video of the interactive object.

[0074] The terminal 101 can send the collected interactive voice and interactive video to the server 102 in real time, so that the server 102 can obtain the interactive voice and interactive video in real time. In the process of obtaining the interactive voice, the server 102 determines multiple sub-voices and the voice number of each sub-voice from the interactive voice in turn based on semantic integrity, wherein the voice number represents the determination order of the sub-voices; allocates an idle processing thread to each sub-voice, and fuses the sub-voice and sub-video through the processing thread to obtain the sub-emotion corresponding to the sub-voice; determines the interactive emotion and the target thread based on the determination time of the sub-emotion and the voice number; and then controls the digital human in the terminal 101 to make interactive actions matching the interactive emotion, and at the same time, the state of the target thread can be adjusted to an idle state.

[0075] The interactive actions performed by the digital human on the terminal 101 can be regarded as responses to the interactive object, thereby achieving that when the interactive object inputs voice, the digital human can respond to the input voice in a timely manner.

[0076] In this embodiment, a digital human interaction method based on emotion is provided. Figure 2 As shown, the specific process of the emotion-based digital human interaction method can be as follows:

[0077] S110: Obtain interactive voice and interactive video.

[0078] Interactive speech refers to the speech input by the interactive object, which may be a user or other intelligent entity that communicates with the digital human. Interactive video refers to a set of images containing the interactive object, and each frame of the interactive video may include the face, limbs, etc. of the interactive object.

[0079] The collection time of interactive voice and interactive video is consistent, that is, the interactive video is the video containing the interactive object collected when the interactive object inputs the interactive voice. As an implementation method, the digital human can be provided with an image collector and a voice collector; the voice collector can be a microphone and other devices, and the image collector can be a camera and other devices. The voice collector can collect the voice input by the interactive object, and the image collector can collect the video within its collection range in real time, or collect the video when it detects that the interactive object appears within its collection range.

[0080] As an implementation method, the interactive object can input voice through the voice collector. When the user inputs voice, the time when the voice starts to be input can be recorded as the starting time of the interactive video; when the user stops inputting voice, the corresponding time is also recorded as the ending time of the interactive video. Using the starting time and the ending time, the interactive video can be captured from the video collected by the image collector.

[0081] As another implementation method, when the interactive object inputs voice through the voice collector, a start collection signal can be sent to the image collector through the server, and the image collector starts to collect the video of the interactive object; when the interactive object stops inputting voice, a stop collection signal can be sent to the image collector through the server, and the image collector stops collecting and continues to collect video, and the video that has been collected is the interactive video.

[0082] S120. In the process of acquiring the interactive speech, a plurality of sub-speech and a speech number of each sub-speech are determined in sequence from the interactive speech based on semantic integrity.

[0083] The interactive object inputting the interactive voice is a continuous process, and the digital human obtaining the interactive voice is also a continuous process. In the process of obtaining the interactive voice, multiple sub-voices and the voice number of each sub-voice can be determined from the interactive voice in sequence based on semantic integrity.

[0084] Among them, the sub-speech is a part of the speech separated from the interactive speech, and the semantics of each sub-speech are complete. The speech number is the number assigned to the sub-speech according to a certain rule. The speech number can be used to characterize the determination order of the sub-speech. In the embodiment of the present invention, the larger the value of the speech number, the larger the determination order of the sub-speech, that is, the later the sub-speech is determined. For example, for speech number 0 and speech number 1, the sub-speech of speech number 0 is determined earlier than the sub-speech of speech number 1, and the input time of the sub-speech of speech number 1 is closer to the current time.

[0085] In the process of acquiring interactive speech, as the interactive speech is input, the semantic integrity of the interactive speech can be detected at a certain frequency, so as to gradually determine multiple sub-speech from the interactive speech and then assign speech numbers to them.

[0086] As an implementation mode, when determining a sub-voice and a voice number of the sub-voice, it may be that in the process of acquiring the interactive voice, if the interactive voice changes, a semantic integrity check is performed on the interactive voice; if the semantics of the interactive voice are complete, the interactive voice is taken as a sub-voice, and the interactive voice acquired after the sub-voice is used to return to execute the step of performing a semantic integrity check on the interactive voice if the interactive voice changes, until the interactive voice is input; the maximum number among the existing voice numbers is obtained, and the voice number of the sub-voice is determined by the maximum number; if the semantics of the interactive voice are incomplete, the step of performing a semantic integrity check on the interactive voice if the interactive voice changes is returned to execute until the interactive voice is input.

[0087] In the process of acquiring interactive speech, as the interactive object gradually inputs interactive speech, the content of the interactive speech will change accordingly. Therefore, it is possible to detect in real time whether the interactive speech has changed. If the interactive speech has changed, the semantic integrity of the interactive speech is checked. Figure 3 , showing a flow chart of determining sub-voice and voice numbers.

[0088] As an implementation method, in order to detect the change of interactive speech in time, the detection frequency can be set, for example, the interactive speech is acquired for detection every 0.5 seconds. Automatic speech recognition (ASR) is used to convert the interactive speech into text, and the length of the text is calculated. When the length of the text is inconsistent with the previous text length, it is determined that the interactive speech has changed. The text length can be the number of characters in the text.

[0089] Then the interactive speech can be tested for semantic integrity. The interactive speech has been converted into text. At this time, the text can be directly input into the semantic detection model to determine whether the semantics of the interactive speech are complete. The semantic detection model can be pre-trained and have the ability to determine the semantic integrity of the input text; if the semantics are complete, the semantic detection model can output a "complete" result; if the semantics are incomplete, the semantic detection model can output an "incomplete" result.

[0090] If the result output by the semantic detection model is "complete", it indicates that the semantics of the interactive speech received at the current moment are complete, and the interactive speech can be directly used as a sub-speech, and then the subsequent interactive speech received is used as a new interactive speech, and the step of detecting the semantic integrity of the interactive speech is returned until the interactive speech input is completed. The interactive speech is "Today's sunset is so beautiful, it is a perfect day". When "Today's sunset is so beautiful" is detected, it has been used as a sub-speech, and the semantic integrity of "It is a perfect day" is subsequently detected.

[0091] After the sub-voice is determined, the corresponding voice number needs to be assigned to the sub-voice. Since the voice number represents the order of determination of the sub-voice, the larger the number value, the later the sub-voice is determined. The maximum number of existing voice numbers can be obtained, and 1 is added to the maximum number as the voice number of the sub-voice. For example, the existing voice numbers include 0, 1, and 2, indicating that 3 sub-voices have been determined, and the maximum number is 3, then the number of the currently determined sub-voice is 3+1=4.

[0092] If the semantics of the interactive speech is incomplete, no other processing can be performed on the interactive speech, but the semantic integrity check can be performed when the interactive speech changes next time. For example, the current interactive speech is "Today's weather is great", which is semantically incomplete. Wait until the next interactive speech change is detected to be "Today's weather is great", and then perform semantic integrity check and subsequent steps until the interactive speech input is completed.

[0093] Among them, if the interactive voice does not change within the preset time, it can be determined that the interactive voice input is completed. The preset time can be set according to actual needs and is not specifically limited here.

[0094] S130, allocating an idle processing thread to each of the sub-voices, and fusing the sub-voices and sub-videos through the processing thread to obtain the sub-emotion corresponding to each sub-voice.

[0095] With the input of interactive speech, multiple sub-speech can be determined. In order to improve the efficiency of determining the sub-emotion corresponding to the sub-speech and realize real-time response, each sub-speech can be assigned its corresponding processing thread, and parallel processing can be achieved through multiple processing threads.

[0096] As an implementation method, a preset thread in an idle state may be selected from a preset thread pool as a processing thread for the sub-voice; for each processing thread, the sub-voice is converted into a first image using the sub-video; the sub-voice is converted into a first voice using the sub-voice; and the first image, the sub-video, the sub-voice and the first voice are integrated to determine the sub-emotion corresponding to the sub-voice.

[0097] The preset thread pool may include multiple preset threads, and the specific number of preset threads can be set based on the actual situation. When allocating processing threads to sub-voices, an idle preset thread can be selected from the thread pool as the processing thread. If there is no idle preset thread in the current thread pool, wait until there is an idle preset thread in the preset thread pool, and then allocate a processing thread to it.

[0098] After the sub-speech is assigned a processing thread, the sub-speech and sub-video can be fused through the processing thread to obtain the sub-emotion corresponding to the sub-speech. Since the sub-speech and sub-video are data of different modalities, in order to better fuse the data of these two modalities and improve the accuracy of subsequent determination of sub-emotions, the sub-speech and sub-video can be converted into each other and then fused to determine the sub-emotion. For example, see Figure 4 , showing a schematic diagram of the process of determining sub-emotions.

[0099] For example, the sub-voice may be converted into a first image using a sub-video so as to incorporate the features of the sub-voice into the sub-video. Optionally, the process may include: performing voice recognition and emotion recognition on the sub-voice to obtain a voice text and voice emotion corresponding to the sub-voice; performing emotion recognition on the voice text to determine a text emotion corresponding to the voice text; fusing the voice emotion and the text emotion to determine a target voice emotion; obtaining an object image of the interactive object from the video to be processed; and adjusting the object image using the target voice emotion to generate a first image.

[0100] Figure 4 In the example, the sub-speech can be converted into corresponding speech text by ASR technology, and the emotion recognition of the sub-speech can be performed to obtain the speech emotion carried in the sub-speech. The speech emotion recognition model can be pre-trained so that the speech emotion recognition model has the ability to analyze the timbre and pitch of the input speech to predict the emotion contained in the speech. Multiple preset emotions can be pre-set, and the speech emotion recognition model can obtain the prediction probability of each preset emotion, and finally the preset emotion with the highest prediction probability is used as the speech emotion. The sub-speech is input into the speech emotion recognition model to obtain the speech emotion corresponding to the sub-speech output by the speech emotion model.

[0101] By performing emotion recognition on speech text, the emotion corresponding to the semantics in the speech text can be obtained. Similarly, when performing emotion recognition on speech text, a text emotion recognition model can be pre-trained, and the speech text can be input into the emotion recognition model to predict the text emotion contained in the speech text.

[0102] Then the text emotion and the voice emotion are integrated to determine the target voice emotion. As an implementation method, if the text emotion and the voice emotion are consistent, the text emotion or the voice emotion can be directly used as the target voice emotion; if the text emotion and the voice emotion are inconsistent, the voice emotion is used as the target voice emotion. For example, the user inputs the voice "Today's weather is so nice" in a relatively low tone. At this time, the text emotion may be happy, but the voice emotion may be sad. Based on the analysis of human emotions, it can be known that the user's emotion is actually more sad at this time. Therefore, when the text emotion and the voice emotion are inconsistent, it is more accurate to use the voice emotion as the target voice emotion.

[0103] As another implementation, when predicting text emotion, the prediction probability corresponding to each preset emotion is obtained, which can be recorded as the text prediction probability; when predicting speech emotion, the prediction probability corresponding to each preset emotion is obtained, which can be recorded as the speech prediction probability. Then different weights are set for speech emotion and text emotion respectively, and the prediction probability corresponding to each preset emotion is recalculated by weighted summation, and the preset emotion corresponding to the maximum prediction probability is used as the target speech emotion.

[0104] The acquisition time of the sub-video and the sub-voice is consistent. After the sub-voice is determined, the start time and the end time of the sub-voice can be obtained, so as to extract the sub-video corresponding to the sub-voice from the interactive video. The sub-video may include multiple continuous frames of images of the interactive object, and the object image corresponding to the interactive object can be obtained from the sub-video. For example, an image containing the interactive object can be randomly selected from the multiple frames of the sub-video as the object image. For another example, an image with a clear front face and body posture can be selected from the multiple frames as the object image.

[0105] Then, the target voice emotion can be used to adjust the object image to obtain the first image that matches the target voice emotion. Under different emotions, humans will show different expressions, which can include facial expressions and body postures. For example, when happy, the body is relaxed, the corners of the mouth are raised, and there may be a smile, while the hands may droop naturally; for example, when sad, the body may lean forward, lower the head and droop the shoulders. In order to make the posture and expression of the interactive object in the first image more matched with the target voice emotion, the expression can be adjusted by adjusting the curvature of the facial features of the interactive object in the object image, such as the curvature of the lips and eyes, and the posture can be adjusted by adjusting the placement of the hands.

[0106] The sub-video can be converted into the first voice by using the sub-voice, so as to incorporate the features of the sub-video into the sub-voice. Optionally, for each frame of the sub-video, feature extraction processing is performed on the image to obtain the object features of the interactive object; the multiple frames of the sub-video are clustered using the object features to obtain multiple clusters; the cluster containing the largest number of images is determined as the target cluster; for each frame of the target cluster, a description text corresponding to the image is generated using a graph-to-text model, and the description text includes an emotional description of the interactive object; the sub-voice is adjusted using the emotional description to generate the first voice.

[0107] The sub-video may include multiple frames of images, and for each frame of the sub-video, feature extraction processing may be performed on the image to extract object features of the interactive object from the image. It is understandable that the object features here may include features of facial features and features of body posture. Optionally, a feature extraction model may be trained in advance with a large amount of training data to extract object features from the image.

[0108] Using the object features extracted from the image, multiple frames of images can be clustered to obtain multiple clusters. There are many ways of clustering, and a suitable clustering method can be selected according to actual needs, such as K-Means clustering. Among the multiple clusters obtained, the number of images contained in each cluster is usually different. By counting the number of images in each cluster, the cluster corresponding to the maximum number of images is determined as the target cluster.

[0109] For each frame of the image in the target cluster, a description text corresponding to the image is generated using a graph-to-text model, and the description text may include an emotional description of the interactive object in the image. The graph-to-text model may be obtained in advance through a large amount of data training, and finally the emotional description corresponding to each frame of the image in the target cluster is used to adjust the sub-speech to generate the first speech.

[0110] The emotions corresponding to the emotional descriptions of the multiple frames of images in the target cluster may be different. Each emotional description can be classified according to the preset emotions, and the preset emotions with the largest number can be selected as the adjusted emotions. Then, the voice features of the sub-speech can be adjusted to generate and adjust the first speech that matches the emotion. The voice features can include pitch, speech speed, pauses, intonation, etc. Optionally, before adjusting the voice features of the sub-speech, an association relationship between each preset emotion and the preset voice feature can be established in advance, and then the preset voice features corresponding to the adjusted emotion can be determined based on the mapping relationship, and then the voice features of the sub-speech can be adjusted to the preset voice features.

[0111] The first image is generated using sub-voice, and the first voice is generated using sub-video. Data of different modalities may also have errors in modal conversion. In order to improve the accuracy of determining sub-emotion, the first image and sub-video can be combined to predict emotion in the image dimension. At the same time, the sub-voice and the first voice can be combined to predict emotion in the voice dimension. Finally, the two prediction results are combined to determine the final sub-emotion.

[0112] Optionally, when determining the sub-emotion corresponding to the sub-voice, it can be: based on the object features of the interactive object, calculating the similarity between the first image and each frame image in the sub-video; if the number of target images is not less than a preset number, predicting a first probability corresponding to each preset emotion based on the first image, the target image is an image with a similarity greater than a preset similarity; if the number of target images is less than a preset number, predicting a first probability corresponding to each preset emotion based on the sub-video; fusing the voice features of the sub-voice and the first voice to predict a second probability corresponding to each preset emotion; and determining the sub-emotion from multiple preset emotions using the first probability and the second probability of each preset emotion.

[0113] The similarity between the first image and each frame image in the sub-video is calculated by using the object features of the interactive object. The similarity may refer to cosine similarity, and the greater the cosine similarity, the more similar the first image and a frame image in the sub-video are.

[0114] The preset similarity is a value set in advance according to actual needs. After calculating the similarity between the first image and each frame image in the sub-video, the similarity corresponding to each frame image can be compared with the preset similarity. If the similarity corresponding to a frame image is greater than the preset similarity, the frame image can be used as the target image.

[0115] The preset number may be determined based on the total number of images in the sub-video, that is, the preset number used in different sub-videos may be different. Optionally, the total number of images in the sub-video may be obtained, and the total number of images may be divided by 2 to obtain a critical number, and the preset number may be set to any number greater than the critical number and less than the total number of images.

[0116] If the number of target images is greater than the preset number, it indicates that most of the images in the sub-video are similar to the target image, and the object features in the first image can represent the entire sub-video. In order to save computing resources and speed up the computing process, the first image can be directly used to predict the first probability corresponding to each preset emotion. Optionally, an image emotion recognition model can be pre-trained, and the first image can be input into the image emotion recognition model. The image emotion recognition model can analyze the first image to predict the first probability of each preset emotion.

[0117] If the number of target images is not greater than the preset number, it indicates that most of the images in the sub-video are not similar to the target image, and the object features in the first image cannot represent the entire sub-video. To ensure the accuracy of recognition, the sub-video may be used to predict the first probability corresponding to each preset emotion. Optionally, a video emotion recognition model may be pre-trained, and the sub-video may be input into the video emotion recognition model. The video emotion recognition model may analyze the sub-video to predict the first probability of each preset emotion.

[0118] After the sub-speech is fused with the first speech, the second probability corresponding to each preset emotion can be predicted. As an implementation method, the sub-speech can be subjected to feature extraction processing to obtain the first speech feature; the first speech can be subjected to feature extraction processing to obtain the second speech feature; the first speech feature and the second speech feature can be fused to obtain the speech feature to be used; the probability of each preset emotion is calculated using the speech feature to be used to obtain the second probability. It should be noted that when the first speech and the sub-speech are subjected to feature extraction processing, the same feature extraction method should be used, and when the first speech feature and the second speech feature are fused, the fusion can be performed by splicing, weighting, etc.

[0119] When determining a sub-emotion from multiple preset emotions using the first probability and the second probability of each preset emotion, for each preset emotion, the first probability may be weighted using a first weight to obtain a first target probability; the second probability may be weighted using a second weight to obtain a second target probability; the first target probability and the second target probability may be summed to obtain the probability to be used of the preset emotion; and the sub-emotion may be determined from multiple preset emotions based on the probability to be used of each preset emotion.

[0120] Based on the foregoing content, it can be known that each preset emotion corresponds to a first probability and a second probability, the first weight is the preset weight of the first probability, and the second probability is the preset weight of the second probability. The first weight and the second weight can be set according to actual needs, and the sum of the first weight and the second weight is 1.

[0121] The first target probability of the preset emotion can be obtained by multiplying the first weight and the first probability, and the second target probability of the preset emotion can be obtained by multiplying the second weight and the second probability. Then, the first target probability of the preset emotion and the second target probability of the preset emotion are added together to obtain the probability to be used of the preset emotion.

[0122] According to this method, the probability of use corresponding to each preset emotion can be calculated, and then the preset emotion with the highest probability of use can be determined as the sub-emotion.

[0123] S140. Determine the interactive emotion and the target thread based on the determination time of the sub-emotion and the voice number.

[0124] As mentioned above, multiple processing threads simultaneously process different sub-voices and sub-videos to output sub-emotions corresponding to the sub-voices. Different sub-voices and sub-videos have different processing times, resulting in different times for different processing threads to determine sub-emotions. The determination of sub-emotions and voice numbers can be used to determine interactive emotions and target threads.

[0125] Among them, interactive emotion refers to the emotion that the digital person has when interacting with the interactive object during the process of interactive voice input, and the target thread is the processing thread that needs additional processing later.

[0126] Optionally, when determining the interactive emotion and the target thread, the sub-emotion and voice number corresponding to the combined sub-voice can be generated, and the combined data is written into the designated queue, and the writing time of the combined data is related to the determination time of the sub-emotion; if it is detected that the designated queue is not empty, the combined data is taken out from the head of the designated queue, and the sub-emotion in the combined data is determined as the interactive emotion, and the voice number in the combined data is used as the designated number; the voice number less than the designated number is obtained as the target number; the combined data containing the target number is deleted from the designated queue, and the processing thread for processing the sub-voice corresponding to the target number is determined as the target thread.

[0127] Sub-emotion and sub-speech are associated, and sub-speech has a corresponding voice number. The sub-emotion and voice number of the sub-speech can be combined to generate combined data, and the generated combined data can be written into the specified queue. It should be noted that the earlier the sub-emotion is determined, the earlier the combined data is generated and written into the specified queue, that is, the writing time of the combined data is positively correlated with the determination time of the sub-emotion.

[0128] The data in the specified queue can be detected in real time. When it is detected that the specified queue is not an empty queue, it indicates that the combined data has been written into the specified queue. The specified queue is a first-in-first-out queue. At this time, the first combined data taken from the head of the specified queue is the earliest combined data written into the specified queue, that is, the sub-emotion in the combined data is the first to be determined. The sub-emotion in the combined data can be used as the interactive emotion, the voice number in the combined data can be used as the specified number, and the voice number less than the specified number can be used as the target number.

[0129] It should be noted that the larger the voice number, the newer the corresponding sub-voice. In order to ensure that the digital human can respond to the latest sub-voice input by the interactive object in a timely manner, it is not necessary to respond to the sub-voice corresponding to the target number. Therefore, the combined data containing the target number in the specified queue can be deleted, and the processing thread that processes the sub-voice corresponding to the target number is determined as the target thread for subsequent processing.

[0130] For example, there are 6 sub-speech numbers from 0 to 5. If the combined data of numbers 4, 5, and 1 are written into the current designated queue in sequence, the combined data of number 4 is taken out from the head of the queue, and the sub-emotion in the combined data of number 4 is the interactive emotion, number 4 is the designated number, and numbers 0 to 3 are all target numbers. At this time, the combined data of number 1 needs to be deleted from the designated queue, and the processing thread for processing the sub-speech corresponding to numbers 0 to 3 is the target thread.

[0131] S150, controlling the digital human to perform an interactive action that matches the interactive emotion, and adjusting the state of the target thread to an idle state.

[0132] After determining the interactive emotion and the target thread, the digital human can be controlled to make interactive actions that match the interactive emotion. In order to save computing resources, the state of the target thread can be adjusted to an idle state. It should be noted that the processing thread usually automatically updates its state to an idle state after determining the sub-emotion. At this time, it is only necessary to adjust the state of the target thread that is still processing the sub-voice and sub-video, terminate the data processing being executed on the target thread, and make it an idle thread in order to process new sub-voices and sub-videos.

[0133] Among them, a mapping relationship between preset emotions and preset actions can be pre-set. After the interactive emotion is determined, the preset action corresponding to the interactive emotion can be determined based on the mapping relationship and used as the interactive action. It should be noted that the interactive action here may include at least one of facial expressions and body movements. For example, when the interactive emotion is happy, the interactive action can be a smiling expression, and when the interactive emotion is sad, the interactive action can be an action of covering the face and crying.

[0134] Among them, the number of threads in the thread pool is usually limited. After determining the sub-voice, if there is no idle processing thread in the thread pool, it is necessary to wait for the idle processing thread to process again. Waiting will reduce the efficiency of determining the sub-emotion. Every time the combined data is taken out from the specified queue, it will trigger the step of determining the target thread and adjusting the state of the target thread to the idle state, so as to release the idle processing thread in time, reduce the waiting time, and improve the efficiency of determining the sub-emotion, so that the digital human can respond in real time when the interactive object inputs the interactive voice.

[0135] The emotion-based digital human interaction method provided by the embodiment of the present invention can be used in a variety of scenarios, for example, companion digital humans, intelligent question-and-answer digital humans, etc. When the user inputs voice, the digital human can imitate human conversation and respond to the acquired voice, making the conversation more natural.

[0136] The emotion-based digital human interaction method provided by the embodiment of the present invention can respond to the interactive object in a timely manner during the process of the interactive object inputting the interactive voice, without waiting for the interactive object to complete the voice input before responding. Such an interactive method is more natural and humanized, and has a better interactive effect. By converting the sub-voice and sub-video modally and then comprehensively analyzing the sub-emotion, the response of the digital human can be more in line with the emotion of the interactive object. In order to achieve real-time response, multiple processing threads and setting an idle state for the target thread can ensure that there are enough idle threads in the thread pool to process the latest sub-voice, thereby ensuring real-time response, improving the naturalness of interaction with the digital human, and making the interaction of the digital human more humanized.

[0137] In order to better implement the above method, the embodiment of the present invention also provides an emotion-based digital human interaction device, which can be integrated into an electronic device, and the electronic device can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0138] For example, in this embodiment, the method of the embodiment of the present invention is described in detail by taking the example of the specific integration of the emotion-based digital human interaction device in the server.

[0139] For example, Figure 5 As shown, the emotion-based digital human interaction device 200 may include an acquisition module 210, a sub-voice determination module 220, a sub-emotion determination module 230, an interaction determination module 240, and an interaction control module 250.

[0140] An acquisition module 210 is used to acquire interactive voice and interactive video, wherein the interactive voice is the voice input by the interactive object, and the acquisition time of the interactive video and the interactive voice is consistent;

[0141] A sub-speech determination module 220 is used to determine a plurality of sub-speech and a speech number of each sub-speech from the interactive speech in sequence based on semantic integrity during the process of acquiring the interactive speech, wherein the speech number represents the determination order of the sub-speech;

[0142] The sub-emotion determination module 230 is used to allocate an idle processing thread to each of the sub-voices, and fuse the sub-voices and sub-videos through the processing thread to obtain the sub-emotion corresponding to each sub-voice, and the acquisition time of the sub-voices and the sub-videos is consistent;

[0143] An interaction determination module 240, configured to determine an interaction emotion and a target thread based on the determination time of the sub-emotion and the voice number;

[0144] The interaction control module 250 is used to control the digital human to perform interaction actions that match the interaction emotion and adjust the state of the target thread to an idle state.

[0145] In some embodiments, the sub-voice determination module 220 is specifically configured to:

[0146] In the process of acquiring the interactive speech, if the interactive speech changes, performing semantic integrity detection on the interactive speech;

[0147] If the interactive speech is semantically complete, the interactive speech is used as a sub-speech, and the interactive speech obtained after the sub-speech is used to return to the step of performing semantic integrity detection on the interactive speech if the interactive speech changes, until the interactive speech is inputted;

[0148] Obtaining the maximum number among the existing voice numbers, and determining the voice number of the sub-voice by using the maximum number;

[0149] If the semantics of the interactive speech is incomplete, the process returns to the step of performing semantic integrity detection on the interactive speech if the interactive speech changes, until the interactive speech is completely input.

[0150] In some embodiments, the sub-emotion determination module 230 is specifically used to:

[0151] Selecting a preset thread in an idle state from a preset thread pool as a processing thread for the sub-voice;

[0152] For each of the processing threads, using the sub-video, converting the sub-speech into a first image;

[0153] Using the sub-speech, converting the sub-video into a first speech;

[0154] The first image, the sub-video, the sub-voice and the first voice are integrated to determine a sub-emotion corresponding to the sub-voice.

[0155] In some embodiments, the sub-emotion determination module 230 is specifically used to:

[0156] Performing speech recognition and emotion recognition on the sub-speech to obtain speech text and speech emotion corresponding to the sub-speech;

[0157] Performing emotion recognition on the speech text to determine the text emotion corresponding to the speech text;

[0158] The speech emotion and the text emotion are integrated to determine the target speech emotion;

[0159] Acquire an object image of the interactive object from the sub-video;

[0160] The object image is adjusted using the target speech emotion to generate a first image.

[0161] In some embodiments, the sub-emotion determination module 230 is specifically used to:

[0162] For each frame of the image in the sub-video, performing feature extraction processing on the image to obtain object features of the interactive object;

[0163] Performing clustering processing on multiple frames of images in the sub-video using the object features to obtain multiple clusters;

[0164] The cluster containing the largest number of images is determined as the target cluster;

[0165] For each frame of the image in the target cluster, a description text corresponding to the image is generated by using an image-to-text model, wherein the description text includes an emotional description of the interactive object;

[0166] The sub-speech is adjusted using the emotion description to generate a first speech.

[0167] In some embodiments, the sub-emotion determination module 230 is specifically used to:

[0168] Calculating the similarity between the first image and each frame image in the sub-video based on the object feature of the interactive object;

[0169] If the number of target images is not less than a preset number, predicting a first probability corresponding to each preset emotion based on the first image, the target image is an image with a similarity greater than a preset similarity;

[0170] If the number of the target images is less than a preset number, predicting a first probability corresponding to each preset emotion based on the sub-video;

[0171] Fusing the speech features of the sub-speech and the first speech to predict a second probability corresponding to each preset emotion;

[0172] A sub-emotion is determined from a plurality of preset emotions using the first probability and the second probability of each preset emotion.

[0173] In some embodiments, the sub-emotion determination module 230 is specifically used to:

[0174] For each preset emotion, weighting the first probability using a first weight to obtain a first target probability;

[0175] Performing weighted processing on the second probability using a second weight to obtain a second target probability;

[0176] Sum the first target probability and the second target probability to obtain a probability of a preset emotion to be used;

[0177] According to the probability of each preset emotion to be used, a sub-emotion is determined from a plurality of preset emotions.

[0178] In some embodiments, the interaction determination module 240 is specifically configured to:

[0179] The sub-emotion and the voice number corresponding to the sub-voice are combined to generate combined data and write it into the specified queue, wherein the writing time of the combined data is related to the determination time of the sub-emotion;

[0180] If it is detected that the designated queue is not empty, taking the combined data from the head of the designated queue, determining the sub-emotion in the combined data as the interactive emotion, and taking the voice number in the combined data as the designated number;

[0181] Acquire a voice number smaller than the specified number as the target number;

[0182] The combined data including the target number is deleted from the designated queue, and the processing thread for processing the sub-voice corresponding to the target number is determined as the target thread.

[0183] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0184] From the above, it can be seen that the emotion-based digital human interaction device of this embodiment can obtain interactive voice and interactive video, and in the process of obtaining interactive voice, determine the sub-voice and voice number from the interactive voice in turn based on semantic integrity, so as to timely find the semantically complete sub-voice from the longer voice input by the user, and then allocate a processing thread to the sub-voice, and fuse the sub-voice and sub-video through the processing thread to quickly and accurately determine the sub-emotion, and then use the determination time of the sub-emotion and the voice number to determine the interactive emotion and target thread, control the digital human to make interactive actions that match the interactive emotion, and adjust the target thread to an idle state to ensure timely response during the user input voice, enhance the naturalness of the interaction, and make the digital human more humanized and the interactive experience better.

[0185] The embodiment of the present invention further provides an electronic device, which may be a terminal, a server, or the like. The terminal may be a mobile phone, a tablet computer, a smart Bluetooth device, a notebook computer, a personal computer, or the like; the server may be a single server or a server cluster composed of multiple servers, or the like.

[0186] In some embodiments, the emotion-based digital human interaction device can also be integrated into multiple electronic devices. For example, the emotion-based digital human interaction device can be integrated into multiple servers, and the emotion-based digital human interaction method of the present invention can be implemented by multiple servers.

[0187] In this embodiment, the electronic device of this embodiment is a server as an example for detailed description, for example, Figure 6 As shown, it shows a schematic diagram of the structure of an electronic device involved in an embodiment of the present invention, specifically:

[0188] The electronic device may include components such as a processor 310 with one or more processing cores, a memory 320 with one or more computer-readable storage media, a power supply 330, an input module 340, and a communication module 350. Those skilled in the art will appreciate that Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0189] The processor 310 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 320, and calling data stored in the memory 320. In some embodiments, the processor 310 may include one or more processing cores; in some embodiments, the processor 310 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 310.

[0190] The memory 320 can be used to store software programs and modules. The processor 310 executes various functional applications and data processing by running the software programs and modules stored in the memory 320. The memory 320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 320 may also include a memory controller to provide the processor 310 with access to the memory 320.

[0191] The electronic device also includes a power supply 330 for supplying power to various components. In some embodiments, the power supply 330 can be logically connected to the processor 310 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 330 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0192] The electronic device may further include an input module 340, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0193] The electronic device may further include a communication module 350. In some embodiments, the communication module 350 may include a wireless module. The electronic device may perform short-range wireless transmission through the wireless module of the communication module 350, thereby providing the user with wireless broadband Internet access. For example, the communication module 350 may be used to help the user send and receive emails, browse web pages, and access streaming media.

[0194] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 310 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 320 according to the following instructions, and the processor 310 will run the application programs stored in the memory 320, thereby implementing the steps in the methods of the embodiments of the present invention.

[0195] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0196] From the above, it can be seen that the electronic device provided by the embodiment of the present invention can obtain interactive voice and interactive video, and in the process of obtaining interactive voice, determine the sub-voice and voice number from the interactive voice in turn based on semantic integrity, so as to timely find the semantically complete sub-voice from the longer voice input by the user, and then allocate a processing thread to the sub-voice, and fuse the sub-voice and sub-video through the processing thread to quickly and accurately determine the sub-emotion, and then use the determination time of the sub-emotion and the voice number to determine the interactive emotion and target thread, control the digital human to make interactive actions that match the interactive emotion, and adjust the target thread to an idle state to ensure timely response during the user input voice, enhance the naturalness of the interaction, and make the digital human more humanized and the interactive experience better.

[0197] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0198] To this end, an embodiment of the present invention provides a computer-readable storage medium, in which multiple instructions are stored, and the instructions can be loaded by a processor to execute the steps in any emotion-based digital human interaction method provided by the embodiment of the present invention.

[0199] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0200] According to one aspect of the present invention, a computer program product or computer program is provided, the computer program product or computer program including a computer program / instruction, the computer program / instruction being stored in a computer-readable storage medium. A processor of an electronic device reads the computer program / instruction from the computer-readable storage medium, and the processor executes the computer program / instruction, so that the electronic device executes the method provided in various optional implementations of the digital human interaction aspect or the model training aspect provided in the above-mentioned embodiments.

[0201] Since the instructions stored in the storage medium can execute the steps in any one of the emotion-based digital human interaction methods provided in the embodiments of the present invention, the beneficial effects that can be achieved by any one of the emotion-based digital human interaction methods provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0202] The above is a detailed introduction to an emotion-based digital human interaction method, device and electronic device provided by an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A digital human interaction method based on emotion, It is characterized in that The method comprises: Acquire interactive voice and interactive video, wherein the interactive voice is the voice input by the interactive object, and the acquisition time of the interactive video and the interactive voice is consistent; In the process of acquiring the interactive speech, a plurality of sub-speech and a speech number of each sub-speech are sequentially determined from the interactive speech based on semantic integrity, wherein the speech number represents the determination order of the sub-speech; Allocate an idle processing thread to each of the sub-voices, fuse the sub-voices and sub-videos through the processing thread to obtain the sub-emotion corresponding to each sub-voice, and the acquisition time of the sub-voices and the sub-videos is consistent; Determine the interactive emotion and the target thread based on the determination time of the sub-emotion and the voice number; Controlling the digital human to perform an interactive action matching the interactive emotion, and adjusting the state of the target thread to an idle state; The step of allocating an idle processing thread to each sub-voice, and fusing the sub-voice and the sub-video through the processing thread to obtain the sub-emotion corresponding to each sub-voice includes: A preset thread in an idle state is selected from a preset thread pool as a processing thread for the sub-voice; for each of the processing threads, the sub-voice is converted into a first image using the sub-video; the sub-voice is converted into a first voice using the sub-voice; the first image, the sub-video, the sub-voice and the first voice are integrated to determine the sub-emotion corresponding to the sub-voice.

2. The method according to claim 1, It is characterized in that In the process of acquiring the interactive speech, determining a plurality of sub-speech and a speech number of each sub-speech from the interactive speech in sequence based on semantic integrity includes: In the process of acquiring the interactive speech, if the interactive speech changes, performing semantic integrity detection on the interactive speech; If the interactive speech is semantically complete, the interactive speech is used as a sub-speech, and the interactive speech obtained after the sub-speech is used to return to the step of performing semantic integrity detection on the interactive speech if the interactive speech changes, until the interactive speech is inputted; Obtaining the maximum number among the existing voice numbers, and determining the voice number of the sub-voice by using the maximum number; If the semantics of the interactive speech is incomplete, the process returns to the step of performing semantic integrity detection on the interactive speech if the interactive speech changes, until the interactive speech is completely input.

3. The method according to claim 1, It is characterized in that The converting the sub-speech into a first image by using the sub-video includes: Performing speech recognition and emotion recognition on the sub-speech to obtain speech text and speech emotion corresponding to the sub-speech; Performing emotion recognition on the speech text to determine the text emotion corresponding to the speech text; The speech emotion and the text emotion are integrated to determine the target speech emotion; Acquire an object image of the interactive object from the sub-video; The object image is adjusted using the target speech emotion to generate a first image.

4. The method according to claim 1, It is characterized in that The converting the sub-video into a first voice by using the sub-voice includes: For each frame of the image in the sub-video, performing feature extraction processing on the image to obtain object features of the interactive object; Performing clustering processing on multiple frames of images in the sub-video using the object features to obtain multiple clusters; The cluster containing the largest number of images is determined as the target cluster; For each frame of the image in the target cluster, a description text corresponding to the image is generated by using an image-to-text model, wherein the description text includes an emotional description of the interactive object; The sub-speech is adjusted using the emotion description to generate a first speech.

5. The method according to claim 1, It is characterized in that The fusing the first image, the sub-video, the sub-voice and the first voice to determine the sub-emotion corresponding to the sub-voice includes: Calculating the similarity between the first image and each frame image in the sub-video based on the object feature of the interactive object; If the number of target images is not less than a preset number, predicting a first probability corresponding to each preset emotion based on the first image, the target image is an image with a similarity greater than a preset similarity; If the number of the target images is less than a preset number, predicting a first probability corresponding to each preset emotion based on the sub-video; Fusing the speech features of the sub-speech and the first speech to predict a second probability corresponding to each preset emotion; A sub-emotion is determined from a plurality of preset emotions using the first probability and the second probability of each preset emotion.

6. The method according to claim 5, It is characterized in that The method of determining a sub-emotion from a plurality of preset emotions by using the first probability and the second probability of each preset emotion comprises: For each preset emotion, weighting the first probability using a first weight to obtain a first target probability; Performing weighted processing on the second probability using a second weight to obtain a second target probability; Sum the first target probability and the second target probability to obtain a probability of a preset emotion to be used; According to the probability of each preset emotion to be used, a sub-emotion is determined from a plurality of preset emotions.

7. The method according to claim 1, It is characterized in that The determining of the interactive emotion and the target thread based on the determination time of the sub-emotion and the voice number includes: The sub-emotion and the voice number corresponding to the sub-voice are combined to generate combined data and write it into the specified queue, wherein the writing time of the combined data is related to the determination time of the sub-emotion; If it is detected that the designated queue is not empty, taking the combined data from the head of the designated queue, determining the sub-emotion in the combined data as the interactive emotion, and taking the voice number in the combined data as the designated number; Acquire a voice number smaller than the specified number as the target number; The combined data including the target number is deleted from the designated queue, and the processing thread for processing the sub-voice corresponding to the target number is determined as the target thread.

8. An emotion-based digital human interaction device, used to implement the emotion-based digital human interaction method according to any one of claims 1 to 7, It is characterized in that The device comprises: An acquisition module, used to acquire interactive voice and interactive video, wherein the interactive voice is the voice input by the interactive object, and the acquisition time of the interactive video and the interactive voice is consistent; A sub-speech determination module is used to determine a plurality of sub-speech and a speech number of each sub-speech from the interactive speech in sequence based on semantic integrity during the process of acquiring the interactive speech, wherein the speech number represents the determination order of the sub-speech; A sub-emotion determination module, used for allocating an idle processing thread to each sub-voice, fusing the sub-voice and the sub-video through the processing thread to obtain the sub-emotion corresponding to each sub-voice, wherein the acquisition time of the sub-voice and the sub-video is consistent; An interaction determination module, used to determine the interaction emotion and the target thread based on the determination time of the sub-emotion and the voice number; An interaction control module, used to control the digital human to make an interaction action matching the interaction emotion, and adjust the state of the target thread to an idle state; The step of allocating an idle processing thread to each sub-voice, and fusing the sub-voice and the sub-video through the processing thread to obtain the sub-emotion corresponding to each sub-voice includes: A preset thread in an idle state is selected from a preset thread pool as a processing thread for the sub-voice; for each of the processing threads, the sub-voice is converted into a first image using the sub-video; the sub-voice is converted into a first voice using the sub-voice; the first image, the sub-video, the sub-voice and the first voice are integrated to determine the sub-emotion corresponding to the sub-voice.

9. An electronic device, It is characterized in that It comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps in the emotion-based digital human interaction method as described in any one of claims 1-7.

Images