Digital human interaction control method and related equipment

Abstract

The embodiment of the invention provides a digital human interaction control method and related equipment. In the embodiment of the invention, in the interaction process of the digital human and the user, the emotion characteristics and the intention information of the user are identified based on the interaction information of the user through the intention identification model, and the multi-modal interaction feedback is carried out based on the emotion characteristics and the intention information of the user through the language model with the multi-modal interaction capability. Therefore, the interaction quality between the digital human and the user is improved, the more natural, smoother and emotional multi-mode interaction experience is realized, the changeable and subdivided personalized dialogue requirements of the user are effectively adapted, the interaction willingness of the user is improved, and particularly in a commodity live broadcast scene, the commodity purchase willingness of the user can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an interactive control method for a digital human and related equipment. Background Art

[0002] With the development of artificial intelligence technology, digital humans are widely used in various video playback scenarios, especially e-commerce live broadcast scenarios. The application of digital humans in e-commerce live broadcasts can not only provide innovative shopping experiences, but also help companies reduce costs, improve efficiency, and enhance user stickiness through interaction with users.

[0003] The interaction between digital humans and users often relies on manually written dialogue scripts, which define some common questions and their corresponding response scripts. Based on the dialogue scripts, digital humans can respond to common questions raised by users. If the questions raised by users do not belong to the common questions defined in the dialogue scripts, digital humans may not be able to correctly understand or respond to users. Current digital humans cannot adapt to users' changing and segmented personalized dialogue needs, which reduces users' willingness to interact and, in turn, their willingness to purchase goods. Summary of the invention

[0004] Multiple aspects of the present application provide a digital human interaction control method and related equipment to improve the quality of interaction between the digital human and the user, and to achieve a more natural, smooth and emotional multimodal interaction experience.

[0005] The embodiment of the present application provides an interactive control method for a digital human, which includes obtaining first interactive information initiated by a target user for the digital human; using an intention recognition model to identify the emotional characteristics and target intention information of the target user from the first interactive information; calling a language model with multimodal interaction capabilities according to the emotional characteristics and the target intention information to generate control information for controlling the digital human to perform multimodal interaction; and controlling the digital human to perform multimodal interaction with the target user according to the control information.

[0006] An embodiment of the present application provides an electronic device, comprising: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to execute the steps in the interactive control method of a digital human.

[0007] An embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps in the interactive control method of a digital human.

[0008] An embodiment of the present application provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is enabled to implement the steps in the interactive control method of a digital human.

[0009] In the embodiment of the present application, during the interaction between the digital human and the user, the user's emotional characteristics and intention information are identified based on the user's interactive information through the intention recognition model, and multimodal interactive feedback is performed based on the user's emotional characteristics and intention information through a language model with multimodal interaction capabilities. As a result, the quality of interaction between the digital human and the user is improved, a more natural, smooth and emotional multimodal interactive experience is achieved, and the user's changing and segmented personalized dialogue needs are effectively adapted, and the user's willingness to interact is improved, especially in the commodity live broadcast scene, which can effectively improve the user's willingness to purchase commodities. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0011] Figure 1 A flowchart of a digital human interactive control method provided in an embodiment of the present application;

[0012] Figure 2 A system architecture diagram of a digital human live broadcast system provided in an embodiment of the present application;

[0013] Figure 3 An exemplary digital human question-and-answer interaction flow chart provided in an embodiment of the present application;

[0014] Figure 4 A schematic diagram of the structure of an interactive control device for a digital human provided in an embodiment of the present application;

[0015] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0017] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the access relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. In the text description of the present application, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship. In addition, in the embodiments of the present application, "first", "second", "third", etc. are only used to distinguish the contents of different objects and have no other special meanings.

[0018] With the development of artificial intelligence technology, digital humans are widely used in various video playback scenarios, especially e-commerce live broadcast scenarios. The application of digital humans in e-commerce live broadcasts can not only provide innovative shopping experiences, but also help companies reduce costs, improve efficiency, and enhance user stickiness through interaction with users.

[0019] The interaction between digital humans and users often relies on manually written dialogue scripts, which define some common questions and their corresponding response scripts. Based on the dialogue scripts, digital humans can respond to common questions raised by users. If the questions raised by users do not belong to the common questions defined in the dialogue scripts, digital humans may not be able to correctly understand or respond to users. Current digital humans cannot adapt to users' changing and segmented personalized dialogue needs, which reduces users' willingness to interact and, in turn, their willingness to purchase goods.

[0020] To this end, the embodiment of the present application provides a method for interactive control of a digital human and related equipment. In the embodiment of the present application, during the interaction between the digital human and the user, the user's emotional characteristics and intention information are identified based on the user's interactive information through the intention recognition model, and multimodal interactive feedback is performed based on the user's emotional characteristics and intention information through a language model with multimodal interaction capabilities. As a result, the quality of interaction between the digital human and the user is improved, a more natural, smooth and emotional multimodal interactive experience is achieved, and the user's changing and segmented personalized dialogue needs are effectively adapted, and the user's willingness to interact is improved, especially in the commodity live broadcast scene, which can effectively improve the user's willingness to purchase commodities.

[0021] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The following is a detailed description of the technical solutions provided by each embodiment of the present application in conjunction with the accompanying drawings.

[0022] Figure 1 This is a flow chart of a digital human interactive control method provided in an embodiment of the present application. Figure 1, the method may include the following steps:

[0023] 101. Obtain the first interactive information initiated by the target user to the digital human.

[0024] 102. Use the intention recognition model to identify the target user's emotional characteristics and target intention information from the first interactive information.

[0025] 103. Calling a language model with multimodal interaction capabilities according to the emotional features and target intention information to generate control information for controlling the digital human to perform multimodal interaction. 104. Controlling the digital human to perform multimodal interaction with the target user according to the control information.

[0026] In this embodiment, the target user can interact with the digital human. For example, in a live broadcast scenario, the digital human can be a virtual anchor, etc. During the target user's viewing of the digital human live broadcast, the target user can interact with the digital human. For another example, in a care service scenario, the digital human can be a couple companion robot, which is a digital human specially designed for a partner relationship, aiming to enhance the interaction and emotional connection between couples. The target user can interact with the couple companion robot. For another example, in a customer service scenario, the digital human can be a customer service robot, and the target user can interact with the customer service robot.

[0027] In actual applications, the target user can initiate any form of interactive information to the digital human, and the interactive information includes but is not limited to: text information, voice information, emoticons, etc.

[0028] Text message: Users can communicate with digital humans by inputting text to ask for information, seek help, or have daily conversations. For example, the text message sent by the user can be, but is not limited to: "What's the weather like today?", "Help me check tomorrow's schedule", "What is the discount price of product A?", etc.

[0029] Emoticons: are an intuitive and expressive way of communicating that can be used to express emotion or enhance the meaning of a text message. For example, use a smiley face emoticon to express friendliness or a surprised emoticon to express surprise, etc.

[0030] Voice message: Users can communicate with digital humans through natural language. This method is suitable for situations where typing is inconvenient. Of course, in situations where typing is convenient, users can also communicate with digital humans through voice to give full play to the convenience and efficiency of voice communication. For example, the voice message sent by the user can be, but is not limited to: "Are there any good movies recently?", "Where is this product produced?", "Is there any discount or discount for this product?", etc.

[0031] In some scenarios, the interactive information between users and digital humans can be extended to various forms, including but not limited to bullet screen messages, comment information, etc. For example, when watching a live broadcast, users can send bullet screen messages or comment information through the input box.

[0032] For ease of understanding and distinction, any interactive information currently initiated by the target user is referred to as first interactive information. After obtaining the first interactive information of the target user, the intention recognition model is used to identify the emotional characteristics and intention information of the target user from the first interactive information.

[0033] In this embodiment, by introducing the intention recognition model, the emotional characteristics and target intention information of the user's first interactive information can be accurately recognized, thereby driving the digital human to make a more natural and personalized response. The intention recognition model can capture subtle emotions and needs from the user's language expression, ensuring that the human-computer dialogue is closely centered around the user's actual interests, greatly improving the depth of personalized dialogue and context perception capabilities.

[0034] In practical applications, the training data set is used to train the model and obtain the intent recognition model. The training data in the training data set includes input information and annotation results, and the annotation results include the emotional features and intent information of the input information. During model training, the input information is input into the intent recognition model to obtain the model output results. The model output results include emotional features and intent information. The loss function is determined based on the model output results and the annotation results, and the model parameters of the intent recognition model are adjusted through the back propagation algorithm to minimize the loss function. Repeat the above operations until the value of the loss function of the intent recognition model meets the convergence requirements or until the model training times reach the specified times.

[0035] In practical applications, there is no restriction on the network structure of the intent recognition model, including but not limited to: Recurrent Neural Network (RNN), Convolutional Neural Network (CNN), Long Short-Term Memory Network (LSTM), Transformer architecture. The Transformer architecture is designed to solve the problems of traditional recurrent neural networks and long short-term memory networks when processing long sequence data, especially the problems of slow training speed and difficulty in parallelization.

[0036] Further optionally, in order to better perform sentiment analysis and intent recognition, the intent recognition model may include a sentiment analysis network and an intent recognition network. The sentiment analysis network is used for sentiment analysis, and the intent recognition network is used for intent recognition.

[0037] In practical applications, the first interactive information may be input into a sentiment analysis network in an intention recognition model for sentiment analysis to identify the emotional characteristics of the target user from the first interactive information.

[0038] In practical applications, the first interaction information may be input into an intent recognition network in an intent recognition model for intent recognition, and target intent information of the target user may be identified from the first interaction information.

[0039] Further optionally, in order to enhance the depth and context perception capability of the personalized dialogue of the digital human, the method of using the intention recognition model to identify the emotional characteristics of the target user and the target intention information from the first interactive information is as follows: the first interactive information is input into the emotion analysis network in the intention recognition model, and the emotional information expressed in the first interactive information is analyzed in combination with the context information as the emotional characteristics; the first interactive information is input into the intention recognition network in the intention recognition model, and the interactive information is semantically understood in combination with the application scenario of the digital human live broadcast to obtain the target intention information.

[0040] In actual applications, during the interaction between the user and the digital human, the digital human can more accurately understand and respond to the user's needs by using context information, thereby providing a more natural and smooth interactive experience. Context information includes, but is not limited to: historical conversation records between the user and the digital human, user behavior data, user identity information, second interactive information initiated by the target user, response information to the second interactive information, third interactive information initiated by other users watching the digital human live broadcast, response information to the third interactive information, and current live broadcast content; wherein the third interactive information is associated with the first interactive information, the second interactive information can be historical interactive information, and the response information refers to the feedback information returned by the digital human in response to the interactive information.

[0041] Further optionally, in order to improve the accuracy of emotion recognition, the emotion information expressed in the first interactive information is analyzed in combination with the context information, and the implementation method of the emotion feature is as follows: the emotion tendency information and the emotion intensity information expressed in the first interactive information are analyzed in combination with the context information; the emotion of the target user is classified according to the emotion tendency information and the emotion intensity information to obtain the emotion category to which the target user belongs as the emotion feature.

[0042] Among them, emotional tendency information usually refers to the emotional direction or polarity expressed by the user, which is mainly divided into the following categories: 1. Positive: expressing positive, optimistic, and satisfied emotions. 2. Negative: expressing negative, pessimistic, and dissatisfied emotions. 3. Neutral: expressing emotions that are neither positive nor negative, which may be more objective or neutral.

[0043] Emotional intensity information refers to the degree or intensity of the user's emotion expression. For example, in descending order of intensity, the emotional intensity information is: very (indicates extremely strong emotions, usually the most extreme emotional situation), medium (indicates that the emotional intensity is moderate, not particularly strong, but not very weak), slight (indicates that the emotion is relatively weak, but there is still a certain tendency), etc.

[0044] Emotional category: A further subdivision or classification of emotional tendencies. It is not limited to the three basic directions of positive, negative, and neutral, but can be further subdivided into more specific emotional categories. Emotional categories are usually used to describe the user's emotional state in more detail. It provides a more detailed classification standard than emotional tendencies. Emotional categories can be used to distinguish emotions of different intensities. Emotional classification is performed by combining emotional tendency information and emotional intensity information. Emotional categories include but are not limited to:

[0045] 1. Very positive: expresses extreme happiness and satisfaction.

[0046] 2. Moderately Positive: Expresses relatively positive emotions but not as strong as Very Positive.

[0047] 3. Express slightly positive emotions.

[0048] 4. Neutral: Expresses emotions that are neither positive nor negative.

[0049] 5. Slightly negative: expresses slightly negative emotions.

[0050] 6. Moderately negative: Expresses relatively negative emotions but not as strong as very negative.

[0051] 7. Very negative: expresses extreme unhappiness and dissatisfaction.

[0052] It is worth noting that by combining contextual information, digital humans can more accurately identify the emotional tendencies and emotional intensity expressed by users, which helps reduce misjudgments and improves the accuracy of emotion recognition. Contextual information can help digital humans understand the emotional changes of users before and after, making the conversation more coherent and natural; combining emotional tendency information and emotional intensity information for emotion classification, digital humans can provide more personalized services based on the emotional category to which the user belongs; with accurate emotion recognition and personalized services, digital humans can better meet the needs of users and improve the overall user experience.

[0053] In practical applications, the first interactive information can be semantically understood in combination with the application scenario of the digital human live broadcast to obtain the target intention information. It is understandable that for the first interactive information in non-text form, it can be converted into the first interactive information in text form before semantic understanding.

[0054] Further optionally, the implementation method of semantically understanding the first interactive information in combination with the application scenario of the digital human live broadcast to obtain the target intention information is: segmenting the first interactive information to obtain multiple segmentations and their parts of speech; selecting at least two segmentations with specific parts of speech from the multiple segmentations to form a segmentation phrase, where the specific part of speech refers to a part of speech with the ability to express intention; semantically understanding the intention information expressed by the segmentation phrase in combination with the application scenario of the digital human live broadcast to obtain the target intention information. Among them, the specific part of speech includes, but is not limited to: verbs, nouns, adjectives, etc.

[0055] In this embodiment, the language model can be any language model with multimodal interaction capabilities, and the language model includes, for example, but is not limited to: a large language model (LLM), a multilingual large-scale pre-trained language model. A large language model refers to a model with powerful natural language processing capabilities trained through massive data, and usually has billions or even tens of billions of parameters. The large language model can understand and generate natural language text, and is widely used in tasks such as text classification, sentiment analysis, and question-answering systems. The multilingual large-scale pre-trained language model has powerful language understanding and generation capabilities, and can be applied to a variety of tasks such as translation, question-answering, and summary generation. It is good at handling multilingual environments and supports seamless conversion and understanding of multiple languages.

[0056] It is worth noting that multimodal interaction capability can refer to the ability to interact with users through multiple input and output modes (such as text, voice, image, video, etc.). Multimodal interaction capability can better understand and respond to the diverse needs of users and provide a richer and more natural user experience. In this embodiment, the language model has multimodal interaction capability, which can be understood as the input information of the language model can be information of multiple modes, or the output information of the language model can also be information of multiple modes.

[0057] In this embodiment, a language model is used to analyze and respond to user interaction information in real time during the live broadcast process, to achieve efficient and flexible text generation and interactive responses, to ensure the real-time nature of the interaction and the quality of the response, and to enable the digital human to maintain coherent and in-depth communication in the ever-changing live broadcast scene.

[0058] In this embodiment, the language model can not only generate responses that are highly consistent with the conversation context, but also combine multimodal feedback such as the emotional color of the voice, speaking rate and pauses, and non-verbal body gestures to give users a vivid and emotional interactive experience. Through multimodal feedback, digital humans can simulate more natural human interactions and enhance the realism and immersion of interactions.

[0059] In this embodiment, the language model is called according to the emotional characteristics and target intention information to generate control information for controlling the digital human to perform multimodal interaction. The control information is, to a certain extent, a multimodal control instruction that can both reflect the user's emotions and accurately respond to the user's needs. The multimodal control information may refer to a combination of information in multiple different modes for controlling the digital human.

[0060] Further optionally, in order to enable the digital human to make appropriate and timely responses according to different scenarios and user behaviors, thereby achieving a higher level of human-computer interaction experience, the language model with multimodal interaction capabilities is called according to the emotional characteristics and target intention information to generate control information for controlling the digital human to perform multimodal interaction as follows: the emotional characteristics and target intention information are input into the language model, and according to the emotional characteristics and target intention information, language control information and non-language control information for controlling the digital human to perform multimodal interaction are generated as control information; wherein the language control information is used for voice control and / or text output control of the digital human; and the non-language control information is used for action control and / or expression control of the digital human.

[0061] In this embodiment, language control information includes, but is not limited to, voice commands and text replies. Voice commands can adjust the pitch, speech speed, volume, emotional color, etc. of the voice information output by the digital human. Voice commands can make the digital human simulate the natural pauses and punctuation in human speech. Text replies are text replies generated by the digital human based on the context. Text replies include, but are not limited to, supplementary information, textual explanations to clarify questions, reply scripts, etc.

[0062] In this embodiment, non-verbal control information includes, but is not limited to, facial expression control instructions and body movement control instructions. Facial expression control instructions are used to control the digital human to simulate various facial expressions such as smiling and frowning; control the direction of eye movement and the position of eyebrows to express doubt or surprise, etc. Body movement control instructions are used to control the digital human to present various body movements, including, but not limited to, waving, clapping, nodding, shaking head, clapping, holding fists, patting chests, jumping, etc.

[0063] For example, in a digital human live shopping scenario, if the user is very negative and asks a question that needs help, the language model may generate a reply that includes comforting words and provides solutions, while adjusting the digital human's tone to make it gentler. It may also let the digital human show a sympathetic expression or make comforting gestures to enhance the user experience and establish a stronger emotional connection. The generation of this multimodal control information is not limited to text replies, but may also include changes in intonation in speech synthesis, changes in facial expressions in facial animation, and even changes in body movements to achieve a more natural and humane effect. In this way, the digital human can respond appropriately and promptly according to different scenarios and user behaviors, thereby achieving a higher level of human-computer interaction experience.

[0064] Further, optionally, in order to better enable the digital human to quickly and accurately adapt to user needs in different interactive scenarios and optimize the interactive experience of the digital human, when generating language control information for interactive control of the digital human, in addition to the target intention information, auxiliary information can also be introduced to improve the accuracy of the generated language control information; and then non-language control information is generated based on the language control information to ensure the accuracy of the generated non-language control information. Based on this, the implementation method of generating language control information and non-language control information for controlling the digital human to perform multimodal interaction according to emotional characteristics and target intention information is as follows: generating language control information according to target intention information and auxiliary information, the auxiliary information includes context information and / or application scenarios of the digital human live broadcast; selecting target action information and / or target expression information adapted to the language control information from the action information and / or expression information supported by the digital human according to the emotional characteristics as non-language control information. Among them, the relevant introduction about context information can be found in the aforementioned embodiments. The application scenarios of digital human live broadcast can be defined as needed. For example, the application scenarios can be defined from one or more attribute dimensions such as the type and category of the live broadcast content. For example, when the live broadcast content involves various commodities, different application scenarios can be defined and distinguished according to various dimensions such as the category of the commodity, the manufacturer of the commodity, and the origin of the commodity.

[0065] In practical applications, initial language control information can be generated based on the target intention information, and then the initial language control information can be adjusted or optimized using auxiliary information to obtain language control information. Auxiliary information includes, but is not limited to: context information and / or application scenarios of digital human live broadcast. It is understandable that after the adjustment or optimization of auxiliary information, the language control information can better meet the requirements of personalized interaction.

[0066] In practical applications, various action information and / or expression information supported by the digital human can be generated in advance, and different action information and expression information have their own corresponding emotion tags. The target action information and / or target expression information whose emotion tags match the emotional characteristics of the target user is selected as the non-language control information adapted to the language control information.

[0067] For example, in the digital human live shopping scene, the user is very negative, and the target action information selected is patting the chest: indicating willingness to take responsibility or provide help. The target expression information selected is gentle eyes and slightly frowning eyebrows, indicating understanding and sympathy. For another example, in the digital human live shopping scene, the user is very positive, and the target action information selected is leaning forward, indicating interest and concentration in the conversation. The target expression information selected is a focused expression, which has firm eyes and indicates that the listener is listening attentively.

[0068] In some optional embodiments, in order to better enable the digital human to quickly adapt to different user needs in different interactive scenarios and optimize the interactive experience of the digital human, multiple interactive control processes can be provided, and different interactive control processes correspond to different intention information, so that the appropriate interactive control process can be selected more quickly according to user needs, and language control information can be generated directly according to the selected interactive control process, and then the digital human can be interactively controlled. Based on this, the implementation method of generating language control information according to the target intention information and auxiliary information is: from multiple interactive control processes, determine the target interactive control process that is adapted to the target intention information; execute the target interactive control process to generate initial language control information; adjust the initial language control information according to the auxiliary information to obtain language control information.

[0069] As an example, if the target intent information is a knowledge question-answering intent related to the first product, the target interaction control process is executed to generate the initial language control information in the following manner:

[0070] The first product is retrieved from the live broadcast product library of the digital person, and the live broadcast product library includes product information that needs to be broadcast live by the digital person; if the first product is retrieved from the live broadcast product library, initial language control information is generated according to the correlation between the first product and the second product currently broadcast live by the digital person; if the first product is not retrieved from the live broadcast product library, initial language control information is generated according to the bottom-up speech or through multiple rounds of dialogue.

[0071] For example, suppose a user asks in the live broadcast room: "What is the camera pixel of this X-brand mobile phone (the first product)?" The digital human needs to find out whether there is information about the X-brand mobile phone in the live broadcast product library. If the X-brand mobile phone is found in the live broadcast product library, and the first product (the X-brand mobile phone) and the second product (the mobile phone currently being broadcasted) are the same mobile phone. The initial language control information that the digital human can generate is as follows: "The X-brand mobile phone you mentioned is one of the products we strongly recommend today. Its camera pixel is very high and it is very suitable for photography enthusiasts." If the X-brand mobile phone is not found in the live broadcast product library, the digital human can use a fallback speech or generate the initial language control information through multiple rounds of dialogue.

[0072] The catch-all phrase is "The phone you mentioned is not currently in our recommended list today, but I can provide you with some general information..." Or, more specific initial language control information can be determined through multiple rounds of dialogue: "You are very interested in camera performance, what specific information do you want to know?" and so on.

[0073] In some optional embodiments, the initial language control information is generated according to the correlation between the first product and the second product currently being broadcast live by the digital human: if the first product is the same as the second product, the initial language control information is generated according to the reply words for the second product; if the first product is different from the second product, and the knowledge question and answer intention indicates to obtain the product details information of the first product, the initial language control information is generated according to the product details information of the first product; if the first product is different from the second product, and the knowledge question and answer intention indicates to explain the first product, the initial language control information is generated according to the payment status of the target user for the first product.

[0074] In actual applications, if the first product is the same as the second product, when generating initial language control information based on the reply words for the second product, the target reply words can be determined directly or through multiple rounds of dialogue from the reply words of the second product currently being explained; and the initial language control information is generated based on the target reply words.

[0075] In this embodiment, suppose a user asks: "What is the camera pixel of this brand A mobile phone (first product)?" The current live broadcast room is also explaining the brand A mobile phone (second product). The digital human directly determines the reply: "This brand A mobile phone is one of the products we recommend today. Its camera pixel is XX megapixels, which is very suitable for friends who like taking pictures." Or, the digital human conducts multiple rounds of dialogue. The digital human asks: "You are interested in the camera performance of this brand A mobile phone. What other specific functions do you want to know?" The user answers: "I also want to know how its battery life is?" The digital human replies: "In addition to excellent camera performance, this phone also has a strong battery life that can meet daily use needs."

[0076] In this embodiment, if the first product is different from the second product, and the knowledge question and answer intends to obtain product detail information of the first product, initial language control information is generated according to the product detail information of the first product.

[0077] For example, suppose a user asks: "What are the main features of this brand B mobile phone (the first product)?" and the current live broadcast room is explaining brand C mobile phone (the second product). Digital Human provides product details: "The brand B mobile phone you mentioned is a high-performance smartphone. It has a XX-megapixel camera, a large-capacity battery, and fast charging function. It is very suitable for users who pursue performance."

[0078] In this embodiment, if the first product is different from the second product, and the knowledge question and answer intends to explain the first product, initial language control information is generated according to the payment status of the target user for the first product.

[0079] For example, suppose a user asks: "Can you introduce the D-brand mobile phone (the first product)?" and the current live broadcast room is explaining the E-brand mobile phone (the second product). Digital people check payment status: If the user has already purchased the D-brand mobile phone, the user is prompted to check the explanation information: "You have already purchased this D-brand mobile phone. Here are some detailed introductions about this phone..." If the user has not yet purchased it, the user can be prompted to pay first.

[0080] Further optionally, in order to improve the interaction efficiency, ensure the accuracy and pertinence of the reply information, and effectively improve the conversion rate of the live broadcast room, an implementation method for generating initial language control information based on the payment status of the target user for the first product is: if the target user has paid for the first product, the information prompting the target user to watch the explanation content of the first product and the explanation content of the first product will be used as the initial language control information; if the target user has not paid for the first product, then when the number of people watching the live broadcast is within a first range, the prompt information prompting the target user to watch the explanation content of the first product later and the explanation content of the first product will be used as the initial language control information; when the number of people watching the live broadcast is within a second range, the order of the explanation content of the first product in the oral broadcast list is adjusted to the first place, and the prompt information prompting the target user to explain the explanation content of the first product next and the explanation content of the first product will be used as the initial language control information.

[0081] In this embodiment, the first number range and the second number range are flexibly set as needed. The first number range indicates that the number of live viewers is small, such as 0 to 2000 people; the second number range indicates that the number of live viewers is large, such as more than 2000 people.

[0082] A spoken list is a form of organization in which a digital person introduces or explains products one by one in a predetermined order during a live broadcast. A spoken list usually contains information about all products planned to be introduced during the live broadcast, including product names, features, prices, discount information, etc., and is arranged in a certain logical order so that the host or digital person can introduce them step by step.

[0083] For example, suppose user D asks about a camera (the first product) in the live broadcast room. The digital human processing flow is as follows:

[0084] 1. Check the user's payment status:

[0085] If user D has already paid for the camera, a prompt message and explanation content will be generated directly. Prompt message: "You have already purchased the camera. Next, I will explain the features and usage of the camera in detail..." Provide explanation content: "This camera has a high-definition camera with XX megapixels, supports 4K video recording, and is equipped with optical image stabilization technology..."

[0086] 2. No payment, check the number of viewers:

[0087] If user D has not purchased the product and the current number of viewers is small, a delayed viewing prompt and explanation content are generated. Generate prompt information: "There are few viewers at the moment. We will explain this camera in detail later. Please wait a moment..." Provide explanation content: "This camera has a high-speed continuous shooting function, supports Wi-Fi transmission, and is easy to operate..."

[0088] If user D has not purchased and there are many viewers, adjust the order of the products and generate prompt information and explanation content. Adjust the order of the products: adjust the explanation content of the camera to the first place in the oral broadcast list. Generate prompt information: "Due to the large number of viewers in the live broadcast room, we will now give priority to explaining this camera to you, please pay attention to the explanation..." Provide explanation content: "This camera supports night scene mode, has multiple built-in filters, and has a long battery life..."

[0089] In some optional embodiments, the digital human can also respond to the live broadcast room information acquisition intention to meet the live broadcast room information acquisition needs. Based on this, if the target intention information is the live broadcast room information acquisition intention, the target interaction control process is executed to generate the initial language control information in the following manner: if the target intention information is the live broadcast room information acquisition intention, the initial language control information is generated according to the live broadcast room information.

[0090] In actual applications, live broadcast room information includes, but is not limited to: whether there are activities, whether there are coupons / discount coupons, customer service, whether there are negative reviews, etc.

[0091] Among them, whether there is an event: refers to any promotional activities going on in the live broadcast room, such as discounts, full discounts, buy one get one free, etc. If there is an event, this usually attracts more users to pay attention and buy products.

[0092] Coupons and discount coupons are common promotional methods. They can reduce product prices to a certain extent and encourage consumers to place orders.

[0093] Customer Service: Good customer service is crucial to solving user questions and handling after-sales issues. High-quality customer service can improve user satisfaction and enhance brand reputation.

[0094] Negative reviews refer to users' negative comments on products or services. Too many negative reviews may affect potential customers' purchasing decisions.

[0095] In some optional embodiments, the digital human can also respond to the product query intention to meet the product query needs. Based on this, the target interaction control process is executed to generate the initial language control information in the following manner: if the target intention information is the product query intention for the third product, and the third product is in the digital human's live broadcast product library, the explanation content of the third product is adjusted to the first place in the oral broadcast list, and the prompt information that prompts the target user to explain the explanation content of the third product next and the explanation content of the third product are used as the initial language control information.

[0096] In some optional embodiments, the digital human can also respond to the chat intention to meet personalized needs. In this way, the digital human can not only better meet the user's chat needs, but also recommend relevant products at the appropriate time, thereby enhancing the interactivity and sales potential of the live broadcast room. Based on this, the target interaction control process is executed to generate the initial language control information in the following way: if the target intention information is the chat intention, the chat content or product recommendation information that meets the personality characteristics of the live broadcast room is used as the initial language control information.

[0097] In some optional embodiments, the digital human can also respond to the intention of product recommendation to meet the demand for product recommendation. The digital human can quickly generate relevant recommendation information based on the user's product recommendation intention to help the user make a purchase decision, while increasing the interactivity and sales conversion rate of the live broadcast room. Based on this, the target interaction control process is executed to generate the initial language control information in the following manner: if the target intention information is a product recommendation intention for the fourth product, the product recommendation information of the fourth product is used as the initial language control information.

[0098] For example, suppose user L leaves a message in the live broadcast room: "I want to buy a backpack suitable for traveling." The product recommendation information generated by the digital human includes: Assuming there is a backpack designed specifically for travel, the digital human will generate the following product recommendation information: "We have a backpack specially designed for travel. It can accommodate a 15-inch laptop, has multiple compartments for easy organization of items, and the material is waterproof and durable."

[0099] In some optional embodiments, the language control information is voice broadcast information, and the non-language control information includes target expression information and target action information. According to the control information, the implementation method of controlling the digital human to perform multimodal interaction with the target user is: controlling the digital human to broadcast voice broadcast information, and controlling the digital human to present expression actions matching the target expression information and body actions matching the target action information. Thus, through multimodal feedback, the digital human can simulate more natural human interaction and enhance the realism and immersion of the interaction.

[0100] The technical solution provided by the embodiment of the present application, in the process of interaction between the digital human and the user, identifies the user's emotional characteristics and intention information based on the user's interactive information through the intention recognition model, and performs multimodal interactive feedback based on the user's emotional characteristics and intention information through a language model with multimodal interaction capabilities. As a result, the quality of interaction between the digital human and the user is improved, a more natural, smooth and emotional multimodal interactive experience is achieved, and the user's changing and segmented personalized dialogue needs are effectively adapted, and the user's willingness to interact is improved, especially in the commodity live broadcast scene, which can effectively improve the user's willingness to purchase commodities.

[0101] In order to better understand the embodiments of the present application, Figure 2 and Figure 3 Introduce a product live broadcast scenario.

[0102] See also Figure 2 The digital human live broadcast system includes a viewer terminal 10, a live broadcast server 20 and an anchor terminal 30. The user watches the live broadcast video on the viewer terminal 10. When the user needs to interact with the digital human (i.e., the virtual anchor), the user enters the interactive information (such as bullet screen information) on the viewer terminal 10. Figure 2 As shown in ① in FIG, the audience terminal 10 sends the interactive information to the live broadcast server 20. Figure 2 As shown in ②, the live broadcast server 20 uses the intention recognition model and the language model to generate control information corresponding to the interactive information. And based on the control information, the digital human is controlled to perform multimodal interaction, and video is recorded during the multimodal interaction of the digital human to generate a digital human video based on the control information. Figure 2 As shown in ③, the anchor terminal 30 obtains the initial live broadcast room video generated by recording the live broadcast room, and the live broadcast server 20 obtains the initial live broadcast room video from the anchor terminal 30. Figure 2 As shown in ④, the live broadcast server 20 synthesizes the digital human video and the initial live broadcast room video to obtain the live broadcast room video. Figure 2 As shown in ⑤, the audience end receives the live broadcast room video returned by the live broadcast server 20. When the audience end plays the live broadcast room video, the audience end 10 presents a live broadcast screen including the digital human. The digital human in the live broadcast screen shows corresponding facial expressions or body movements, and performs corresponding voice broadcasts or text outputs, thereby realizing multimodal feedback of the digital human and providing a richer live broadcast experience.

[0103] Combine the following Figure 3 This shows that digital humans respond in a targeted manner to different situations.

[0104] In actual applications, digital humans can be triggered by behaviors or systems to conduct Q&A interactions. For example, user behaviors that can trigger digital humans to conduct Q&A interactions include, but are not limited to: entering a live broadcast room, liking, following, placing an order, sharing, or meeting viewing time requirements, etc. For example, system information that can trigger digital humans to conduct Q&A interactions includes, but is not limited to: the number of people in the live broadcast room meets requirements, the broadcast time meets requirements, the inventory of the product is low, or there are red envelope discounts in the live broadcast room, etc.

[0105] In actual applications, when the digital human is triggered by behavior or the system to conduct question-and-answer interactions, the digital human can conduct targeted question-and-answer interactions according to the response script in the conversation script.

[0106] In actual applications, interactive information (such as bullet message) can also be used to trigger the digital human to conduct question-and-answer interaction. In some scenarios, after receiving the user's interactive information, the digital human can search for the reply words for the interactive information in the knowledge base. If the reply words are found in the knowledge base, the digital human outputs the reply words in the knowledge base to the user. If the reply words are not found in the knowledge base (that is, the reply words in the knowledge base are not hit), the digital human uses the intent recognition model and language model to conduct question-and-answer interaction. See Figure 3 , digital humans conduct targeted question-and-answer interactions based on different intentions.

[0107] If the user intends to obtain live broadcast room information, the digital human will obtain live broadcast room information such as whether there are activities, coupons / discount coupons, customer service, whether there are bad reviews, etc., and the digital human will voice broadcast or text output this live broadcast room information. The digital human can also present corresponding expression information or body movements.

[0108] If the user's intention is to ask questions about knowledge, and if the product that the user intends to see is not found in the live product library, the digital human will use a back-up speech or a multi-round dialogue method to interact with the user in a question-and-answer process. During the question-and-answer interaction, the digital human can also present corresponding facial expressions or body movements. If the product that the user intends to see is found in the live product library, the digital human will also distinguish whether the product that the user intends to see is related to the current live product. If it is related to the current live product, the user can be prompted to listen to the explanation of the current live product. If it is not related to the current live product, it will also distinguish whether to return the product details or whether to explain the product. During the question-and-answer interaction, the digital human can also present corresponding facial expressions or body movements.

[0109] If the user intends to search for products, and if the products intended by the user are not found in the live product library, the digital human will use a back-up speech or a multi-round dialogue method to interact with the user in a question-and-answer manner. During the question-and-answer interaction, the digital human can also present corresponding facial expressions or body movements. If the products intended by the user are found in the live product library, the digital human will also distinguish whether the products intended by the user are related to the current live product. If they are related to the current live product, the user can be prompted to listen to the explanation of the current live product. If the user intends to chat, the digital human can chat in a way that is consistent with the personality characteristics of the live room, or recommend products. If the user intends to recommend products, the digital human will recommend products.

[0110] In this embodiment, by introducing the intention recognition model and the language model, the following advantages are achieved during the digital human live broadcast process:

[0111] 1. Multimodal fusion interactive feedback technology: Digital humans can not only generate words that are highly consistent with the context of the conversation, but also combine multimodal feedback such as the emotional color of the voice, speaking rate and pauses, and non-verbal body postures to give users a vivid and emotional interactive experience. Through multimodal feedback, digital humans can simulate more natural human interactions and enhance the realism and immersion of interactions.

[0112] 2. Real-time interactive capabilities based on language models: Use language models to analyze and respond to interactive information such as user bullet screen messages in real time during live broadcasts, achieving efficient and flexible text generation and interactive responses. This technology ensures the real-time nature of the interaction and the quality of the response, enabling digital humans to maintain coherent and in-depth communication in the ever-changing live broadcast scenes.

[0113] 3. Dynamically adjusted dialogue strategies: Digital humans can dynamically adjust dialogue strategies based on real-time user feedback and data analysis. This innovation ensures flexibility of interaction and accuracy of response, enabling digital humans to quickly adapt to user needs and optimize user experience in different interaction scenarios.

[0114] 4. Allow the digital human to understand and respond to the user's emotional state during the interaction process, not only to answer questions, but also to provide emotional support and resonance, deepening the emotional connection between the user and the digital human.

[0115] Figure 4 This is a schematic diagram of the structure of a digital human interactive control device provided in an embodiment of the present application. Figure 4 , the device may include:

[0116] An acquisition module 41 is used to acquire the first interactive information initiated by the target user to the digital human;

[0117] An identification module 42, configured to identify the target user's emotional characteristics and target intention information from the first interaction information using an intention identification model;

[0118] A generation module 43 is used to call a language model with multimodal interaction capabilities according to the emotional characteristics and target intention information, and generate control information for controlling the digital human to perform multimodal interaction;

[0119] The interactive module 44 is used to control the digital human to perform multimodal interaction with the target user according to the control information.

[0120] Optionally, the intention recognition model includes a sentiment analysis network and an intention recognition network; the recognition module 42 is specifically used to: input the first interactive information into the sentiment analysis network in the intention recognition model, and analyze the emotional information expressed by the first interactive information in combination with the context information as an emotional feature; input the first interactive information into the intention recognition network in the intention recognition model, and perform semantic understanding of the interactive information in combination with the application scenario of digital human live broadcast to obtain target intention information.

[0121] Optionally, the identification module 42 analyzes the emotional information expressed by the first interactive information in combination with the context information as an emotional feature, and is specifically used to: analyze the emotional tendency information and emotional intensity information expressed by the first interactive information in combination with the context information; and perform emotional classification of the target user according to the emotional tendency information and the emotional intensity information to obtain the emotional category to which the target user belongs as an emotional feature.

[0122] Optionally, the context information includes at least one of the following information: second interactive information initiated by the target user, response information to the second interactive information, third interactive information initiated by other users watching the digital human live broadcast, response information to the third interactive information, and current live broadcast content; wherein the third interactive information is associated with the first interactive information.

[0123] Optionally, when the recognition module 42 performs semantic understanding on the first interactive information in combination with the application scenario of the digital human live broadcast to obtain the target intention information, it is specifically used to: perform word segmentation processing on the first interactive information to obtain multiple word segments and their parts of speech; select at least two word segments with specific parts of speech from the multiple word segments to form a word segmentation phrase, where the specific part of speech refers to a part of speech with the ability to express intention; perform semantic understanding on the intention information expressed by the word segmentation phrase in combination with the application scenario of the digital human live broadcast to obtain the target intention information.

[0124] Optionally, the generation module 43 is specifically used to: input the emotional characteristics and target intention information into the language model, and generate language control information and non-language control information for controlling the digital human to perform multimodal interaction as control information based on the emotional characteristics and target intention information; wherein the language control information is used to perform voice control and / or text output control on the digital human; and the non-language control information is used to perform action control and / or expression control on the digital human.

[0125] Optionally, the generation module 43 is specifically used to: generate language control information according to target intention information and auxiliary information, the auxiliary information including context information and / or application scenarios of the digital human live broadcast; select target action information and / or target expression information adapted to the language control information from the action information and / or expression information supported by the digital human according to emotional characteristics as non-language control information.

[0126] Optionally, when the generation module 43 generates language control information, it is specifically used to: determine a target interaction control process that is compatible with the target intent information from multiple interaction control processes, where different interaction control processes correspond to different intent information; execute the target interaction control process to generate initial language control information; and adjust the initial language control information according to the auxiliary information to obtain the language control information.

[0127] Optionally, when the target intention information is a knowledge question-and-answer intention related to the first product, the generation module 43 executes the target interaction control process to generate initial language control information, which is specifically used to: retrieve the first product from the digital human's live broadcast product library, and the live broadcast product library includes product information that needs to be broadcast live by the digital human; if the first product is retrieved from the live broadcast product library, generate initial language control information based on the correlation between the first product and the second product currently broadcast live by the digital human; if the first product is not retrieved from the live broadcast product library, generate initial language control information based on the bottom-up speech or through multiple rounds of dialogue.

[0128] Optionally, when the generation module 43 generates the initial language control information according to the correlation between the first product and the second product currently being broadcast live by the digital human, it is specifically used to: if the first product is the same as the second product, then the initial language control information is generated according to the reply words for the second product; if the first product is different from the second product, and the knowledge question and answer intention indicates to obtain the product details information of the first product, then the initial language control information is generated according to the product details information of the first product; if the first product is different from the second product, and the knowledge question and answer intention indicates to explain the first product, then the initial language control information is generated according to the payment status of the target user for the first product.

[0129] Optionally, when the generation module 43 generates the initial language control information according to the payment status of the target user for the first product, it is specifically used for: if the target user has paid for the first product, then the information prompting the target user to watch the explanation content of the first product and the explanation content of the first product are used as the initial language control information; if the target user has not paid for the first product, then when the number of people watching the live broadcast is within a first range, the prompt information prompting the target user to watch the explanation content of the first product later and the explanation content of the first product are used as the initial language control information; when the number of people watching the live broadcast is within a second range, the order of the explanation content of the first product in the oral broadcast list is adjusted to the first place, and the prompt information prompting the target user to explain the explanation content of the first product next and the explanation content of the first product are used as the initial language control information.

[0130] Optionally, when the generation module 43 executes the target interaction control process to generate the initial language control information, it is specifically used for: if the target intention information is the live broadcast room information acquisition intention, then the initial language control information is generated according to the live broadcast room information; if the target intention information is the product query intention for the third product, and the third product is in the live broadcast product library of the digital person, then the explanation content of the third product is adjusted to the first place in the order of the products in the oral broadcast list, and the prompt information that prompts the target user to explain the explanation content of the third product next and the explanation content of the third product are used as the initial language control information; if the target intention information is the chat intention, the chat content or product recommendation information that conforms to the personality characteristics of the live broadcast room is used as the initial language control information; if the target intention information is the product recommendation intention for the fourth product, the product recommendation information of the fourth product is used as the initial language control information.

[0131] Optionally, the language control information is voice broadcast information, and the non-language control information includes target expression information and target action information. The interactive module 44 is specifically used to: control the digital human to broadcast the voice broadcast information, and control the digital human to present expression actions that match the target expression information and body actions that match the target action information.

[0132] Figure 4 The device shown can perform Figure 1 The method shown in the embodiment shown, its implementation principle and technical effect are not described in detail. Figure 4 The specific manner in which each module and unit of the device shown performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0133] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 104 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of steps 103 and 104 can be device B; and so on.

[0134] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0135] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0136] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device includes: a memory 51 and a processor 52;

[0137] The memory 51 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc.

[0138] The memory 51 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0139] The processor 52 is coupled to the memory 51 and is used to execute the computer program in the memory 51 to execute the steps in the interactive control method of the digital human.

[0140] Optional, such as Figure 5 As shown, the electronic device also includes: a communication component 53, a display 54, a power component 55, an audio component 56 and other components. Figure 5 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 5 In addition, Figure 5 The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the electronic device. The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT (Internet of Things) device, or a server device such as a conventional server, a cloud server, or a server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 5 If the electronic device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Figure 5 Components within the dashed box.

[0141] For the detailed implementation process of the processor executing each action, please refer to the relevant description in the aforementioned method embodiment or device embodiment, which will not be repeated here.

[0142] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by an electronic device in the above method embodiment.

[0143] Accordingly, an embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is enabled to implement each step in the above method embodiment that can be executed by an electronic device.

[0144] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi (Wireless Fidelity), 2G (2 Generation, 2 Generation), 3G (3 Generation, 3 Generation), 4G (4 Generation, 4 Generation) / LTE (long Term Evolution), 5G (5 Generation, 5 Generation) and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (Near Field Communication, NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0145] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0146] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0147] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0148] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0149] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0152] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0153] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0154] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, Phase Change RAM (PRAM), Static Random-Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0155] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0156] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A digital human interactive control method, characterized in that: include: Obtain the first interactive information initiated by the target user towards the digital human; Using an intention recognition model to identify the target user's emotional characteristics and target intention information from the first interactive information; Invoking a language model with multimodal interaction capability according to the emotional features and the target intention information to generate control information for controlling the digital human to perform multimodal interaction; According to the control information, the digital human is controlled to perform multimodal interaction with the target user.

2. The method according to claim 1, characterized in that The intention recognition model includes a sentiment analysis network and an intention recognition network; Using the intention recognition model to identify the emotional characteristics and target intention information of the target user from the first interactive information includes: Inputting the first interactive information into the sentiment analysis network in the intention recognition model, and analyzing the sentiment information expressed by the first interactive information in combination with context information as the sentiment feature; The first interactive information is input into the intent recognition network in the intent recognition model, and the interactive information is semantically understood in combination with the application scenario of digital human live broadcast to obtain the target intent information.

3. The method according to claim 2, characterized in that Analyzing the emotional information expressed by the first interactive information in combination with the context information as the emotional feature includes: Analyzing the emotional tendency information and emotional intensity information expressed by the first interactive information in combination with the context information; The target user is emotionally classified according to the emotional tendency information and the emotional intensity information to obtain the emotional category to which the target user belongs as the emotional feature.

4. The method according to claim 3, characterized in that The context information includes at least one of the following information: The second interactive information initiated by the target user, the response information to the second interactive information, the third interactive information initiated by other users watching the digital human live broadcast, the response information to the third interactive information and the current live broadcast content; wherein the third interactive information is associated with the first interactive information.

5. The method according to claim 2, characterized in that: Combining the application scenario of the digital human live broadcast with semantic understanding of the first interactive information to obtain the target intention information includes: Performing word segmentation processing on the first interactive information to obtain a plurality of word segments and their parts of speech; Selecting at least two participles with specific parts of speech from the multiple participles to form a participle phrase, wherein the specific part of speech refers to a part of speech with the ability to express an intention; The intention information expressed by the participle phrase is semantically understood in combination with the application scenario of the digital human live broadcast to obtain the target intention information.

6. The method according to any one of claims 1 to 5, characterized in that: The method calls a language model with multimodal interaction capability according to the emotional features and the target intention information to generate control information for controlling the digital human to perform multimodal interaction, including: Inputting the emotion feature and the target intention information into the language model, and generating language control information and non-language control information for controlling the digital human to perform multimodal interaction as the control information according to the emotion feature and the target intention information; The language control information is used to perform voice control and / or text output control on the digital human; and the non-language control information is used to perform action control and / or expression control on the digital human.

7. The method according to claim 6, characterized in that Generating language control information and non-language control information for controlling the digital human to perform multimodal interaction according to the emotional characteristics and the target intention information, including: Generate the language control information according to the target intention information and the auxiliary information, wherein the auxiliary information includes context information and / or an application scenario of the digital human live broadcast; According to the emotional characteristics, target action information and / or target expression information adapted to the language control information is selected from the action information and / or expression information supported by the digital human as the non-language control information.

8. The method according to claim 7, characterized in that Generating the language control information according to the target intention information and the auxiliary information includes: Determine, from a plurality of interactive control processes, a target interactive control process adapted to the target intention information, where different interactive control processes correspond to different intention information; The target interactive control process is executed to generate initial language control information; and the initial language control information is adjusted according to the auxiliary information to obtain the language control information.

9. The method according to claim 8, characterized in that If the target intention information is a knowledge question-answering intention related to the first product, the target interaction control process is executed to generate initial language control information, including: Retrieving the first commodity from the digital human's live broadcast commodity library, wherein the live broadcast commodity library includes commodity information that needs to be broadcast live by the digital human; If the first product is retrieved in the live product library, initial language control information is generated according to the relevance between the first product and the second product currently being live-broadcasted by the digital human; If the first product is not found in the live broadcast product library, initial language control information is generated based on a catch-all speech or through multiple rounds of dialogue.

10. The method according to claim 9, characterized in that According to the correlation between the first commodity and the second commodity currently being broadcast live by the digital human, initial language control information is generated, including: If the first product is the same as the second product, generating initial language control information according to the reply words for the second product; If the first product is different from the second product, and the knowledge question and answer intention indicates obtaining product details information of the first product, generating initial language control information according to the product details information of the first product; If the first product is different from the second product, and the knowledge question and answer intends to indicate an explanation of the first product, initial language control information is generated according to the payment status of the target user for the first product.

11. The method according to claim 9, characterized in that According to the payment status of the target user for the first commodity, initial language control information is generated, including: If the target user has paid for the first product, the information prompting the target user to watch the explanation content of the first product and the explanation content of the first product are used as the initial language control information; If the target user has not paid for the first product, then when the number of live broadcast viewers is within the first number range, prompt information prompting the target user to watch the explanation content of the first product later and the explanation content of the first product are used as the initial language control information; When the number of people watching the live broadcast is within the second range, the order of the explanation content of the first product in the oral broadcast list is adjusted to the first place, and the prompt information that prompts the target user to explain the explanation content of the first product next and the explanation content of the first product are used as the initial language control information.

12. The method according to claim 8, characterized in that Executing the target interactive control process to generate initial language control information also includes: If the target intention information is the live broadcast room information acquisition intention, initial language control information is generated according to the live broadcast room information; If the target intention information is a product query intention for a third product, and the third product is in the live broadcast product library of the digital human, the explanation content of the third product is adjusted to the first place in the oral broadcast list, and the prompt information prompting the target user to explain the explanation content of the third product next and the explanation content of the third product are used as the initial language control information; If the target intention information is a chat intention, the chat content or product recommendation information that matches the personality characteristics of the live broadcast room is used as the initial language control information; If the target intention information is a product recommendation intention for a fourth product, the product recommendation information of the fourth product is used as the initial language control information.

13. The method according to any one of claims 1 to 6, characterized in that: The language control information is voice broadcast information, and the non-language control information includes target expression information and target action information. According to the control information, the digital human is controlled to perform multimodal interaction with the target user, including: The digital human is controlled to broadcast the voice broadcast information, and the digital human is controlled to present an expression action matching the target expression information and a body action matching the target action information.

14. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer programs; The processor is coupled to the memory and configured to execute the computer program to perform the steps of the method according to any one of claims 1 to 13.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is enabled to implement the steps in the method according to any one of claims 1 to 13.

16. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, enables the processor to implement the steps of the method according to any one of claims 1 to 13.

Images