A data processing method, apparatus, device, medium, and product

Abstract

The application discloses a data processing method, device, equipment, medium and product in the technical field of Internet, which comprises the following steps: after the client receives the first voice (such as the voice describing the user's problem) input by the user, the client performs voice recognition on the first voice to obtain the first text (such as the text describing the user's problem), and sends the first text to the server, so that the server can process and feedback the first text, so that after the client receives the second voice, the expression sequence and the action sequence sent by the server, the client generates the image sequence according to the expression sequence and the action sequence, and the client plays the second voice and displays the image sequence, so that the client can automatically reply to the user's problem, thereby reducing the reply cost and improving the user experience.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a data processing method, apparatus, device, medium, or product. Background Technology

[0002] In some scenarios, there is a need to answer user questions in order to resolve user doubts.

[0003] In addition, to achieve the above requirements, relevant personnel can interact with users to answer their questions. However, this approach requires users to learn a significant amount of information beforehand, making the response cost relatively high. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, medium, and product that helps reduce response costs.

[0005] To achieve the above objectives, the technical solution provided in this application is as follows:

[0006] This application provides a data processing method applied to a client. The method includes: receiving a first voice message; performing speech recognition processing on the first voice message to obtain first text; and sending the first text message to a server; receiving a second voice message, an expression sequence, and an action sequence sent by the server, wherein the second voice message indicates the content of a virtual person's speech in response to the first voice message, the expression sequence indicates the changes in the virtual person's facial expressions in response to the first voice message, and the action sequence indicates the changes in the virtual person's actions in response to the first voice message; generating an image sequence based on the expression sequence and the action sequence, wherein the image sequence describes the changes of the virtual person in at least one dimension, the at least one dimension including facial expressions and / or actions; playing the second voice message; and displaying the image sequence.

[0007] In one possible implementation, the client is a holographic capsule.

[0008] In one possible implementation, the method further includes: receiving emotional information sent by the server, the emotional information indicating the emotional state of the virtual person when responding to the first voice; generating an image sequence based on the expression sequence and the action sequence includes: generating an image sequence based on the emotional information, the expression sequence, and the action sequence.

[0009] In one possible implementation, the facial expression sequence indicates lip movements; generating an image sequence based on the emotion information, the facial expression sequence, and the action sequence includes: generating facial expressions based on the emotion information, the facial expressions indicating facial states of the virtual person other than lip movements when responding to the first speech; and generating the image sequence based on the facial expressions, the facial expression sequence, and the action sequence.

[0010] In one possible implementation, the emotion information includes the predicted probability of at least one candidate emotion; generating a facial expression based on the emotion information includes: determining a target emotion from the at least one candidate emotion based on the predicted probability of the at least one candidate emotion, wherein the predicted probability of the target emotion is not less than the predicted probabilities of other emotions besides the target emotion among the at least one candidate emotions; and performing a weighted fusion of the expression template corresponding to the target emotion and the expression template corresponding to the default emotion based on the predicted probability of the target emotion to obtain the facial expression, wherein the predicted probability of the target emotion indicates the weight of the expression template corresponding to the target emotion in the weighted fusion, the expression template corresponding to the target emotion indicates the facial state under the target emotion, and the expression template corresponding to the default emotion indicates the facial state under the default emotion.

[0011] In one possible implementation, the client includes a state machine that indicates the switching relationship between different states of the virtual human; after receiving the second voice, facial expression sequence and action sequence sent by the server, the method further includes: updating the state information of the virtual human according to the state machine, wherein the updated state information indicates that the virtual human switches to the speaking state.

[0012] In one possible implementation, the image sequence is also generated based on at least one body feature corresponding to the updated state information, the at least one body feature including blink frequency, head orientation, eye orientation, and part or all of the body orientation.

[0013] In one possible implementation, the step of performing speech recognition processing on the first speech to obtain the first text includes: in response to detecting that the first speech does not have a preset feature, performing speech recognition processing on the first speech to obtain the first text, wherein the preset feature indicates the features of the speech responded by the virtual human through a broadcasting method; before receiving the second speech, facial expression sequence, and action sequence sent by the server, the method further includes: in response to detecting that the first speech has the preset feature, sending the text corresponding to the preset feature to the server.

[0014] In one possible implementation, the client includes a first database; sending the first text to the server includes: in response to the first text not satisfying the search conditions of the first database, sending the first text to the server; before receiving the second voice, facial expression sequence, and action sequence sent by the server, the method further includes: in response to the first text satisfying the search conditions of the first database, sending the search results of the first text in the first database to the server.

[0015] In one possible implementation, the method further includes: receiving second text sent by the server;

[0016] The action sequence includes an action prediction result based on the second speech and an action matched from an action library based on the second text, wherein the second text includes the insertion time of the matched action, and the insertion time indicates the time when the virtual human performs the matched action;

[0017] The image sequence includes the image corresponding to each frame of the action prediction result and the image corresponding to the matched action;

[0018] The method further includes: during the display of the image sequence, in response to playing the image corresponding to the matching action, the client determines whether the insertion time of the matching action in the second text is at the end of a sentence; if so, the image corresponding to the matching action is skipped; if not, the image corresponding to the matching action and its adjacent images in the image sequence are smoothed to obtain a smoothed result of the image corresponding to the matching action and a smoothed result of the adjacent images, wherein the adjacent images include a previous frame image and / or a next frame image. The smoothed result of the image corresponding to the matching action and the smoothed result of the next frame image are played sequentially, wherein the next frame image is positioned later in the image sequence than the image corresponding to the matching action, and the previous frame image is positioned earlier in the image sequence than the image corresponding to the matching action.

[0019] This application provides a data processing method applied to a server. The method includes: receiving first text sent by a client; determining second text based on the first text; generating second speech based on the second text; determining an action sequence based on the second speech and the second text; predicting an expression sequence based on the second speech; and sending the second speech, the expression sequence, and the action sequence to the client.

[0020] In one possible implementation, determining the action sequence based on the second speech and the second text includes: for any word in the second text, searching for an action that matches the word from a pre-built action library to obtain a matching result for the word; and in response to the matching result indicating that there is an action that matches the word in the action library, determining the start and end times of the audio segment corresponding to the word from the second speech; and fusing the matched action and the action prediction result generated based on the second speech according to the start and end times to obtain the action sequence.

[0021] In one possible implementation, fusing the matched action and the action prediction result generated based on the second speech according to the start and end times to obtain the action sequence includes: determining the insertion time of the matched action according to the start and end times; in response to the position indicated by the insertion time in the second text not being the end of a sentence, concatenating the matched action and the action prediction result according to the insertion time to obtain a concatenated result; and smoothing the concatenated result to obtain the action sequence.

[0022] In one possible implementation, the method further includes: predicting emotion information based on the second speech and the second text; predicting an expression sequence based on the second speech includes: predicting an expression sequence based on the emotion information and the second speech; and / or, sending the second speech, the expression sequence, and the action sequence to the client includes: sending the emotion information, the second speech, the expression sequence, and the action sequence to the client.

[0023] In one possible implementation, the server includes a second database; determining the second text based on the first text includes: in response to the first text satisfying the retrieval conditions of the second database, retrieving the first document in the second database to obtain the second text; in response to the first text not satisfying the retrieval conditions of the second database, inputting the first text into a pre-built response model to obtain the second text output by the response model.

[0024] In one possible implementation, the first text indicates the semantic content carried by the first speech; the method further includes: receiving an emotion recognition result sent by the client, the emotion recognition result indicating the emotion information carried by the first speech; the step of determining the second text based on the first text includes: inputting the first text and the emotion recognition result into a pre-built response model to obtain the second text output by the response model.

[0025] This application provides a data processing apparatus, comprising: a first unit, configured to receive a first voice, perform speech recognition processing on the first voice to obtain a first text, and send the first text to a server; a receiving unit, configured to receive a second voice, an expression sequence, and an action sequence sent by the server, wherein the second voice indicates the content of a virtual person's speech in response to the first voice, the expression sequence indicates the changes in the virtual person's facial expressions in response to the first voice, and the action sequence indicates the changes in the virtual person's actions in response to the first voice; a generating unit, configured to generate an image sequence based on the expression sequence and the action sequence, wherein the image sequence describes the changes of the virtual person in at least one dimension, the at least one dimension including facial expressions and / or actions; and a playing unit, configured to play the second voice and display the image sequence.

[0026] This application provides a data processing apparatus, comprising: a second unit for receiving first text sent by a client, determining second text based on the first text, and generating second speech based on the second text; a processing unit for determining an action sequence based on the second speech and the second text, and predicting an expression sequence based on the second speech; and a sending unit for sending the second speech, the expression sequence, and the action sequence to the client.

[0027] This application provides an electronic device, the device comprising: a processor and a memory; the memory for storing instructions or computer programs; the processor for executing the instructions or computer programs in the memory, so that the electronic device performs the data processing method provided in this application.

[0028] This application provides a computer-readable medium storing instructions or computer programs that, when executed on a device, cause the device to perform the data processing method provided in this application.

[0029] This application provides a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the data processing method provided in this application.

[0030] Compared with related technologies, this application has at least the following advantages:

[0031] In the technical solution provided in this application, for a client capable of interacting with a user, after the client receives the first voice input by the user (such as voice describing the user's question), the client first performs speech recognition processing on the first voice to obtain the first text (such as text describing the user's question), and sends the first text to the server so that the server can process and provide feedback on the first text. After the client receives the second voice, facial expression sequence, and action sequence sent by the server, the client generates an image sequence based on the facial expression sequence and the action sequence, and plays the second voice and displays the image sequence. This enables the client to automatically answer the user's question, thereby reducing the answering cost. Specifically, because the facial expression sequence can indicate the changes in the virtual person's facial expression when responding to the first voice, and the action sequence can indicate the changes in the virtual person's actions when responding to the first voice, the image sequence generated based on the facial expression sequence and the action sequence can describe the changes in the virtual person in at least one dimension (such as facial expression and / or action), thereby enabling the image sequence to represent the changes (such as changes in facial expression, changes in action, etc.) exhibited by the virtual person when responding to the first voice; furthermore, because the second voice can indicate the content of the virtual person's speech when responding to the first voice, this application presents a vivid and expressive response from the virtual person to the user by playing the second voice and displaying the image sequence, thereby improving the user experience. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A flowchart illustrating a data processing method applied to a client, as provided in this application embodiment;

[0034] Figure 2 A schematic diagram illustrating a data processing flow in a question-and-answer scenario provided in an embodiment of this application;

[0035] Figure 3 A schematic diagram of a complete one-round dialogue provided for an embodiment of this application;

[0036] Figure 4 A schematic diagram of a state machine provided for an embodiment of this application;

[0037] Figure 5 A schematic diagram of a state switching loop provided in an embodiment of this application;

[0038] Figure 6 A schematic diagram illustrating a data processing flow in an automatic broadcast scenario provided in an embodiment of this application;

[0039] Figure 7 A flowchart illustrating a data processing method applied to a server, as provided in this application embodiment;

[0040] Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0041] Figure 9 This is a schematic diagram of another data processing apparatus provided in an embodiment of this application;

[0042] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0043] Research has found that, in order to reduce costs, technologies such as machine learning models can be used to provide automated response services to users, thereby reducing response costs and improving user experience.

[0044] The study also found that although the aforementioned automated response service can improve the user experience in some aspects, it has at least the defects shown in ①-② below, which makes many users prefer to ask and answer questions manually.

[0045] ① Poor service capability is due to the fact that the above-mentioned automatic reply service responds to user questions based on pre-set question-and-answer pairs. This makes it impossible for the automatic reply service to flexibly analyze and provide effective suggestions (or replies) for user-submitted questions that have some variation or for user-submitted needs that are made on an ad hoc basis.

[0046] ② Insufficient emotional expression. The reason is that the above-mentioned automatic reply service lacks emotion when replying, which makes it impossible for users to have a vivid communication and empathy with the automatic reply service, thus affecting the user experience.

[0047] Based on the above research, in order to better improve the user experience, this application provides a data processing method applied to a client. The method includes: for a client capable of interacting with a user, after the client receives a first voice input by the user (such as a voice describing a user's question), the client first performs speech recognition processing on the first voice to obtain a first text (such as text describing the user's question), and sends the first text to a server so that the server can process and provide feedback on the first text. After the client receives a second voice, an expression sequence, and an action sequence sent by the server, the client generates an image sequence based on the expression sequence and the action sequence, and the client plays the second voice and displays the image sequence. This enables the client to automatically answer user questions, thereby reducing the response cost. Specifically, because the facial expression sequence can indicate the changes in the virtual person's facial expression when responding to the first voice, and the action sequence can indicate the changes in the virtual person's actions when responding to the first voice, the image sequence generated based on the facial expression sequence and the action sequence can describe the changes in the virtual person in at least one dimension (such as facial expression and / or action), thereby enabling the image sequence to represent the changes (such as changes in facial expression, changes in action, etc.) exhibited by the virtual person when responding to the first voice; furthermore, because the second voice can indicate the content of the virtual person's speech when responding to the first voice, this application presents a vivid and expressive response from the virtual person to the user by playing the second voice and displaying the image sequence, thus effectively overcoming the shortcomings of the aforementioned automatic response service and improving the user experience.

[0048] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0049] To better understand the technical solution provided in this application, the data processing method provided in this application will be explained below with reference to some accompanying drawings. For example... Figure 1 As shown, the data processing method applied to the client provided in this application includes S101-S104 below.

[0050] S101: The client receives the first voice message, performs speech recognition processing on the first voice message to obtain the first text message, and sends the first text message to the server.

[0051] The client can provide interactive services to users, such as question-and-answer services; and this application does not limit the implementation method of the client. For example, the client can be implemented using any type of terminal device.

[0052] The first voice refers to the voice input by the user to the client (e.g., ...). Figure 2 The first voice (as shown in the question voice) is used to express the user's needs; and this application does not limit the implementation of the first voice, for example, the first voice carries a target question so that the first voice can express the user's inquiry needs.

[0053] The first text refers to the result obtained by performing speech recognition processing on the first speech (e.g., Figure 2 The first text (as shown in the question text) is used to represent the semantic content carried by the first speech (such as the target question mentioned above). It should be noted that this application does not limit the implementation of the speech recognition process. For example, the speech recognition process can be implemented using any method that can convert a speech into text, such as Automatic Speech Recognition (ASR) or any pre-built machine learning model that can perform speech recognition.

[0054] The server and client can communicate data to enable the server to provide services to the client, such as data processing services. It should be noted that this application does not limit the implementation method of the server; for example, it can be implemented using any type of server, such as a standalone server, a cluster server, or a cloud server.

[0055] Based on the above content related to S101, it can be seen that for the client (such as...) Figure 2 For the client shown), when the client receives the first voice input from the user (such as...) Figure 2 After the first speech (as shown in the image), the client first performs ASR on the first speech to obtain the first text (such as...). Figure 2 The first text (as shown) is used to represent the semantic content carried by the first speech; then the client sends the first text to the server so that the server can process the first text (e.g., Figure 2 The processing shown is performed by the server.

[0056] S102: The client receives a second voice, an expression sequence, and an action sequence sent by the server. The second voice indicates the content of the virtual person's speech when responding to the first voice, the expression sequence indicates the changes in the virtual person's facial expressions when responding to the first voice, and the action sequence indicates the changes in the virtual person's actions when responding to the first voice.

[0057] The second voice refers to the voice fed back from the server to the client, so that the second voice can indicate the content of the virtual person's speech when responding to the first voice, so that when the first voice carries a target question, the second voice can indicate the answer given by the virtual person to the target question.

[0058] An expression sequence refers to the expression information fed back from the server to the client, so that the expression sequence can indicate the changes in the virtual person's expression (such as changes in lip shape) when responding to the first voice, so that the expression sequence can represent the changes in the virtual person's expression when playing the second voice, and thus the expression sequence can represent the emotions presented by the virtual person when speaking according to the second voice to a certain extent.

[0059] An action sequence refers to action information fed back from the server to the client, so that the action sequence can indicate the changes in the virtual person's actions (such as changes in body posture) when responding to the first voice, thereby enabling the action sequence to represent the changes in the virtual person's actions during the playback of the second voice, and thus enabling the action sequence to represent, to a certain extent, the body posture of the virtual person when speaking according to the second voice.

[0060] S103: The client generates an image sequence based on the facial expression sequence and the action sequence. The image sequence describes the changes of the virtual human in at least one dimension, which includes facial expressions and / or actions.

[0061] The image sequence is generated based on an expression sequence and a motion sequence, ensuring that the facial expression changes presented in the image sequence are consistent with the facial expression changes indicated by the expression sequence, and that the motion changes presented in the image sequence are consistent with the motion changes indicated by the motion sequence. This allows the image sequence to describe changes in the virtual human in at least one dimension (such as facial expression and / or motion), thereby enabling the image sequence to represent the driving results for the virtual human. It should be noted that this application does not limit the implementation of the at least one dimension; for example, the at least one dimension may include at least facial expression and / or motion.

[0062] Furthermore, this application does not limit the implementation of the above-mentioned S103. For example, it can be implemented using any method that can perform image generation processing (or video generation processing) based on facial expression sequences and action sequences, such as any virtual human driving method or any pre-built machine learning model that can achieve the aforementioned functions.

[0063] Based on the above content of S103, it can be seen that for the client, after obtaining the facial expression sequence and action sequence from the server, it can perform virtual human driving (such as video generation processing) based on these two sequences to obtain an image sequence. This ensures that the facial expression changes presented in the image sequence are consistent with the facial expression changes indicated by the facial expression sequence, and that the action changes presented in the image sequence are consistent with the action changes indicated by the action sequence. This allows the image sequence to better represent the multi-dimensional changes presented by the virtual human when responding to the first voice, and further, it allows the image sequence to better represent the multi-dimensional changes presented by the virtual human during the playback of the second voice. Thus, the image sequence can better represent the multi-dimensional changes presented by the virtual human when speaking according to the second voice, thereby enabling the virtual human to respond to the user with vivid expression through the image sequence, thereby improving the user experience.

[0064] S104: The client plays a second audio message and displays an image sequence.

[0065] It should be noted that this application does not limit the implementation of the above-described S104. For example, it can specifically be: while playing the second voice on the client, it can also be done through a certain method (such as... Figure 2 The rendering process shown above displays the image sequence to the user, allowing the user to perceive the client's response to the user's first voice input through both visual and auditory means, thus improving the user experience.

[0066] Additionally, in some scenarios, to better improve the user experience, the above S104 may include: the client playing a second voice message, displaying an image sequence, and displaying subtitles corresponding to the second voice message (such as...). Figure 2 (as shown in the caption). Because this caption indicates the semantic content carried by the second voice, it represents the speech content used by the virtual person when speaking according to the second voice. This allows the user to better understand the client's response to the user's input of the first voice, thus improving the user experience.

[0067] Therefore, in one possible implementation, when the client plays the second voice, it is necessary not only to display the image sequence but also to display the subtitles corresponding to the second voice so that the user can more clearly understand the client's response to the user's first voice input through the subtitles, thereby improving the user experience.

[0068] It should be noted that this application does not limit the method of obtaining the subtitles corresponding to the second speech. For example, the subtitles can be obtained by the client through speech recognition processing of the second speech. Alternatively, the second speech can be generated by the server through speech generation processing of the second text (e.g.,...). Figure 2 When the text-to-speech result shown is obtained, the subtitle can refer to a second text (such as text sent from the server to the client) Figure 2 The response text shown is an example. Text-to-speech (TTS), also known as speech synthesis, is a technique that converts text information into speech signals, such as Pulse Code Modulation (PCM) signals. PCM signals can record and transmit audio signals.

[0069] Based on the above-mentioned content of S101 to S104, it can be seen that for a client capable of interacting with a user, after the client receives the first voice input by the user (such as voice describing the user's question), the client first performs speech recognition processing on the first voice to obtain the first text (such as text describing the user's question), and sends the first text to the server so that the server can process and provide feedback on the first text. After the client receives the second voice, facial expression sequence, and action sequence sent by the server, the client generates an image sequence based on the facial expression sequence and the action sequence, and plays the second voice and displays the image sequence. In this way, the client can automatically answer the user's question, thereby reducing the answering cost. Specifically, because the facial expression sequence can indicate the changes in the virtual person's facial expression when responding to the first voice, and the action sequence can indicate the changes in the virtual person's actions when responding to the first voice, the image sequence generated based on the facial expression sequence and the action sequence can describe the changes in the virtual person in at least one dimension (such as facial expression and / or action), thereby enabling the image sequence to represent the changes (such as changes in facial expression, changes in action, etc.) exhibited by the virtual person when responding to the first voice; furthermore, because the second voice can indicate the content of the virtual person's speech when responding to the first voice, this application presents a vivid and expressive response from the virtual person to the user by playing the second voice and displaying the image sequence, thus effectively overcoming the shortcomings of the aforementioned automatic response service and improving the user experience.

[0070] In addition, to further enhance the user experience, this application also provides a possible implementation of the aforementioned client. In this implementation, the client can be a holographic capsule, so that when a virtual human is projected onto the holographic capsule via a 3D laser, the capsule's structure can utilize dynamic light and shadow with multiple reflections and 3D spatial illusions to ensure that the virtual human presents a 3D stereoscopic effect with a field of view exceeding 180 degrees. This allows the user to interact with the virtual human in an immersive way, reducing the spatial barrier between the user and the virtual human, thereby improving the user experience.

[0071] It should be noted that a holographic capsule is a naked-eye 3D display device created using technologies such as holographic imaging. Specifically, this holographic capsule uses the principle of spatial illusion to perform layered perspective processing on two-dimensional (2D) images, combining this with the sense of spatial depth to generate a naked-eye 3D effect. Furthermore, the holographic capsule is equipped with a high-resolution holographic projection device that uses laser and interference principles to encode light information and reconstruct a three-dimensional image, thus providing realistic 3D images and an immersive experience within a closed space.

[0072] In addition, to further improve the user experience, this application also provides a possible implementation of the aforementioned client. In this implementation, the client can at least satisfy the following constraints: the client can be a holographic capsule, and the screen size of the client is determined based on the height of a real person, so that the virtual person presented by the client can restore the height of a real person at a 1:1 scale, thereby enabling the user to have an immersive voice conversation with the virtual person up close and face-to-face, which is conducive to improving the user experience.

[0073] It should be noted that the "real person's height" mentioned above refers to the result obtained by analyzing the heights of a large number of real people, so that the "real person's height" can better represent the height characteristics of people in the real world.

[0074] Research has found that if a virtual human lacks the ability to express emotions, it will be difficult for users to empathize with the virtual human, thus affecting the user experience.

[0075] Based on the above research, in order to better improve the user experience, this application also provides a possible implementation of the above-mentioned data processing method applied to the client. In this implementation, the data processing method may include at least the following steps: the client receives emotional information sent by the server, the emotional information indicating the emotional state of the virtual person when responding to the first voice; an image sequence is generated based on the emotional information, the above-mentioned facial expression sequence, and the above-mentioned action sequence, so that the emotional state of the virtual person in the image sequence is consistent with the emotional state indicated by the emotional information, thereby enabling the virtual person to respond to the first voice in the emotional state, and thus enabling the virtual person to have the ability to express emotions, so as to effectively overcome the defects caused by the lack of emotional expression ability of the virtual person, which is conducive to improving the user experience.

[0076] It should be noted that this application does not limit the execution time of the server sending the emotion information, as long as it is completed before the client performs the image sequence generation process. In addition, in order to better save communication resources, the server can send the emotion information together with other information (such as second voice, facial expression sequence, action sequence, etc.) to the client.

[0077] Therefore, in one possible implementation, after receiving the first voice input by the user, the client first performs speech recognition processing on the first voice to obtain the first text, and then sends the first text to the server, so that the server can process and provide feedback on the first text, so that the client can receive the second voice, facial expression sequence, action sequence, and emotional information (such as...) sent by the server. Figure 2 After the emotional information shown, the client generates an image sequence based on the emotional information, the expression sequence, and the action sequence. The client also plays the second voice and displays the image sequence. This enables the virtual human on the client to respond to user questions according to the emotional state specified by the emotional information, making it easy for the user to empathize with the virtual human and thus improving the user experience.

[0078] It should be noted that this application does not limit the implementation of the above-mentioned step of "generating an image sequence based on the emotion information, the expression sequence and the action sequence". For example, it can be implemented by any machine learning model that can achieve the aforementioned function or any virtual human driving method that can achieve the aforementioned function.

[0079] In addition, to further improve the user experience, this application also provides a possible implementation of the above-mentioned image sequence generation process. In this method, when the above-mentioned expression sequence indicates a change in lip shape, the image sequence generation process may include: generating a facial expression based on the above-mentioned emotional information, the facial expression indicating the virtual person's facial state other than lip shape when responding to the first speech; generating an image sequence based on the facial expression, the above-mentioned expression sequence, and the above-mentioned action sequence, so that the changes in the virtual person's lip shape in the image sequence are consistent with the changes in lip shape indicated by the expression sequence, and so that the other facial states of the virtual person in the image sequence, other than lip shape, are consistent with the state indicated by the facial expression, thereby enabling the image sequence to better present the facial state presented by the virtual person when responding according to the emotional state indicated by the emotional information, thereby ensuring that the virtual person presents a better emotional expression ability, so as to better improve the user experience.

[0080] As can be seen, in one possible implementation, after the client receives the second voice, facial expression sequence, action sequence, and emotional information from the server in response to the first voice feedback, the client first generates a facial expression based on the emotional information so that the facial expression can express the emotional state indicated by the emotional information (such as happiness). Then, an image sequence is generated based on the facial expression, the facial expression sequence, and the action sequence so that the lip movements of the virtual human in the image sequence are consistent with the lip movements indicated by the facial expression sequence, and so that the other facial states of the virtual human in the image sequence, except for the lip movements, are consistent with the corresponding states indicated by the facial expression. This allows the virtual human in the image sequence to achieve synchronization with the lip movements of the second voice while expressing the emotional state, thereby enabling the virtual human to simulate the emotional expression of a normal person. This makes it difficult for the user to perceive any abnormalities (such as inaccurate lip movements, incorrect emotions, etc.) in the virtual human, thus improving the user experience.

[0081] It should be noted that this application does not limit the implementation of the above-mentioned step of "generating an image sequence based on the facial expression, the expression sequence, and the action sequence". For example, it can be implemented using any machine learning model or any virtual human driving method that can achieve the aforementioned function.

[0082] It should also be noted that this application does not limit the implementation of the above-mentioned step of "generating facial expressions based on the emotional information". For example, it can be implemented using any machine learning model or any expression generation method that can achieve the aforementioned function.

[0083] For example, in some scenarios, when the aforementioned emotional information includes the predicted probability (such as a prediction score) of at least one candidate emotion, the aforementioned facial expression generation process may include: first, determining the target emotion with the highest probability from the at least one candidate emotion based on the predicted probability of the at least one candidate emotion; then, obtaining a pre-set expression template for the target emotion so that the expression template can represent the facial muscle contraction state under the target emotion; and then, using the expression template as the facial expression so that the facial expression can more accurately present the target emotion.

[0084] It should be noted that the above-mentioned at least one candidate emotion refers to a pre-set emotion that the virtual human may exhibit; and this application does not limit the at least one candidate emotion. For example, the at least one candidate emotion may include some or all of the default emotion (also known as calm emotion), joyful emotion (also known as happy emotion), sad emotion, angry emotion, and horrified emotion.

[0085] It should also be noted that, for any candidate emotion, the predicted probability of the candidate emotion is determined by the server so that the predicted probability can indicate the likelihood of the virtual person adopting the candidate emotion when responding to the first speech; and this application does not limit the implementation of the predicted probability, for example, it can be implemented by means of a prediction score.

[0086] It should also be noted that, for any candidate emotion, the corresponding expression template refers to an expression template pre-set for that target emotion, so that the expression template can represent the facial expression under that candidate emotion; and this application does not limit the method of obtaining the expression template, for example, it can be obtained through manual annotation. Alternatively, the expression template can refer to a virtual facial expression animation pre-recorded for that candidate emotion.

[0087] Research has found that for any given emotion, a specific facial expression template can represent the state of facial muscle contraction under extreme conditions. However, because the state of muscle relaxation varies dynamically from person to person at different times, the facial expression an individual displays when expressing the same emotion may differ at different times.

[0088] Based on the above research, in order to better improve the effect of emotion expression, this application also provides a possible implementation of the above-mentioned facial expression generation process. In this method, when the above-mentioned emotion information includes the predicted probability of at least one candidate emotion, the facial expression generation process may include: determining a target emotion from the at least one candidate emotion based on the predicted probability of the at least one candidate emotion, wherein the predicted probability of the target emotion is not less than the predicted probability of other emotions besides the target emotion among the at least one candidate emotion; and performing a weighted fusion of the expression template corresponding to the target emotion and the expression template corresponding to the default emotion based on the predicted probability of the target emotion to obtain the facial expression. The predicted probability of the target emotion indicates the weight of the expression template corresponding to the target emotion in the weighted fusion, the expression template corresponding to the target emotion indicates the facial state under the target emotion, and the expression template corresponding to the default emotion indicates the facial state under the default emotion. In this way, the expression of the current emotion can be optimized by superimposing the emotion template corresponding to the current emotion on the regular expression, thereby facilitating a better expression of the current emotion.

[0089] As can be seen, in one possible implementation, after the client receives the predicted probabilities of each candidate emotion sent by the server, as shown in formulas (1)-(2) below, the client first selects the emotion with the highest probability value from these candidate emotions based on these predicted probabilities; then, the client uses the highest probability value as a weight to superimpose the expression template corresponding to the emotion with the highest probability value onto the expression template corresponding to the default emotion to obtain the above facial expression, so that the facial expression can better represent the emotion with the highest probability value.

[0090] α = arg max i∈{1,2,…,N} score i (1)

[0091] Exp(output) = α × Exp(emotion) score＝α )+(1-α)×Exp(natural) (2)

[0092] In the formula, Exp(output) represents the facial expression mentioned above; α represents the maximum predicted probability of at least one candidate emotion; and score i This represents the predicted probability of the i-th candidate emotion among the at least one candidate emotion, where i is a positive integer and i ≤ the number of emotions among the at least one candidate emotion; score＝α This represents the target emotion with the highest predicted probability among at least one candidate emotion; Exp(emotion) score＝αThis represents the emoji template corresponding to the target emotion. For example, when the target emotion is the default emotion, this Exp(emotion) represents the emoji template. score＝α ) is Exp(natural), when the target emotion is a happy emotion, this Exp(emotion) is... score＝α ) is Exp(happy), which is the value when the target emotion is sadness. score＝α ) is Exp(sad), which is the value of Exp(emotion) when the target emotion is anger. score＝α ) is Exp(angry), which is the value of Exp(emotion) when the target emotion is fear. score＝α ) represents Exp(surprise); natural represents the default emotion among the at least one candidate emotion; Exp(natural) represents the expression template corresponding to the default emotion.

[0093] Research has revealed that, for a client-side application, the virtual avatar displayed by the client may exhibit multiple states during interaction with the user. To facilitate understanding, examples are provided below.

[0094] As an example, consider a complete human-computer dialogue process implemented based on the aforementioned client and server (such as...). Figure 3 In the context of the dialogue process shown, this process can include: After the user launches the client (such as a holographic capsule), if no operation is performed within 10 seconds, the client interface is in an inactive state. During this time, because the user has not triggered a dialogue for an extended period, the virtual person displayed by the client is in a standby state (i.e., waiting for the user to speak), thus exhibiting a smiling and natural posture. Then, the user clicks the record button and inputs the question "What are the features of this product?" through the microphone. At this point, the virtual person displayed by the client enters a listening state, exhibiting a slightly tilted body, a slight nod, and focused gaze. Next, after the user finishes speaking, the user enters a state of waiting for the virtual person's response. Since the client receives the signal that speaking has ended, the virtual person displayed by the client enters a thinking state, exhibiting occasional gestures such as resting their chin on their hand or frowning, indicating thoughtfulness. Furthermore, because the question includes the keyword "product features," it triggers the Frequently Asked Questions (FAQ) search mode (e.g.,...). Figure 3The search mode shown indicates that the server (or client) retrieves relevant information from the database corresponding to that search mode to enter a response state, providing the user with detailed answers to their questions. Furthermore, while the user waits for an answer, after a certain period, the information is successfully returned, at which point the user can see and hear the response provided by the client. Additionally, in the response state, the virtual person displayed by the client can perform a series of actions, such as speaking while using gestures to explain the various features of the product. Simultaneously, relevant product feature introduction videos or graphic information may be inserted into the client's interface to create a vivid and complete dialogue with the user.

[0095] It is evident that for a complete human-computer dialogue process implemented based on the aforementioned client and server (such as...) Figure 3 In the context of the dialogue process shown, the process can include a first action, a second action, and a third action. The first action occurs at the application physical layer of the client, enabling it to collect information such as user-triggered operations and user-inputted voice commands. The second action occurs on the server side, enabling it to understand the dialogue intent and generate responses and feedback information for different scenarios based on the question. The third action occurs at the display layer of the client, enabling it to drive the virtual human to perform scripted actions or answer questions; this third action may occur actively or passively.

[0096] Based on the above example, in order to better improve the service effect, this application also provides a possible implementation of the above client, in which the client includes a state machine (e.g., Figure 4 The state machine shown indicates the switching relationship between different states of the virtual human displayed by the client. In this way, the client can be guaranteed to support the state transition of the virtual human in different scenarios, so as to improve flexibility.

[0097] It should be noted that this application does not limit the implementation method of the above-described state machine; for example, such as... Figure 4As shown, this state machine can indicate the switching relationships between the power-on state, greeting state, standby state, listening state, thinking state, and speaking state. The power-on state refers to the initial state of the virtual human displayed by the client after the client is started. Upon wake-up, it can enter the greeting state, and immediately enter the standby state after greeting. The standby state refers to the state where the virtual human waits for user commands. After receiving user voice input, it enters the listening state; this standby state can also be returned from the speaking state. If the standby state is maintained for a certain period, it will return to the power-on state, at which point the virtual human needs to be woken up again. The listening state refers to the state entered after receiving user voice input, allowing it to enter the thinking state after the voice input ends. The thinking state refers to the state when the client and server process user input, allowing it to enter the speaking state after processing is complete; this thinking state can be interrupted by listening or speaking. The speaking state refers to the state in which the virtual human responds to the user; after the speaking state ends, if no user voice input is detected within a preset time, it enters the standby state; however, if user voice input is detected within a preset time, it enters the listening state.

[0098] It should also be noted that for the state machine described above, after completing one round of interaction from the listening state → thinking state → speaking state, it can return to the standby state loop, as follows: Figure 5 The loop is shown. Simultaneously, after being woken up, the user greets the user before entering standby mode; when user input is processed by the client and server, the user transitions from listening mode to thinking mode, so that when the client responds to the user's input via a virtual avatar, the user transitions from thinking mode to speaking mode.

[0099] Based on the aforementioned state machine content, in order to better improve the user experience, this application also provides a possible implementation of the aforementioned data processing method applied to the client. In this method, when the client includes a state machine, and the state machine indicates the switching relationship between different states of the virtual human displayed by the client, the data processing method may include at least the following steps: after the client receives some information sent by the server (such as second voice, facial expression sequence, action sequence, emotion information, etc.), the client updates the state information of the virtual human according to the state machine. The updated state information indicates that the virtual human switches to the speaking state. In this way, it can automatically switch from the listening state to the speaking state based on the information fed back by the server, thereby improving the flexibility of the virtual human's state transition.

[0100] It can be seen that, in one possible implementation, for the client (such as...) Figure 2For the client shown, after receiving some information (such as second voice, facial expression sequence, action sequence, emotional information, etc.) from the server in response to the user's first voice input, the client first switches the virtual human's state information from listening state to speaking state; then the client provides this information to the user from both visual and auditory perspectives by driving the virtual human based on this information.

[0101] Research has found that to enhance the realism of virtual humans, certain physical characteristics (such as eye movement control) can be manipulated. For example, in standby mode, the virtual human's blinking frequency can be controlled to match the blinking frequency in a real scene. In listening, thinking, or speaking mode, the virtual human can be controlled to blink once every 2.2 to 3 seconds on average. Furthermore, to better guide the user, in listening, thinking, or speaking mode, the virtual human's head, eyes, and even body can be controlled to face the camera (such as a virtual camera used during rendering).

[0102] Based on the above research, in order to better improve the user experience, this application also provides a possible implementation of the above-mentioned data processing method applied to the client. In this method, when the client includes a state machine, and the state machine indicates the switching relationship between different states of the virtual person displayed by the client, the data processing method may include at least the following steps: After the client receives some information (such as facial expression sequence, action sequence, emotion information, etc.) sent by the server, the client first updates the state information of the virtual person according to the state machine, so that the updated state information indicates that the virtual person switches to the speaking state; then, based on this information and at least one body feature corresponding to the updated state information, an image sequence is generated. The at least one body feature includes blinking frequency, head orientation, eye orientation, and part or all of the body orientation, so that the virtual person in the image sequence generated based on these facial features can meet these body features, thereby making the virtual person in the image sequence more realistic and better improving the user experience.

[0103] It is evident that for the client (such as...) Figure 2For the client shown, after receiving the second voice, facial expression sequence, action sequence, and emotional information from the server in response to the user's first voice input, the client first switches the virtual human's state from listening to speaking. Then, the client searches a pre-built mapping relationship for body features (such as blinking frequency, head orientation, eye orientation, and body orientation) that match the speaking state, so that these body features can represent some physical characteristics of the virtual human displayed by the client in the speaking state. Next, the client generates an image sequence based on these body features, the facial expression sequence, the action sequence, and the emotional information, so that the virtual human in the image sequence at least presents these body features. Finally, the client simultaneously plays the second voice and the image sequence, so that the user can receive a response from both visual and auditory perspectives. The mapping relationship records some body features corresponding to various states; and this mapping relationship can be pre-built according to the virtual human requirements of the actual application scenario.

[0104] It should be noted that this application does not limit the implementation of the above-mentioned step of "generating an image sequence based on these body features, the expression sequence, the action sequence, and the emotional information". For example, it can be implemented using any pre-built machine learning model that can achieve the aforementioned function.

[0105] Research has found that in some scenarios, the client can not only passively respond to user-inputted questions, but also actively broadcast content, such as the virtual person displayed by the client initiating chats, introducing products, and performing certain scheduled tasks.

[0106] Based on the above research, and to better meet the requirements of the above section, this application also provides a possible implementation of the aforementioned data processing method applied to a client. In this implementation, the data processing method may include at least the following steps: the client receives a first voice message; in response to detecting that the first voice message does not have a preset feature, the client performs speech recognition processing on the first voice message to obtain a first text message (e.g., ...). Figure 2 The client, in response to detecting that the first speech has the preset feature, sends the text corresponding to the preset feature (such as the problem text shown), and sends the first text to the server; the client, in response to detecting that the first speech has the preset feature, sends the text corresponding to the preset feature (such as the problem text shown), and sends the first text to the server; Figure 6 The text to be broadcast is sent to the server, and the preset feature indicates the voice (such as...) that the virtual person displayed by the client responds via broadcast. Figure 6 The features of the wake-up voice shown enable the virtual human displayed by the client to not only have question-and-answer functions but also automatic broadcast functions, which helps to better meet the needs of users in different scenarios and thus improve the user experience.

[0107] It should be noted that the aforementioned preset features refer to user voice that requires a response from the client via broadcast (e.g., ...). Figure 6 The wake-up voice shown has the following characteristics, such as carrying a wake-up word and carrying pre-set information (such as vocabulary, events, tone, etc.); moreover, this application does not limit the implementation of the preset features. For example, the preset features can be determined according to the automatic broadcasting requirements of the current application scenario.

[0108] It should also be noted that, for the text corresponding to the aforementioned preset features, this text refers to pre-defined text that is bound to the preset feature and needs to be displayed to the user through automatic playback, such as... Figure 6 The text to be broadcast is shown. Furthermore, this application does not limit the implementation method of this text. For example, the text can be a script or document edited according to the rules of Speech Synthesis Markup Language (SSML), so that the text not only records the virtual human's speech content, but also some speech characteristics of the virtual human, such as pronunciation characteristics, actions, events, etc. SSML is a markup language based on Extensible Markup Language (XML), which can mark and annotate text in a speech synthesis system to better control the way and effect of speech output. In addition, this application does not limit the server-side processing procedure for this text, for example, such as... Figure 6 As shown, the processing procedure may include: after the server receives the text, it first converts the text into spoken language using TTS; then, it performs facial expression prediction (such as lip-reading prediction) on the spoken language to obtain the facial expression sequence corresponding to the text; and then, the server provides the spoken language and the facial expression sequence to the client. Furthermore, the client can use an action library to determine the action sequence corresponding to the text. Moreover, the action sequence corresponding to the text can be obtained by fusing the result of the server using an action prediction model to predict the actions of the spoken language with the result obtained by the client querying the text from the action library. This application does not limit the method of obtaining the action sequence; it can be determined according to the actual application scenario.

[0109] Based on the above three paragraphs, it can be seen that in one possible implementation, for the client, after receiving the first voice input from the user, if the client detects that the user's request indicated by the first voice is a question-and-answer request, the client can first perform speech recognition processing on the first voice to obtain the first text, so that the first text can represent the question described by the first voice; then the client sends the first text to the server, so that the server can generate a response voice for the question based on the first text (e.g., ...). Figure 2 The response voice shown) and its related information (such as Figure 2 The client receives feedback on the emotional information, facial expression sequences, and action sequences shown in the first voice message for use; however, if the client detects that the user's request indicated by the first voice message is a wake-up request, the client can obtain pre-set text for wake-up and send it to the server, so that the server can generate corresponding voice based on the "pre-set text for wake-up" (such as...). Figure 6 The client receives the broadcast voice and related information (such as facial expression sequences, action sequences, etc.) for use, thus enabling the client to support not only question-and-answer functions but also automatic broadcast functions, thereby meeting the usage needs of more scenarios.

[0110] Research has found that in some scenarios, such as those with high data security requirements, a database capable of answer retrieval can be deployed on the client side to protect the data stored in that database.

[0111] Based on the above research, and to better meet the needs described above, this application also provides a possible implementation of the aforementioned data processing method applied to a client. In this implementation, when the client includes a first database (such as...), Figure 2 When using the local database shown in Information Retrieval 1), the data processing method may include at least the following steps: The client receives a first voice message and performs speech recognition processing on the first voice message to obtain a first text message; in response to the first text message not meeting the retrieval conditions of the first database, the client sends the first text message to the server, so that the server can generate feedback information (such as the aforementioned second voice message, facial expression sequence, action sequence, emotion information, etc.); in response to the first text message meeting the retrieval conditions of the first database, the client sends the retrieval results of the first text message in the first database to the server, so that the server can generate feedback information (such as the aforementioned second voice message, facial expression sequence, action sequence, emotion information, etc.) based on the retrieval results. Figure 2 The feedback information is generated by generating the output data of Information Retrieval 1 shown. This allows the security of some data to be protected by deploying a database locally on the client, thus enabling the client to better meet the needs of scenarios with high data security requirements.

[0112] It should be noted that the aforementioned first database refers to a database deployed on the client that stores some question-and-answer pairs, enabling the client to perform FAQ-mode retrieval processing locally using this first database, thereby allowing subsequent retrieval of answers to some questions from this first database (such as...). Figure 2 (The output data of information retrieval 1 shown). Furthermore, this application does not limit the first database; it can be manually created by relevant personnel on the client or automatically constructed by the client based on a large amount of data provided by the user. Moreover, the retrieval criteria for the first database refer to the conditions that must be met for a question to retrieve an answer from the first database; and this application does not limit the retrieval criteria. For example, the retrieval criteria could be: at least one keyword in the user-input question belongs to a keyword existing in the first database.

[0113] Based on the above two paragraphs, in one possible implementation, for the client, when the client includes a local database (such as the first database mentioned above), after the client receives the first voice input from the user, the client first performs speech recognition processing on the first voice to obtain the first text; then the client determines whether each word in the first text belongs to a keyword existing in the local database; if it is determined that none of the words in the first text belong to a keyword existing in the local database, it can be determined that the answer to the first text does not exist in the local database, so the client sends the first text to the server, so that the server can generate a response voice and related information based on the first text. (e.g., emotional information, facial expression sequences, action sequences, etc.) are fed back to the client for use; however, if it is determined that at least one word in the first text belongs to a keyword existing in the local database, it can be determined that the answer to the first text exists in the local database. Therefore, the client can perform an answer search for the first text in the local database to obtain search results, and the client can send the search results to the server so that the server can generate a response voice and related information based on the search results and feed it back to the client for use. In this way, the security of some data can be protected by the local database, so that the client can better meet the needs of scenarios with high data security requirements.

[0114] It should be noted that this application does not limit the implementation of the step "the server can generate a response voice and related information based on the search result". For example, this step may include: other processes besides the answer text generation process in any implementation of the step "the server can generate a response voice and related information based on the first text", such as... Figure 6 The process shown is the implementation of the server processing the output data of information retrieval 1.

[0115] Based on the aforementioned client-related content, the client provided in this application has at least the following functions: (i) The client can process user input, specifically: the client can collect user voice through a microphone and call the ASR service to convert the user's voice into text information. (ii) The client can process information provided by the server, specifically: the client can analyze data such as audio (PCM), facial expressions (Blend Shape, BS), skeletal motion (Bone), and event signals (such as event signals recorded in the broadcast text) fed back by the server. (iii) The client can perform rendering and interaction, specifically: the client can combine state machine, facial expression template, and subtitle information, and finally render and display it through the rendering module to achieve real-time visual interaction with the user.

[0116] It's important to note that Shapeshifting (BS) is a technique that creates different shapes by deforming the vertices of a base model (such as a facial model); moreover, BS can blend these shapes with different weights to achieve smooth transitions and deformation effects. The Bone indicates the rotation and offset of the body's skeleton. The Render refers to the virtual human renderer, which receives drive signals and instructions to control the virtual human's movements and expressions.

[0117] It should also be noted that, for the aforementioned client, the pre-recorded animation can be played by default in standby, listening, or thinking states. Furthermore, the animation in standby state can be obtained by recording long-shot data of real people and using long-template animation. In addition, this application will also pre-create a nodding animation for an interactive scene, so that in thinking state, the client can select whether to add the nodding animation, whether to increase the nodding speed, and whether to increase the nodding amplitude based on the audio rhythm indicated by signals such as the onset signal and peak signal. Here, onset refers to the starting point of the signal, that is, the moment when the signal begins to change. Peak refers to the maximum value or highest point reached by the signal at a certain moment.

[0118] Based on the aforementioned client-side content, in order to better provide services to this client, this application also provides a data processing method applied to the server side, such as... Figure 7 As shown, the method includes S701-S703 below.

[0119] S701: The server receives the first text sent by the client, determines the second text based on the first text, and generates the second speech based on the second text.

[0120] The relevant content of the first text can be found above, and will not be repeated here for the sake of brevity.

[0121] The second text refers to the answer determined by the server in response to the question indicated by the first text, so that the second text can represent the content of speech used in response to the first text (or first speech); and this application does not limit the method of determining the second text. For example, it can be implemented using any method that can generate an answer to a question, such as any machine learning model that can achieve the aforementioned function.

[0122] The second speech refers to speech data generated based on the second text, enabling the second speech to represent the semantic content carried by the second text through audio, thereby allowing the second speech to represent the spoken audio used in response to the aforementioned first text (or first speech). It should be noted that this application does not limit the generation method of the second speech; for example, it can be implemented using any method capable of generating speech based on text, such as TTS, or any machine learning model capable of achieving the aforementioned functions.

[0123] Based on the above-mentioned content of S701, it can be seen that, for the server, after receiving the first text sent by the user that can represent the question, the server first performs answer determination processing on the first text to obtain the second text, so that the second text can represent the answer to the question indicated by the first text in text form; then, TTS is used to perform speech generation processing on the second text to obtain the second speech, so that the second speech can represent the answer in audio form.

[0124] S702: The server determines the action sequence based on the second speech and the second text, and predicts the expression sequence based on the second speech.

[0125] The action sequence refers to a series of actions determined based on the second speech and the second text, so that the action sequence can indicate the changes in the actions of the virtual human during the playback of the second speech, thereby enabling the action sequence to represent the changes in actions presented in response to the first text (or the first speech).

[0126] Furthermore, this application does not limit the process for determining the aforementioned action sequence. For example, it can be implemented using any method capable of generating actions based on speech and text, such as any machine learning model capable of achieving the aforementioned functions, or an Artificial Intelligence (AI) Motion service. Here, AIMotion refers to a method that can drive virtual human limb movements through speech signals and AI prediction. Alternatively, the action sequence can be determined by querying and matching actions from an action database.

[0127] An expression sequence refers to some facial information (such as lip movements) predicted based on the second speech, so that the expression sequence can indicate the changes in facial expressions of the virtual human during the playback of the second speech, thereby enabling the expression sequence to represent the changes in facial expressions presented in response to the first text (or the first speech).

[0128] Furthermore, this application does not limit the prediction process of the aforementioned facial expression sequence. For example, it can be implemented using any method capable of predicting facial expressions based on speech, such as any machine learning model capable of achieving the aforementioned functions, any lip-shape prediction model, or an AI face service. Here, AIFace refers to a method capable of driving virtual human lip shapes and facial expressions through speech signals and AI prediction techniques.

[0129] Furthermore, this application does not limit the characteristics of the lip shape indicated by the above-mentioned facial expression sequence. For example, the lip shape can be implemented using the Arkit standard so that the facial expression sequence can be predicted by a lip shape driven model that has been trained in advance according to the training data under the Arkit standard.

[0130] For example, in order to better present a more expressive and standard speaking state, the above mouth shape can adopt the Iclone standard, so that the expression sequence can be predicted by the mouth shape driven model that has been trained in advance according to the training data under the Iclone standard, so that the expression sequence can present stronger facial muscle expressiveness and differentiability of different pronunciation mouth shapes.

[0131] S703: The server sends the second voice, facial expression sequence, and action sequence to the client.

[0132] It should be noted that this application does not limit the implementation of the above-described S703. For example, in order to better save communication resources, S703 can be: first, the server summarizes the second voice, expression sequence and action sequence to obtain a summary result; then, the server sends the summary result to the client.

[0133] Based on the above content in S701 to S703, it can be seen that, for the server, after receiving the first text sent by the client (such as the problem text shown in 2), the server first determines the second text (such as...) based on the first text. Figure 2 The second text (shown as the response text) is used to represent the response to the question indicated by the first text; then, a second speech is generated based on the second text (such as...). Figure 2 The second voice (shown as the response voice) is used to represent the content spoken in response to the first text; then, the server determines the action sequence (such as...) based on the second voice and the second text. Figure 2 The action sequence shown is used to express the changes in action presented in response to the first text, and the server predicts the facial expression sequence based on the second voice (such as...). Figure 2 The server uses a sequence of facial expressions (as shown) to express the changes in facial expressions when responding to the first text. Finally, the server sends the second voice, the facial expression sequence, and the action sequence to the client for use, enabling the client to respond to the first text with expressive voice based on this information, thus improving the user experience. Since both the action sequence and the facial expression sequence are determined based on the second voice, the changes in actions indicated by the action sequence and the changes in actions indicated by the facial expression sequence are highly correlated with the content of the speech indicated by the second voice. This results in a more human-like virtual persona when the client responds based on this information, further enhancing the user experience.

[0134] In addition, to further improve the action determination effect, this application also provides a possible implementation of the above-mentioned "determining the action sequence based on the second speech and the second text" step. In this method, the step may include the following steps: for any word in the second text, searching for an action that matches the word from a pre-built action library to obtain a matching result for the word, so that the matching result can indicate whether there is an action that matches the word in the action library; and in response to the matching result indicating that there is an action that matches the word in the action library, determining the start and end times (such as start time and end time) of the audio segment corresponding to the word from the second speech, so that the start and end times can indicate the playback time of the audio segment corresponding to the word at the time of response, thereby enabling the start and end times to indicate the usage time of the word at the time of response, and further enabling the start and end times to indicate to some extent the display time of the action that matches the word at the time of response; according to the start and end times, fusing the matched action and the action prediction result generated based on the second speech to obtain an action sequence, so that the action sequence can better represent the action changes at the time of response, which is beneficial to improving the accuracy of action determination.

[0135] It should be noted that the aforementioned action library refers to a pre-constructed database that records actions of a large number of words (such as words with relatively high word frequency). Furthermore, this application does not limit the method of constructing the action library. For example, it could specifically involve: first, analyzing a large number of words to obtain some high-frequency words (also known as hot words); then, creating animation clips for each high-frequency word so that the animation clips can represent the actions required when the corresponding words appear; then, based on the correspondence between these high-frequency words and these action clips, constructing an action library so that the action library can record the actions associated with each high-frequency word. Additionally, this application does not limit the implementation method of the animation clips. For example, the average duration of the animation clip is 2-4 seconds. Also, the start and end poses in the animation clip remain consistent.

[0136] It should also be noted that the aforementioned action prediction result refers to the action sequence obtained by performing action prediction on the second speech, so that the action prediction result can represent an action sequence that satisfies some characteristics of the second speech, such as rhythm and other audio characteristics. Therefore, in one possible implementation, the action prediction result can indicate that the second speech carries information other than semantic information.

[0137] It should also be noted that this application does not limit the implementation of the step of "fusing the matched action and the action prediction result generated based on the second speech according to the start and end times to obtain the action sequence". For example, it can be implemented using any method that can synthesize two action sequences into one action sequence (such as splicing). Alternatively, this step can be: inserting the matched action into the action prediction result according to the start and end times to obtain the action sequence.

[0138] Research has found that for some words (such as "hello"), the word should ideally appear at the beginning of a sentence, not the end, so that the action associated with that word should also not appear at the end. It should be noted that "end of sentence" refers to the end of a complete sentence.

[0139] Based on the above research, in order to better meet the above needs, this application also provides a possible implementation of the step of "fusing the matched action and the action prediction result generated based on the second speech according to the start and end times to obtain an action sequence". In this way, the step may include: determining the insertion time of the matched action according to the start and end times, so that the insertion time can indicate the timing of the display of the matched action in the response; in response to the fact that the position indicated by the insertion time in the second text is not the end of the sentence, concatenating the matched action with the action prediction result according to the insertion time to obtain a concatenated result, and smoothing the concatenated result to obtain the action sequence. This can effectively avoid inserting actions at some inappropriate positions, thereby helping to improve the accuracy of action determination.

[0140] It should be noted that, for any given word, the insertion time of the action matching that word is determined based on the start and end times of the corresponding audio segment in the second speech, so that the insertion time can indicate the timing of the display of the matching action during the response. Furthermore, this application does not limit the implementation of the insertion time. For example, when the start and end times include a start time and an end time, the insertion time refers to a randomly selected moment within the time period from the start time to the end time. As another example, since some actions matching words are suitable to be displayed after the word is displayed, some actions matching words are suitable to be displayed at the same time as the word begins to be displayed, and some actions matching words are suitable to be displayed after a short wait after the word begins to be displayed, in order to better improve the action display effect, the insertion time can be determined based on the start and end times and the action display requirements of the word, so that the insertion time satisfies the action display requirements. The action display requirements can indicate the relative relationship between the timing of the display of the action matching the word and the display time period of the word.

[0141] Based on the above seven paragraphs, it can be seen that for the server, after obtaining the second text and the second speech, the server uses an action library to perform action matching on some words in the second text to obtain semantic actions, and uses an action prediction model to predict actions in the second speech to obtain rhythmic and other actions. The server then merges these two types of actions into an action sequence and provides it to the client for use. Because this action sequence is obtained through action selection and smoothing, it can accurately represent the changes in actions during a response. Therefore, the client can use it directly without needing to perform selection and smoothing on the action sequence itself, thus improving the client's display effect.

[0142] In addition, in some scenarios, the aforementioned action selection and smoothing can also be performed by the client. Therefore, in one possible implementation, this application also provides a scheme for action matching, insertion, and splicing based on high-frequency words. This scheme includes at least the following steps: First, create an animation segment for each high-frequency word, with an average duration of 2-4 seconds. The start and end postures of this animation segment are consistent, so that the server can subsequently find the animation segments associated with the high-frequency words appearing in the second text from these animation segments. The start and end times corresponding to the high-frequency words appearing in the second text are found on the audio track, and these start and end times are recorded using SSML action tags. Then, after the server sends this information to the client, when the client plays the second audio to the designated position of the animation segment, it determines whether the insertion timing of the animation segment is at the end of a sentence. If so, the animation segment is discarded; otherwise, it is inserted using a smoothing algorithm. Since the starting posture in each animation segment is fixed, this ensures that the basic postures of the actions before and after the keyframes indicating the word's action are largely consistent, thereby improving coherence.

[0143] Based on the above, in one possible implementation, for a client with abundant computing resources, the working principle of the client may include at least the following steps: After the client receives information such as second text, second speech, facial expression sequence, and action sequence sent by the server, if the action sequence includes action prediction results based on the second speech and actions matched from the action library based on the second text, and the second text includes the insertion time of the matched action recorded by SSML, then the client generates the aforementioned image sequence based on this information sent by the server, so that the image sequence includes the image corresponding to each frame of action in the action prediction result and the matched action... The client creates a corresponding image so that, during the display of the image sequence, in response to playing the image corresponding to the matched action, it determines whether the insertion time of the matched action in the second text is at the end of a sentence. If so, the image corresponding to the matched action is skipped; otherwise, the image corresponding to the matched action and its adjacent images in the image sequence (such as the previous frame and / or the next frame) are smoothed to obtain the smoothed result of the image corresponding to the matched action and the smoothed result of the next frame, which are then played sequentially. This allows the client to merge actions from different sources, thereby reducing the processing pressure on the server.

[0144] It should be noted that the aforementioned "previous frame image" refers to an image in the image sequence that is adjacent to the aforementioned "image corresponding to the matched action" and whose arrangement position is earlier than that of the aforementioned "image corresponding to the matched action". The aforementioned "next frame image" refers to an image in the image sequence that is adjacent to the aforementioned "image corresponding to the matched action" and whose arrangement position is later than that of the aforementioned "image corresponding to the matched action".

[0145] In addition, to further improve the expression prediction results, this application also provides a possible implementation of the above-mentioned data processing method applied to the server. In this implementation, the data processing method may include at least the following steps: the server predicts emotional information based on the second speech and the second text, so that the emotional information can indicate the emotional state presented during the playback of the second speech; based on the emotional information and the second speech, the server predicts an expression sequence, so that the expression sequence can present the expression changes in the emotional state indicated by the emotional information, thereby enabling the expression sequence to more accurately represent the expression changes (such as lip movements) presented during the playback of the second speech, which is beneficial to improving the accuracy of expression prediction.

[0146] It should be noted that this application does not limit the implementation of the above-mentioned step of "predicting emotion information based on second speech and second text". For example, it can be implemented using any method that can identify emotions based on speech and text, such as any machine learning model that can achieve the aforementioned function.

[0147] It should also be noted that this application does not limit the implementation of the above-mentioned step of "predicting the expression sequence based on the emotion information and the second speech". For example, it can be implemented by any method that can generate expressions (such as lip movements) based on emotions and speech, such as any machine learning model that can achieve the aforementioned function.

[0148] In addition, to further improve the action prediction results, this application also provides a possible implementation of the above-mentioned data processing method applied to the server. In this implementation, the data processing method may include at least the following steps: the server predicts emotion information based on the second speech and the second text, so that the emotion information can indicate the emotional state presented during the playback of the second speech; and determines an action sequence based on the emotion information, the second speech, and the second text, so that the action sequence can present the action changes in the emotional state indicated by the emotion information, thereby enabling the action sequence to accurately represent the action changes presented during the playback of the second speech, which is beneficial to improving the accuracy of action prediction.

[0149] It should be noted that this application does not limit the implementation of the above-mentioned step of "determining the action sequence based on the emotion information, the second voice, and the second text". For example, it can be implemented using any method that can generate actions based on emotion, voice, and text, such as any machine learning model that can achieve the aforementioned function.

[0150] For example, the step of "determining the action sequence based on the emotion information, the second speech, and the second text" can specifically be: first, generating an action prediction result based on the emotion information and the second speech; then, determining the action sequence based on the action prediction result and the action matched against the second text from the action library. This application does not limit the implementation of the step of "generating an action prediction result based on the emotion information and the second speech." For example, it can be implemented using any method capable of predicting actions based on emotion and speech, such as any machine learning model capable of achieving the aforementioned function.

[0151] In addition, to improve the client-side display effect, this application also provides a possible implementation of the above-mentioned data processing method applied to the server. In this implementation, the data processing method may include at least the following steps: the server predicts emotional information based on the second voice and the second text; the server sends the emotional information, the second voice, the above-mentioned facial expression sequence and the above-mentioned action sequence to the client, so that the client can respond with emotion based on this information, which is beneficial to improving the client's emotional expression ability.

[0152] Therefore, in one possible implementation, after receiving the first text (such as the question text shown in Figure 2) sent by the client, the server first determines the second text (such as...) based on the first text. Figure 2 The server then generates a second voice message based on the second text (as shown in the response text); Figure 2 The server then predicts emotional information (such as the response voice shown) based on the second voice and the second text. Figure 2 The server uses the emotional information shown and the second voice to predict the facial expression sequence (e.g., the emotional information shown). Figure 2 The emotional sequence shown), and the action sequence determined by the server based on the second speech and the second text (such as...). Figure 2 (The action sequence shown); Finally, the server summarizes the second voice, the expression sequence, the action sequence, and the emotional information and sends them to the client for use, so that the client can respond to the first text with expression based on this information, thereby improving the user experience.

[0153] Furthermore, in some scenarios, to better improve the answer determination effect, this application also provides a possible implementation of the above-mentioned data processing method applied to the server, in which the server includes a second database (such as...). Figure 2 When using the database shown in Information Retrieval 2, the data processing method may include at least the following steps: the server receives a first text sent by the client; in response to the first text satisfying the retrieval conditions of the second database, the server retrieves the second text from the first text in the second database; in response to the first text not satisfying the retrieval conditions of the second database, the server inputs the first text into a pre-built response model (such as...). Figure 2 The response model shown is used to obtain the second text output by the response model. This allows for answer determination through multiple methods, thereby improving the accuracy of the answer.

[0154] It should be noted that the aforementioned second database refers to a database deployed on the server that stores some question-and-answer pairs. This second database enables FAQ-style retrieval processing on the server, allowing subsequent retrieval of answers to some questions from the second database (e.g., ...). Figure 2 (The output data of information retrieval 2 shown). Furthermore, this application does not limit the second database; it can be manually created by relevant personnel on the server, or it can be automatically constructed by the server based on a large amount of data provided by the user. In addition, the search criteria for the second database refer to the conditions that must be met for a question to retrieve an answer from the second database; and this application does not limit the search criteria. For example, the search criteria could be: at least one keyword in the user-input question belongs to a keyword existing in the second database.

[0155] Based on the above two paragraphs, in one possible implementation, when the server includes a second database, after receiving the first text sent by the client, the server first determines whether each word in the first text belongs to a keyword existing in the second database. If it is determined that none of the words in the first text belong to a keyword existing in the second database, it can be determined that the answer to the first text does not exist in the second database. Therefore, the server inputs the first text into a pre-built response model so that the response model can perform answer prediction processing on the first text to obtain and output the second text. However, if it is determined that at least one word in the first text belongs to a keyword existing in the second database, it can be determined that the answer to the first text exists in the second database. Therefore, the server can perform an answer retrieval on the first text in the second database to obtain the second text. In this way, the second text can be obtained through retrieval mode and model processing mode (e.g., ...). Figure 3The combination of the model processing modes shown in the diagram can better improve the accuracy of the answers.

[0156] In addition, to further improve the accuracy of answers, this application also provides a possible implementation of the above-mentioned data processing method applied to the server. In this implementation, when the first text indicates the semantic content carried by the first speech, the data processing method may include at least the following steps: the server receives the emotion recognition result sent by the client, the emotion recognition result indicating the emotional information carried by the first speech; the server inputs the first text and the emotion recognition result into a pre-built response model to obtain the second text output by the response model, so that the answer given by the second text can better correspond to the emotional state indicated by the emotional information, so that the response based on the second answer is more humanized.

[0157] It should be noted that the emotion recognition result shown above is obtained through analysis of the first speech; moreover, this application does not limit the method of obtaining the emotion recognition result. For example, it can be implemented using any method capable of recognizing emotional states from speech. Furthermore, since ASR has emotion recognition functionality, in one possible implementation, both the first text and the emotion recognition result are obtained through speech recognition processing (such as ASR) on the first speech.

[0158] Based on the above-mentioned server-side content, it can be seen that the server-side can include the following characteristics: (1) The server-side can provide some services to the client, such as interactive central control services, etc. Specifically, after the server-side receives the question text provided by the client, it first performs FAQ retrieval and judgment; if the question text meets the requirements of the FAQ retrieval, it will accurately match the answer through information retrieval; at the same time, it will pass the matched answer to the subsequent processing module for processing; however, if the question text does not meet the requirements of the FAQ retrieval, it will input the question text into the response model for processing, and pass the output data of the response model to the subsequent processing module for processing. (2) The server-side can drive signal aggregation and data integration, specifically, after the server-side obtains a lot of information from some processing modules, it will aggregate this information so that the aggregation result includes multiple types of data, so that this data can be provided to the client for rendering. (3) The server-side can also cooperate to implement automatic broadcasting function, specifically, after the client sends the pre-set broadcasting text (such as text and script edited according to SSML rules) to the server-side, the server-side processes the broadcasting text (such as text and script edited according to SSML rules). Figure 6 The server performs the processing shown in the diagram and provides feedback so that the client can automatically broadcast the virtual human based on the feedback.

[0159] Based on the data processing method provided in the embodiments of this application, the embodiments of this application also provide a data processing apparatus, which will be described below in conjunction with... Figure 8 Explanation and clarification will be provided. Among them, Figure 8 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. It should be noted that for technical details of the data processing apparatus provided in this application embodiment, please refer to the relevant content of the data processing method above.

[0160] like Figure 8 As shown, the data processing apparatus 800 provided in this embodiment includes:

[0161] The first unit 801 is used to receive the first voice, perform voice recognition processing on the first voice to obtain the first text, and send the first text to the server.

[0162] The receiving unit 802 is configured to receive a second voice, an expression sequence, and an action sequence sent by the server. The second voice indicates the content of the virtual person's speech when responding to the first voice, the expression sequence indicates the changes in the virtual person's expression when responding to the first voice, and the action sequence indicates the changes in the virtual person's action when responding to the first voice.

[0163] The generation unit 803 is configured to generate an image sequence based on the expression sequence and the action sequence, the image sequence describing the changes of the virtual human in at least one dimension, the at least one dimension including expression and / or action;

[0164] The playback unit 804 is used to play the second voice and display the image sequence.

[0165] In one possible implementation, the data processing device 200 is deployed on the client; or, the data processing device 200 is independent of the client, but the data processing device 200 and the client are capable of data communication.

[0166] In one possible implementation, the data processing device 200 may be implemented using software or hardware (such as a circuit board).

[0167] In one possible implementation, the client is a holographic capsule.

[0168] In one possible implementation, the receiving unit 802 is further configured to: receive emotional information sent by the server, the emotional information indicating the emotional state of the virtual person when responding to the first voice;

[0169] The generation unit 803 is specifically used to generate an image sequence based on the emotion information, the expression sequence, and the action sequence.

[0170] In one possible implementation, the facial expression sequence indicates changes in lip shape;

[0171] The generation unit 803 is specifically used for: generating facial expressions based on the emotion information, wherein the facial expressions indicate the facial states of the virtual person other than mouth shapes when responding to the first voice; and generating the image sequence based on the facial expressions, the expression sequence, and the action sequence.

[0172] In one possible implementation, the emotion information includes the predicted probability of at least one candidate emotion;

[0173] The generation unit 803 is specifically configured to: determine a target emotion from the at least one candidate emotion based on the predicted probability of the at least one candidate emotion, wherein the predicted probability of the target emotion is not less than the predicted probability of other emotions besides the target emotion among the at least one candidate emotion; and perform a weighted fusion of the expression template corresponding to the target emotion and the expression template corresponding to the default emotion based on the predicted probability of the target emotion to obtain the facial expression, wherein the predicted probability of the target emotion indicates the weight of the expression template corresponding to the target emotion in the weighted fusion, the expression template corresponding to the target emotion indicates the facial state under the target emotion, and the expression template corresponding to the default emotion indicates the facial state under the default emotion.

[0174] In one possible implementation, the client includes a state machine that indicates the switching relationship between different states of the virtual human;

[0175] The data processing device 800 further includes an update unit, configured to update the state information of the virtual human according to the state machine after receiving the second voice, expression sequence and action sequence sent by the server, wherein the updated state information instructs the virtual human to switch to the speaking state.

[0176] In one possible implementation, the image sequence is also generated based on at least one body feature corresponding to the updated state information, the at least one body feature including blink frequency, head orientation, eye orientation, and part or all of the body orientation.

[0177] In one possible implementation, the first unit 801 is specifically used to: in response to detecting that the first speech does not have a preset feature, perform speech recognition processing on the first speech to obtain the first text, wherein the preset feature indicates the features of the speech responded by the virtual human through a broadcasting method;

[0178] The first unit 801 is further configured to, in response to detecting that the first voice has the preset feature, send the text corresponding to the preset feature to the server before receiving the second voice, facial expression sequence and action sequence sent by the server.

[0179] In one possible implementation, the client includes a first database;

[0180] The first unit 801 is specifically used to: in response to the first text not meeting the search conditions of the first database, send the first text to the server;

[0181] The first unit 801 is further configured to: before receiving the second voice, facial expression sequence and action sequence sent by the server, in response to the first text satisfying the retrieval conditions of the first database, send the retrieval results of the first text in the first database to the server.

[0182] In one possible implementation, the receiving unit 802 is further configured to: receive second text sent by the server; the action sequence includes an action prediction result based on the second speech prediction and an action matched from the action library based on the second text, the second text including the insertion time of the matched action, the insertion time indicating the time when the virtual human presents the matched action; the image sequence includes images corresponding to each frame of the action prediction result and images corresponding to the matched action;

[0183] The playback unit 804 is further configured to: during the display of the image sequence, in response to the playback of the image corresponding to the matching action, determine whether the insertion time of the matching action in the second text is at the end of a sentence; if so, skip the image corresponding to the matching action; if not, smooth the image corresponding to the matching action and its adjacent images in the image sequence to obtain a smoothed result of the image corresponding to the matching action and a smoothed result of the adjacent images, wherein the adjacent images include a previous frame image and / or a next frame image, and sequentially play the smoothed result of the image corresponding to the matching action and the smoothed result of the next frame image, wherein the next frame image is positioned later in the image sequence than the image corresponding to the matching action, and the previous frame image is positioned earlier in the image sequence than the image corresponding to the matching action.

[0184] Based on the aforementioned data processing device 800, its working principle includes: after receiving a first voice input by the user (such as a voice describing the user's question), the device 800 first performs speech recognition processing on the first voice to obtain a first text (such as text describing the user's question), and sends the first text to the server so that the server can process and provide feedback on the first text. After receiving a second voice, facial expression sequence, and action sequence sent by the server, the device 800 generates an image sequence based on the facial expression sequence and the action sequence, and plays the second voice and displays the image sequence. This enables the device 800 to automatically answer user questions, thereby reducing response costs. Specifically, because the facial expression sequence can indicate the changes in the virtual person's facial expression when responding to the first voice, and the action sequence can indicate the changes in the virtual person's actions when responding to the first voice, the image sequence generated based on the facial expression sequence and the action sequence can describe the changes in the virtual person in at least one dimension (such as facial expression and / or action), thereby enabling the image sequence to represent the changes (such as changes in facial expression, changes in action, etc.) exhibited by the virtual person when responding to the first voice; furthermore, because the second voice can indicate the content of the virtual person's speech when responding to the first voice, this application presents a vivid and expressive response from the virtual person to the user by playing the second voice and displaying the image sequence, thereby improving the user experience.

[0185] Based on the data processing method provided in the embodiments of this application, the embodiments of this application also provide a data processing apparatus, which will be described below in conjunction with... Figure 9 Explanation and clarification will be provided. Among them, Figure 9 This is a schematic diagram of another data processing apparatus provided in an embodiment of this application. It should be noted that for technical details of the data processing apparatus provided in this application embodiment, please refer to the relevant content of the data processing method above.

[0186] like Figure 9 As shown, the data processing apparatus 900 provided in this application embodiment includes:

[0187] The second unit 901 is used to receive a first text sent by the client, determine a second text based on the first text, and generate a second voice based on the second text;

[0188] Processing unit 902 is configured to determine an action sequence based on the second speech and the second text, and to predict an expression sequence based on the second speech;

[0189] The sending unit 903 is used to send the second voice, the expression sequence, and the action sequence to the client.

[0190] In one possible implementation, the processing unit 902 is specifically configured to: for any word in the second text, search for an action matching the word from a pre-built action library to obtain a matching result for the word; and in response to the matching result indicating that there is an action matching the word in the action library, determine the start and end times of the audio segment corresponding to the word from the second speech; and according to the start and end times, fuse the matching action and the action prediction result generated based on the second speech to obtain the action sequence.

[0191] In one possible implementation, the processing unit 902 is specifically configured to: determine the insertion time of the matched action based on the start and end times; in response to the fact that the position indicated by the insertion time in the second text is not the end of a sentence, concatenate the matched action with the action prediction result according to the insertion time to obtain a concatenation result, and smooth the concatenation result to obtain the action sequence.

[0192] In one possible implementation, the processing unit 902 is further configured to: predict emotion information based on the second speech and the second text;

[0193] The processing unit 902 is specifically used to: predict an expression sequence based on the emotion information and the second voice;

[0194] And / or,

[0195] The sending unit 903 is specifically used to send the emotion information, the second voice, the expression sequence, and the action sequence to the client.

[0196] In one possible implementation, the server includes a second database;

[0197] The second unit 901 is specifically used for: in response to the first text satisfying the retrieval conditions of the second database, retrieving the first document in the second database to obtain the second text; in response to the first text not satisfying the retrieval conditions of the second database, inputting the first text into a pre-built response model to obtain the second text output by the response model.

[0198] In one possible implementation, the first text indicates the semantic content carried by the first speech;

[0199] The second unit 901 is further configured to: receive the emotion recognition result sent by the client, wherein the emotion recognition result indicates the emotion information carried by the first voice;

[0200] The second unit 901 is specifically used to: input the first text and the emotion recognition result into a pre-built response model to obtain the second text output by the response model.

[0201] Based on the aforementioned data processing device 900, its working principle includes: after receiving a first text (as shown in Figure 2) sent by a client, the device 900 first determines a second text based on the first text, so that the second text can represent the response to the question indicated by the first text; then, it generates a second speech based on the second text, so that the second speech can represent the speech content used in responding to the first text; next, the device 900 determines an action sequence based on the second speech and the second text, so that the action sequence can express the changes in actions presented in responding to the first text, and the device 900 predicts an expression sequence based on the second speech, so that the expression sequence can express the changes in expressions presented in responding to the first text; finally, the device 900 sends the second speech, the expression sequence, and the action sequence to the client for use, so that the client can respond to the first text with expression and emotion based on this information, thereby improving the user experience. Since both the action sequence and the facial expression sequence are determined based on the second voice, the changes in actions indicated by the action sequence and the changes in actions indicated by the facial expression sequence are highly correlated with the content of the speech indicated by the second voice. This allows the virtual human presented when the client responds based on this information to exhibit better anthropomorphic expression, thereby improving the user experience.

[0202] In addition, this application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory so that the electronic device performs any implementation of the data processing method provided in this application.

[0203] See Figure 10 The diagram illustrates a structural schematic of an electronic device 1000 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0204] like Figure 10 As shown, the electronic device 1000 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the electronic device 1000. The processing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0205] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 An electronic device 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0206] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from storage device 1008, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0207] The electronic device provided in this embodiment belongs to the same inventive concept as the method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0208] This application also provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform any implementation of the data processing method provided in this application.

[0209] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0210] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0211] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0212] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to perform the aforementioned methods.

[0213] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0214] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0215] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0216] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0217] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0218] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0219] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0220] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0221] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0222] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized in that, The method is applied to a client, and the method includes: Receive a first voice message, perform speech recognition processing on the first voice message to obtain a first text message, and send the first text message to the server. The system receives a second voice, an expression sequence, and an action sequence sent by the server. The second voice indicates the content of the virtual person's speech when responding to the first voice, the expression sequence indicates the changes in the virtual person's facial expressions when responding to the first voice, and the action sequence indicates the changes in the virtual person's actions when responding to the first voice. An image sequence is generated based on the expression sequence and the action sequence, the image sequence describing the changes of the virtual human in at least one dimension, the at least one dimension including expression and / or action; Play the second audio and display the image sequence.

2. The method according to claim 1, characterized in that, The client is a holographic capsule.

3. The method according to claim 1, characterized in that, The method further includes: Receive emotional information sent by the server, the emotional information indicating the emotional state of the virtual person when responding to the first voice; The step of generating an image sequence based on the expression sequence and the action sequence includes: An image sequence is generated based on the emotional information, the facial expression sequence, and the action sequence.

4. The method according to claim 3, characterized in that, The facial expression sequence indicates changes in lip shape; The step of generating an image sequence based on the emotion information, the expression sequence, and the action sequence includes: Facial expressions are generated based on the emotional information, and the facial expressions indicate the facial states of the virtual person other than mouth movements when responding to the first voice. The image sequence is generated based on the facial expression, the expression sequence, and the action sequence.

5. The method according to claim 4, characterized in that, The emotion information includes the predicted probability of at least one candidate emotion; The step of generating facial expressions based on the emotional information includes: Based on the predicted probability of the at least one candidate emotion, a target emotion is determined from the at least one candidate emotion, wherein the predicted probability of the target emotion is not less than the predicted probability of other emotions besides the target emotion among the at least one candidate emotions. Based on the predicted probability of the target emotion, the expression template corresponding to the target emotion and the expression template corresponding to the default emotion are weighted and fused to obtain the facial expression. The predicted probability of the target emotion indicates the weight of the expression template corresponding to the target emotion in the weighted fusion. The expression template corresponding to the target emotion indicates the facial state under the target emotion, and the expression template corresponding to the default emotion indicates the facial state under the default emotion.

6. The method according to claim 1, characterized in that, The client includes a state machine, which indicates the switching relationship between different states of the virtual human; After receiving the second voice, facial expression sequence, and action sequence sent by the server, the method further includes: The state information of the virtual human is updated according to the state machine, and the updated state information instructs the virtual human to switch to the speaking state.

7. The method according to claim 6, characterized in that, The image sequence is also generated based on at least one body feature corresponding to the updated state information, the at least one body feature including blink frequency, head orientation, eye orientation, and part or all of the body orientation.

8. The method according to claim 1, characterized in that, The step of performing speech recognition processing on the first speech to obtain the first text includes: In response to detecting that the first speech does not have a preset feature, the first speech is processed by speech recognition to obtain the first text, wherein the preset feature indicates the features of the speech responded by the virtual human by a broadcast method; Before receiving the second voice, facial expression, and action sequence sent by the server, the method further includes: In response to detecting that the first speech has the preset feature, the text corresponding to the preset feature is sent to the server.

9. The method according to any one of claims 1-8, characterized in that, The client includes a first database; Sending the first text to the server includes: In response to the first text not meeting the search criteria of the first database, the first text is sent to the server; Before receiving the second voice, facial expression, and action sequence sent by the server, the method further includes: In response to the first text satisfying the search criteria of the first database, the search results of the first text in the first database are sent to the server.

10. A data processing method, characterized in that, The method is applied to the server side, and the method includes: Receive a first text sent by the client, determine a second text based on the first text, and generate a second speech based on the second text; Determine an action sequence based on the second speech and the second text, and predict an expression sequence based on the second speech; The second voice, the facial expression sequence, and the action sequence are sent to the client.

11. The method according to claim 10, characterized in that, The step of determining the action sequence based on the second speech and the second text includes: For any word in the second text, search for an action that matches the word from a pre-built action library to obtain a matching result for the word, and in response to the matching result indicating that there is an action that matches the word in the action library, determine the start and end times of the audio segment corresponding to the word from the second speech. According to the start and end times, the matched action and the action prediction result generated based on the second speech are fused to obtain the action sequence.

12. The method according to claim 11, characterized in that, The step of fusing the matched action and the action prediction result generated based on the second speech according to the start and end times to obtain the action sequence includes: Based on the start and end times, determine the insertion time of the matching action; In response to the fact that the position indicated by the insertion time in the second text is not the end of the sentence, the matched action and the action prediction result are concatenated according to the insertion time to obtain a concatenated result, and the concatenated result is smoothed to obtain the action sequence.

13. The method according to claim 10, characterized in that, The method further includes: Predict emotional information based on the second speech and the second text; The step of predicting the facial expression sequence based on the second speech includes: Based on the emotional information and the second speech, predict the facial expression sequence; And / or, Sending the second voice, the facial expression sequence, and the action sequence to the client includes: The emotional information, the second voice, the facial expression sequence, and the action sequence are sent to the client.

14. The method according to claim 10, characterized in that, The server includes a second database; Determining the second text based on the first text includes: In response to the first text satisfying the search criteria of the second database, the first file is searched in the second database to obtain the second text; In response to the first text not meeting the search criteria of the second database, the first text is input into a pre-built response model to obtain the second text output by the response model.

15. The method according to claim 10, characterized in that, The first text indicates the semantic content carried by the first speech; The method further includes: Receive the emotion recognition result sent by the client, the emotion recognition result indicating the emotion information carried by the first voice; Determining the second text based on the first text includes: The first text and the emotion recognition result are input into a pre-built response model to obtain the second text output by the response model.

16. A data processing apparatus, characterized in that, include: The first unit is used to receive the first voice, perform voice recognition processing on the first voice to obtain the first text, and send the first text to the server. The receiving unit is configured to receive a second voice, an expression sequence, and an action sequence sent by the server. The second voice indicates the content of the virtual person's speech when responding to the first voice, the expression sequence indicates the changes in the virtual person's facial expressions when responding to the first voice, and the action sequence indicates the changes in the virtual person's actions when responding to the first voice. A generation unit is configured to generate an image sequence based on the expression sequence and the action sequence, the image sequence describing the changes of the virtual human in at least one dimension, the at least one dimension including expression and / or action; The playback unit is used to play the second voice and display the image sequence.

17. A data processing apparatus, characterized in that, include: The second unit is used to receive a first text sent by the client, determine a second text based on the first text, and generate a second voice based on the second text; The processing unit is configured to determine an action sequence based on the second speech and the second text, and to predict an expression sequence based on the second speech; A sending unit is used to send the second voice, the facial expression sequence, and the action sequence to the client.

18. An electronic device, characterized in that, The device includes: a processor and a memory; The memory is used to store instructions or computer programs; The processor is configured to execute the instructions or computer program in the memory to cause the electronic device to perform the method according to any one of claims 1-15.

19. A computer-readable medium, characterized in that, The computer-readable medium stores instructions or computer programs that, when executed on the device, cause the device to perform the method according to any one of claims 1-15.

20. A computer program product, characterized in that, It includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method according to any one of claims 1-15.

Images