Role interaction method, device, electronic device and storage medium

Abstract

The present application discloses a method, device, electronic device and storage medium for role interaction; the present application can determine the target role from the roles corresponding to the target video; obtain the user's interactive text for the target role; generate the target role's reply text for the interactive text, and the reply text conforms to the dialogue logic when the target role talks with other roles; generate a reply voice corresponding to the reply text, and the reply voice conforms to the voice characteristics of the target role; play the reply voice so that the user can interact with the target role. In the present application, the target role can be selected from the roles corresponding to the target video for interaction, and the target role's reply conforms to the dialogue logic and voice characteristics when the target role talks with other roles, which can make the user have an immersive experience when watching the video, and increase the interactive mode when watching the video. Therefore, this solution can improve user stickiness.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a role interaction method, device, electronic device and storage medium. Background Art

[0002] Video player refers to software that can play videos stored in the form of digital signals, and also refers to electronic devices that have the function of playing videos. Video players carry decoders to restore compressed media files, and video players can also have a set of built-in conversion frequency and buffering algorithms. And video players can also support the playback of audio files.

[0003] However, users' interaction methods when watching videos played by video players are relatively simple, resulting in low user stickiness. Summary of the invention

[0004] The present application provides a role interaction method, device, electronic device and storage medium, which can make interactive responses more accurate.

[0005] The present application provides a role interaction method, comprising:

[0006] Determine a target role from roles corresponding to the target video;

[0007] Get the user's interactive text for the target role;

[0008] Generate a reply text of the target character to the interactive text, the reply text conforms to the dialogue logic when the target character talks with other characters, and the other characters are the characters corresponding to the target video except the target character;

[0009] Generate a reply voice corresponding to the reply text, and the reply voice conforms to the voice characteristics of the target role;

[0010] Play the reply voice so that the user can interact with the target character.

[0011] The present application also provides a character interaction device, comprising:

[0012] A determination unit, used to determine a target role from roles corresponding to a target video;

[0013] An acquisition unit, used for acquiring the user's interactive text for the target role;

[0014] A first generating unit is used to generate a reply text of the target character to the interactive text, where the reply text conforms to the dialogue logic when the target character talks with other characters, where the other characters are characters corresponding to the target video except the target character;

[0015] The second generating unit is used to generate a reply voice corresponding to the reply text, and the reply voice conforms to the voice characteristics of the target role;

[0016] The playback unit is used to play the reply voice so that the user can interact with the target character.

[0017] In some embodiments, the first generating unit is specifically configured to:

[0018] Obtain a dialogue logic model. The dialogue logic model is trained using dialogue data between the target role and other roles.

[0019] Generate the target character's response text to the interactive text, including:

[0020] The dialogue logic model is used to generate the target role's response text to the interactive text.

[0021] In some embodiments, the dialogue logic model includes a text semantic analysis network and a generation network. The dialogue logic model is used, and the first generation unit is specifically used to:

[0022] Use text semantic analysis network to perform semantic analysis on interactive text and obtain interactive semantic features;

[0023] Based on the interactive semantic features, a generative network is used to generate reply text.

[0024] In some embodiments, the text semantic analysis network includes an analysis self-attention layer and a feedforward layer, and the first generation unit is specifically used to:

[0025] Perform word embedding processing on the interactive text to obtain multiple word vectors;

[0026] Use the analytical self-attention layer to adjust the word vector to obtain the adjusted word vector;

[0027] The feed-forward layer is used to extract semantic features from the adjusted word vectors to obtain interactive semantic features.

[0028] In some embodiments, the generating network includes generating a self-attention layer, a multi-attention layer, and an output layer. Based on the interactive semantic features, the first generating unit is specifically used to:

[0029] Determine the position vector corresponding to this iteration;

[0030] The generated self-attention layer is used to adjust the attention of the current input vector to obtain the current input features. The current input vector includes the position vector and the current input vector and generated word vector corresponding to the last iteration;

[0031] Use multiple attention layers to perform multi-attention adjustments on the interactive semantic features and the current input features to obtain generated features;

[0032] Use the output layer to perform convolution operation on the generated features, output the generated word vector corresponding to this iteration, return to the execution step to determine the position vector corresponding to this iteration, until the generated word vector corresponding to this iteration is the preset vector;

[0033] All generated word vectors output by the output layer are processed by text conversion to obtain the reply text.

[0034] In some embodiments, the second generating unit is specifically configured to:

[0035] Obtain the voice features of the target character;

[0036] Generate the reply voice corresponding to the reply text, including:

[0037] Based on the speech characteristics of the target character, the reply text is speech synthesized to obtain the reply speech.

[0038] In some embodiments, the speech feature of the target character includes an energy feature, and the second generating unit is specifically configured to:

[0039] Obtain the dialogue voice when the target character talks with other characters in the target video;

[0040] Perform discrete Fourier transform on the conversation speech to obtain amplitudes corresponding to multiple frequency bands;

[0041] Filter the amplitude corresponding to the frequency band to obtain the logarithmic energy;

[0042] The logarithmic energy is processed to obtain the energy characteristics.

[0043] In some embodiments, the speech feature of the target character includes a Mel-frequency cepstral coefficient, and the second generating unit is specifically used for:

[0044] Perform inverse discrete Fourier transform on the energy features to obtain Mel-frequency cepstrum coefficients.

[0045] In some embodiments, the second generating unit is specifically configured to:

[0046] Pre-emphasize the dialogue voice to obtain enhanced dialogue voice;

[0047] Perform frame division and windowing processing on the enhanced conversational speech to obtain speech signals corresponding to multiple windows;

[0048] Perform discrete Fourier transform on the conversational speech to obtain the amplitudes corresponding to multiple frequency bands, including:

[0049] The speech signal corresponding to each window is subjected to discrete Fourier transform to obtain the amplitudes of multiple frequency bands corresponding to each window.

[0050] In some embodiments, the acquisition unit is specifically used to:

[0051] Obtain the user's interactive voice for the target role;

[0052] Perform speech recognition on the interactive speech to obtain the interactive text.

[0053] In some embodiments, the determining unit is specifically configured to:

[0054] Displaying a playback interface of the target video, the playback interface including character interaction controls;

[0055] In response to a triggering operation on the character interaction control, an interaction prompt control is displayed on the playback interface, and the interaction prompt control corresponds to the character in the target video one by one;

[0056] In response to a triggering operation on the interactive prompt control, the character corresponding to the interactive prompt control is determined as the target character.

[0057] In some embodiments, the determining unit is specifically configured to:

[0058] Determine the video type of the target video;

[0059] When the video type is a plot video, the target role is determined from the roles corresponding to the target video.

[0060] The present application also provides an electronic device, including a memory and a processor, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps in any one of the role interaction methods provided in the present application.

[0061] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for a processor to load to execute the steps in any one of the role interaction methods provided in the present application.

[0062] The present application can determine the target role from the roles corresponding to the target video; obtain the user's interactive text for the target role; generate the target role's reply text for the interactive text, the reply text conforming to the dialogue logic when the target role talks with other roles, and the other roles are the roles corresponding to the target video except the target role; generate a reply voice corresponding to the reply text, the reply voice conforming to the voice characteristics of the target role; play the reply voice so that the user can interact with the target role.

[0063] In this application, when watching a video, the user can select a target character from the characters corresponding to the target video to have a conversation; the electronic device can generate a reply voice of the target character to the user, and the reply voice of the target character conforms to the dialogue logic and voice characteristics of the target character when talking to other characters, which can make the interactive reply more accurate and more in line with the characteristics of the target character, so that the user can have an immersive experience when watching the video, and increase the interactive mode of the user when watching the video. Therefore, this solution can improve user stickiness. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0065] Figure 1a It is a scene diagram of the role interaction method provided by this application;

[0066] Figure 1b It is a flowchart of the role interaction method provided by this application;

[0067] Figure 1c is a schematic diagram of the interactive prompt control provided by this application;

[0068] Figure 2a It is a flowchart of applying the role interaction method provided by this application to a plot video;

[0069] Figure 2b is a schematic diagram of the dialogue logic model provided by this application;

[0070] Figure 2c This is a flowchart of obtaining speech features provided by this application;

[0071] Figure 3 It is a structural schematic diagram of a role interaction device provided by the present application;

[0072] Figure 4 It is a structural schematic diagram of the electronic device provided by this application. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0074] The present application provides a character interaction method, device, electronic device and storage medium.

[0075] The role interaction device can be integrated into an electronic device, which can be a terminal, a server, or other device. The terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop computer, a desktop computer, a smart TV, a smart car terminal, or other devices; the server can be a single server, or a server cluster composed of multiple servers, or a cloud server, etc. In some embodiments, the server can also be implemented in the form of a terminal.

[0076] In some embodiments, the character interaction device may also be integrated into a plurality of electronic devices. For example, the character interaction device may be integrated into a terminal and a server, and the terminal and the server jointly implement the character interaction method of the present application.

[0077] For example, Figure 1a As shown, Figure 1a : is a scene diagram of the role interaction method provided by an embodiment of the present invention. The scene of the role interaction method may include a terminal and a server; wherein the number of servers and terminals is not limited, and only one example is given here. The server may be a server (server) corresponding to a video player. The terminal is equipped with a video player (client), and the playback interface of the target video played by the video player can be displayed on the display screen of the terminal. Among them, the encapsulation format of the video that the video player can support for playback is not limited, and may include but is not limited to the formats such as mp4, mkv, rmvb, flv, avi, etc.

[0078] In some embodiments, the terminal can determine the target role from the roles corresponding to the target video; obtain the user's interactive text for the target role; generate a reply text for the target role to the interactive text, the reply text conforming to the dialogue logic when the target role talks with other roles, and the other roles are the roles corresponding to the target video except the target role; generate a reply voice corresponding to the reply text, the reply voice conforming to the voice characteristics of the target role; play the reply voice so that the user can interact with the target role.

[0079] In this embodiment, when watching a video through a video player, the user can select a target character from the characters corresponding to the target video to have a conversation; the terminal can generate a reply voice of the target character to the user in response to the user's interactive text, and the reply voice of the target character conforms to the dialogue logic and voice characteristics of the target character when talking to other characters, which can make the user have an immersive experience when watching the video and increase the interactive mode of the user when watching the video. Therefore, this solution can improve user stickiness.

[0080] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0081] Artificial Intelligence (AI) is a technology that uses digital computers to simulate human perception of the environment, acquire knowledge, and use knowledge. This technology can enable machines to have functions similar to human perception, reasoning, and decision-making. Artificial Intelligence technology mainly includes computer vision technology, speech processing technology, natural language processing technology (NLP), machine learning, deep learning, and other major directions.

[0082] Among them, natural language processing includes two core tasks, namely natural language understanding (NLU) and natural language generation (NLG). The main focus of natural language understanding is to use human language (natural language) as input and output machine-readable semantic representation after processing; while natural language generation is to express semantic information in a human-readable natural language form, select and execute certain grammatical and semantic rules to generate natural language text.

[0083] In this embodiment, a role interaction method is provided, which can be executed by an electronic device. Figure 1b As shown, the specific process of the role interaction method can be as follows:

[0084] 110. Determine a target role from roles corresponding to the target video.

[0085] The target video may be a video file in any format and may be played online or locally through a video player; the content of the target video may be formed by multiple characters communicating and interacting with each other in different scenarios, and the characters may be human characters, animal characters, or virtual characters, etc.

[0086] The target character can be any character in the target video; this embodiment is applicable to all characters in the target video, and only the target character is used as an example. In some embodiments, the target video can be played in a video player in response to a user triggering operation of a play control for the target video; and then the video type of the target video can be determined; when the video type is a plot video, the target character is determined from the characters corresponding to the target video; for example, plot videos can include movies, TV series, etc.

[0087] In some embodiments, when playing the target video, determining the target role from the role corresponding to the target video may include but is not limited to the following steps:

[0088] i. Displaying a playback interface of the target video, the playback interface including a character interaction control, wherein the character interaction control may be a control of any shape; for example, the character interaction control may be a button.

[0089] ii. In response to the triggering operation of the character interaction control, the interactive prompt control is displayed on the playback interface, and the interactive prompt control corresponds to the character in the target video one by one. Among them, the triggering operation in this embodiment is not limited, and can be a long press, short press, click, drag, etc.; the interactive prompt control can be a control of any shape, for example, the prompt control can display the name, avatar, character model, photo, etc. of the character; in some embodiments, the interactive prompt control can be displayed within a certain range near the character; in some embodiments, the interactive prompt controls can be arranged and displayed on the playback interface.

[0090] iii. In response to the triggering operation of the interactive prompt control, the character corresponding to the interactive prompt control is determined as the target character. Figure 1c , which is a schematic diagram of an interactive prompt control provided by the present application. The interactive prompt control is displayed near the character, and when the user clicks the "character 1" control, the corresponding character is determined to be the target character.

[0091] In some embodiments, when the target video is determined to be a plot video, the user's voice command can be monitored, and when the user's voice command is received, the target role is determined from the roles corresponding to the target video. For example, the voice command can be a role name, and when the user's voice "role 1" is monitored, role 1 is determined to be the target role.

[0092] In some embodiments, the target video in this article can also be replaced by audio, for example, it can be a commentary or reading audio of a novel, a classic, a textbook, a TV series, a movie, etc.

[0093] 120. Obtain the user's interactive text for the target role.

[0094] The interactive text may be a text for the user to interact with the target character; the content of the interactive text is not limited, for example, it may be a plot inquiry, character analysis, scene analysis, etc., such as "What is your relationship with character 2?" After determining the target character, the user can have a dialogue with the target character.

[0095] In some embodiments, the user's interactive speech for the target character can be obtained; the interactive speech is recognized to obtain interactive text. The speech recognition method is not limited, and can be a combination of one or more of the model matching method, the probability statistics method, and the discriminator classification method; the model matching method can include vector quantization, dynamic time warping, etc.; the probability statistics method can include Gaussian mixture model, hidden Markov model, etc.; the discriminator classification method can include support vector machine, artificial neural network, and deep neural network, etc.

[0096] In some embodiments, an input control may be provided through which a user may enter interactive text.

[0097] 130. Generate a reply text of the target character to the interactive text. The reply text conforms to the dialogue logic when the target character talks with other characters. The other characters are the characters corresponding to the target video except the target character.

[0098] Among them, the reply text is generated from the perspective of the target character to reply to the user's interactive text. Therefore, the reply text conforms to the dialogue logic when the target character talks with other characters, which will appear more real.

[0099] In some embodiments, a dialogue logic model can be obtained, and the dialogue logic model can be used to generate the target character's reply text for the interactive text. Among them, the dialogue logic model is trained using the dialogue data (training set) between the target character and other characters; the dialogue data can be lines, scripts, etc. Using the dialogue data of the target character and other characters for training, the dialogue logic model can learn the dialogue logic when the target character and other characters are talking. In some embodiments, the server can pre-train the dialogue logic model using the dialogue data, and provide an application programming interface (Application Programming Interface, API), the terminal can call the API, and the server executes the dialogue logic model to generate the target character's reply text for the interactive text. In some embodiments, the trained dialogue logic model can also be obtained locally at the terminal, and the terminal executes the dialogue logic model to generate the target character's reply text for the interactive text.

[0100] In some embodiments, the dialogue logic model may include but is not limited to a text semantic analysis network (encoder) and a generation network (decoder); the number of semantic analysis networks and generation networks is not limited, and the dialogue logic model may include multiple semantic analysis networks and generation networks. This embodiment will take a text semantic analysis network and a generation network as an example to illustrate the use of a dialogue logic model to generate a target role's reply text for an interactive text, which may include but is not limited to the following steps:

[0101] 1) Use the text semantic analysis network to perform semantic analysis on the interactive text and obtain interactive semantic features.

[0102] In some embodiments, the text semantic analysis network includes an analysis self-attention layer and a feed-forward layer.

[0103] i. The interactive text can be processed by word embedding to obtain multiple word vectors. The specific implementation of the word embedding process is not limited, and one or more combinations of methods such as One-hot encoding, information retrieval (IR) technology, distributed representation, matrix-based distribution representation, clustering-based distribution representation, neural network-based distribution representation (such as Word2vec) can be used.

[0104] It should be noted that when there are multiple text semantic analysis networks, word embedding processing generally acts on the underlying text semantic analysis network. And all text semantic analysis networks can receive a vector list. The input of the underlying text semantic analysis network is the word vector, but the input of other text semantic analysis networks is the output of the text semantic analysis network of the next layer. Among them, the vector list size is a hyperparameter that can be set according to the actual application situation. For example, it can be set according to the length of the longest sentence in the training set, such as 512 dimensions.

[0105] ii. Use the analytical self-attention layer to adjust the word vector to obtain the adjusted word vector. The analytical self-attention layer enables information exchange between word vectors, that is, each word in the interactive text pays attention to other words in the interactive text, and clarifies which words among other words should be paid attention to.

[0106] iii. Using a feedforward layer to extract semantic features from the adjusted word vector to obtain interactive semantic features. The feedforward layer may be a neural network, for example, a fully connected network. In some embodiments, each word at each position in the interactive text corresponds to a feedforward layer.

[0107] 2) Based on the interactive semantic features, a generative network is used to generate reply text.

[0108] In some embodiments, the generating network includes generating a self-attention layer, a multi-attention layer, and an output layer. In some embodiments, the generating network may also include a feed-forward layer.

[0109] i. Determine the position vector corresponding to this iteration; the position of a single word in the entire text can be clarified so that when the attention layer performs self-attention adjustment, it can not only know which word to focus on, but also know the relative positions between words.

[0110] ii. Use the generated self-attention layer to adjust the attention of the current input vector to obtain the current input features. The current input vector includes the position vector and the current input vector and generated word vector corresponding to the last iteration. The current input vector at the first iteration is the vector corresponding to the start symbol.

[0111] iii. Use a multi-attention layer to perform multi-attention adjustment on the interactive semantic features and the current input features to obtain the generated features; in some embodiments, the interactive semantic features are used as the key and value of the generated network, and the current input features are used as the query, and then Self-Attention is performed. In some embodiments, a feedforward layer can also be used to extract features from the generated features.

[0112] iv. Use the output layer to perform convolution operation on the generated features, output the generated word vector corresponding to this iteration, return to the execution step to determine the position vector corresponding to this iteration, until the generated word vector corresponding to this iteration is a preset vector. The preset vector is the vector corresponding to the terminator. The output layer may include but is not limited to a linear transformation sublayer and a Softmax sublayer. The linear transformation sublayer may be a fully connected neural network that can project the generated features into a vector of logits.

[0113] v. Perform text conversion on all generated word vectors output by the output layer to obtain the reply text.

[0114] In some embodiments, the lines of dialogue between the target character and other characters can be obtained, and a knowledge base can be extracted for the target character based on the dialogue lines. After obtaining the interactive text, a rule engine, pattern matching, or a classifier trained by machine learning can be used to select an optimal reply text from the knowledge base.

[0115] 140. Generate a reply voice corresponding to the reply text, and the reply voice conforms to the voice characteristics of the target role.

[0116] Among them, the reply voice can be generated according to the reply text.

[0117] In some embodiments, the speech features of the target character are obtained; based on the speech features of the target character, speech synthesis is performed on the reply text to obtain a reply speech. The specific implementation of speech synthesis is not limited, and a combination of one or more methods in a traditional speech synthesis system or speech synthesis based on deep learning can be used. By allowing different characters to communicate with users using their own speech features and speaking styles, the user experience can be improved.

[0118] In some embodiments, the speech feature of the target character includes an energy feature. Specific implementations of obtaining the speech feature of the target character may include but are not limited to:

[0119] i. Obtain the dialogue audio when the target character talks with other characters in the target video.

[0120] ii. Perform discrete Fourier transform on the conversation speech to obtain the amplitudes corresponding to multiple frequency bands;

[0121] In some embodiments, before performing discrete Fourier transform on the conversation speech, the following steps may be performed:

[0122] Pre-emphasize the conversational speech to obtain enhanced conversational speech; for example, the energy of the easily attenuated high-frequency signal is increased through a high-pass filter. The loss of high-frequency signals will cause the resonance peaks of phonemes to be unclear, and enhancing their energy is convenient for subsequent signal processing.

[0123] The enhanced conversational speech is processed by framing and windowing to obtain speech signals corresponding to multiple windows. In some embodiments, the framing process can be to segment the conversational speech according to a preset time interval, each segment is called a frame. In order to maintain continuity, the two frames before and after the framing are often partially overlapped to achieve a smooth transition; the preset time interval can be customized according to the actual application situation, such as 25ms. Since the conversational speech has non-steady-state and time-varying characteristics, but in a short time, it can be considered to be steady and time-invariant. By framing the conversational speech and segmenting it, the impact of non-steady-state time-varying speech in a long time range can be reduced. Windowing is to process the conversational signal into a periodic signal, so as to reduce the leakage of the speech signal during subsequent processing. For example, a Hamming window can be used to make the signal approximately zero at the window boundary, so that the signal approaches a periodic signal. The Hamming window function is as follows:

[0124]

[0125] Where L is the Hamming window length.

[0126] The speech signal corresponding to each window is subjected to discrete Fourier transform to obtain the amplitudes of multiple frequency bands corresponding to each window, and then the amplitudes of multiple frequency bands corresponding to each window are subjected to the following filtering processing.

[0127] iii. Filter the amplitude corresponding to the frequency band to obtain logarithmic energy. In some embodiments, a Mel filter group can be used to filter the amplitude corresponding to the frequency band to obtain logarithmic energy; for example, the starting, middle and cutoff frequencies of each triangular filter in the Mel domain are converted to the linear frequency domain, and the amplitude corresponding to the frequency band is filtered to obtain logarithmic energy. Humans have different perception abilities for speech of different frequencies: for frequencies below 1kHz, it is linearly related to the frequency, and for frequencies above 1kHz, it is logarithmically related to the frequency. The higher the frequency, the worse the perception ability. The Mel scale is a nonlinear scale unit defined based on frequency, which represents the human ear's sense of pitch changes. In the Mel frequency domain, human perception is linear. If the Mel frequencies of two segments of speech differ by twice, then the perception of humans is also twice as different.

[0128] iv. Perform logarithmic processing on the logarithmic energy to obtain energy features, where the energy features may be Fbank (Filter Bank) features.

[0129] In some embodiments, the speech features of the target character include Mel-frequency cepstral coefficients. After the logarithmic energy is logarithmically processed to obtain the energy features, the energy features may be inversely discrete Fourier transformed to obtain the Mel-frequency cepstral coefficients.

[0130] In some embodiments, the server may pre-acquire the target character's conversation voice, and perform pre-emphasis, frame windowing, filtering, etc. on the conversation voice to obtain voice features. When generating a reply voice, the terminal acquires the target character's voice features from the server.

[0131] 150. Play the reply voice so that the user can interact with the target character.

[0132] In some embodiments, the reply text may also be displayed on the playback interface, while the reply voice is played through the video player.

[0133] In some embodiments, after playing the reply voice, the user's interactive voice or reply text for the target character's reply voice can also be obtained, and steps 120 to 150 can be re-executed.

[0134] In some embodiments, the target character's behavior when talking to other characters in the target video can also be obtained, and the deep learning model can be trained to obtain a character model, so that the character model learns the behavior habits of the target character when talking to other characters. When playing the reply voice, the virtual model corresponding to the target character can be obtained, and the virtual model can imitate the behavior of the target character and be displayed on the playback interface.

[0135] In this embodiment, when watching a target video, the user can select a target character from the characters corresponding to the target video to have a conversation; based on the user's interactive text for the target character, a dialogue logic model can be used to generate the target character's reply text for the user, so that the generated reply text conforms to the dialogue logic when the target character talks to other characters; then the reply text and the voice features of the target character are synthesized to obtain a reply voice, so that the reply voice conforms to the voice features of the target character; based on the multi-dimensional features of the characters in the video (dialogue logic and voice features, etc.) to interact with the user, the interactive reply can be more accurate and more in line with the characteristics of the target character, so that the user can have an immersive experience when watching the video, and increase the user's interactive mode when watching the video. Therefore, this solution can improve user stickiness.

[0136] The character interaction scheme provided by this application can be applied to various scenarios of interacting with characters in audio and video. For example, taking the target video as a plot video, the target character can be determined from the characters corresponding to the target video; the user's interactive text for the target character can be obtained; the target character's reply text for the interactive text is generated, and the reply text conforms to the dialogue logic when the target character talks with other characters, and the other characters are the characters corresponding to the target video except the target character; the reply voice corresponding to the reply text is generated, and the reply voice conforms to the voice characteristics of the target character; the reply voice is played so that the user can interact with the target character. The scheme provided by this application can increase the interactive methods when watching videos, thereby improving user stickiness.

[0137] According to the method described in the above embodiment, the following will be further described in detail. In this embodiment, a role interaction method is provided, which can be executed by an electronic device. Figure 2a As shown, the specific process of a role interaction method is as follows:

[0138] 210. In response to a user triggering an operation on a play control of a target video, the target video is played in a video player.

[0139] 220. Determine the video type of the target video; when the video type is a plot video, determine the target role from roles corresponding to the target video.

[0140] Drama videos can include TV series and movies.

[0141] 230. Obtain the user's interactive voice for the target role; perform voice recognition on the interactive voice to obtain interactive text.

[0142] 240. Obtain a dialogue logic model; and use the dialogue logic model to generate a response text of the target role to the interactive text.

[0143] The dialogue logic model may include but is not limited to a text semantic analysis network (encoder) and a generation network (decoder). The dialogue logic model is trained using dialogue data when the target character talks to other characters in the target video. Figure 2b The figure shows a schematic diagram of the dialogue logic model. When encoding each word in the interactive text, the encoder can pay attention to other words in the interactive text through the self-attention layer (generated self-attention layer). The output of the self-attention layer is passed to the feedforward layer. Each word at each position corresponds to a feedforward layer. The feedforward layer can be a neural network with the same structure. The decoder also has a self-attention layer (generated self-attention layer) and a feedforward layer. In addition, there can be an encoding-decoding attention layer (multi-attention layer) between the two layers, which can pay attention to the relevant semantic information of the interactive text during decoding.

[0144] The encoder encoding includes: converting each word of the interactive text into a word vector through word embedding processing. The word embedding processing only occurs in the bottom encoder. And all encoders can receive a vector list. The input of the encoder at the bottom layer is the word vector of the interactive text, but the input of other encoders is the output of the encoder at the next layer. Among them, the vector list size is a hyperparameter that can be set according to the actual application. For example, it can be set according to the length of the longest sentence in the training set, such as 512 dimensions. The word vector of each word in the interactive text will flow through the self-attention layer and feedforward layer in the encoder to obtain interactive semantic features.

[0145] Decoder decoding includes: inputting interactive semantic features into the encoder-decoder attention layer, and the decoder finally outputs a real vector. The decoder can also include a fully connected neural network and a Softmax layer. The linear transformation layer can turn floating point numbers into a word vector, and finally the Softmax layer will turn those scores into probabilities. Select the word with the highest probability and get the reply text.

[0146] 250. Acquire the speech features of the target character; perform speech synthesis on the reply text based on the speech features of the target character to obtain a reply speech.

[0147] like Figure 2c As shown, obtaining the speech features of the target character includes the following steps:

[0148] Get the dialogue voice when the target character talks with other characters in the target video.

[0149] Pre-emphasize the conversational speech to obtain enhanced conversational speech; for example, the energy of the easily attenuated high-frequency signal is increased through a high-pass filter. The loss of high-frequency signals will cause the resonance peaks of phonemes to be unclear, and enhancing their energy is convenient for subsequent signal processing.

[0150] The enhanced conversational speech is processed by frame division and windowing to obtain speech signals corresponding to multiple windows.

[0151] The speech signal corresponding to each window is subjected to discrete Fourier transform to obtain the amplitudes of multiple frequency bands corresponding to each window.

[0152] The amplitudes of multiple frequency bands corresponding to each window are filtered using a Mel filter bank to obtain logarithmic energy.

[0153] The logarithmic energy is processed to obtain the Fbank (Filter Bank) feature.

[0154] Perform inverse discrete Fourier transform on the energy features to obtain Mel frequency cepstral coefficients (MFCC)

[0155] 260. Play the reply voice so that the user can interact with the target character.

[0156] As can be seen from the above, in this embodiment, when watching a target video, the user can select a target character from the characters corresponding to the target video to have a conversation; the user's interactive voice is converted into interactive text, the encoder can perform natural voice understanding on the user's interactive text to obtain interactive semantic features, and the decoder can decode the interactive semantic features to obtain a reply text, so that the generated reply text conforms to the dialogue logic when the target character talks with other characters; then the reply text is voice synthesized using the voice features of the target character to obtain a reply voice, so that the reply voice conforms to the voice features of the target character; based on the multi-dimensional features of the characters in the video to interact with the user, the interactive reply can be made more accurate and more in line with the characteristics of the target character, so that the user can have an immersive experience when watching the video, and increase the user's interactive mode when watching the video. Therefore, this solution can improve user stickiness.

[0157] In order to better implement the above method, the present application also takes the role interaction device being specifically integrated into an electronic device as an example to explain the method of the present application in detail.

[0158] For example, Figure 3 As shown, the character interaction device may include a determination unit 301, an acquisition unit 302, a first generation unit 303, a second generation unit 304, and a playback unit 305, as follows:

[0159] (I) Determine unit 301

[0160] The determination unit 301 is used to determine a target role from roles corresponding to a target video.

[0161] (II) Acquisition Unit 302

[0162] The acquisition unit 302 is used to acquire the user's interactive text for the target character.

[0163] (III) First Generation Unit 303

[0164] The first generating unit 303 is used to generate a reply text of the target character to the interactive text, where the reply text complies with the dialogue logic when the target character talks with other characters, and the other characters are the characters corresponding to the target video except the target character.

[0165] (IV) Second Generation Unit 304

[0166] The second generating unit 304 is used to generate a reply voice corresponding to the reply text, and the reply voice conforms to the voice characteristics of the target role.

[0167] (V) Playback Unit 305

[0168] The playing unit 305 is used to play the reply voice so that the user can interact with the target character.

[0169] In some embodiments, the first generating unit 303 is specifically used to:

[0170] Obtain a dialogue logic model. The dialogue logic model is trained using dialogue data between the target role and other roles.

[0171] Generate the target character's response text to the interactive text, including:

[0172] The dialogue logic model is used to generate the target role's response text to the interactive text.

[0173] In some embodiments, the dialogue logic model includes a text semantic analysis network and a generation network. Using the dialogue logic model, the first generation unit 303 is specifically used to:

[0174] Use text semantic analysis network to perform semantic analysis on interactive text and obtain interactive semantic features;

[0175] Based on the interactive semantic features, a generative network is used to generate reply text.

[0176] In some embodiments, the text semantic analysis network includes an analysis self-attention layer and a feedforward layer, and the first generation unit 303 is specifically used to:

[0177] Perform word embedding processing on the interactive text to obtain multiple word vectors;

[0178] Use the analytical self-attention layer to adjust the word vector to obtain the adjusted word vector;

[0179] The feed-forward layer is used to extract semantic features from the adjusted word vectors to obtain interactive semantic features.

[0180] In some embodiments, the generating network includes generating a self-attention layer, a multi-attention layer, and an output layer. Based on the interactive semantic features, the first generating unit 303 is specifically used to:

[0181] Determine the position vector corresponding to this iteration;

[0182] The generated self-attention layer is used to adjust the attention of the current input vector to obtain the current input features. The current input vector includes the position vector and the current input vector and generated word vector corresponding to the last iteration;

[0183] Use multiple attention layers to perform multi-attention adjustments on the interactive semantic features and the current input features to obtain generated features;

[0184] Use the output layer to perform convolution operation on the generated features, output the generated word vector corresponding to this iteration, return to the execution step to determine the position vector corresponding to this iteration, until the generated word vector corresponding to this iteration is the preset vector;

[0185] All generated word vectors output by the output layer are processed by text conversion to obtain the reply text.

[0186] In some embodiments, the second generating unit 304 is specifically configured to:

[0187] Obtain the voice features of the target character;

[0188] Generate the reply voice corresponding to the reply text, including:

[0189] Based on the speech characteristics of the target character, the reply text is speech synthesized to obtain the reply speech.

[0190] In some embodiments, the speech feature of the target character includes an energy feature, and the second generating unit 304 is specifically used to:

[0191] Obtain the dialogue voice when the target character talks with other characters in the target video;

[0192] Perform discrete Fourier transform on the conversation speech to obtain amplitudes corresponding to multiple frequency bands;

[0193] Filter the amplitude corresponding to the frequency band to obtain the logarithmic energy;

[0194] The logarithmic energy is processed to obtain the energy characteristics.

[0195] In some embodiments, the speech feature of the target character includes Mel-frequency cepstral coefficients, and the second generating unit 304 is specifically used to:

[0196] Perform inverse discrete Fourier transform on the energy features to obtain Mel-frequency cepstrum coefficients.

[0197] In some embodiments, the second generating unit 304 is specifically configured to:

[0198] Pre-emphasize the dialogue voice to obtain enhanced dialogue voice;

[0199] Perform frame division and windowing processing on the enhanced conversational speech to obtain speech signals corresponding to multiple windows;

[0200] Perform discrete Fourier transform on the conversational speech to obtain the amplitudes corresponding to multiple frequency bands, including:

[0201] The speech signal corresponding to each window is subjected to discrete Fourier transform to obtain the amplitudes of multiple frequency bands corresponding to each window.

[0202] In some embodiments, the acquisition unit 302 is specifically used to:

[0203] Obtain the user's interactive voice for the target role;

[0204] Perform speech recognition on the interactive speech to obtain the interactive text.

[0205] In some embodiments, the determining unit 301 is specifically configured to:

[0206] Displaying a playback interface of the target video, the playback interface including character interaction controls;

[0207] In response to a triggering operation on the character interaction control, an interaction prompt control is displayed on the playback interface, and the interaction prompt control corresponds to the character in the target video one by one;

[0208] In response to a triggering operation on the interactive prompt control, the character corresponding to the interactive prompt control is determined as the target character.

[0209] In some embodiments, the determining unit 301 is specifically configured to:

[0210] Determine the video type of the target video;

[0211] When the video type is a plot video, the target role is determined from the roles corresponding to the target video.

[0212] In specific implementation, each of the above units can be implemented as an independent entity, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of each of the above units can be referred to the previous method embodiments, which will not be repeated here. It should be noted that (1) (2) (3) (4) (5) in this application, the first, second, third, etc. are not limitations on the execution order.

[0213] As can be seen from the above, the role interaction device of this embodiment can select a target role from the roles corresponding to the target video to have a conversation when the user is watching a video; the role interaction device can generate a reply voice of the target role to the user, and the reply voice of the target role conforms to the dialogue logic and voice characteristics of the target role when talking to other roles, which can make the interactive reply more accurate and more in line with the characteristics of the target role, so that the user can have an immersive experience when watching the video, and increase the interactive mode of the user when watching the video. Therefore, this solution can improve user stickiness.

[0214] In this embodiment, the electronic device will be described in detail, for example, Figure 4 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0215] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0216] The processor 401 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, it performs various functions of the electronic device and processes data, thereby performing overall detection of the electronic device. In some embodiments, the processor 401 may include one or more processing cores; in some embodiments, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0217] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0218] The electronic device also includes a power supply 403 for supplying power to various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0219] In some embodiments, the electronic device may further include an input module 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0220] The electronic device may further include a communication module 405. In some embodiments, the communication module 405 may include a wireless module. The electronic device may perform short-range wireless transmission through the wireless module of the communication module 405, thereby providing the user with wireless broadband Internet access. For example, the communication module 405 may be used to help the user send and receive emails, browse web pages, and access streaming media.

[0221] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:

[0222] Determine a target role from roles corresponding to the target video;

[0223] Obtaining the user's interactive text for the target role;

[0224] Generate a reply text of the target character to the interactive text, wherein the reply text conforms to the dialogue logic when the target character dialogues with other characters, wherein the other characters are characters corresponding to the target video except the target character;

[0225] Generate a reply voice corresponding to the reply text, wherein the reply voice conforms to the voice characteristics of the target character;

[0226] The reply voice is played so that the user can interact with the target character.

[0227] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0228] As can be seen from the above, when the user is watching a video, the electronic device can select the target character from the characters corresponding to the target video to have a conversation; the electronic device can generate the target character's reply voice for the user, and the target character's reply voice conforms to the dialogue logic and voice characteristics of the target character when talking to other characters, which can make the interactive reply more accurate and more in line with the characteristics of the target character, so that the user can have an immersive experience when watching the video, and increase the user's interactive methods when watching the video. Therefore, this solution can improve user stickiness.

[0229] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0230] To this end, the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any of the role interaction methods provided in the present application. For example, the instructions can execute the following steps:

[0231] Determine a target role from roles corresponding to the target video;

[0232] Obtaining the user's interactive text for the target role;

[0233] Generate a reply text of the target character to the interactive text, wherein the reply text conforms to the dialogue logic when the target character dialogues with other characters, wherein the other characters are characters corresponding to the target video except the target character;

[0234] Generate a reply voice corresponding to the reply text, wherein the reply voice conforms to the voice characteristics of the target character;

[0235] The reply voice is played so that the user can interact with the target character.

[0236] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0237] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the method provided in various optional implementations of the character interaction aspect provided in the above-mentioned embodiment or the character interaction aspect based on a plot-based video.

[0238] Since the instructions stored in the storage medium can execute the steps in any one of the role interaction methods provided in the present application, the beneficial effects that can be achieved by any one of the role interaction methods provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0239] The above is a detailed introduction to a role interaction method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A role interaction method, characterized in that: include: Determine a target role from roles corresponding to the target video; Obtaining the user's interactive text for the target role; Generate a reply text of the target character to the interactive text, wherein the reply text conforms to the dialogue logic when the target character dialogues with other characters, wherein the other characters are characters corresponding to the target video except the target character, including: Using a text semantic analysis network to perform semantic analysis on the interactive text to obtain interactive semantic features; Based on the interactive semantic features, a generative network is used to generate the reply text, wherein the generative network includes a self-attention layer, a multi-attention layer and an output layer; The step of generating the reply text based on the interactive semantic features by using a generative network includes: Determine the position vector corresponding to this iteration; Using the generated self-attention layer to adjust the attention of the current input vector to obtain the current input feature, the current input vector includes the position vector and the current input vector and the generated word vector corresponding to the last iteration; Using the multi-attention layer to perform multi-attention adjustment on the interactive semantic feature and the current input feature to obtain a generated feature; Using the output layer to perform a convolution operation on the generated features, outputting the generated word vector corresponding to the current iteration, returning to the execution step to determine the position vector corresponding to the current iteration, until the generated word vector corresponding to the current iteration is a preset vector; Perform text conversion processing on all generated word vectors output by the output layer to obtain the reply text; Generate a reply voice corresponding to the reply text, wherein the reply voice conforms to the voice characteristics of the target character; The reply voice is played so that the user can interact with the target character.

2. The role interaction method according to claim 1, characterized in that: Before generating the target character's reply text to the interactive text, the method further includes: Acquire a dialogue logic model, wherein the dialogue logic model is trained using dialogue data between the target role and the other roles, and the dialogue logic model includes the text semantic analysis network and the generation network; The step of generating a reply text of the target character to the interactive text includes: The dialogue logic model is used to generate a reply text of the target role to the interactive text.

3. The role interaction method according to claim 1, characterized in that: The text semantic analysis network includes an analysis self-attention layer and a feedforward layer. The text semantic analysis network is used to perform semantic analysis on the interactive text to obtain interactive semantic features, including: Performing word embedding processing on the interactive text to obtain multiple word vectors; Using the analysis self-attention layer to adjust the attention of the word vector to obtain an adjusted word vector; The feed-forward layer is used to extract semantic features from the adjusted word vector to obtain interactive semantic features.

4. The role interaction method according to claim 1, characterized in that: Before generating the reply voice corresponding to the reply text, the method further includes: Acquire the voice features of the target character; The generating a reply voice corresponding to the reply text includes: Based on the voice feature of the target character, the reply text is subjected to speech synthesis to obtain the reply voice.

5. The role interaction method according to claim 4, characterized in that: The speech characteristics of the target character include energy characteristics, The acquiring of the voice feature of the target character includes: Acquire the dialogue voice when the target character talks with other characters in the target video; Performing discrete Fourier transform on the conversation speech to obtain amplitudes corresponding to multiple frequency bands; filtering the amplitude corresponding to the frequency band to obtain logarithmic energy; The logarithmic energy is logarithmically processed to obtain the energy feature.

6. The role interaction method according to claim 5, characterized in that: The speech feature of the target character includes a Mel frequency cepstral coefficient, and after the logarithmic energy is logarithmically processed to obtain the energy feature, the following further includes: Perform an inverse discrete Fourier transform on the energy feature to obtain the Mel-frequency cepstrum coefficient.

7. The role interaction method according to claim 5, characterized in that: Before performing discrete Fourier transform on the conversation speech, the method further includes: Pre-emphasize the dialogue speech to obtain enhanced dialogue speech; Performing frame division and windowing processing on the enhanced conversational speech to obtain speech signals corresponding to multiple windows; The performing discrete Fourier transform on the conversation speech to obtain amplitudes corresponding to multiple frequency bands includes: The speech signal corresponding to each window is subjected to discrete Fourier transform to obtain the amplitudes of multiple frequency bands corresponding to each window.

8. The role interaction method according to claim 1, characterized in that: The obtaining of the user's interactive text for the target role includes: Acquire the user's interactive voice for the target character; Performing speech recognition on the interactive speech to obtain the interactive text.

9. The role interaction method according to claim 1, characterized in that: Determining the target role from the roles corresponding to the target video includes: Displaying a playback interface of the target video, wherein the playback interface includes a character interaction control; In response to a triggering operation on the character interaction control, an interaction prompt control is displayed on the playback interface, wherein the interaction prompt control corresponds one-to-one to the character in the target video; In response to a triggering operation on the interactive prompt control, the character corresponding to the interactive prompt control is determined as the target character.

10. The role interaction method according to any one of claims 1 to 8, characterized in that: Determining the target role from the roles corresponding to the target video includes: Determine the video type of the target video; When the video type is a plot video, the target role is determined from roles corresponding to the target video.

11. A character interaction device, characterized in that: include: A determination unit, used to determine a target role from roles corresponding to a target video; An acquisition unit, used for acquiring the user's interactive text for the target role; The first generating unit is used to generate a reply text of the target character to the interactive text, wherein the reply text conforms to the dialogue logic when the target character dialogues with other characters, and the other characters are characters corresponding to the target video except the target character, including: Using a text semantic analysis network to perform semantic analysis on the interactive text to obtain interactive semantic features; Based on the interactive semantic features, a generative network is used to generate the reply text, wherein the generative network includes a self-attention layer, a multi-attention layer and an output layer; The step of generating the reply text based on the interactive semantic features by using a generative network includes: Determine the position vector corresponding to this iteration; Using the generated self-attention layer to adjust the attention of the current input vector to obtain the current input feature, the current input vector includes the position vector and the current input vector and the generated word vector corresponding to the last iteration; Using the multi-attention layer to perform multi-attention adjustment on the interactive semantic feature and the current input feature to obtain a generated feature; Using the output layer to perform a convolution operation on the generated features, outputting the generated word vector corresponding to the current iteration, returning to the execution step to determine the position vector corresponding to the current iteration, until the generated word vector corresponding to the current iteration is a preset vector; Perform text conversion processing on all generated word vectors output by the output layer to obtain the reply text; A second generating unit is used to generate a reply voice corresponding to the reply text, wherein the reply voice conforms to the voice characteristics of the target role; The playing unit is used to play the reply voice so that the user can interact with the target character.

12. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps in the role interaction method as claimed in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the character interaction method according to any one of claims 1 to 10.

Images