Virtual human interaction method and device, related equipment and computer program product

Abstract

The invention discloses a virtual human interaction method and device, related equipment and a computer program product, and relates to the technical field of artificial intelligence, and the method supports the creation of more than two virtual human roles, can respond to a virtual human role creation request, can create more than two virtual human roles according to role identifiers, and improves the user experience. And respectively generating video streams of the virtual human roles through different virtual human engines. More than two virtual human roles can be controlled at the same time, the corresponding video frames in the video streams of more than two virtual human roles are merged, and the merged target video stream is generated and pushed to the client for playing. According to the method, more than two virtual human roles can be combined into one video stream, the same-screen interaction of more than two virtual humans is realized, the number of the interactive virtual human roles is increased, the diversity of virtual human display is improved, the expressive force of the virtual humans is enriched, the virtual humans can be more vivid in a more complex scene, and the user experience is improved. And personalized requirements of users are met.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a virtual human interaction method, apparatus, related equipment and computer program product. Background Art

[0002] Interactive virtual humans are digital characters created through computer graphics, artificial intelligence and other technologies, with high realism and interactive capabilities. Interactive virtual humans are widely used in entertainment, games, commercial marketing, e-commerce, finance, education, medical care and other fields.

[0003] Existing interactive virtual human solutions can generally only process and control one virtual human character. In some scenarios, the diversity and flexibility are slightly insufficient, especially for some complex scenes, the expressiveness is somewhat lacking, and it cannot meet the personalized needs of different users and different scenarios. Summary of the invention

[0004] In view of the above problems, this application is proposed to provide a virtual human interaction method, apparatus, related equipment and computer program product to achieve multi-role virtual human interaction on the same screen, improve the diversity of virtual human interaction, and meet the personalized needs of different users and different scenarios. The specific solution is as follows:

[0005] In a first aspect, a virtual human interaction method is provided, comprising:

[0006] Obtaining a virtual human role creation request, the request including role identifiers of two or more virtual humans to be created;

[0007] Creating two or more corresponding virtual human characters according to the character identifiers, and generating video streams of the virtual human characters through a virtual human engine corresponding to each virtual human character;

[0008] Merging corresponding video frames in the video streams of the two or more virtual human characters, and using the merged video frames as the target video stream;

[0009] The target video stream is pushed to the client for playback.

[0010] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the video stream of the virtual human character generated by the virtual human engine is stored in a streaming media service module;

[0011] The process of merging corresponding video frames in the video streams of the two or more virtual human characters and using the merged video frames as the target video stream includes:

[0012] Pull the video streams of the two or more virtual human characters from the streaming media service module;

[0013] The corresponding video frames in the pulled video streams of the two or more virtual human characters are merged, and the merged video frames are used as target video streams and stored in the streaming media service module so that the client can pull the target video stream from the streaming media service module.

[0014] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of generating a video stream of a virtual human character by a virtual human engine corresponding to each virtual human character includes:

[0015] Determine the dialogue text for each virtual human character;

[0016] The dialogue text of each virtual human character is synthesized into speech, and the virtual human engine is driven based on the speech to generate a video stream of the virtual human character.

[0017] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of determining the dialogue text of each virtual human character includes:

[0018] Acquire script information input by the client, wherein the script information includes dialogue text configured for each virtual human character;

[0019] The dialogue text of each virtual human character is extracted from the script information.

[0020] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of determining the dialogue text of each virtual human character includes:

[0021] Acquire interactive voice input by the client, and recognize the interactive voice into interactive text;

[0022] A natural language processing model is called to determine a target virtual human role that needs to respond based on the interactive text, and response information of the target virtual human role is generated as a dialogue text of the target virtual human role.

[0023] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of determining the dialogue text of each virtual human character includes:

[0024] Get the interactive text entered by the client;

[0025] A natural language processing model is called to determine a target virtual human role that needs to respond based on the interactive text, and response information of the target virtual human role is generated as a dialogue text of the target virtual human role.

[0026] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of determining the dialogue text of each virtual human character includes:

[0027] Acquire multimodal interaction information input by the client, wherein the multimodal interaction information includes information of two or more modes among voice, facial expression, and gesture;

[0028] The multimodal processing model is called to determine the target virtual human role that needs to respond based on the multimodal interaction information, and the response information of the target virtual human role is generated as the dialogue text of the target virtual human role.

[0029] In a possible design, in another implementation of the first aspect of the embodiment of the present application, before the virtual human engine corresponding to each virtual human character generates a video stream of the virtual human character, the process further includes:

[0030] Receive video control information input by a client, where the video control information includes at least one of subtitle control information, virtual human position control information, background / foreground control information, interaction control information, and rendering control information;

[0031] The process of generating a video stream of a virtual human character by using a virtual human engine corresponding to each virtual human character includes:

[0032] The video stream of the virtual human character is generated according to the video control information through the virtual human engine corresponding to each virtual human character.

[0033] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of creating two or more corresponding virtual human roles according to the role identifier includes:

[0034] Searching for virtual human role resources corresponding to each role identifier in each configured virtual human role resource, and creating two or more corresponding virtual human roles using the found virtual human role resources;

[0035] or,

[0036] Determining parameter information of two or more virtual human characters to be created according to the character identifiers;

[0037] The virtual human generation engine is called, and the corresponding virtual human roles are created using the parameter information to obtain two or more created virtual human roles.

[0038] In a second aspect, a virtual human interaction device is provided, comprising:

[0039] A request acquisition unit, used to acquire a virtual human role creation request, wherein the request includes role identifiers of two or more virtual humans to be created;

[0040] A virtual human creation unit, used to create two or more corresponding virtual human roles according to the role identifier;

[0041] A virtual human video generation unit, used to generate a video stream of a virtual human character through a virtual human engine corresponding to each virtual human character;

[0042] A virtual human merging processing unit, used for merging corresponding video frames in the video streams of the two or more virtual human characters, and using each merged video frame as a target video stream;

[0043] The video streaming unit is used to push the target video stream to the client for playback.

[0044] In a third aspect, an electronic device is provided, comprising: a memory and a processor;

[0045] The memory is used to store programs;

[0046] The processor is used to execute the program to implement each step of the virtual human interaction method described in any one of the first aspects of the present application.

[0047] In a fourth aspect, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the various steps of the virtual human interaction method described in any one of the first aspects of the present application are implemented.

[0048] In a fifth aspect, a computer program product is provided, including a computer program, which, when executed by a processor, implements the various steps of the virtual human interaction method described in any one of the first aspects of the present application.

[0049] By means of the above technical solution, the present application supports the creation of more than two virtual human characters, can respond to a virtual human character creation request, create more than two corresponding virtual human characters according to the character identifier, and generate video streams of the virtual human characters through different virtual human engines respectively. More than two virtual human characters can be controlled at the same time, the corresponding video frames in the video streams of more than two virtual human characters are merged and processed, and the merged target video stream is generated and pushed to the client for playback. The present application solution can combine more than two virtual human characters into one video stream, realize the same-screen interaction of more than two virtual people, increase the number of interactive virtual human characters, and then increase the diversity of virtual human display, enrich the expressiveness of virtual people, and make virtual people more vivid in more complex scenes, meeting the personalized needs of different users and different scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0051] Figure 1 A schematic diagram of a system architecture for implementing the virtual human interaction method provided in an embodiment of the present application;

[0052] Figure 2 A flowchart of a virtual human interaction method provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of another virtual human interaction method flow chart provided in an embodiment of the present application;

[0054] Figure 4 A flowchart of another virtual human interaction method provided in an embodiment of the present application;

[0055] Figure 5 A flowchart of another virtual human interaction method provided in an embodiment of the present application;

[0056] Figure 6 A flowchart of another virtual human interaction method provided in an embodiment of the present application;

[0057] Figure 7 A flowchart of another virtual human interaction method provided in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of the structure of a virtual human interaction device provided in an embodiment of the present application;

[0059] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0061] The virtual human interaction solution of this application involves the following technologies:

[0062] 1. Real-time rendering technology:

[0063] 3D modeling and texture mapping: Use 3D modeling software to create the appearance and form of the virtual human model. Use texture mapping technology to add realistic skin, clothing, hair and other details to the virtual human model to improve the visual effect.

[0064] Skeletal animation and skinning: Create a skeletal system for the virtual human model and define the motion relationship between the bones.

[0065] The virtual human model is bound to the skeleton using skinning technology to achieve the deformation effect of the virtual human model following the movement of the skeleton.

[0066] Real-time rendering engine: A high-performance real-time rendering engine is used to achieve efficient rendering of virtual humans in a real-time environment. The lighting, shadow, reflection and other special effects provided by the engine are used to enhance the realism and immersion of virtual humans.

[0067] Performance optimization: To meet the performance requirements of real-time rendering, optimization measures such as model simplification and texture compression are carried out.

[0068] Use the performance analysis tools provided by the engine to identify performance bottlenecks and perform targeted optimization.

[0069] 2. Interactive Technology

[0070] Speech recognition and synthesis: Integrate speech recognition technology to enable virtual humans to understand the user's voice commands.

[0071] Through speech synthesis technology, the virtual person's response is converted into natural and fluent speech output.

[0072] Natural language processing: Natural language processing technology is used to enable virtual humans to understand and generate natural language.

[0073] Through the dialogue management system, appropriate responses are generated based on the user's intent and context information.

[0074] Facial expression capture and synthesis: Use facial expression capture technology to capture real people's facial expression changes in real time. Convert the captured expression data into virtual people's facial expressions to achieve more delicate emotional expression.

[0075] Gesture recognition and interaction: Integrate gesture recognition technology to enable the virtual human to recognize the user's gesture commands. Through gesture interaction, users can interact with the virtual human more intuitively, such as controlling the virtual human's movements, selecting menu items, etc.

[0076] Multimodal interaction: Combine multiple interaction methods such as voice, facial expressions, gestures, etc. to achieve a richer and more natural interactive experience. Through multimodal interaction, virtual humans can more accurately understand the user's intentions and emotional state, improving the accuracy and efficiency of interaction.

[0077] The virtual human interaction solution of this application can be applied to many different fields such as entertainment, games, commercial marketing, e-commerce, finance, education, and medical care. Some possible application scenarios are as follows:

[0078] In entertainment scenarios, it is possible to provide interactions between multiple virtual idols, between virtual idols and users,

[0079] In online education scenarios, virtual lecturers and virtual teaching assistants can be provided on the screen at the same time, and the two can interact in real time for online teaching, enhancing students' online learning experience and sense of reality.

[0080] In a live broadcast scenario, more than two anchor virtual human characters can be provided on the live broadcast side, and different anchor virtual human characters can interact with each other, for example, interactive games, interactive product promotion, etc.

[0081] Of course, the above examples only illustrate some possible application scenarios of the virtual human interaction solution of the present application. In addition, the virtual human interaction solution of the present application can also be applied to many other scenarios.

[0082] This application provides a virtual human interaction method, which can be applied to Figure 1 The system architecture shown in FIG. 1 may include a client 100 and a server 200. The server 200 may include one or more servers.

[0083] The client 100 or the server 200 can be used alone to execute the virtual human interaction method provided in the embodiment of the present application. In addition, the client 100 or the server 200 can also be used in conjunction to execute the virtual human interaction method provided in the embodiment of the present application.

[0084] The client 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a teaching screen, a wearable device, a vehicle-mounted device, a conference terminal, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0085] The client 100 has a display unit, which can be used to display information input by the user or information provided to the user, various menus of the client 100, interactive interfaces, file display and / or playback of any multimedia file. In the embodiment of the present application, the display unit can be used to display various interactive interfaces in the virtual human interaction method and a synthesized target video containing more than two virtual human characters.

[0086] The embodiment of the present application provides a virtual human interaction method, which is illustrated by applying the method to a computer device. The computer device may be Figure 1 The server 200 in. Figure 2 , the virtual human interaction method specifically includes the following steps:

[0087] Step S100: Obtain a virtual human role creation request, wherein the request includes role identifiers of two or more virtual humans to be created.

[0088] Specifically, the user can initiate a virtual human role creation request through the client, and specifically can request to create two or more virtual human roles. The request includes the role identifiers of the two or more virtual humans to be created. The role identifier of the virtual human is used to uniquely represent a virtual human role.

[0089] Step S110: creating two or more corresponding virtual human roles according to the role identifiers, and generating a video stream of the virtual human role through a virtual human engine corresponding to each virtual human role.

[0090] Specifically, after receiving the virtual human role creation request, the server can create a corresponding virtual human role according to the role identifier carried in the request.

[0091] In an optional embodiment, the server may be configured with resource information of virtual human roles corresponding to different role identifiers. When the server receives a virtual human role creation request, it may query the virtual human role resources corresponding to the requested role identifier, and then use the found virtual human role resources to create the corresponding virtual human role. When the virtual human role creation request contains more than two role identifiers, the server may create more than two virtual human roles.

[0092] In another optional embodiment, when receiving a request to create a virtual human role, the server reads each role identifier from the request. Further, according to the role identifier, the server determines the parameter information of each of the two or more virtual human roles to be created, such as appearance parameters, body parameters, action parameters, and other parameters. Then, the server calls the virtual human generation engine, creates the corresponding virtual human role using the parameter information, and finally obtains the two or more virtual human roles created.

[0093] After creating two or more virtual human characters requested by the user, a virtual human engine may be configured for each virtual human character, and a video stream of the virtual human character may be generated by the virtual human engine.

[0094] The virtual human engine can drive the virtual human character based on voice, which is the voice to be broadcast by the virtual human character. The broadcast voice, mouth shape and body movements of the virtual human character can be driven by voice to obtain the generated video stream of the virtual human character.

[0095] The voice that drives the virtual human character can be input by the user in the form of text, voice, etc., or it can be the reply voice of the virtual human character determined based on natural language understanding technology in the scenario where the virtual human character is communicating with the user.

[0096] Step S120: merging corresponding video frames in the video streams of the two or more virtual human characters, and using the merged video frames as target video streams.

[0097] Specifically, in the previous step, a corresponding video stream is generated for each virtual human character, and the video stream corresponding to each virtual human character only contains the image of a single virtual human character. In this step, the video streams of each virtual human character can be merged to achieve the purpose of playing more than two virtual people on the same screen. Specifically, the video frames of the same time in the video streams of more than two virtual human characters are merged, and the merged video frame containing more than two virtual human characters can be obtained by image synthesis technology. Among them, the process of merging two video frames includes but is not limited to: image synthesis methods such as layer overlay and mixing.

[0098] Furthermore, each video frame after the merging process is used as a target video stream. The target video stream is a video stream containing more than two virtual human characters.

[0099] Step S130: Push the target video stream to the client for playback.

[0100] Specifically, the server can actively push the target video stream to the client for playback. Alternatively, the server can inform the client of the storage address of the target video stream, and the client can actively pull the target video stream based on the storage address. The user can then see the virtual human character video streams of more than two characters on the client.

[0101] The method provided in the embodiment of the present application supports the creation of more than two virtual human characters, can respond to a virtual human character creation request, create more than two corresponding virtual human characters according to the character identifier, and generate video streams of the virtual human characters through different virtual human engines respectively. More than two virtual human characters can be controlled at the same time, and the corresponding video frames in the video streams of more than two virtual human characters are merged and processed to generate a merged target video stream and push it to the client for playback. The present application scheme can combine more than two virtual human characters into one video stream, realize the same-screen interaction of more than two virtual people, increase the number of interactive virtual human characters, and then increase the diversity of virtual human display, enrich the expressiveness of virtual people, and make virtual people more vivid in more complex scenes to meet the personalized needs of different users and different scenes.

[0102] In one possible implementation, the data transmission process between the client and the server can use protocols such as WebRTC and WebSocket to achieve real-time data transmission, improve the timeliness of data transmission, and thus enhance the interaction effect.

[0103] Reference Figure 3 , provides another implementation process of virtual human interaction and supports the creation of more than two virtual human characters. Figure 3 In this example, we only create two virtual characters:

[0104] The client can initiate a multi-role session request. The request can include the role identifiers of the two virtual humans to be initiated. When the server receives the multi-role session request, it initiates role 1 and role 2 (corresponding to the role identifiers of the two virtual humans, respectively). Afterwards, the virtual human engine corresponding to role 1 generates the role 1 video stream, and the virtual human engine corresponding to role 2 generates the role 2 video stream.

[0105] In this embodiment, the video stream of the virtual human character generated by the virtual human engine can be stored in the streaming media service module. The streaming media service module can be integrated in the server, or can be other equipment independent of the server.

[0106] The generated video streams of character 1 and character 2 are stored in the streaming service module. The server pulls the video streams of two or more virtual human characters from the streaming service module (i.e., pulls the video stream of character 1 and the video stream of character 2). The corresponding video frames in each pulled video stream are merged, and the merged video frames are used as the target video stream (i.e., video stream 3) to achieve multi-character merging. The generated target video stream is stored in the streaming service module.

[0107] In an optional example, the server can perform the above steps of pulling and merging the video streams of the virtual human characters through the configured virtual human post-processing engine to obtain the target video stream. The storage addresses of the character 1 video stream and the character 2 video stream in the streaming media service module can be pre-specified. Therefore, the virtual human post-processing engine can pull the character 1 video stream and the character 2 video stream in the streaming media service module based on the two input pull stream addresses (i.e., the storage addresses of the character 1 video stream and the character 2 video stream in the streaming media service module).

[0108] The client can pull the target video stream for playback based on the storage address of the target video stream in the streaming media service module.

[0109] In this embodiment, a streaming media service module is set up to temporarily store video streams of multiple different virtual human characters for the virtual human post-processing engine to pull the video streams for merging and processing, and to store the merged target video stream. On this basis, the client can pull the target video stream to achieve download and playback, and the client can start watching the video without waiting for the entire video to be downloaded. At the same time, the server performs different task processing through multiple different processing engines, such as generating video streams of different virtual human characters through different virtual human engines, and merging the video streams of multiple different characters through the virtual human post-processing engine, so that the processing flow of the server is more simplified and clear.

[0110] Of course, the above Figure 3 Only two virtual human characters are used as examples for illustration. In addition, the present invention can be extended to scenarios with more virtual human characters, which will not be described in detail in this embodiment.

[0111] In some embodiments of the present application, the process of generating a video stream of a virtual human character by a virtual human engine corresponding to each virtual human character in the aforementioned step S110 is further described.

[0112] The virtual human engine can generate a video stream of a voice-driven character based on the voice corresponding to the dialogue text of the virtual human character, that is, the text to be broadcast by the virtual human character.

[0113] In an optional example, the dialogue text of each virtual human role is first determined, and then the dialogue text of each virtual human role is synthesized into speech, and a virtual human engine is driven based on the speech to generate a video stream of the virtual human role.

[0114] The dialogue text of each virtual human character can be obtained in a variety of ways according to different usage scenarios.

[0115] In an example scenario, the user can control the dialogue texts of different virtual human characters in the form of scripts through the client. For example, in a virtual human live broadcast scenario, the user can input the dialogue texts to be broadcasted by different virtual human characters in the form of scripts through the client.

[0116] Reference Figure 4 As shown, the process of determining the dialogue text of each virtual human role may include:

[0117] The server obtains the script information input by the client, and the script information includes the dialogue text configured for each virtual human role. A virtual human role identifier can be added to the script information, and the dialogue texts of different virtual human roles can be distinguished by different role identifiers.

[0118] Furthermore, different dialogue texts in the script information may also be marked with a broadcasting order, thereby facilitating the server to control the speech synthesis of different dialogue texts and the driving of the virtual human character.

[0119] The server extracts the dialogue text of each virtual human character from the script information.

[0120] The server can extract the dialogue text of each virtual human character from the script information according to the identifier of each virtual human character. The dialogue text is further converted into speech, so as to drive different virtual human characters through speech. The subsequent processing flow can refer to the relevant introduction in the previous article, which will not be repeated here.

[0121] The solution provided in this embodiment can support users to configure the dialogue texts of different virtual human characters in the form of scripts, and then control multiple different virtual human characters to interact on the same screen. For example, in a virtual human live broadcast scenario, users can control the dialogue texts to be broadcast by different virtual human characters in the form of scripts, and the interaction between multiple different virtual human characters can be completed under user control.

[0122] In another example scenario, a user can interact with multiple different virtual human characters through a client. For example, in a scenario where multiple virtual characters chat, a user can input the conversation content in voice or text form through the client, and each virtual human character generates a reply content and synthesizes it into a target video stream for playback.

[0123] Reference Figure 5 As shown, it illustrates the process of a user having a conversation and chatting with multiple different virtual human characters in the form of voice. The process of determining the conversation text of each virtual human character may include:

[0124] The server obtains the interactive voice input by the client and recognizes the interactive voice into interactive text.

[0125] A natural language processing model is called to determine the target virtual human role that needs to respond based on the interactive text, and response information of the target virtual human role is generated as the dialogue text of the target virtual human role.

[0126] Among them, the user can input interactive voice through the client, and the interactive voice is a conversation voice initiated by the user for one or more characters among multiple different virtual human characters. Take the scenario where the user chats with two virtual human characters as an example, and define virtual human characters A and B. Then the interactive voice input by the user can be "The weather is so nice today, everyone, tell me where we should go to play.", and the interactive voice can trigger characters A and B to reply respectively. For another example, the interactive voice input by the user can be "The weather is so nice today, A, where are you going to play?", and the interactive voice can trigger character A to reply alone.

[0127] After receiving the interactive voice input by the user, the server first recognizes it as interactive text. Further, the natural language processing model configured by the server can be called, such as a large model, to determine the target virtual human role that needs to respond in this round based on the interactive text. The target virtual human role is any one or more roles among all virtual human roles.

[0128] Furthermore, the server may call a natural language processing model to generate response information of the target virtual human character as the dialogue text of the target virtual human character.

[0129] In an optional example, the server can call the big model to determine the target virtual human character that needs to respond in this round and the response information of the target virtual human character based on the dialogue history and the user's interaction text.

[0130] Reference Figure 6 As shown, it illustrates the process of a user having a conversation with multiple different virtual human characters in text form. The process of determining the conversation text of each virtual human character may include:

[0131] Get the interactive text entered by the client.

[0132] A natural language processing model is called to determine a target virtual human role that needs to respond based on the interactive text, and response information of the target virtual human role is generated as a dialogue text of the target virtual human role.

[0133] Compared with the form of user input through voice, in this embodiment, the user can input the interactive text in text form. At this time, the voice-to-text processing link can be omitted, and the natural language processing model can be called to directly determine the target virtual human role that needs to respond and its response information based on the input interactive text as the dialogue text of the target virtual human role.

[0134] The above embodiments respectively introduce scenarios in which users input conversation content in the form of voice or text. The server can determine the target virtual human character and its response information that need to be responded to through voice recognition technology and natural speech understanding technology, and then synthesize it into voice through speech synthesis technology, and drive the generation of video stream of the target virtual human character through voice, so as to support users to flexibly select the interaction form.

[0135] Reference Figure 7 As shown, it illustrates the process of a user having a conversation with multiple different virtual human characters in the form of multimodal input. The process of determining the conversation text of each virtual human character may include:

[0136] Multimodal interaction information input by a client is obtained, where the multimodal interaction information includes information of two or more modes among voice, facial expression, and gesture.

[0137] The multimodal processing model is called to determine the target virtual human role that needs to respond based on the multimodal interaction information, and the response information of the target virtual human role is generated as the dialogue text of the target virtual human role.

[0138] Compared with the aforementioned embodiment, the present embodiment supports multimodal interaction, that is, the user can use more than two modes of interactive input such as voice, facial expressions, gestures, etc. For example, the client can obtain the user's input in two modes, voice and facial expressions. On this basis, the server supports the processing of multimodal information. Specifically, a multimodal processing model can be called to process the input of a variety of different modal information, so as to determine the target virtual human role that needs to respond, and its response information. By supporting multimodal interaction, the present embodiment can achieve a richer and more natural interactive experience. Through multimodal interaction, the virtual human role can more accurately understand the user's intentions and emotional state, thereby improving the accuracy and efficiency of the interaction.

[0139] In some embodiments of the present application, another implementation process of the virtual human interaction method is provided. During the virtual human interaction process, the user can control the style of the generated virtual human character video stream through the client. Specifically, before the aforementioned step S110 generates the video stream of the virtual human character through the virtual human engine corresponding to each virtual human character, the following processing steps may also be included:

[0140] Receive video control information input by the client.

[0141] On this basis, the process of generating a video stream of a virtual human character through a virtual human engine corresponding to each virtual human character includes:

[0142] The video stream of the virtual human character is generated according to the video control information through the virtual human engine corresponding to each virtual human character.

[0143] The video control information may include at least one of subtitle control information, virtual human position control information, background / foreground control information, interaction control information, and rendering control information.

[0144] Video control information is used to describe and manage the presentation of video content.

[0145] in:

[0146] Subtitle control information, used to manage the display content, position, font, color, size, appearance time, etc. of subtitles. Examples: subtitle text content (such as dialogue text); subtitle style (such as font, color, background transparency); subtitle timeline (such as start time, end time).

[0147] Virtual human position control information is used to control the position, size, rotation angle, movement, etc. of the virtual human in the video screen. Examples: virtual human coordinate position (such as X, Y, Z axis coordinates); virtual human action instructions (such as waving, nodding); virtual human expression changes (such as smiling, frowning).

[0148] Background control information is used to manage the content, color, dynamic effects, etc. of the video background. Examples: background image or video material; background dynamic effects (such as gradient, blur, particle effects); background switching time point.

[0149] Foreground control information is used to manage the display mode of foreground elements (such as virtual humans, props, special effects). Examples: transparency, size, and position of foreground elements; foreground special effects (such as halo and shadow); interaction between foreground and background (such as occlusion relationship).

[0150] Timeline control information is used to manage the appearance time, duration, and synchronization relationship of each element in the video. Examples: the synchronization time point of subtitles and virtual human actions; the triggering time of background switching and foreground special effects.

[0151] Interaction control information is used to manage the interaction between the virtual human and the user or other elements. Examples: feedback action after the user clicks on the virtual human; the virtual human dynamically adjusts the conversation content based on the user input.

[0152] Rendering control information, used to manage the video rendering method, such as resolution, frame rate, lighting effects, etc. Examples: video resolution (such as 1080p, 4K); lighting parameters (such as light source position, intensity); configuration parameters of the rendering engine.

[0153] In the method provided in this embodiment, the user can control the synthesis control information of the virtual human character video flow through the client, such as subtitles, virtual human position, foreground, background information, etc., thereby improving the presentation of the synthesized virtual human character video stream.

[0154] In other possible implementations, after determining the dialogue text of the virtual human character, the server can automatically determine the relevant control information of the synthesized video stream based on the natural language processing model, and then generate the video stream of the virtual human character according to the determined video control information through the virtual human engine corresponding to each virtual human character.

[0155] Exemplarily, the server can generate matching video control information based on the dialogue text of the virtual human character by calling the capabilities of the large model.

[0156] Example scenario:

[0157] Assume that a user is having a conversation with a virtual character A among multiple virtual characters. The conversation text is as follows:

[0158] User: "Can you take me to see the beautiful sunset at the seaside?"

[0159] Virtual character A: "Of course, let's enjoy the beautiful sunset at the seaside together!"

[0160] The server uses the big model to analyze the conversation text and identify key information, such as "seaside" and "sunset". Based on the identified key information, it searches for video clips related to "seaside sunset" from the preset video resource library. These video clips may contain different seaside sunset scenes, and one or more clips that best match the user's description can be selected.

[0161] Foreground and background matching:

[0162] In the selected video clips, the big model further distinguishes foreground and background information. The foreground may be dynamic elements such as waves, pedestrians or animals on the beach; the background is static or slowly changing elements such as the sky and the sea surface. The big model automatically adjusts the presentation of the foreground and background according to the context of the dialogue text and the user's expectations.

[0163] After determining the video control information such as foreground and background information, the server can call the virtual human engine to generate a video stream of the virtual human character according to the video control information.

[0164] Example results:

[0165] Finally, a video containing a conversation between a virtual person and a user is generated. In the video, the virtual person stands by the sea with a beautiful sunset in the background. As the conversation progresses, the movements and expressions of the virtual person are synchronized with the voice, giving the user an immersive feeling.

[0166] In the method provided in this embodiment, there is no need for the user to manually control the relevant parameter information of the virtual human character video stream synthesis process. The server can call on the large model and other capabilities to automatically match the appropriate video control information based on the dialogue text, thereby realizing more intelligent virtual human interaction.

[0167] The virtual human interaction device provided in an embodiment of the present application is described below. The virtual human interaction device described below and the virtual human interaction method described above can be referenced to each other.

[0168] See also Figure 8 , Figure 8 A schematic diagram of the structure of a virtual human interaction device disclosed in an embodiment of the present application.

[0169] like Figure 8 As shown, the device may include:

[0170] A request acquisition unit 11 is used to acquire a virtual human role creation request, wherein the request includes role identifiers of two or more virtual humans to be created;

[0171] A virtual human creation unit 12, configured to create two or more corresponding virtual human roles according to the role identifier;

[0172] A virtual human video generating unit 13 is used to generate a video stream of a virtual human character through a virtual human engine corresponding to each virtual human character;

[0173] A virtual human merging processing unit 14 is used to merge corresponding video frames in the video streams of the two or more virtual human characters, and use the merged video frames as the target video stream;

[0174] The video streaming unit 15 is used to push the target video stream to the client for playback.

[0175] In a possible implementation, the virtual human video generation unit stores the video stream of the virtual human character generated by the virtual human engine to the streaming media service module. Then, the virtual human merging processing unit merges the corresponding video frames in the video streams of the two or more virtual human characters, and uses the merged video frames as the target video stream, including:

[0176] Pull the video streams of the two or more virtual human characters from the streaming media service module;

[0177] The corresponding video frames in the pulled video streams of the two or more virtual human characters are merged, and the merged video frames are used as target video streams and stored in the streaming media service module so that the client can pull the target video stream from the streaming media service module.

[0178] In a possible implementation, the process of the virtual human video generation unit generating a video stream of a virtual human character through a virtual human engine corresponding to each virtual human character includes:

[0179] Determine the dialogue text for each virtual human character;

[0180] The dialogue text of each virtual human character is synthesized into speech, and the virtual human engine is driven based on the speech to generate a video stream of the virtual human character.

[0181] In a possible implementation, the process of determining the dialogue text of each virtual human character by the virtual human video generation unit includes:

[0182] Acquire script information input by the client, wherein the script information includes dialogue text configured for each virtual human character;

[0183] The dialogue text of each virtual human character is extracted from the script information.

[0184] In another possible implementation, the process of the virtual human video generation unit determining the dialogue text of each virtual human role includes:

[0185] Acquire interactive voice input by the client, and recognize the interactive voice into interactive text;

[0186] A natural language processing model is called to determine a target virtual human role that needs to respond based on the interactive text, and response information of the target virtual human role is generated as a dialogue text of the target virtual human role.

[0187] In another possible implementation, the process of the virtual human video generation unit determining the dialogue text of each virtual human role includes:

[0188] Get the interactive text entered by the client;

[0189] A natural language processing model is called to determine a target virtual human role that needs to respond based on the interactive text, and response information of the target virtual human role is generated as a dialogue text of the target virtual human role.

[0190] In another possible implementation, the process of the virtual human video generation unit determining the dialogue text of each virtual human role includes:

[0191] Acquire multimodal interaction information input by the client, wherein the multimodal interaction information includes information of two or more modes among voice, facial expression, and gesture;

[0192] The multimodal processing model is called to determine the target virtual human role that needs to respond based on the multimodal interaction information, and the response information of the target virtual human role is generated as the dialogue text of the target virtual human role.

[0193] In a possible implementation, the device of the present application may further include:

[0194] The video control information receiving unit is used to receive video control information input by the client before the virtual human engine corresponding to each virtual human character generates the video stream of the virtual human character, wherein the video control information includes at least one of subtitle control information, virtual human position control information, background / foreground control information, interaction control information, and rendering control information. Then, the process in which the virtual human video generating unit generates the video stream of the virtual human character through the virtual human engine corresponding to each virtual human character includes:

[0195] The video stream of the virtual human character is generated according to the video control information through the virtual human engine corresponding to each virtual human character.

[0196] In a possible implementation, the process of the virtual human creation unit creating two or more corresponding virtual human roles according to the role identifier includes:

[0197] Searching for virtual human role resources corresponding to each role identifier in each configured virtual human role resource, and creating two or more corresponding virtual human roles using the found virtual human role resources;

[0198] or,

[0199] Determining parameter information of two or more virtual human characters to be created according to the character identifiers;

[0200] The virtual human generation engine is called, and the corresponding virtual human roles are created using the parameter information to obtain two or more created virtual human roles.

[0201] The present application also provides an electronic device in an embodiment. Fig. 9 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, tablet computers, vehicle-mounted terminals, teaching large screens, wearable devices, etc. Fig. 9 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0202] like Fig. 9As shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603 to implement the virtual human interaction method of the aforementioned embodiment of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0203] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0204] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any virtual human interaction method provided in the embodiment of the present application.

[0205] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any virtual human interaction method provided in the embodiment of the present application.

[0206] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0207] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0208] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0209] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0210] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

Claims

1. A virtual human interaction method, characterized in that: include: Obtaining a virtual human role creation request, the request including role identifiers of two or more virtual humans to be created; Creating two or more corresponding virtual human characters according to the character identifiers, and generating a video stream of the virtual human character through a virtual human engine corresponding to each virtual human character; Merging corresponding video frames in the video streams of the two or more virtual human characters, and using the merged video frames as the target video stream; The target video stream is pushed to the client for playback.

2. The method according to claim 1, characterized in that The video stream of the virtual human character generated by the virtual human engine is stored in a streaming media service module; The corresponding video frames in the video streams of the two or more virtual human characters are merged, and each merged video frame is used as the target video stream, including: Pulling the video streams of the two or more virtual human characters from the streaming media service module; The corresponding video frames in the pulled video streams of the two or more virtual human characters are merged, and the merged video frames are used as the target video stream and stored in the streaming media service module so that the client can pull the target video stream from the streaming media service module.

3. The method according to claim 1, characterized in that The process of generating a video stream of a virtual human character by using a virtual human engine corresponding to each virtual human character includes: Determine the dialogue text for each virtual human character; The dialogue text of each virtual human character is synthesized into speech, and the virtual human engine is driven based on the speech to generate a video stream of the virtual human character.

4. The method according to claim 3, characterized in that The process of determining the dialogue text for each virtual human character includes: Acquire script information input by the client, wherein the script information includes dialogue text configured for each virtual human character; The dialogue text of each virtual human character is extracted from the script information.

5. The method according to claim 3, characterized in that: The process of determining the dialogue text for each virtual human character includes: Acquire interactive voice input by the client, and recognize the interactive voice into interactive text; A natural language processing model is called to determine a target virtual human role that needs to respond based on the interactive text, and response information of the target virtual human role is generated as a dialogue text of the target virtual human role.

6. The method according to claim 3, characterized in that The process of determining the dialogue text for each virtual human character includes: Get the interactive text entered by the client; A natural language processing model is called to determine a target virtual human role that needs to respond based on the interactive text, and response information of the target virtual human role is generated as a dialogue text of the target virtual human role.

7. The method according to claim 3, characterized in that The process of determining the dialogue text for each virtual human character includes: Acquire multimodal interaction information input by the client, wherein the multimodal interaction information includes information of two or more modes among voice, facial expression, and gesture; The multimodal processing model is called to determine the target virtual human role that needs to respond based on the multimodal interaction information, and the response information of the target virtual human role is generated as the dialogue text of the target virtual human role.

8. The method according to claim 1, characterized in that Before the virtual human engine corresponding to each virtual human character generates a video stream of the virtual human character, the method further includes: Receive video control information input by a client, where the video control information includes at least one of subtitle control information, virtual human position control information, background / foreground control information, interaction control information, and rendering control information; The process of generating a video stream of a virtual human character by using a virtual human engine corresponding to each virtual human character includes: The video stream of the virtual human character is generated according to the video control information through the virtual human engine corresponding to each virtual human character.

9. The method according to any one of claims 1 to 8, characterized in that: The process of creating two or more corresponding virtual human roles according to the role identifier includes: Searching for virtual human role resources corresponding to each role identifier in each configured virtual human role resource, and creating two or more corresponding virtual human roles using the found virtual human role resources; or, Determining parameter information of two or more virtual human characters to be created according to the character identifiers; The virtual human generation engine is called, and the corresponding virtual human roles are created using the parameter information to obtain two or more created virtual human roles.

10. A virtual human interaction device, characterized in that: include: A request acquisition unit, used to acquire a virtual human role creation request, wherein the request includes role identifiers of two or more virtual humans to be created; A virtual human creation unit, used to create two or more corresponding virtual human roles according to the role identifier; A virtual human video generation unit, used to generate a video stream of a virtual human character through a virtual human engine corresponding to each virtual human character; A virtual human merging processing unit, used for merging corresponding video frames in the video streams of the two or more virtual human characters, and using each merged video frame as a target video stream; The video streaming unit is used to push the target video stream to the client for playback.

11. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the virtual human interaction method as described in any one of claims 1 to 9.

12. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the virtual human interaction method according to any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the virtual human interaction method as described in any one of claims 1 to 9 is implemented.

Images