Playing method and device

Abstract

The invention discloses a playing method and device, and belongs to the technical field of artificial intelligence, and the method comprises the steps: in a process of obtaining user voice information when a user carries out a video call with a virtual object, playing a first virtual object audio-visual image in a first display region of a video call interface; a video stream request is sent to the server side, the video stream request carries a virtual object response text corresponding to the user voice information and display data of the virtual object, and the video stream request is used for requesting the server side to construct virtual object video stream data matched with the image of the virtual object based on the display data; generating virtual object response video stream data according to the virtual object video stream data, the virtual object response text and the virtual object response audio of the virtual object response text; and receiving virtual object response video stream data sent by the server, and switching the first virtual object played in the first display area into a virtual object response audio-visual picture rendered by the virtual object response video stream data.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a playback method and a device thereof. Background Art

[0002] With the development of artificial intelligence technology, electronic devices can support users to make video calls with virtual objects created by artificial intelligence (AI) technology. During the video call, the virtual object is required to answer the user's questions like a real user, and the visual image of the virtual object is required to be visible throughout the video call. For example, the virtual object must have corresponding body movements, facial expressions, speaking sounds and lip shapes.

[0003] In the related art, in order to ensure that the visual image of the virtual object is visible throughout the video call, the server needs to continuously draw the video stream data of the virtual object during the video call, and render and display the visual image of the virtual object corresponding to the video stream data through the client. In addition, in the question-and-answer interaction between the virtual object and the user, the client needs to continuously identify the virtual object's speaking state, and trigger the acquisition of the user's voice information when it detects that the speech is finished, and send it to the server so that the server can draw the virtual object's answering video stream data. However, continuously drawing the virtual object's video stream data will increase the server load, and continuously identifying the virtual object's speaking state will increase the client's performance overhead. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a playback method, device, electronic device and storage medium, which can reduce the server load and the client performance overhead.

[0005] In a first aspect, an embodiment of the present application provides a playback method, including:

[0006] In the process of acquiring user voice information when the user is making a video call with the virtual object, playing the first virtual object audio-visual picture in the first display area of ​​the video call interface;

[0007] Sending a video stream request to the server, the video stream request carrying the virtual object response text corresponding to the user voice information and the display data of the virtual object, the video stream request being used to request the server to construct virtual object video stream data matching the image of the virtual object based on the display data, and to generate virtual object response video stream data according to the virtual object video stream data, the virtual object response text, and the virtual object response audio of the virtual object response text;

[0008] Receive the virtual object response video stream data sent by the server;

[0009] The first virtual object audio-visual picture played in the first display area is switched to a virtual object response audio-visual picture, where the virtual object response audio-visual picture is rendered by the virtual object response video stream data.

[0010] In a second aspect, an embodiment of the present application provides a playback device, including:

[0011] A display module, configured to play an audio-visual picture of a first virtual object in a first display area of ​​a video call interface during a process of acquiring user voice information when the user makes a video call with the virtual object;

[0012] a sending module, used to send a video stream request to a server, the video stream request carrying a virtual object response text corresponding to the user voice information and display data of the virtual object, the video stream request being used to request the server to construct virtual object video stream data matching the image of the virtual object based on the display data, and to generate virtual object response video stream data according to the virtual object video stream data, the virtual object response text, and the virtual object response audio of the virtual object response text;

[0013] A receiving module, used for receiving the virtual object response video stream data sent by the server;

[0014] The processing module is used to switch the first virtual object audio-visual picture played in the first display area to a virtual object response audio-visual picture, where the virtual object response audio-visual picture is rendered by the virtual object response video stream data.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and when the program or instruction is executed by the processor, the steps of the playback method shown in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the playback method shown in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a display interface, the display interface and the processor are coupled, and the processor is used to run a program or instruction to implement the steps of the playback method shown in the first aspect.

[0018] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the playback method shown in the first aspect.

[0019] In an embodiment of the present application, considering that the virtual object only needs to make some default body movements in the listening stage and the thinking stage, and no audio and lip shape are required to match them, so in the process of obtaining the user voice information when the user has a video call with the virtual object, the first virtual object audio-visual picture displayed in the virtual object display area of ​​the video call interface replaces the process of the server continuously rendering the virtual object picture. In this way, there is no need for the server to continuously render the virtual object picture, and intermittently send video stream requests to the server, which reduces the time for the server to draw the virtual object video stream and reduces the load on the server; and, when the virtual object response video stream data sent by the server is received, the first virtual object audio-visual picture played in the virtual object display area is switched to the virtual object response audio-visual picture rendered by the virtual object response video stream data. In this way, there is no need for the client to continuously identify the speaking state of the virtual object, which reduces the high performance overhead of the client to identify that the virtual object has finished speaking, and reduces the frequency of data transmission between the server and the client, which can reduce the transmission cost of network resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a playback method provided for some embodiments of the present application;

[0021] Figure 2 A schematic diagram of an interface of a playback method provided in some embodiments of the present application;

[0022] Figure 3 One of the interface schematic diagrams of the playback method provided in some embodiments of the present application;

[0023] Figure 4 A second schematic diagram of an interface of a playback method provided in some embodiments of the present application;

[0024] Figure 5 A flowchart of a server-based playback method provided for some embodiments of the present application;

[0025] Figure 6 A schematic diagram of the interactive process of the playback method provided in some embodiments of the present application;

[0026] Figure 7 A schematic diagram of the structure of a playback device provided in some embodiments of the present application;

[0027] Figure 8 A schematic diagram of the structure of an electronic device provided for some embodiments of the present application;

[0028] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided for some embodiments of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0030] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0031] In related technologies, in order to ensure that the visual image of the virtual object is visible throughout the video call, the server needs to continuously draw the video stream data of the virtual object during the video call. Based on this, the process of continuously transmitting the video stream data of the virtual object between the client and the server is too dependent on the transmission network. If the quality of the transmission network is low, the transmission delay will increase, making the client display the virtual object picture unsmooth, and also reducing the quality and efficiency of the user's interaction with the virtual object.

[0032] In order to solve the problems in the related art, the embodiments of the present application provide a playback method, device and electronic device. Figures 1 to 9 , the playback method provided in the embodiment of the present application is described in detail through specific embodiments and application scenarios.

[0033] First, combine Figure 1 A playback method provided in an embodiment of the present application is described in detail.

[0034] Figure 1 A flowchart of a playback method provided for some embodiments of the present application.

[0035] like Figure 1 As shown, the playback method provided in the embodiment of the present application can be applied to electronic devices. Based on this, the playback method can include steps 110 and 130 as shown below.

[0036] Step 110 , in the process of obtaining user voice information when the user makes a video call with the virtual object, playing the first virtual object audio-visual picture in the first display area of ​​the video call interface.

[0037] Step 120, sending a video stream request to the server, the video stream request carries the virtual object response text corresponding to the user voice information and the display data of the virtual object, the video stream request is used to request the server to construct virtual object video stream data that matches the image of the virtual object based on the display data, and generate virtual object response video stream data based on the virtual object video stream data, the virtual object response text, and the virtual object response audio of the virtual object response text.

[0038] Step 130: Receive the virtual object response video stream data sent by the server.

[0039] Step 140, when receiving the virtual object response video stream data sent by the server, the first virtual object audio-visual picture displayed in the virtual object display area is switched to a virtual object response audio-visual picture, and the virtual object response audio-visual picture is rendered by the virtual object response video stream data.

[0040] Therefore, in the embodiment of the present application, it is taken into account that the virtual object only needs to make some default body movements in the listening stage and the thinking stage, and there is no need for audio and lip shape to match it. Therefore, in the process of obtaining the user voice information when the user has a video call with the virtual object, the first virtual object audio-visual picture displayed in the virtual object display area of ​​the video call interface replaces the process of the server continuously rendering the virtual object picture. In this way, there is no need for the server to continuously render the virtual object picture, and intermittently send video stream requests to the server, which reduces the time for the server to draw the virtual object video stream and reduces the load on the server; and, when the virtual object response video stream data sent by the server is received, the first virtual object audio-visual picture played in the virtual object display area is switched to the virtual object response audio-visual picture rendered by the virtual object response video stream data. In this way, there is no need for the client to continuously identify the speaking state of the virtual object, which reduces the high performance overhead of the client to identify that the virtual object has finished speaking, and reduces the frequency of data transmission between the server and the client, which can reduce the transmission cost of network resources.

[0041] The above steps are described in detail below, as shown below.

[0042] First, referring to step 110 , in some embodiments of the present application, before step 110 , the playback method may further include steps 1501 to 1503 .

[0043] Step 1501: Receive a first input from a user to initiate a video call with a virtual object.

[0044] Step 1502, in response to the first input, play the video call interface, the video call interface includes a first display area.

[0045] For example, Figure 2 As shown, a video call interface can be displayed, which includes a first display area 20, and the first display area 20 is used to display a preset virtual object audio-visual picture, such as a first virtual object audio-visual picture, a second virtual object audio-visual picture or a third virtual object audio-visual picture, and the first display area 20 is also used to display a virtual object answer audio-visual picture.

[0046] Step 1503: Play the audio-visual image of the third virtual object in the first display area and record the audio.

[0047] Among them, the third virtual object audio-visual picture may include at least two third virtual object audio-visual pictures. At this time, at least two third virtual object audio-visual pictures can be played according to the preset virtual object audio-visual picture playback order, and the third virtual object audio-visual picture includes a picture used to represent at least one of the following: a picture of the virtual object in a listening state, and a picture of the virtual object in a thinking state.

[0048] Based on this, step 110 may specifically include:

[0049] When a voice signal is detected in the audio through a voice activation detection algorithm, user voice information is obtained when the user makes a video call with a virtual object, and the third virtual object audio-visual picture played in the first display area is switched to the first virtual object audio-visual picture.

[0050] For example, a voice activity detection algorithm (VAD) can be used to detect whether the recorded audio includes a voice signal. If the recorded audio includes a voice signal, the third virtual object audio-visual picture played in the first display area is switched to the first virtual object audio-visual picture. Here, the first virtual object audio-visual picture includes a picture for representing at least one of the following: a picture of the virtual object listening, and a picture of the virtual object in a thinking state.

[0051] In this way, the user can intuitively browse the status of the virtual object through the virtual object images of the virtual object at different stages in the first virtual object video screen and the third virtual object video screen, thereby prompting the user whether the virtual object is listening to the user's questions or waiting to answer questions during the current video call, thereby improving the interaction efficiency between the user and the virtual object.

[0052] Here, before playing the first virtual object audio-visual image in the first display area of ​​the video call interface, the playback method also includes a process of obtaining user voice information. Based on this, before step 110, the playback method may also include steps 1601 and 1602.

[0053] Step 1601, receiving the user's voice audio.

[0054] Step 1602: convert the user's voice audio through an audio signal speech recognition conversion algorithm to obtain user voice information.

[0055] Here, the embodiment of the present application provides a method for obtaining user voice information based on the user's video image in the video call interface for some special groups, such as deaf-mute users, users who are sick and cannot speak, the elderly, etc., that is, before step 110, the playback method can also include steps 1603 and 1606.

[0056] Step 1603, receiving the user's voice audio, and capturing the user's video image in the video call interface.

[0057] Step 1604, converting the user's voice audio through an audio signal speech recognition conversion algorithm to obtain a first text; and in the case where the video image includes the user's sign language movements, generating a second text representing the meaning of the sign language movements based on the sign language movements in the video image.

[0058] Step 1605: Generate user voice information based on the first text and the second text.

[0059] In this way, for some special groups of people, user voice information can also be accurately obtained, broadening the applicable scenarios for users to make video calls with virtual objects.

[0060] Next, step 120 is involved. In the embodiment of the present application, the video stream data can be sent to the server through the WebSocket connection between the client and the server.

[0061] Based on this, in some embodiments of the present application, the display data includes the display coordinates of the virtual object in the first display area and the display size of the virtual object display area in the video call interface, wherein the display coordinates can be specifically the coordinates of the center point of the virtual object relative to the coordinate system of the virtual object display area. Based on this, before step 120, the playback method can also include steps 1201 and 1202.

[0062] Step 1201, based on the user voice information, determine the virtual object response text corresponding to the user voice information.

[0063] Step 1202, construct a video stream request based on the virtual object response text, display size and display coordinates.

[0064] It should be noted that if the virtual object has at least two images, such as virtual object 1 and virtual object 2, the display data may also include an image identifier of the virtual object, such as 1 or 2. In this way, the server can select the virtual object video stream data corresponding to the image identifier from at least two images based on the video stream request carrying the image identifier, so that the virtual object video stream data is consistent with the image of the virtual object on the electronic device side, and based on the display size and display coordinates, the virtual object video stream data can be made consistent with the size of the virtual object on the electronic device side, thereby avoiding frame skipping and achieving seamless connection.

[0065] Furthermore, the above step 1201 may specifically include the following steps 12011 to 12014.

[0066] Step 12011, fill the user voice information and the virtual object identity data of the virtual object into the preset artificial intelligence text assembly template to obtain text assembly content. Among them, the virtual object identity data may include basic personality settings and video call tasks.

[0067] Exemplarily, after obtaining the user voice information of the user speaking, the electronic device can play the preset virtual object audio-visual picture in the above stage. At the same time, through artificial intelligence tools such as prompt, the user voice information and the virtual object identity data of the virtual object are filled into the preset artificial intelligence text assembly template. Here, the basic person setting of the virtual object such as {name}, the video call task such as {companyName} and the user voice information, that is, the user question such as {question} are filled into the preset artificial intelligence text assembly template to obtain the text assembly content, as shown below:

[0068] You need to perform a role-playing task, playing the role of a human female named {name}. The following are the specific settings of {name}: Your basic information: You are a 27-year-old woman, 165 cm tall and 50 kg in weight; you are calm, rigorous and considerate; you are currently working for {companyName} as a project manager. You now start to play the role of {name} and chat with users. You cannot call yourself an artificial intelligence. You need to answer users' questions as {name}. You can use some more obscure metaphors or statements to answer questions that are beyond your character cognition. Now answer the user's question "{question}". Step 12012, send a response data request to the large model server. The response data request carries text assembly content. The response data request is used to request the large model server to stream the text assembly content and obtain streaming keywords related to the text assembly content.

[0069] Here, it should be noted that the large model server performs streaming processing on the text assembly content, which means that when the model generates content, it does not output all the results at once, but generates output step by step and continuously. This method is similar to the thinking process of humans when they are communicating or writing, that is, they think and express at the same time. Streaming processing can include streaming output, which is a technology that allows the model to generate text, not output all the text at once, but generate output step by step and continuously in the form of a stream.

[0070] Step 12013, upon receiving at least two streaming keywords sent by the large model server, store the at least two streaming keywords into the buffer.

[0071] Exemplarily, the text assembly content, i.e., prompt query text, is used to request the virtual object response text, i.e., video stream driven text, from the large model server. Here, when the streaming keywords returned by the large model streaming are received, they can be converted into sentences through the creation buffer created by the electronic device and stored in the buffer.

[0072] Step 12014, when it is identified in the buffer that a target sentence constructed by at least two streaming keywords meets a preset sentence composition condition, a virtual object response text is generated based on the target sentence.

[0073] Exemplarily, the buffer is checked by punctuation marks to see whether a complete sentence is contained, and when a complete sentence is detected, it is output, that is, the target sentence is obtained, and the target sentence can be subjected to text semantic processing to obtain a virtual object response text.

[0074] Furthermore, the above step 1202 may specifically include:

[0075] According to the preset data structure, the virtual object response text, display size and display coordinates are packaged and processed to obtain a video stream request.

[0076] Among them, the display size may include the width of the virtual object display area in the video call interface and the height of the virtual object display area in the video call interface.

[0077] For example, Figure 3 As shown, according to the preset data structure, the width W, height H, display coordinates (X, Y) of the virtual object display area in the video call interface and the virtual object response text are packaged and processed to obtain a video stream request, that is, Figure 3 The data in the preset data structure is as follows:

[0078] {width: w,

[0079] Height: h,

[0080] Display coordinates: (x, y)

[0081] Virtual object reply text: text}.

[0082] In this way, based on the display size and display coordinates, the virtual object video stream data can be made consistent with the size of the virtual object on the electronic device end, thereby avoiding frame skipping and achieving seamless connection.

[0083] In other embodiments of the present application, the video stream request may also carry the playback order of the virtual object audio-visual images, so as to obtain more accurate virtual object video stream data.

[0084] Based on this, the first virtual object audio-visual screen in the embodiment of the present application includes at least two first virtual object audio-visual screens, and the at least two first virtual object audio-visual screens are played according to the preset playback order of the virtual object audio-visual screens; the video stream request also carries the playback order of the first target virtual object audio-visual screen in the at least two first virtual object audio-visual screens; the video stream request is also used to request the server to obtain the display form of the virtual object in the first target virtual object audio-visual screen corresponding to the playback order, and based on the display form of the virtual object in the first target virtual object audio-visual screen, construct virtual object video stream data matching the virtual object image of the virtual object. Based on this, step 1202 can specifically include:

[0085] Construct a video stream request based on the play order, virtual object response text, display size and display coordinates.

[0086] For example, Figure 3 As shown, the preset virtual object video screen (such as the first virtual object video screen) built into the electronic device can be an animation resource or a video. In order to make the virtual object image of the virtual object in the preset virtual object video screen consistent with the virtual object image of the virtual object video stream data drawn by the server, the virtual object displayed in the preset virtual object video screen is in a default body movement of listening, or in a default body movement of thinking. The client program in the electronic device can initialize the virtual object container, set the width W, height H, and the coordinates (X, Y) of the center coordinates of the virtual object relative to the origin (0, 0).

[0087] Therefore, if the first virtual object video screen is realized by playing a picture sequence frame scheme and can be played in a loop, the first virtual object video screen can include at least two first virtual object video screens, that is, the animation resource is sliced ​​into fragment 1, fragment 2, fragment 3,..., and the time of each fragment is equal and relatively short, which can meet the condition of loop playback, that is, the last fragment can be seamlessly connected with fragment 1 to achieve the beginning and the end connection and avoid frame skipping.

[0088] Furthermore, the above step 1202 may specifically include:

[0089] According to the preset data structure, the play order, virtual object response text, display size and display coordinates are packaged and processed to obtain a video stream request.

[0090] Exemplarily, according to a preset data structure, the play order n, width W, height H, display coordinates (X, Y) and virtual object response text are packaged and processed. The preset data structure is exemplified as follows:

[0091] {width: w

[0092] Height: h

[0093] Coordinates: (x, y)

[0094] Play order: n

[0095] Driver text: text}.

[0096] Thus, based on the playback order, display size and display coordinates, the virtual object video stream data can be made consistent with the size of the virtual object on the electronic device end, and based on the playback order, the virtual object response audio-visual picture in the requested virtual object response video stream data can be seamlessly connected with the preset virtual object audio-visual picture, avoiding frame skipping, thereby improving the display effect of the virtual object audio-visual picture when making a video call with the virtual object.

[0097] Then, involving step 140, the virtual object response video stream data sent by the server can be received through the real-time communication connection channel of the web page. Based on this, before step 140, the playback method can also include steps 1701 to 1703.

[0098] Step 1701, based on the WebSocket connection with the server, obtain the communication address of the web page real-time communication corresponding to the video stream request, and the WebSocket connection is used to send the video stream request to the server.

[0099] Step 1702: Establish a real-time web page communication connection channel with the server according to the communication address.

[0100] Step 1703: Receive the virtual object response video stream data sent by the server through the web real-time communication connection channel.

[0101] Exemplarily, the electronic device can establish a Websocket connection with the server to obtain the communication address of the web real-time communication connection channel (Web Real-Time Communications, WebRTC) of the corresponding video stream; according to the communication address, establish WebRTC and receive the virtual object response video stream data sent by the server through WebRTC.

[0102] Here, it should be noted that in an embodiment of the present application, after the playback of the virtual object response audio-visual screen is completed, the Websocket connection can be disconnected and switched back to the first virtual object audio-visual screen preset in the electronic device, such as the animation of animation program segment 1, or a video segment, thereby completing a dialogue cycle.

[0103] In this way, there is no need for the server to continuously render the virtual object screen. By presetting the virtual object audio-visual screen, the server can be intermittently requested to respond to the video stream request of the virtual object.

[0104] Based on this, an embodiment of the present application provides the following method for playing a virtual object response audio-visual screen, as shown below.

[0105] In some embodiments of the present application, the virtual object response video stream data includes data of the virtual object response audio-visual picture connected with the first target virtual object audio-visual picture. Based on this, step 140 may specifically include steps 1401 to 1403.

[0106] Step 1401, splicing the virtual object response audio-visual picture after the first target virtual object audio-visual picture.

[0107] Step 1402: Switch the preset virtual object audio-visual picture currently being played in the first display area to the first target virtual object audio-visual picture.

[0108] Step 1403: When the first target virtual object audio-visual picture is finished playing, play the virtual object response audio-visual picture.

[0109] Exemplarily, the client program initializes the virtual object container at the bottom of the animation container and enters the video stream waiting to be played state. The client program detects the playing progress of the first virtual object audio-visual picture, and when the first target virtual object audio-visual picture is finished playing, the top first target virtual object audio-visual picture is hidden, the bottom container is displayed, and the virtual object response audio-visual picture is started to be played.

[0110] So, like Figure 4 As shown, a seamless replacement of the preset virtual object response audio-visual picture and the response video picture generated by the server can be achieved without the user's perception.

[0111] In some other embodiments of the present application, the virtual object response video stream data includes data of the virtual object response audio-visual picture connected with the first target virtual object audio-visual picture. Based on this, step 140 may specifically include step 1404.

[0112] Step 1404, when the picture similarity between the first target virtual object audio-visual picture and the second target virtual object audio-visual picture currently playing in the first display area is greater than or equal to a preset threshold, after the playback of the second target virtual object audio-visual picture is finished, the virtual object response audio-visual picture is played in the first display area.

[0113] The second target virtual object audio-visual picture is the first target virtual object audio-visual picture or a preset virtual object audio-visual picture displayed after the first target virtual object audio-visual picture.

[0114] For example, still taking the above client program as an example, the client program initializes the virtual object container at the bottom of the animation container and enters the video stream waiting to be played state. The client program detects the playback progress of the first virtual object audio-visual picture. When the first target virtual object audio-visual picture is finished playing, the top client animation is hidden, the bottom container is displayed, and the virtual object response audio-visual picture is started to play. Figure 4 As shown, a seamless replacement of the preset virtual object response audio-visual picture and the response video picture generated by the server can be achieved without the user's perception.

[0115] In some other embodiments of the present application, the virtual object response video stream data includes data of a virtual object response audio-visual picture connected with the first target virtual object audio-visual picture; the first target virtual object audio-visual picture is a picture played after the third target virtual object audio-visual picture among at least two first virtual object audio-visual pictures, and the playback interval between the third target virtual object audio-visual picture and the first target virtual object audio-visual picture is a preset interval; the third target virtual object audio-visual picture is the preset virtual object audio-visual picture currently playing in the first display area when the video stream request is generated, and the preset interval is determined by the empirical duration of the virtual object response video stream data fed back by the server. Based on this, step 140 may specifically include steps 1405 and 1406.

[0116] Step 1405: splice the virtual object response audio-visual picture after the third target virtual object audio-visual picture.

[0117] Exemplarily, the posture, position, portrait size and other information of the virtual object in the third target virtual object audio-visual picture are extracted and converted into json data, and the posture, position, portrait size and other information of the virtual object in the virtual object response audio-visual picture are processed by openCV, and the information of the virtual object in the virtual object response audio-visual picture is mapped to the information of the virtual object in the third target virtual object audio-visual picture, ensuring that the content of the virtual object in the virtual object response audio-visual picture is consistent with that of the virtual object in the third target virtual object audio-visual picture and can be smoothly transitioned, and the natural connection is ensured by means of video editing, cross-fading and the like, such as Figure 4 As shown, the icon frames and the picture frames are connected to improve the smoothness of playback.

[0118] Here, it should be noted that Figure 4 As shown, the embodiment of the present application converts the switching of video and video between the preset virtual object audio-visual screen and the virtual object response audio-visual screen into the switching of image frames and image frames, thereby improving the controllability of the screen switching and reducing the technical complexity.

[0119] Step 1407: after the third target virtual object audio-visual picture is played, the virtual object response audio-visual picture is played in the first display area.

[0120] For example, still taking the above client program as an example, the client program initializes the virtual object container at the bottom of the animation container and enters the video stream waiting to be played state. The client program detects the playback progress of the first virtual object audio-visual picture, and when the first target virtual object audio-visual picture is finished playing, the top client animation is hidden, the bottom container is displayed, and the virtual object response audio-visual picture is started to be played.

[0121] So, like Figure 4 As shown, a seamless replacement of the preset virtual object response audio-visual picture and the response video picture generated by the server can be achieved without the user's perception.

[0122] It should be noted that the embodiment of the present application provides a method for determining the preset interval, which is specifically shown as follows.

[0123] Based on the experience value, the duration of the virtual object response video generated by the server is determined; based on the duration and the preset virtual object audio-visual picture currently being played corresponding to when the video stream request is sent to the server, the first target preset virtual object video picture expected to be displayed when the virtual object response video is received is determined.

[0124] In addition, after step 140, the playback method may further include step 180, in which the first virtual object audio-visual picture is played in the first display area when the virtual object responds that the audio-visual picture playback is finished.

[0125] Exemplarily, when the virtual object responds to the audio-visual screen playback, it switches back to the default virtual object audio-visual screen of the client, destroys the virtual object video stream data container, and disconnects the Websocket connection. The system immediately switches back to the first virtual object audio-visual screen played by the client, such as the animation of the default animation program segment 1, and thus completes a conversation cycle. The system re-enters the stage of turning on the recording function of the electronic device and converting the recorded audio into user voice information through the audio signal speech recognition conversion algorithm, and starts the next round of conversation, and so on.

[0126] In this way, there is no need for the server to continuously render the virtual object screen, and intermittent sending of video stream requests to the server is implemented, thereby reducing the time for the server to draw the virtual object video stream and reducing the load on the server; and, when the virtual object response video stream data sent by the server is received, the first virtual object audio-visual screen played in the virtual object display area is switched to the virtual object response audio-visual screen rendered by the virtual object response video stream data. In this way, there is no need for the client to continuously identify the speaking state of the virtual object, thereby reducing the high performance overhead of the client to identify that the virtual object has finished speaking, and reducing the frequency of data transmission between the server and the client, which can reduce the transmission cost of network resources.

[0127] In addition, the embodiment of the present application also provides a process for generating virtual object response video stream data based on video stream request, which can be specifically combined with Figure 5 Describe the steps performed by the server.

[0128] like Figure 5 As shown, the server of the embodiment of the present application can execute steps 510 to 570, as shown below.

[0129] Step 510: The server receives a video stream request, which may carry at least one of the following: a virtual object response text corresponding to the user voice information, display data of the virtual object, and a play order of the first target virtual object audio-visual picture in at least two first virtual object audio-visual pictures. The display data may include at least one of the following: display coordinates of the virtual object in the first display area, display size of the virtual object display area in the video call interface, and image identification of the virtual object.

[0130] Step 520, parse the video stream request, and extract at least one of the following from the video stream request: display data, virtual object response text, display data, virtual object response text, and playback order.

[0131] Step 5301, constructing virtual object video stream data based on display data and playback order, the virtual object video data including home page virtual object response video stream data connected with the first target virtual object audio-visual picture matching the image of the virtual object and related to the playback order.

[0132] Exemplarily, openPose is used to extract the posture, position, size and other information of the virtual object in the first target virtual object audio-visual picture and convert it into json data, which is then processed by openCV to generate virtual object video stream data to ensure that the home page virtual object audio-visual picture corresponding to the home page virtual object response video stream data is consistent with the content of the first target virtual object audio-visual picture, thereby achieving a smooth transition.

[0133] Step 5302, based on the virtual object video stream data, the virtual object response text and the virtual object response audio of the virtual object response text, generates data of the main virtual object audio-visual picture connected to the home page virtual object audio-visual picture corresponding to the home page virtual object response video stream data.

[0134] Specifically, the server uses Text To Speech (TTS) technology to convert the virtual object response text into virtual object response audio, and uses Wav2Lip to perform voice-driven portrait to generate data of the main virtual object audio-visual picture connected to the home page virtual object audio-visual picture.

[0135] Step 540, synthesizing the home page virtual object audio-visual picture and the main virtual object audio-visual picture by video trimming to obtain virtual object response video stream data.

[0136] Step 550: Connectivity detection to ensure a smooth transition.

[0137] Exemplarily, by means of video editing, cross-fading, etc., natural connection is ensured, the video frames of the virtual object response video stream data are detected, and adjustments and optimizations are repeatedly made until the smoothness value of the connection between the first target virtual object audio-visual picture and the home page virtual object audio-visual picture is greater than or equal to the preset smoothness, and step 560 can be executed.

[0138] It should be noted that the detection of video frames can convert the video frames into image frames, so that all contents can be converted into image frames, thereby improving controllability and reducing technical complexity.

[0139] The playback method provided in the embodiment of the present application can be applied to interactions such as conversations with three-dimensional (3D) cute pets, video conferencing, and the like.

[0140] Step 560: Send the virtual object response video stream data to the electronic device.

[0141] Therefore, the embodiment of the present application provides a solution that uses a hybrid client to play preset animations and a server to draw video streams, which can effectively reduce the server load, eliminate the need for client decoding, VAD detection or lip shape detection and other technologies, and avoid the high performance overhead of the client.

[0142] The embodiment of the present application breaks down the video call process with a virtual object into the stages of virtual object listening (user speaking), virtual object thinking, and virtual object speaking. In the virtual object listening and thinking stages, since the virtual object only needs to make some default body movements and does not require audio or lip shape, a relatively fixed animation can be used instead. This stage uses animation resources preset in the client and is played by the client. During the virtual object speaking stage, the server is switched to play a video stream that is consistent in size and position with the client and can be connected. After the virtual object finishes speaking (the speech is completed when the video stream is played, and there is no need to use recognition technology), it is switched to the animation preset by the client, and so on, to achieve seamless connection.

[0143] Based on the above content, in order to better illustrate the playback method provided by the embodiment of the present application, the following can be combined with Figure 6 Provide detailed explanation.

[0144] like Figure 6 As shown, in an embodiment of the present application, a virtual object video call system can be established. The interaction of the virtual object video call system and the user's video call system is similar. The difference is that the user's video call system is a video call between real users, while the virtual object call system is a video call between people and virtual objects. The virtual object call system provides functions corresponding to three stages for a dialogue cycle between the user and the virtual object. Among them, a dialogue cycle means that the user speaks once and the virtual object speaks once. The three stages can include the stage where the virtual object listens to the user's speech, that is, the P1 stage, the stage where the virtual object thinks, that is, the P2 stage, and the stage where the virtual object speaks, that is, the P3 stage. The functions corresponding to each stage can be the functions of the virtual object listening stage, the functions of the virtual object thinking stage, and the functions of the virtual object speaking stage.

[0145] Combine the following Figure 6 The playback method provided by this application is described in detail.

[0146] The P1 stage may include step 601 and step 602 .

[0147] Step 601: The user starts speaking, and the electronic device can start the recording function and convert the recorded audio into user voice information through an audio signal speech recognition conversion algorithm (Automatic Speech Recognition, ASR).

[0148] During this process, the first virtual object audio-visual picture can be played in the first display area of ​​the video call interface. Here, the virtual object image of the virtual object in the first virtual object audio-visual picture is used to represent at least one of the following states: the virtual object is in a listening state, and the virtual object is in a thinking state.

[0149] Step 602: the electronic device fills the user voice information and the virtual object identity data of the virtual object into the preset artificial intelligence text assembly template to obtain text assembly content such as prompt query text, and establishes a WebSocket connection with the server.

[0150] The P2 stage may include steps 603 to 612 .

[0151] Step 603, the electronic device obtains the text assembly content based on the aforementioned P1 stage, and sends a response data request to the large model server. The response data request carries the text assembly content, and the response data request is used to request the large model server to stream the text assembly content to obtain streaming keywords related to the text assembly content.

[0152] Step 604: the large model server performs streaming processing on the text assembly content to obtain streaming keywords related to the text assembly content.

[0153] Step 605: The large model server sends at least two streaming keywords to the electronic device.

[0154] Step 606: When the electronic device receives at least two streaming keywords sent by the large model server, the electronic device stores the at least two streaming keywords in a buffer.

[0155] Step 607 : When the electronic device recognizes in the buffer that a target sentence constructed by at least two streaming keywords meets a preset sentence composition condition, the electronic device generates a virtual object response text based on the target sentence.

[0156] Step 608: The electronic device constructs a video stream request based on the virtual object response text and display data.

[0157] The display data includes the display coordinates of the virtual object in the first display area and the display size of the first display area in the video call interface.

[0158] In step 609, the electronic device can establish a web real-time communication connection channel, such as a webrtc connection channel, with the server through a WebSocket connection with the server, and send a video stream request to the server through the webrtc connection channel. The video stream request carries a virtual object response text and display data of the virtual object corresponding to the user's voice information.

[0159] Step 610 : The server constructs virtual object video stream data matching the virtual object image of the virtual object based on the display data in the received video stream request.

[0160] Step 611, the server generates virtual object response video stream data according to the virtual object video stream data, the virtual object response text and the virtual object response audio of the virtual object response text.

[0161] Step 612: The server sends the virtual object response video stream data to the electronic device.

[0162] Therefore, at this stage, there is a certain amount of time spent in the process of the electronic device and the server and the large model server performing network transmission respectively, the large model server generating the virtual object response text, and the server generating the virtual object response video stream data. Therefore, the virtual object image of the virtual object in the first virtual object audio-visual screen of the electronic device at this stage is used to represent at least one of the following states: the virtual object is in a listening state, and the virtual object is in a thinking state.

[0163] The P3 stage may include step 613 .

[0164] Step 613, the electronic device receives the virtual object response video stream data sent by the server through the webrtc connection channel, renders the virtual object response video stream data, obtains the virtual object response audio-visual picture, and switches the first virtual object video picture played in the first display area to the virtual object response audio-visual picture.

[0165] Therefore, in this stage, the virtual object responds according to the user's voice information. The virtual object image of the virtual object in this stage can be a presentation of content related to the user's response text and the virtual object's response audio, including the virtual object's body shape, lip shape, expression, and sound.

[0166] The above three stages constitute a cycle, and the following three stages are repeated in a loop to realize the scenario of a user making a call or video call with a virtual object.

[0167] In this way, in the embodiment of the present application, the three stages of virtual object video playback can be disassembled, and the relatively fixed actions of the virtual object can be replaced by the preset animation of the electronic device, and the video stream can be retained in the speech stage. The mixed switching of the two not only retains all the virtual object's body movements, lip shapes, sounds, facial expressions and other contents, but also minimizes the time from the whole process to the virtual object speech stage to the server playing the virtual object video stream, reducing the server load and reducing the network transmission overhead. In addition, the client in the electronic device does not need to decode the video stream, avoids the use of high-performance overhead technologies such as real-time VAD detection and lip shape recognition to identify that the virtual object has finished speaking, greatly improving the performance of the client, and cleverly mixes the use of animation playback and video streaming playback to reduce the technical complexity. By subdividing by fragments, the animation and video playback are seamlessly connected without a sense of separation or jump. There is no loss of effect, and the performance overhead is greatly reduced.

[0168] It should be noted that the playback method provided in the embodiments of the present application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, wearable devices, etc. In some embodiments of the present application, the playback method provided in the embodiments of the present application is described by taking the electronic device as the execution subject to execute the playback method as an example.

[0169] The playback method provided in the embodiment of the present application can be executed by a playback device. In the embodiment of the present application, a playback device executing the playback method is taken as an example to illustrate the device of the playback method provided in the embodiment of the present application.

[0170] The present application also provides a playback device. Figure 7 Provide detailed explanation.

[0171] Figure 7 A schematic diagram of the structure of a playback device provided in some embodiments of the present application.

[0172] like Figure 7 As shown, the playback device 70 can be applied to an electronic device, and the playback device 70 can specifically include:

[0173] The display module 701 is used to play the audio-visual picture of the first virtual object in the first display area of ​​the video call interface during the process of acquiring the user voice information when the user makes a video call with the virtual object;

[0174] The sending module 702 is used to send a video stream request to the server, the video stream request carries the virtual object response text corresponding to the user voice information and the display data of the virtual object, and the video stream request is used to request the server to construct virtual object video stream data matching the image of the virtual object based on the display data, and generate virtual object response video stream data according to the virtual object video stream data, the virtual object response text, and the virtual object response audio of the virtual object response text;

[0175] The receiving module 703 is used to receive the virtual object response video stream data sent by the server;

[0176] The processing module 704 is used to switch the first virtual object audio-visual picture played in the first display area to a virtual object response audio-visual picture, where the virtual object response audio-visual picture is rendered by the virtual object response video stream data.

[0177] The playback device 70 in the embodiment of the present application is described in detail below, as shown below.

[0178] In some embodiments of the present application, the playback device 70 further includes a determination module for determining, according to the user voice information, a virtual object response text corresponding to the user voice information when the display data includes the display coordinates of the virtual object in the first display area and the display size of the first display area in the video call interface;

[0179] The playback device 70 also includes a construction module for constructing a video stream request based on the virtual object response text, display size and display coordinates.

[0180] In some embodiments of the present application, the determination module is specifically used to fill the user voice information and the virtual object identity data of the virtual object into the preset artificial intelligence text assembly template to obtain the text assembly content;

[0181] The sending module 702 is also used to send a response data request to the large model server, the response data request carries the text assembly content, and the response data request is used to request the large model server to perform streaming processing on the text assembly content to obtain streaming keywords related to the text assembly content;

[0182] The playback device 70 further includes a storage module for storing the at least two streaming keywords in a buffer upon receiving the at least two streaming keywords sent by the large model server;

[0183] The playback device 70 further includes a generation module for generating a virtual object response text based on the target sentence when a target sentence constructed by at least two streaming keywords is identified in the buffer and satisfies a preset sentence composition condition.

[0184] In some embodiments of the present application, the construction module is specifically used to, when the first virtual object audio-visual screen includes at least two first virtual object audio-visual screens, and the at least two first virtual object audio-visual screens are played according to the playback order of the preset virtual object audio-visual screens; the video stream request also carries the playback order of the first target virtual object audio-visual screen in the at least two first virtual object audio-visual screens, and the video stream request is also used to request the server to obtain the display form of the virtual object in the first target virtual object audio-visual screen corresponding to the playback order, and based on the display form of the virtual object in the first target virtual object audio-visual screen, construct a virtual object video stream data that matches the virtual object image of the virtual object, and construct the video stream request based on the playback order, the virtual object response text, the display size and the display coordinates.

[0185] In some embodiments of the present application, the playback device 70 further includes a splicing module for splicing the virtual object response audio-visual picture after the first target virtual object audio-visual picture when the virtual object response video stream data includes data of the virtual object response audio-visual picture connected with the first target virtual object audio-visual picture;

[0186] The playback device 70 further includes a switching module for switching the preset virtual object audio-visual picture currently being played in the first display area to the first target virtual object audio-visual picture;

[0187] The display module 701 is also used to play the virtual object response audio-visual picture when the playback of the first target virtual object audio-visual picture is completed.

[0188] In some embodiments of the present application, the display module 701 is also used to play the virtual object response audio-visual picture in the first display area after the playback of the second target virtual object audio-visual picture is finished, when the virtual object response video stream data includes data of a virtual object response audio-visual picture connected with the first target virtual object audio-visual picture, and the picture similarity between the first target virtual object audio-visual picture and the second target virtual object audio-visual picture currently playing in the first display area is greater than or equal to a preset threshold.

[0189] In some embodiments of the present application, the playback device 70 further includes a splicing module for splicing the virtual object response audio-visual picture after the third target virtual object audio-visual picture when the virtual object response video stream data includes data of a virtual object response audio-visual picture connected with the first target virtual object audio-visual picture; the first target virtual object audio-visual picture is a picture played after the third target virtual object audio-visual picture among at least two first virtual object audio-visual pictures, the playback interval between the third target virtual object audio-visual picture and the first target virtual object audio-visual picture is a preset interval, the third target virtual object audio-visual picture is the preset virtual object audio-visual picture currently played in the first display area when the video stream request is generated, and the preset interval is determined by the empirical duration of the virtual object response video stream data fed back by the server end;

[0190] The display module 701 is further configured to play the virtual object response audio-visual picture in the first display area after the third target virtual object audio-visual picture is played.

[0191] In some embodiments of the present application, the display module 701 is further configured to play the first virtual object audio-visual picture in the first display area when the virtual object responds to the end of the audio-visual picture playback.

[0192] The playback device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0193] The playback device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0194] The playback device provided in the embodiment of the present application can achieve Figures 1 to 6 The various processes implemented in the illustrated playback method embodiment achieve the same technical effect, and will not be described again here to avoid repetition.

[0195] Based on this, the playback device provided in the embodiment of the present application takes into account that the virtual object only needs to make some default body movements in the listening stage and the thinking stage, and does not need audio and lip shape to match it. Therefore, in the process of obtaining the user voice information when the user has a video call with the virtual object, the first virtual object audio-visual picture displayed in the virtual object display area of ​​the video call interface replaces the process of the server continuously rendering the virtual object picture. In this way, there is no need for the server to continuously render the virtual object picture, and intermittently send video stream requests to the server, thereby reducing the time for the server to draw the virtual object video stream and reducing the load on the server; and, when receiving the virtual object response video stream data sent by the server, the first virtual object audio-visual picture played in the virtual object display area is switched to the virtual object response audio-visual picture rendered by the virtual object response video stream data. In this way, there is no need for the client to continuously identify the speaking state of the virtual object, which reduces the high performance overhead of the client to identify that the virtual object has finished speaking, and reduces the frequency of data transmission between the server and the client, which can reduce the transmission cost of network resources.

[0196] Optional, such as Figure 8 As shown, an embodiment of the present application also provides an electronic device 80, including a processor 801 and a memory 802, wherein the memory 802 stores programs or instructions that can be executed on the processor 801, and when the program or instructions are executed by the processor 801, the various steps of the above-mentioned playback method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0197] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0198] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0199] The electronic device 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, a processor 910 and other components.

[0200] Those skilled in the art will appreciate that the electronic device 900 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 910 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system.

[0201] Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0202] Among them, in the embodiment of the present application, the display unit 906 is used to play the first virtual object audio-visual picture in the first display area of ​​the video call interface during the process of obtaining the user voice information when the user makes a video call with the virtual object. The network module 902 is used to send a video stream request to the server, the video stream request carries the virtual object response text corresponding to the user voice information and the display data of the virtual object, and the video stream request is used to request the server to construct virtual object video stream data matching the image of the virtual object based on the display data, and generate virtual object response video stream data according to the virtual object video stream data, the virtual object response text, and the virtual object response audio of the virtual object response text. The network module 902 can also be used to receive the virtual object response video stream data sent by the server. The processor 910 is used to switch the first virtual object audio-visual picture played in the first display area to the virtual object response audio-visual picture, and the virtual object response audio-visual picture is rendered by the virtual object response video stream data.

[0203] The electronic device 900 is described in detail below, as shown below.

[0204] In some embodiments of the present application, the processor 910 is configured to determine, according to the user voice information, a virtual object response text corresponding to the user voice information when the display data includes the display coordinates of the virtual object in the first display area and the display size of the first display area in the video call interface;

[0205] Construct a video stream request based on the virtual object response text, display size, and display coordinates.

[0206] In some embodiments of the present application, the processor 910 is used to fill the user voice information and the virtual object identity data of the virtual object into the preset artificial intelligence text assembly template to obtain the text assembly content;

[0207] The sending module 702 is also used to send a response data request to the large model server, the response data request carries the text assembly content, and the response data request is used to request the large model server to perform streaming processing on the text assembly content to obtain streaming keywords related to the text assembly content;

[0208] The memory 909 is used to store the at least two streaming keywords into a buffer when receiving the at least two streaming keywords sent by the large model server;

[0209] The processor 910 is configured to generate a virtual object response text based on the target sentence when it is identified in the buffer that the target sentence constructed by at least two streaming keywords meets the preset sentence composition condition.

[0210] In some embodiments of the present application, the processor 910, when the first virtual object audio-visual screen includes at least two first virtual object audio-visual screens, and the at least two first virtual object audio-visual screens are played in a preset playback order of the virtual object audio-visual screen; the video stream request also carries the playback order of the first target virtual object audio-visual screen in the at least two first virtual object audio-visual screens, and the video stream request is also used to request the server to obtain the display form of the virtual object in the first target virtual object audio-visual screen corresponding to the playback order, and based on the display form of the virtual object in the first target virtual object audio-visual screen, construct a virtual object video stream data that matches the virtual object image of the virtual object, and construct a video stream request based on the playback order, the virtual object response text, the display size and the display coordinates.

[0211] In some embodiments of the present application, the processor 910 is configured to, when the virtual object response video stream data includes data of a virtual object response audio-visual picture connected to the first target virtual object audio-visual picture, splice the virtual object response audio-visual picture after the first target virtual object audio-visual picture;

[0212] Switching the preset virtual object audio-visual picture currently being played in the first display area to the first target virtual object audio-visual picture;

[0213] The display unit 906 is further configured to play the virtual object response audio-visual picture when the playback of the first target virtual object audio-visual picture is completed.

[0214] In some embodiments of the present application, the display unit 906 is also used to play the virtual object response audio-visual picture in the first display area after the playback of the second target virtual object audio-visual picture is finished, when the virtual object response video stream data includes data of a virtual object response audio-visual picture connected with the first target virtual object audio-visual picture, and the picture similarity between the first target virtual object audio-visual picture and the second target virtual object audio-visual picture currently playing in the first display area is greater than or equal to a preset threshold.

[0215] In some embodiments of the present application, the processor 910 is used to splice the virtual object response audio-visual picture after the third target virtual object audio-visual picture when the virtual object response video stream data includes data of a virtual object response audio-visual picture connected with the first target virtual object audio-visual picture; the first target virtual object audio-visual picture is a picture played after the third target virtual object audio-visual picture among at least two first virtual object audio-visual pictures, the playback interval between the third target virtual object audio-visual picture and the first target virtual object audio-visual picture is a preset interval, the third target virtual object audio-visual picture is the preset virtual object audio-visual picture currently played in the first display area when the video stream request is generated, and the preset interval is determined by the empirical duration of the virtual object response video stream data fed back by the server;

[0216] The display unit 906 is further configured to play the virtual object response audio-visual picture in the first display area after the playback of the third target virtual object audio-visual picture is finished.

[0217] In some embodiments of the present application, the display unit 906 is further configured to play the first virtual object audio-visual picture in the first display area when the virtual object responds to the end of the audio-visual picture playback.

[0218] It should be understood that the input unit 904 may include a graphics processing unit (GPU) 9041 and a microphone 9042, and the graphics processor 9041 processes the image data of a static image or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 906 may include a display panel, and the display panel may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes a touch panel 9071 and at least one of other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch display. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as a volume display button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0219] The memory 909 can be used to store software programs and various data. The memory 909 can mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area can store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 909 can include a volatile memory or a non-volatile memory, or the memory 909 can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0220] The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless display signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 910.

[0221] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned playback method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0222] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0223] In addition, an embodiment of the present application further provides a chip, which includes a processor and a display interface, the display interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned playback method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0224] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0225] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned playback method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0226] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0227] In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0228] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0229] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A playback method, characterized in that: include: In the process of acquiring user voice information when the user is making a video call with the virtual object, playing the first virtual object audio-visual picture in the first display area of ​​the video call interface; Sending a video stream request to a server, the video stream request carrying a virtual object response text corresponding to the user voice information and display data of the virtual object, the video stream request being used to request the server to construct virtual object video stream data matching the image of the virtual object based on the display data, and generating virtual object response video stream data according to the virtual object video stream data, the virtual object response text, and the virtual object response audio of the virtual object response text; Receiving the virtual object response video stream data sent by the server; The first virtual object audio-visual picture played in the first display area is switched to a virtual object response audio-visual picture, where the virtual object response audio-visual picture is rendered by the virtual object response video stream data.

2. The method according to claim 1, characterized in that The display data includes the display coordinates of the virtual object in the first display area and the display size of the first display area in the video call interface; Before sending the video stream request to the server, the method further includes: Determining, according to the user voice information, a virtual object response text corresponding to the user voice information; The video stream request is constructed based on the virtual object response text, the display size and the display coordinates.

3. The method according to claim 2, characterized in that The first virtual object audio-visual picture includes at least two first virtual object audio-visual pictures, and the at least two first virtual object audio-visual pictures are played according to a preset playback order of the virtual object audio-visual pictures; the video stream request also carries the playback order of the first target virtual object audio-visual picture in the at least two first virtual object audio-visual pictures, and the video stream request is also used to request the server to obtain the display form of the virtual object in the first target virtual object audio-visual picture corresponding to the playback order, and construct the virtual object video stream data matching the virtual object image of the virtual object based on the display form of the virtual object in the first target virtual object audio-visual picture; The constructing the video stream request based on the virtual object response text, the display size and the display coordinates includes: The video stream request is constructed based on the playback order, the virtual object response text, the display size and the display coordinates.

4. The method according to claim 3, characterized in that The virtual object response video stream data includes data of a virtual object response audio-visual picture connected with the first target virtual object audio-visual picture; The step of switching the first virtual object audio-visual picture played in the first display area to a virtual object response audio-visual picture includes: splicing the virtual object response audio-visual picture after the first target virtual object audio-visual picture; Switching the preset virtual object audio-visual picture currently being played in the first display area to the first target virtual object audio-visual picture; When the playback of the first target virtual object audio-visual image is completed, the virtual object response audio-visual image is played.

5. The method according to claim 3, characterized in that: The virtual object response video stream data includes data of a virtual object response audio-visual picture connected with the first target virtual object audio-visual picture; The step of switching the first virtual object audio-visual picture played in the first display area to a virtual object response audio-visual picture includes: When the picture similarity between the first target virtual object audio-visual picture and the second target virtual object audio-visual picture currently playing in the first display area is greater than or equal to a preset threshold, after the playback of the second target virtual object audio-visual picture is finished, the virtual object response audio-visual picture is played in the first display area.

6. A playback device, characterized in that: include: A display module, configured to play an audio-visual picture of a first virtual object in a first display area of ​​a video call interface during a process of acquiring user voice information when the user makes a video call with the virtual object; a sending module, configured to send a video stream request to a server, wherein the video stream request carries a virtual object response text corresponding to the user voice information and display data of the virtual object, wherein the video stream request is used to request the server to construct virtual object video stream data matching the image of the virtual object based on the display data, and to generate virtual object response video stream data according to the virtual object video stream data, the virtual object response text, and the virtual object response audio of the virtual object response text; A receiving module, used for receiving the virtual object response video stream data sent by the server; The processing module is used to switch the first virtual object audio-visual picture played in the first display area to a virtual object response audio-visual picture, where the virtual object response audio-visual picture is rendered by the virtual object response video stream data.

7. The device according to claim 6, characterized in that The playback device further includes a determination module for determining, according to the user voice information, a virtual object response text corresponding to the user voice information when the display data includes the display coordinates of the virtual object in the first display area and the display size of the first display area in the video call interface; The playback device also includes a construction module for constructing the video stream request based on the virtual object response text, the display size and the display coordinates.

8. The device according to claim 7, characterized in that The construction module is specifically used to, when the first virtual object audio-visual screen includes at least two first virtual object audio-visual screens, and the at least two first virtual object audio-visual screens are played according to the preset playback order of the virtual object audio-visual screen; the video stream request also carries the playback order of the first target virtual object audio-visual screen in the at least two first virtual object audio-visual screens, and the video stream request is also used to request the server to obtain the display form of the virtual object in the first target virtual object audio-visual screen corresponding to the playback order, and based on the display form of the virtual object in the first target virtual object audio-visual screen, construct the virtual object video stream data that matches the virtual object image of the virtual object, and construct the video stream request based on the playback order, the virtual object response text, the display size and the display coordinates.

9. The device according to claim 7, characterized in that The playback device further includes a splicing module for splicing the virtual object response audio-visual picture after the first target virtual object audio-visual picture when the virtual object response video stream data includes data of a virtual object response audio-visual picture connected with the first target virtual object audio-visual picture; The playback device further includes a switching module for switching the preset virtual object audio-visual picture currently being played in the first display area to the first target virtual object audio-visual picture; The display module is further configured to play the virtual object response audio-visual picture when the playing of the first target virtual object audio-visual picture is finished.

10. The device according to claim 7, characterized in that The display module is also used to play the virtual object response audio-visual picture in the first display area after the second target virtual object audio-visual picture is played, if the virtual object response video stream data includes data of a virtual object response audio-visual picture connected with the first target virtual object audio-visual picture, and the picture similarity between the first target virtual object audio-visual picture and the second target virtual object audio-visual picture currently playing in the first display area is greater than or equal to a preset threshold.

Images