Playback method and device

Abstract

This application discloses a playback method and apparatus, belonging to the field of artificial intelligence technology. The method includes: during the process of acquiring user voice information when a user is making a video call with a virtual object, playing the audio-visual image of a first virtual object in a first display area of ​​the video call interface; sending a video stream request to a server, the video stream request carrying virtual object response text corresponding to the user voice information and display data of the virtual object, for requesting the server to construct virtual object video stream data matching the image of the virtual object based on the display data, and generating virtual object response video stream data based on the virtual object video stream data, the virtual object response text, and the virtual object response audio of the virtual object response text; receiving the virtual object response video stream data sent by the server, and switching the first virtual object played in the first display area to the virtual object response audio-visual image rendered by the virtual object response video stream data.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a playback method and apparatus. Background Technology

[0002] With the development of artificial intelligence technology, electronic devices can support users to make video calls with virtual objects created by artificial intelligence (AI) technology. During the video call, the virtual object is required to act like a real user, answering the user's questions, and the virtual object's visual image is required to be visible throughout the video call. For example, the virtual object should have corresponding body movements, facial expressions, voice, and lip movements.

[0003] In related technologies, to ensure the visual appearance of virtual objects is visible throughout a video call, the server needs to continuously render the video stream data of the virtual objects during the call. The client then renders and displays the visual appearance of the virtual objects corresponding to the video stream data. Furthermore, during the question-and-answer interaction between the virtual objects and the user, the client needs to continuously recognize the speaking state of the virtual objects and, upon detecting the completion of speaking, trigger the acquisition of the user's voice information and send it to the server so that the server can render the response video stream data of the virtual objects. However, continuously rendering the video stream data of the virtual objects increases the server load, and continuously recognizing the speaking state of the virtual objects increases the performance overhead on the client side. Summary of the Invention

[0004] The purpose of this application is to provide a playback method, apparatus, electronic device, and storage medium that can reduce server load and client performance overhead.

[0005] In a first aspect, embodiments of this application provide a playback method, including:

[0006] During the process of acquiring user voice information during a video call between the user and the virtual object, the audio-visual image of the first virtual object is played in the first display area of ​​the video call interface.

[0007] Send a video stream request to the server. The video stream request carries the virtual object response text corresponding to the user's voice information and the display data of the virtual object. The video stream request is used to request the server to construct virtual object video stream data that matches the image of the virtual object based on the display data, and generate virtual object response video stream data based on the virtual object video stream data, the virtual object response text, and the virtual object response audio of the virtual object response text.

[0008] Receive virtual object response video stream data sent by the server;

[0009] The first virtual object audio-visual image played in the first display area is switched to the virtual object response audio-visual image, which is rendered from the virtual object response video stream data.

[0010] Secondly, embodiments of this application provide a playback device, including:

[0011] The display module is used to play the audio-visual image of the first virtual object in the first display area of ​​the video call interface during the process of acquiring user voice information when the user is making a video call with the virtual object.

[0012] The sending module is used to send a video stream request to the server. The video stream request carries the virtual object response text corresponding to the user's voice information and the display data of the virtual object. The video stream request is used to request the server to construct virtual object video stream data that matches the image of the virtual object based on the display data, and generate virtual object response video stream data based on the virtual object video stream data, the virtual object response text, and the virtual object response audio of the virtual object response text.

[0013] The receiving module is used to receive virtual object response video stream data sent by the server;

[0014] The processing module is used to switch the first virtual object audio-visual image played in the first display area to the virtual object response audio-visual image, which is rendered from the virtual object response video stream data.

[0015] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the playback method as described in the first aspect.

[0016] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, the steps of the playback method as shown in the first aspect are implemented.

[0017] Fifthly, embodiments of this application provide a chip, which includes a processor and a display interface, the display interface and the processor being coupled together, the processor being used to run programs or instructions to implement the steps of the playback method as shown in the first aspect.

[0018] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the steps of the playback method as described in the first aspect.

[0019] In this embodiment, considering that the virtual object only needs to perform some default body movements during the listening and thinking phases without requiring audio and lip movements, the process of acquiring user voice information during a video call between the user and the virtual object replaces the continuous rendering of the virtual object's image by displaying the first virtual object's audiovisual image in the virtual object display area of ​​the video call interface. This eliminates the need for the server to continuously render the virtual object's image, allowing for intermittent sending of video stream requests to the server, reducing the time the server spends rendering the virtual object's video stream, and lowering the server's load. Furthermore, upon receiving the virtual object's response video stream data sent by the server, the first virtual object's audiovisual image played in the virtual object display area is switched to the virtual object's response audiovisual image rendered from the virtual object's response video stream data. This eliminates the need for the client to continuously recognize the virtual object's speaking state, reducing the high performance overhead of client recognition of the virtual object's speech completion, and also reduces the data transmission frequency between the server and client, thereby lowering the transmission cost of network resources. Attached Figure Description

[0020] Figure 1 Flowcharts of playback methods provided for some embodiments of this application;

[0021] Figure 2 A schematic diagram of the interface for a playback method provided in some embodiments of this application;

[0022] Figure 3 One of the schematic diagrams of the interface for a playback method provided in some embodiments of this application;

[0023] Figure 4 A second schematic diagram of the interface for a playback method provided in some embodiments of this application;

[0024] Figure 5 A schematic flowchart of a server-based playback method provided for some embodiments of this application;

[0025] Figure 6 A flowchart illustrating the playback method provided in some embodiments of this application;

[0026] Figure 7 Schematic diagrams of the playback device provided for some embodiments of this application;

[0027] Figure 8 Schematic diagrams of the structure of electronic devices provided for some embodiments of this application;

[0028] Figure 9 A schematic diagram of the hardware structure of an electronic device provided for some embodiments of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0030] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0031] In related technologies, to ensure the visual appearance of virtual objects remains visible throughout a video call, the server needs to continuously render the video stream data of the virtual objects during the call. Therefore, the continuous transmission of the virtual object's video stream data between the client and server relies heavily on the transmission network. If the network quality is low, it increases transmission latency, resulting in choppy display of the virtual objects on the client and reducing the quality and efficiency of user interaction with the virtual objects.

[0032] To address the problems in related technologies, embodiments of this application provide a playback method, apparatus, and electronic device. The following description, in conjunction with the appendix... Figures 1 to 9 The playback method provided in this application will be described in detail through specific embodiments and application scenarios.

[0033] First, combined Figure 1 A playback method provided in the embodiments of this application will be described in detail.

[0034] Figure 1 A flowchart illustrating a playback method provided for some embodiments of this application.

[0035] like Figure 1 As shown, the playback method provided in this application embodiment can be applied to electronic devices. Based on this, the playback method may include steps 110 and 130, as detailed below.

[0036] Step 110: During the process of obtaining user voice information during a video call between the user and the virtual object, the audiovisual image of the first virtual object is played in the first display area of ​​the video call interface.

[0037] Step 120: Send a video stream request to the server. The video stream request carries the virtual object response text corresponding to the user's voice information and the display data of the virtual object. The video stream request is used to request the server to construct virtual object video stream data that matches the image of the virtual object based on the display data, and generate virtual object response video stream data based on the virtual object video stream data, the virtual object response text, and the virtual object response audio of the virtual object response text.

[0038] Step 130: Receive the virtual object response video stream data sent by the server.

[0039] Step 140: Upon receiving the virtual object response video stream data sent by the server, the first virtual object audio-visual image displayed in the virtual object display area is switched to the virtual object response audio-visual image, which is rendered from the virtual object response video stream data.

[0040] Therefore, in this embodiment, considering that the virtual object only needs to perform some default body movements during the listening and thinking phases without the need for audio and lip movements, the process of obtaining user voice information during a video call between the user and the virtual object replaces the continuous rendering of the virtual object's image by displaying the first virtual object's audiovisual image in the virtual object display area of ​​the video call interface. This eliminates the need for the server to continuously render the virtual object's image, allowing for intermittent sending of video stream requests to the server, reducing the time the server spends rendering the virtual object's video stream, and lowering the server's load. Furthermore, upon receiving the virtual object's response video stream data sent by the server, the first virtual object's audiovisual image played in the virtual object display area is switched to the virtual object's response audiovisual image rendered from the virtual object's response video stream data. This eliminates the need for the client to continuously recognize the virtual object's speaking state, reducing the high performance overhead of the client recognizing when the virtual object has finished speaking, and also reduces the data transmission frequency between the server and client, thus lowering the transmission cost of network resources.

[0041] The steps described above are explained in detail below.

[0042] First, regarding step 110, in some embodiments of this application, the playback method may further include steps 1501 to 1503 before step 110.

[0043] Step 1501: Receive the user's first input to initiate a video call with the virtual object.

[0044] Step 1502: In response to the first input, play the video call interface, which includes a first display area.

[0045] For example, such as Figure 2 As shown, a video call interface can be displayed. The video call interface includes a first display area 20, which is used to display the audio-visual screen of a preset virtual object, such as the audio-visual screen of a first virtual object, a second virtual object, or a third virtual object. The first display area 20 is also used to display the audio-visual screen of a virtual object response.

[0046] Step 1503: Play the audio-visual image of the third virtual object in the first display area and record the audio.

[0047] The third virtual object audiovisual screen may include at least two third virtual object audiovisual screens. At this time, at least two third virtual object audiovisual screens may be played in a preset virtual object audiovisual screen playback order. The third virtual object audiovisual screen includes a screen for representing at least one of the following: a screen in which the virtual object is in a listening state, and a screen in which the virtual object is in a thinking state.

[0048] Based on this, step 110 may specifically include:

[0049] When the voice activation detection algorithm detects that the audio contains voice signals, the user's voice information during the video call between the user and the virtual object is obtained, and the audio-visual image of the third virtual object played in the first display area is switched to the audio-visual image of the first virtual object.

[0050] For example, a Voice Activity Detection (VAD) algorithm can be used to detect whether the recorded audio contains a speech signal. If the recorded audio contains a speech signal, the third virtual object audiovisual image playing in the first display area is switched to the first virtual object audiovisual image. Here, the first virtual object audiovisual image includes images representing at least one of the following: an image of the virtual object listening, or an image of the virtual object thinking.

[0051] In this way, users can intuitively view the status of the virtual object through the virtual object images at different stages in the first and third virtual object video frames, thereby indicating to the user whether the virtual object is listening to the user's questions or waiting for answers during the current video call, thus improving the interaction efficiency between the user and the virtual object.

[0052] Here, before playing the audiovisual image of the first virtual object in the first display area of ​​the video call interface, the playback method also includes the process of obtaining user voice information. Based on this, before step 110, the playback method may also include steps 1601 and 1602.

[0053] Step 1601: Receive the user's voice audio.

[0054] Step 1602: The user's voice audio is converted using an audio signal speech recognition and conversion algorithm to obtain the user's voice information.

[0055] Here, this application embodiment provides a method for obtaining user voice information based on the user's video image in the video call interface for some special groups, such as deaf and mute users, users who are unable to speak due to illness, the elderly, etc. That is, before step 110, the playback method may also include steps 1603 and 1606.

[0056] Step 1603: Receive the user's voice audio and capture the user's video image from the video call interface.

[0057] Step 1604: The user's voice audio is converted and processed using an audio signal speech recognition and conversion algorithm to obtain the first text; and if the video image includes the user's sign language gestures, a second text representing the meaning of the sign language gestures is generated based on the sign language gestures in the video image.

[0058] Step 1605: Generate user voice information based on the first text and the second text.

[0059] In this way, user voice information can be accurately obtained for certain special groups, expanding the applicable scenarios for users to make video calls with virtual objects.

[0060] Next, regarding step 120, in this embodiment of the application, video stream data can be sent to the server through a WebSocket connection between the client and the server.

[0061] Based on this, in some embodiments of this application, the display data includes the display coordinates of the virtual object in the first display area and the display size of the virtual object display area in the video call interface, wherein the display coordinates can specifically be the coordinates of the center point of the virtual object relative to the coordinate system of the virtual object display area. Based on this, before step 120, the playback method may further include steps 1201 and 1202.

[0062] Step 1201: Determine the virtual object response text corresponding to the user's voice information based on the user's voice information.

[0063] Step 1202: Construct a video stream request based on the virtual object response text, display size, and display coordinates.

[0064] It should be noted that if there are at least two virtual objects, such as virtual object 1 and virtual object 2, the display data can also include the image identifier of the virtual object, such as 1 or 2. In this way, the server can select the virtual object video stream data corresponding to the image identifier from at least two images based on the video stream request carrying the image identifier. This ensures that the virtual object video stream data is consistent with the image of the virtual object on the electronic device. Furthermore, based on the display size and display coordinates, the virtual object video stream data can be made consistent with the size of the virtual object on the electronic device, thereby avoiding frame skipping and achieving seamless transition.

[0065] Furthermore, step 1201 above may specifically include steps 12011 to 12014.

[0066] Step 12011: Populate the user's voice information and the virtual object's identity data into a preset AI text assembly template to obtain the assembled text content. The virtual object's identity data may include basic persona and video call task.

[0067] For example, after obtaining the user's voice information, the electronic device can play a preset virtual object's audiovisual content in the same manner as described above. Simultaneously, using an artificial intelligence tool such as prompt, the user's voice information and the virtual object's identity data are populated into a preset AI text assembly template. Here, the basic persona of the virtual object, such as {name}, the video call task, such as {companyName}, and the user's voice information, i.e., the user's question, such as {question}, are populated into the preset AI text assembly template to obtain the assembled text content, as shown in the example below:

[0068] You need to perform a role-playing task, assuming the role of a human female named {name}. Here are the specific settings for {name}: Your basic information: You are a 27-year-old woman, 165 cm tall, weighing 50 kg; calm, meticulous, and understanding; you currently work at {companyName} as a project manager. You will now begin chatting with the user as {name}. You cannot identify yourself as artificial intelligence; you need to answer the user's questions as {name}. You can use somewhat subtle metaphors or expressions to answer questions that are outside your established persona. Now answer the user's question "{question}". Step 12012: Send a response data request to the large model server. The response data request carries the assembled text content and is used to request the large model server to perform streaming processing on the assembled text content to obtain streaming keywords related to the assembled text content.

[0069] It's important to clarify here that streaming processing of text assembly content by large model servers means that the model doesn't output all the results at once, but rather generates the output gradually and continuously. This is similar to the human thought process when engaging in oral communication or writing—thinking and expressing simultaneously. Streaming processing can include streaming output, a technique that allows the model to generate text gradually and continuously in a stream, rather than outputting all the text at once.

[0070] Step 12013: Upon receiving at least two streaming keywords from the large model server, store at least two streaming keywords in the buffer.

[0071] For example, the text assembly content, i.e. the prompt query text, requests the virtual object response text, i.e. the video stream-driven text, from the large model server. Here, when the streaming keywords returned by the large model stream are received, they can be converted into sentences through the creation buffer created by the electronic device and stored in the buffer.

[0072] Step 12014: If at least two target sentences constructed from streaming keywords are identified in the buffer and meet the preset sentence composition conditions, generate virtual object response text based on the target sentences.

[0073] For example, the buffer is checked for the presence of a complete sentence by punctuation marks. When a complete sentence is detected, it is output, which is the target sentence. The target sentence can be processed to obtain the virtual object response text.

[0074] Furthermore, step 1202 above may specifically include:

[0075] According to the preset data structure, the virtual object's response text, display size, and display coordinates are packaged and processed to obtain the video stream request.

[0076] The display size may include the width and height of the virtual object display area in the video call interface.

[0077] For example, such as Figure 3 As shown, according to the preset data structure, the width W, height H, display coordinates (X, Y) of the virtual object display area in the video call interface, and the virtual object's response text are packaged and processed to obtain the video stream request. Figure 3 The data in the example, with a preset data structure, is as follows:

[0078] Width: w,

[0079] Height: h

[0080] Display coordinates: (x, y)

[0081] Virtual object response text: text}.

[0082] In this way, based on the display size and display coordinates, the video stream data of the virtual object can be kept consistent with the size of the virtual object on the electronic device, thereby avoiding frame skipping and achieving seamless connection.

[0083] In other embodiments of this application, the video stream request may also carry the playback order of the virtual object's audiovisual images in order to obtain more accurate virtual object video stream data.

[0084] Based on this, the first virtual object audiovisual screen in this embodiment includes at least two first virtual object audiovisual screens, which are played in a preset playback order; the video stream request also carries the playback order of the first target virtual object audiovisual screens in the at least two first virtual object audiovisual screens; the video stream request is also used to request the server to obtain the display form of the virtual object in the first target virtual object audiovisual screen corresponding to the playback order, and based on the display form of the virtual object in the first target virtual object audiovisual screen, construct virtual object video stream data matching the virtual object image of the virtual object. Based on this, step 1202 may specifically include:

[0085] A video stream request is constructed based on the playback order, the virtual object's response text, the display size, and the display coordinates.

[0086] For example, such as Figure 3 As shown, the preset virtual object video frame built into the electronic device (such as the first virtual object video frame) can be either an animation resource or a video. To ensure that the virtual object image in the preset virtual object video frame is consistent with the virtual object image in the virtual object video stream data drawn by the server, the virtual object displayed in the preset virtual object video frame is in a default listening posture or a default thinking posture. The client program in the electronic device can initialize the virtual object container, setting the width W, height H, and the coordinates (X, Y) of the virtual object center relative to the origin (0, 0).

[0087] Therefore, if the first virtual object video frame is implemented by playing a sequence of image frames, it can be played in a loop. In this case, the first virtual object video frame can include at least two first virtual object video frames, that is, the animation resource slices are segment 1, segment 2, segment 3, ..., and each segment has an equal and short duration, which can meet the condition for loop playback. That is, the last segment can be seamlessly connected with segment 1 to achieve a seamless connection between the beginning and the end and avoid frame skipping.

[0088] Furthermore, step 1202 above may specifically include:

[0089] According to the preset data structure, the playback order, virtual object response text, display size, and display coordinates are packaged and processed to obtain the video stream request.

[0090] For example, according to a preset data structure, the playback order n, width W, height H, display coordinates (X, Y), and virtual object response text are packaged and processed. An example of the preset data structure is as follows:

[0091] Width: w

[0092] Height: h

[0093] Coordinates: (x, y)

[0094] Playback order: n

[0095] Driven text: text}.

[0096] Therefore, based on the playback order, display size, and display coordinates, the size of the virtual object video stream data can be kept consistent with that of the virtual object on the electronic device. At the same time, based on the playback order, the virtual object response video stream data can be seamlessly connected with the preset virtual object video stream, avoiding frame skipping and improving the display effect of the virtual object video stream when making video calls with the virtual object.

[0097] Then, in step 140, the virtual object response video stream data sent by the server can be received through the web real-time communication connection channel. Based on this, before step 140, the playback method may also include steps 1701 to 1703.

[0098] Step 1701: Based on the WebSocket connection with the server, obtain the communication address for real-time web communication corresponding to the video stream request. The WebSocket connection is used to send the video stream request to the server.

[0099] Step 1702: Establish a real-time web page communication connection channel with the server based on the communication address.

[0100] Step 1703: Receive the virtual object response video stream data sent by the server through the real-time web communication connection channel.

[0101] For example, an electronic device can establish a WebSocket connection with a server to obtain the communication address of the Web Real-Time Communications (WebRTC) connection channel for the corresponding video stream; based on the communication address, it can establish a WebRTC connection and receive virtual object response video stream data sent by the server through WebRTC.

[0102] It should be noted that in this embodiment of the application, after playing the virtual object response audio-visual screen, the Websocket connection can be disconnected and the preset first virtual object audio-visual screen in the electronic device can be switched back, such as the animation of animation program segment 1, or a video segment, thus completing a dialogue cycle.

[0103] In this way, the server does not need to continuously render the virtual object's screen. By pre-setting the virtual object's audiovisual screen, the server can be intermittently requested to respond to the virtual object's video stream request.

[0104] Based on this, the embodiments of this application provide the following method for playing virtual object response audio-visual images, as detailed below.

[0105] In some embodiments of this application, the virtual object response video stream data includes data of the virtual object response audio-visual image connected with the audio-visual image of the first target virtual object. Based on this, step 140 may specifically include steps 1401 to 1403.

[0106] Step 1401: Attach the virtual object response audiovisual screen after the first target virtual object audiovisual screen.

[0107] Step 1402: Switch the currently playing preset virtual object audiovisual image in the first display area to the first target virtual object audiovisual image.

[0108] Step 1403: After the first target virtual object's audiovisual screen finishes playing, play the virtual object's response audiovisual screen.

[0109] For example, the client program initializes a virtual object container at the bottom of the animation container and enters a video stream playback state. The client program detects the playback progress of the first virtual object's audiovisual content. When the first target virtual object's audiovisual content finishes playing, it hides the top first target virtual object's audiovisual content, displays the bottom container, and begins playing the virtual object's response audiovisual content.

[0110] Thus, as Figure 4 As shown, it can seamlessly replace the preset virtual object response audio-visual screen with the response video screen generated by the server, without the user noticing.

[0111] In some other embodiments of this application, the virtual object response video stream data includes data of the virtual object response audio-visual image connected to the first target virtual object audio-visual image. Based on this, step 140 may specifically include step 1404.

[0112] Step 1404: If the similarity between the first target virtual object audiovisual image and the second target virtual object audiovisual image currently playing in the first display area is greater than or equal to a preset threshold, after the second target virtual object audiovisual image finishes playing, a virtual object response audiovisual image is played in the first display area.

[0113] The second target virtual object audiovisual screen is either the first target virtual object audiovisual screen or a preset virtual object audiovisual screen displayed after the first target virtual object audiovisual screen.

[0114] For example, using the aforementioned client program as an example, the client program initializes a virtual object container at the bottom of the animation container and enters a video stream playback state. The client program detects the playback progress of the first virtual object's audiovisual content. When the first target virtual object's audiovisual content finishes playing, it hides the top client animation, displays the bottom container, and begins playing the virtual object's response audiovisual content. Thus, as... Figure 4 As shown, it can seamlessly replace the preset virtual object response audio-visual screen with the response video screen generated by the server, without the user noticing.

[0115] In some other embodiments of this application, the virtual object response video stream data includes data of the virtual object response audio-visual screen that is connected to the first target virtual object audio-visual screen; the first target virtual object audio-visual screen is a screen played after the third target virtual object audio-visual screen in at least two first virtual object audio-visual screens, the playback interval between the third target virtual object audio-visual screen and the first target virtual object audio-visual screen is a preset interval, the third target virtual object audio-visual screen is the preset virtual object audio-visual screen currently playing in the first display area when the video stream request is generated, and the preset interval is determined by the empirical duration of the server feedback of the virtual object response video stream data. Based on this, step 140 may specifically include steps 1405 and 1406.

[0116] Step 1405: Splice the virtual object's response audiovisual image after the third target virtual object's audiovisual image.

[0117] For example, information such as the pose, position, and image size of the virtual object in the third target virtual object's audiovisual image is extracted and converted into JSON data. OpenCV is then used to process this information in the virtual object's response audiovisual image, mapping it to the information in the third target virtual object's audiovisual image. This ensures that the content of the virtual object in the response audiovisual image matches the content in the third target virtual object's audiovisual image and allows for a smooth transition. Video editing and crossfading techniques are used to ensure a natural transition. Figure 4 As shown, connecting icon frames and image frames improves playback smoothness.

[0118] Here, it should be noted that, as Figure 4 As shown, this application embodiment converts the switching between video and video between preset virtual object audio-visual screens and virtual object response audio-visual screens into switching between image frames, thereby improving the controllability of screen switching and reducing technical complexity.

[0119] Step 1407: After the third target virtual object's audiovisual screen finishes playing, the virtual object's response audiovisual screen is played in the first display area.

[0120] For example, using the client program described above, the client program initializes a virtual object container at the bottom of the animation container and enters a video stream playback state. The client program detects the playback progress of the first virtual object's audiovisual content. When the first target virtual object's audiovisual content finishes playing, the top client animation is hidden, the bottom container is displayed, and the virtual object's response audiovisual content begins to play.

[0121] Thus, as Figure 4 As shown, it can seamlessly replace the preset virtual object response audio-visual screen with the response video screen generated by the server, without the user noticing.

[0122] It should be noted that the embodiments of this application provide a method for determining the preset interval, as shown below.

[0123] The duration of the virtual object response video generated by the server is determined based on empirical values. Based on this duration and the currently playing preset virtual object audiovisual image when the video stream request is sent to the server, the first target preset virtual object video image to be displayed when the virtual object response video is expected to be received is determined.

[0124] Furthermore, after step 140, the playback method may also include step 180, in which the first virtual object audiovisual image is played in the first display area after the virtual object response audiovisual image playback has ended.

[0125] For example, when the virtual object's response audiovisual playback ends, the system switches back to the client's default virtual object audiovisual playback, destroys the virtual object video stream data container, and disconnects the WebSocket connection. The system immediately switches back to the client's first virtual object audiovisual playback, such as the animation of the default animation program segment 1, thus completing one dialogue cycle. The system then transitions back to the stage where the electronic device activates its recording function and converts the recorded audio into user voice information using an audio signal speech recognition conversion algorithm, initiating the next round of dialogue, and so on.

[0126] In this way, the server does not need to continuously render the virtual object screen, enabling intermittent sending of video stream requests to the server, reducing the time the server spends rendering the virtual object video stream, and lowering the server load. Furthermore, upon receiving the virtual object response video stream data sent by the server, the first virtual object audiovisual screen playing in the virtual object display area is switched to the virtual object response audiovisual screen rendered from the virtual object response video stream data. This eliminates the need for the client to continuously recognize the virtual object's speaking status, reducing the high performance overhead of the client recognizing when the virtual object has finished speaking, and also reduces the frequency of data transmission between the server and the client, thereby lowering the transmission cost of network resources.

[0127] Furthermore, this application embodiment also provides a process for generating virtual object response video stream data based on video stream requests, which can be specifically combined with... Figure 5 The steps executed on the server side are explained.

[0128] like Figure 5 As shown, the server in the embodiment of this application can execute steps 510 to 570, as detailed below.

[0129] Step 510: The server receives a video stream request. The video stream request may carry at least one of the following: virtual object response text corresponding to the user's voice information, virtual object display data, and the playback order of the first target virtual object's audiovisual images in at least two first virtual object audiovisual images. The display data may include at least one of the following: the virtual object's display coordinates in the first display area, the virtual object's display area size in the video call interface, and the virtual object's visual identifier.

[0130] Step 520: Parse the video stream request and extract at least one of the following from the video stream request: display data, virtual object response text, display data, virtual object response text, and playback order.

[0131] Step 5301: Based on the display data and playback order, construct virtual object video stream data. The virtual object video data includes homepage virtual object response video stream data that matches the image of the virtual object and is related to the playback order, connecting the audiovisual images of the first target virtual object.

[0132] For example, OpenPose is used to extract information such as the pose, position, and size of the virtual object in the first target virtual object's audiovisual screen and convert it into JSON data. OpenCV is then used to process and generate virtual object video stream data to ensure that the content of the homepage virtual object's audiovisual screen corresponding to the homepage virtual object's response video stream data matches the content of the first target virtual object's audiovisual screen, thus achieving a smooth transition.

[0133] Step 5302: Based on the virtual object video stream data, the virtual object response text, and the virtual object response audio of the virtual object response text, generate the main virtual object audio-visual screen data that is connected to the homepage virtual object audio-visual screen corresponding to the homepage virtual object response video stream data.

[0134] Specifically, the server uses Text-to-Speech (TTS) technology to convert the virtual object's response text into virtual object response audio, and uses Wav2Lip to drive the human image through voice, generating data for the main virtual object's audio-visual screen that is connected to the virtual object's audio-visual screen on the homepage.

[0135] Step 540: Combine the audio-visual images of the virtual object on the homepage with the audio-visual images of the main virtual object through video cropping to obtain the virtual object response video stream data.

[0136] Step 550: Connectivity test to ensure a smooth transition.

[0137] For example, by using video editing, crossfading and other methods to ensure a natural transition, the video frame of the virtual object response video stream data is detected and repeatedly adjusted and optimized until the smoothness value of the connection between the first target virtual object audio-visual screen and the homepage virtual object audio-visual screen is greater than or equal to the preset smoothness value, then step 560 can be executed.

[0138] It should be noted that the detection of video frames can convert them into image frames, thus turning all content into image frames, improving controllability and reducing technical complexity.

[0139] The playback method provided in this application embodiment can be applied to interactive applications such as dialogue with 3D (Three Dimensions, 3D) cute pets, video conferencing, etc.

[0140] Step 560: Send virtual object response video stream data to the electronic device.

[0141] Therefore, this application provides a solution that uses a hybrid approach of client-side playback of preset animations and server-side video streaming, which can effectively reduce server load, eliminate the need for client-side decoding, VAD detection, or lip-sync detection technologies, and avoid the high performance overhead of the client.

[0142] This application's embodiments break down the video call process with a virtual object into three stages: virtual object listening (user speaking), virtual object thinking, and virtual object speaking. During the listening and thinking stages, since the virtual object only needs to perform some default body movements and requires no audio or lip movements, relatively fixed animations can be used instead. This stage uses pre-installed animation resources on the client side, which are played by the client. During the virtual object speaking stage, the system switches to playing a video stream generated by the server that is the same size and position as the client and can be seamlessly connected. After the virtual object finishes speaking (the completion of the video stream signifies the end of the speaking, without the need for recognition technology), the system switches back to the client's pre-installed animation, and so on, achieving a seamless transition.

[0143] Based on the above, in order to better illustrate the playback method provided in the embodiments of this application, the following can be combined with... Figure 6 Please provide a detailed explanation.

[0144] like Figure 6 As shown in the embodiments of this application, a virtual object video call system can be established. This virtual object video call system interacts similarly to a user's video call system, the difference being that a user's video call system involves real users interacting with each other, while the virtual object video call system involves a person interacting with a virtual object. This virtual object video call system provides three stages of functionality for one dialogue cycle between the user and the virtual object. One dialogue cycle refers to the user speaking once, and the virtual object speaking once. The three stages can include a virtual object listening to the user (P1 stage), a virtual object thinking stage (P2 stage), and a virtual object speaking stage (P3 stage). The functions corresponding to each stage can be the functions of the virtual object listening stage, the virtual object thinking stage, and the virtual object speaking stage.

[0145] The following is combined Figure 6 The playback method provided in this application is described in detail.

[0146] Phase P1 may include steps 601 and 602.

[0147] Step 601: The user begins to speak, and the electronic device can activate the recording function and convert the recorded audio into the user's voice information through an Automatic Speech Recognition (ASR) algorithm.

[0148] During this process, the audiovisual image of the first virtual object can be played in the first display area of ​​the video call interface. Here, the virtual object image of the virtual object in the audiovisual image of the first virtual object is used to represent at least one of the following states: the virtual object is in a listening state, or the virtual object is in a thinking state.

[0149] In step 602, the electronic device fills the user's voice information and the virtual object's identity data into a preset AI text assembly template to obtain the assembled text content, such as the prompt query text. Additionally, it establishes a WebSocket connection with the server.

[0150] The P2 stage may include steps 603 to 612.

[0151] Step 603: Based on the text assembly content obtained in the aforementioned P1 stage, the electronic device sends a response data request to the large model server. The response data request carries the text assembly content and is used to request the large model server to perform streaming processing on the text assembly content to obtain streaming keywords related to the text assembly content.

[0152] Step 604: The large model server performs streaming processing on the assembled text content to obtain streaming keywords related to the assembled text content.

[0153] Step 605: The large model server sends at least two streaming keywords to the electronic device.

[0154] Step 606: Upon receiving at least two streaming keywords from the large model server, the electronic device stores at least two streaming keywords in a buffer.

[0155] Step 607: If the electronic device recognizes that at least two target sentences constructed from streaming keywords in the buffer meet the preset sentence composition conditions, it generates virtual object response text based on the target sentences.

[0156] Step 608: The electronic device constructs a video stream request based on the virtual object response text and display data.

[0157] The display data includes the display coordinates of the virtual object in the first display area and the display size of the first display area in the video call interface.

[0158] Step 609: The electronic device can establish a real-time web communication connection channel, such as a WebRTC connection channel, with the server through a WebSocket connection. The electronic device can then send a video stream request to the server through the WebRTC connection channel. The video stream request carries the virtual object response text corresponding to the user's voice information and the display data of the virtual object.

[0159] Step 610: The server constructs virtual object video stream data that matches the virtual object image based on the display data in the received video stream request.

[0160] Step 611: The server generates virtual object response video stream data based on the virtual object video stream data, the virtual object response text, and the virtual object response audio of the virtual object response text.

[0161] Step 612: The server sends virtual object response video stream data to the electronic device.

[0162] Therefore, since there is a certain amount of time spent in the process of network transmission between the electronic device and the server and the large model server, the large model server generating virtual object response text, and the server generating virtual object response video stream data, the virtual object image of the first virtual object audiovisual screen of the electronic device in this stage is used to represent at least one of the following states: the virtual object is in the listening state, and the virtual object is in the thinking state.

[0163] The P3 stage may include step 613.

[0164] Step 613: The electronic device receives the virtual object response video stream data sent by the server through the WebRTC connection channel, renders the virtual object response video stream data to obtain the virtual object response audiovisual screen, and switches the first virtual object video screen playing in the first display area to the virtual object response audiovisual screen.

[0165] Therefore, this stage involves the virtual object responding to the user's voice information. The virtual object's image in this stage can be a presentation of content related to the user's response text and the virtual object's response audio, including the virtual object's body shape, lip shape, facial expression, and voice.

[0166] The above three stages constitute one cycle, and this process is repeated cyclically to enable scenarios where users can make or video calls with virtual objects.

[0167] Thus, in this embodiment, the virtual object video playback can be broken down into three stages, with pre-set animations on the electronic device replacing the relatively fixed movements of the virtual object, while the speaking stage retains the video stream. This hybrid approach preserves all aspects of the virtual object, including body movements, lip movements, voice, and facial expressions, while minimizing the time from the entire process to the virtual object's speaking stage and the playback of the virtual object's video stream. This reduces server load and network transmission overhead. Furthermore, the client on the electronic device does not need to decode the video stream, avoiding the use of high-performance technologies such as real-time VAD detection and lip recognition to identify when the virtual object has finished speaking, significantly improving client performance. The clever hybrid use of animation and video stream playback reduces technical complexity, and the segmented approach ensures seamless transitions between animation and video playback without any abrupt changes or interruptions. There is no loss of effect, and performance overhead is significantly reduced.

[0168] It should be noted that the playback method provided in this application can be executed by electronic devices such as mobile phones, tablets, laptops, PDAs, and wearable devices. Some embodiments of this application use electronic devices as the executing entity to illustrate the playback method provided in this application.

[0169] The playback method provided in this application can be executed by a playback device. This application uses a playback device executing the playback method as an example to illustrate the apparatus for the playback method provided in this application.

[0170] This application also provides a playback device. (Specifically combined with...) Figure 7 Please provide a detailed explanation.

[0171] Figure 7 A schematic diagram of the structure of a playback device provided for some embodiments of this application.

[0172] like Figure 7 As shown, the playback device 70 can be applied to electronic devices, and the playback device 70 may specifically include:

[0173] Display module 701 is used to play the audio-visual image of the first virtual object in the first display area of ​​the video call interface during the process of acquiring user voice information when the user is making a video call with the virtual object.

[0174] The sending module 702 is used to send a video stream request to the server. The video stream request carries the virtual object response text corresponding to the user's voice information and the display data of the virtual object. The video stream request is used to request the server to construct virtual object video stream data that matches the image of the virtual object based on the display data, and to generate virtual object response video stream data based on the virtual object video stream data, the virtual object response text, and the virtual object response audio of the virtual object response text.

[0175] The receiving module 703 is used to receive virtual object response video stream data sent by the server;

[0176] The processing module 704 is used to switch the first virtual object audio-visual image played in the first display area to a virtual object response audio-visual image, which is rendered from the virtual object response video stream data.

[0177] The playback device 70 in the embodiments of this application will be described in detail below.

[0178] In some embodiments of this application, the playback device 70 further includes a determining module, configured to determine the virtual object response text corresponding to the user's voice information based on the user's voice information, provided that the display data includes the display coordinates of the virtual object in the first display area and the display size of the first display area in the video call interface.

[0179] The playback device 70 also includes a building module for constructing a video stream request based on virtual object response text, display size, and display coordinates.

[0180] In some embodiments of this application, the determining module is specifically used to fill the user's voice information and the virtual object identity data of the virtual object into a preset artificial intelligence text assembly template to obtain the text assembly content;

[0181] The sending module 702 is also used to send a response data request to the large model server. The response data request carries the text assembly content and is used to request the large model server to perform streaming processing on the text assembly content to obtain streaming keywords related to the text assembly content.

[0182] The playback device 70 also includes a storage module for storing at least two streaming keywords into a buffer upon receiving at least two streaming keywords from a large model server.

[0183] The playback device 70 also includes a generation module, which generates virtual object response text based on the target sentence when at least two streaming keywords in the buffer are identified as meeting the preset sentence composition conditions.

[0184] In some embodiments of this application, the construction module is specifically used to construct a video stream request based on the playback order, virtual object response text, display size, and display coordinates, when the first virtual object audiovisual screen includes at least two first virtual object audiovisual screens, and the at least two first virtual object audiovisual screens are played in a preset playback order; the video stream request also carries the playback order of the first target virtual object audiovisual screens in the at least two first virtual object audiovisual screens; and the video stream request is further used to request the server to obtain the display form of the virtual object in the first target virtual object audiovisual screen corresponding to the playback order, and to construct virtual object video stream data matching the virtual object image based on the display form of the virtual object in the first target virtual object audiovisual screen.

[0185] In some embodiments of this application, the playback device 70 further includes a splicing module, used to splice the virtual object response audio-visual image after the first target virtual object audio-visual image when the virtual object response video stream data includes data of the virtual object response audio-visual image connected to the first target virtual object audio-visual image.

[0186] The playback device 70 also includes a switching module for switching the preset virtual object audio-visual image currently playing in the first display area to the first target virtual object audio-visual image;

[0187] The display module 701 is also used to play the virtual object response audio-visual screen when the first target virtual object audio-visual screen finishes playing.

[0188] In some embodiments of this application, the display module 701 is further configured to, after the second target virtual object audio-visual image has finished playing, play the virtual object response audio-visual image in the first display area when the virtual object response video stream data includes data of the virtual object response audio-visual image connected to the first target virtual object audio-visual image, and the similarity between the first target virtual object audio-visual image and the second target virtual object audio-visual image currently playing in the first display area is greater than or equal to a preset threshold.

[0189] In some embodiments of this application, the playback device 70 further includes a splicing module, used to splice the virtual object response video stream data after the third target virtual object video stream data when the virtual object response video stream data includes data of a virtual object response audio-visual image connected to the first target virtual object audio-visual image; the first target virtual object audio-visual image is a screen played after a third target virtual object audio-visual image in at least two first virtual object audio-visual images; the playback interval between the third target virtual object audio-visual image and the first target virtual object audio-visual image is a preset interval; the third target virtual object audio-visual image is a preset virtual object audio-visual image currently playing in the first display area when the video stream request is generated; and the preset interval is determined by the empirical duration of the virtual object response video stream data fed back by the server.

[0190] The display module 701 is also used to play the virtual object response audio-visual screen in the first display area after the third target virtual object audio-visual screen has finished playing.

[0191] In some embodiments of this application, the display module 701 is further configured to play the first virtual object audio-visual image in the first display area when the virtual object response audio-visual image playback ends.

[0192] The playback device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the scope of the device.

[0193] The playback device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0194] The playback device provided in this application embodiment can achieve... Figures 1 to 6 The various processes implemented in the playback method embodiments shown achieve the same technical effect, and will not be described again here to avoid repetition.

[0195] Based on this, the playback device provided in this application embodiment considers that the virtual object only needs to perform some default body movements during the listening and thinking phases, without the need for audio and lip movements to accompany them. Therefore, in the process of obtaining user voice information during a video call between the user and the virtual object, the process of continuously rendering the virtual object's image by the server is replaced by displaying the first virtual object audio-visual image in the virtual object display area of ​​the video call interface. In this way, the server does not need to continuously render the virtual object's image, realizing the intermittent sending of video stream requests to the server, reducing the time for the server to draw the virtual object's video stream, and reducing the server's load. Furthermore, upon receiving the virtual object response video stream data sent by the server, the first virtual object audio-visual image played in the virtual object display area is switched to the virtual object response audio-visual image rendered by the virtual object response video stream data. In this way, the client does not need to continuously recognize the virtual object's speaking state, reducing the high performance overhead of the client recognizing the virtual object's speech completion, and reducing the data transmission frequency between the server and the client, which can reduce the transmission cost of network resources.

[0196] Optional, such as Figure 8 As shown, this application embodiment also provides an electronic device 80, including a processor 801 and a memory 802. The memory 802 stores a program or instructions that can run on the processor 801. When the program or instructions are executed by the processor 801, they implement the various steps of the above-described playback method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0197] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0198] Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0199] The electronic device 900 includes, but is not limited to, components such as: radio frequency unit 901, network module 902, audio output unit 903, input unit 904, sensor 905, display unit 906, user input unit 907, interface unit 908, memory 909, and processor 910.

[0200] Those skilled in the art will understand that the electronic device 900 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 910 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system.

[0201] Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0202] In this embodiment, the display unit 906 is used to play a first virtual object audio-visual image in the first display area of ​​the video call interface during the process of acquiring user voice information when the user is making a video call with a virtual object. The network module 902 is used to send a video stream request to the server. The video stream request carries virtual object response text corresponding to the user's voice information and virtual object display data. The video stream request requests the server to construct virtual object video stream data matching the virtual object's image based on the display data, and to generate virtual object response video stream data based on the virtual object video stream data, the virtual object response text, and the virtual object response audio of the virtual object response text. The network module 902 can also be used to receive virtual object response video stream data sent by the server. The processor 910 is used to switch the first virtual object audio-visual image played in the first display area to a virtual object response audio-visual image, which is rendered from the virtual object response video stream data.

[0203] The electronic device 900 will be described in detail below.

[0204] In some embodiments of this application, the processor 910 is configured to determine the virtual object response text corresponding to the user's voice information based on the user's voice information, provided that the display data includes the display coordinates of the virtual object in the first display area and the display size of the first display area in the video call interface.

[0205] A video stream request is constructed based on the virtual object's response text, display size, and display coordinates.

[0206] In some embodiments of this application, processor 910 is used to fill user voice information and virtual object identity data of virtual objects into a preset artificial intelligence text assembly template to obtain text assembly content;

[0207] The sending module 702 is also used to send a response data request to the large model server. The response data request carries the text assembly content and is used to request the large model server to perform streaming processing on the text assembly content to obtain streaming keywords related to the text assembly content.

[0208] The memory 909 is used to store at least two streaming keywords into a buffer upon receiving at least two streaming keywords sent by the large model server.

[0209] The processor 910 is used to generate virtual object response text based on the target sentence when it is found that at least two target sentences constructed from streaming keywords in the buffer meet the preset sentence composition conditions.

[0210] In some embodiments of this application, the processor 910, when the first virtual object audiovisual screen includes at least two first virtual object audiovisual screens, and the at least two first virtual object audiovisual screens are played in a preset playback order; the video stream request also carries the playback order of the first target virtual object audiovisual screens in the at least two first virtual object audiovisual screens, and the video stream request is also used to request the server to obtain the display form of the virtual object in the first target virtual object audiovisual screen corresponding to the playback order, and to construct virtual object video stream data matching the virtual object image based on the display form of the virtual object in the first target virtual object audiovisual screen, constructs a video stream request based on the playback order, virtual object response text, display size, and display coordinates.

[0211] In some embodiments of this application, the processor 910 is configured to splice the virtual object response video stream after the first target virtual object video stream when the virtual object response video stream data includes data of the virtual object response video stream that is connected to the first target virtual object video stream.

[0212] Switch the currently playing preset virtual object audio-visual image in the first display area to the first target virtual object audio-visual image;

[0213] The display unit 906 is also used to play the virtual object response audio-visual screen when the first target virtual object audio-visual screen playback ends.

[0214] In some embodiments of this application, the display unit 906 is further configured to, after the second target virtual object audio-visual image has finished playing, play the virtual object response audio-visual image in the first display area when the virtual object response video stream data includes data of the virtual object response audio-visual image connected to the first target virtual object audio-visual image, and the similarity between the first target virtual object audio-visual image and the second target virtual object audio-visual image currently playing in the first display area is greater than or equal to a preset threshold.

[0215] In some embodiments of this application, the processor 910 is configured to splice the virtual object response video stream data after the third target virtual object video stream data when the virtual object response video stream data includes data of a virtual object response audio-visual screen that is connected to the first target virtual object audio-visual screen; the first target virtual object audio-visual screen is a screen played after a third target virtual object audio-visual screen in at least two first virtual object audio-visual screens; the playback interval between the third target virtual object audio-visual screen and the first target virtual object audio-visual screen is a preset interval; the third target virtual object audio-visual screen is a preset virtual object audio-visual screen currently playing in the first display area when the video stream request is generated; and the preset interval is determined by the empirical duration of the virtual object response video stream data fed back by the server.

[0216] The display unit 906 is also used to play the virtual object response audio-visual screen in the first display area after the third target virtual object audio-visual screen has finished playing.

[0217] In some embodiments of this application, the display unit 906 is further configured to play the first virtual object audio-visual image in the first display area when the virtual object response audio-visual image playback ends.

[0218] It should be understood that the input unit 904 may include a graphics processing unit (GPU) 9041 and a microphone 9042. The GPU 9041 processes image data of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 906 may include a display panel, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch display. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume display buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0219] The memory 909 can be used to store software programs and various data. The memory 909 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 909 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 909 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0220] Processor 910 may include one or more processing units; optionally, processor 910 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless display signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 910.

[0221] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described playback method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0222] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0223] In addition, this application embodiment provides another chip, which includes a processor and a display interface. The display interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above playback method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0224] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0225] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the playback method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0226] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0227] Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0228] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0229] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A playing method characterized by comprising: The method comprises the following steps: In the process of acquiring user voice information when the user has a video call with a virtual object, playing a first virtual object audio-visual picture in a first display area of a video call interface; Sending a video stream request to a server, the video stream request carrying virtual object response text corresponding to the user voice information and display data of the virtual object, the video stream request being used to request the server to construct virtual object video stream data matching the image of the virtual object based on the display data, and generate virtual object response video stream data according to the virtual object video stream data, the virtual object response text, and virtual object response audio of the virtual object response text; the display data comprising display coordinates of the virtual object in the first display area and display size of the first display area in the video call interface; Receiving the virtual object response video stream data sent by the server; Switching the first virtual object audio-visual picture played in the first display area to a virtual object response audio-visual picture, the virtual object response audio-visual picture being rendered from the virtual object response video stream data.

2. The method of claim 1, wherein, Before the step of sending the video stream request to the server, the method further comprises the following steps: Determining virtual object response text corresponding to the user voice information according to the user voice information; Constructing the video stream request based on the virtual object response text, the display size and the display coordinates.

3. The method of claim 2, wherein, The first virtual object audio-visual picture comprises at least two first virtual object audio-visual pictures, the at least two first virtual object audio-visual pictures being played in a preset virtual object audio-visual picture playing order; the video stream request further carries a playing order of a first target virtual object audio-visual picture in the at least two first virtual object audio-visual pictures, and the video stream request is further used to request the server to acquire a display form of a virtual object in the first target virtual object audio-visual picture corresponding to the playing order, and construct the virtual object video stream data matching the virtual object image of the virtual object based on the display form of the virtual object in the first target virtual object audio-visual picture; The step of constructing the video stream request based on the virtual object response text, the display size and the display coordinates comprises the following step: Constructing the video stream request based on the playing order, the virtual object response text, the display size and the display coordinates.

4. The method of claim 3, wherein, The virtual object response video stream data comprises data of a virtual object response audio-visual picture connected with the first target virtual object audio-visual picture; The step of switching the first virtual object audio-visual picture played in the first display area to the virtual object response audio-visual picture comprises the following steps: Splicing the virtual object response audio-visual picture after the first target virtual object audio-visual picture; Switching a preset virtual object audio-visual picture currently played in the first display area to the first target virtual object audio-visual picture; In the case that the playing of the first target virtual object audio-visual picture ends, playing the virtual object response audio-visual picture.

5. The method of claim 3, wherein, The virtual object response video stream data includes data of a virtual object response audio-visual picture that is spliced with the first target virtual object audio-visual picture; The first virtual object audio-visual picture played in the first display area is switched to a virtual object response audio-visual picture, including: In a case where a picture similarity between the first target virtual object audio-visual picture and a second target virtual object audio-visual picture currently played in the first display area is greater than or equal to a preset threshold, the virtual object response audio-visual picture is played in the first display area after the second target virtual object audio-visual picture is played.

6. A playing device, characterized in that Including: The display module is configured to play a first virtual object audio-visual picture in a first display area of a video call interface in a process of obtaining user voice information when the user has a video call with a virtual object; The sending module is configured to send a video stream request to a server, the video stream request carrying virtual object response text corresponding to the user voice information and display data of the virtual object, the video stream request being used to request the server to construct virtual object video stream data matched with an image of the virtual object based on the display data, and generate virtual object response video stream data according to the virtual object video stream data, the virtual object response text, and virtual object response audio of the virtual object response text, the display data including display coordinates of the virtual object in the first display area and display dimensions of the first display area in the video call interface; The receiving module is configured to receive the virtual object response video stream data sent by the server; The processing module is configured to switch the first virtual object audio-visual picture played in the first display area to a virtual object response audio-visual picture, the virtual object response audio-visual picture being rendered by the virtual object response video stream data.

7. The apparatus of claim 6, wherein, The playing device further includes a determination module configured to determine virtual object response text corresponding to the user voice information according to the user voice information; The playing device further includes a construction module configured to construct the video stream request based on the virtual object response text, the display dimensions and the display coordinates.

8. The apparatus of claim 7, wherein, The construction module is specifically configured to, in a case where the first virtual object audio-visual picture includes at least two first virtual object audio-visual pictures, the at least two first virtual object audio-visual pictures being played in a preset virtual object audio-visual picture playing order, the video stream request further carrying a playing order of a first target virtual object audio-visual picture in the at least two first virtual object audio-visual pictures, the video stream request being further used to request the server to obtain a display form of a virtual object in the first target virtual object audio-visual picture corresponding to the playing order, and construct the virtual object video stream data matched with the virtual object image of the virtual object based on the display form of the virtual object in the first target virtual object audio-visual picture, construct the video stream request based on the playing order, the virtual object response text, the display dimensions and the display coordinates.

9. The apparatus of claim 8, wherein, The playing device further comprises a splicing module, configured to splice the virtual object response audio-visual picture after the first target virtual object audio-visual picture in a case that the virtual object response video stream data comprises data of a virtual object response audio-visual picture which is continuous with the first target virtual object audio-visual picture; The playing device further comprises a switching module, configured to switch the preset virtual object audio-visual picture currently played in the first display area to the first target virtual object audio-visual picture; The display module is further configured to play the virtual object response audio-visual picture in a case that the playing of the first target virtual object audio-visual picture ends.

10. The apparatus of claim 8, wherein, The display module is further configured to play the virtual object response audio-visual picture in the first display area after the playing of the second target virtual object audio-visual picture ends in a case that the virtual object response video stream data comprises data of a virtual object response audio-visual picture which is continuous with the first target virtual object audio-visual picture, and a picture similarity between the first target virtual object audio-visual picture and the second target virtual object audio-visual picture currently played in the first display area is greater than or equal to a preset threshold.

Images