Display device, server and live broadcast interaction method
By receiving and processing reply videos sent by the server in live video, and using accelerated speed playback technology, the problem of viewers missing live content is solved, and the viewing experience is improved.
Patent Information
- Application Number
- CN202310512509.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-08
- Publication Date
- 2024-11-08
AI Technical Summary
During the live video broadcast process, viewers are prone to missing the original live content when playing and replying to the video after posting interactive information, resulting in poor viewing experience.
By receiving the reply video sent by the server on the display device, obtaining its playback time and start time stamp, and calculating the second time length based on the acceleration multiple, and playing the live video at the acceleration multiple speed after the reply video is played, thereby catching up with the live broadcast progress and ensuring that The audience didn't miss the live content.
It effectively avoids the problem of missing live content caused by playing and replying videos, and improves the audience experience of live videos.
Smart Images

Figure CN118921497A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video live broadcast technology, and in particular to a display device, a server and a live broadcast interactive method. Background Art
[0002] With the development of live broadcast technology, anchors can broadcast live videos through various live broadcast apps (Applications), and viewers can also watch live broadcasts through various live broadcast apps. During the live broadcast, viewers can post interactive information in the live broadcast room at any time. After detecting the interactive information of the audience, corresponding replies will be generated and fed back to the audience.
[0003] When there is too much interactive information during a live video broadcast, a corresponding reply video can be generated based on the interactive information and fed back to the audience. However, while playing the reply video, the audience will miss the original live video content, resulting in a poor live viewing experience. Summary of the invention
[0004] The exemplary implementation of the present application provides a display device, a server and a live broadcast interactive method for improving the audience viewing experience during live video broadcast.
[0005] The technical solutions provided by the embodiments of this application are as follows:
[0006] In a first aspect, a display device is provided, comprising:
[0007] The communicator is configured to: receive a reply video sent by a server during a live video broadcast, wherein the reply video is a video related to the interactive information in the live video broadcast;
[0008] The controller is configured to: play the reply video and obtain a first duration and a target timestamp, wherein the first duration is the playback duration of the reply video and the target timestamp is the timestamp at which the playback of the reply video starts; obtain an acceleration multiple; and obtain a second duration according to the first duration and the acceleration multiple;
[0009] The display is configured to: after finishing playing the reply video, play the live video for the second duration at the acceleration multiple starting from the target timestamp.
[0010] In a second aspect, a live interactive method is provided, comprising:
[0011] The communicator is configured to: receive interactive information sent by the display device; the interactive information is: interactive information input by the user during the live video broadcast;
[0012] The controller is configured as:
[0013] Obtaining a reply text corresponding to the interactive information;
[0014] Generate a reply audio based on the reply text;
[0015] Acquire a response video based on the response audio;
[0016] The communicator is further configured to: send the reply video to the display device.
[0017] In a third aspect, the present invention provides a live interactive method, comprising:
[0018] During the live video broadcast, a reply video sent by a server is received, wherein the reply video is a video related to the interactive information in the live video broadcast;
[0019] Play the reply video and obtain a first duration and a target timestamp, wherein the first duration is the playback duration of the reply video and the target timestamp is the timestamp for starting to play the reply video;
[0020] Obtaining an acceleration multiple; and obtaining a second duration according to the first duration and the acceleration multiple;
[0021] After finishing playing the reply video, playing the live video for the second duration at the acceleration multiple starting from the target timestamp.
[0022] In a fourth aspect, the present invention provides another live interactive method, comprising:
[0023] Receiving interactive information sent by a display device; the interactive information is: interactive information input by a user during a live video broadcast;
[0024] Obtaining a reply text corresponding to the interactive information;
[0025] Generate a reply audio based on the reply text;
[0026] Acquire a response video based on the response audio;
[0027] The reply video is sent to the display device.
[0028] In a fifth aspect, the present invention provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a controller, the live broadcast interactive method shown in the third aspect and the fourth aspect is implemented.
[0029] In a sixth aspect, the present invention provides a computer program product, comprising: when the computer program product runs on a computer, the computer implements the live interactive method shown in the third and fourth aspects.
[0030] It can be seen from the above technical solutions that the display device, server and live interactive method provided by the embodiment of the present application receive the reply video sent by the server through the communicator during the video live broadcast, and the reply video is a video related to the interactive information in the video live broadcast; the reply video is played through the controller, and the first duration and the target timestamp are obtained, the first duration is the playback duration of the reply video, and the target timestamp is the timestamp for starting to play the reply video; the acceleration multiple is obtained; and the second duration is obtained according to the first duration and the acceleration multiple; finally, after the playback of the reply video is ended, the display plays the live video at the acceleration multiple times from the target timestamp. The embodiment of the present application can continue to play the live video of the second duration from the target timestamp at the acceleration multiple after the reply video is played from the target timestamp, thereby avoiding the situation in which the audience misses the live video content due to the playback of the reply video during the video live broadcast. Therefore, the embodiment of the present application can improve the audience experience during the video live broadcast. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the implementation methods in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technology descriptions. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.
[0032] Figure 1 A scene architecture diagram of a live interactive method provided by an embodiment of the present invention is shown;
[0033] Figure 2 A hardware configuration block diagram of a control device provided by an embodiment of the present invention is shown;
[0034] Figure 3 A hardware configuration block diagram of a display device provided by an embodiment of the present invention is shown;
[0035] Figure 4 A schematic diagram of a network architecture of a display device and a server provided in an embodiment of the present invention is shown;
[0036] Figure 5 A flowchart of a live interactive method provided by an embodiment of the present invention is shown;
[0037] Figure 6 A flowchart showing another live interactive method provided by an embodiment of the present invention is shown;
[0038] Figure 7 A flowchart showing another live interactive method provided by an embodiment of the present invention is shown;
[0039] Figure 8 A flowchart showing another live interactive method provided by an embodiment of the present invention is shown;
[0040] Fig. 9 A flowchart of a live interactive method provided by an embodiment of the present invention is shown;
[0041] Fig.10 An interactive flow chart of a live interactive method provided by an embodiment of the present invention is shown;
[0042] Fig.11 A logical block diagram of a live broadcast interactive method provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0043] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.
[0044] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.
[0045] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.
[0046] Figure 1 This is a schematic diagram of the scene architecture of the live interactive method provided in the embodiment of the present application. Figure 1 As shown, the scenario architecture provided by the embodiment of the present application includes: a server 400 and a display device 300.
[0047] The display device 300 provided in the embodiment of the present application can have various implementation forms, for example, it can be a smart speaker, a television, a refrigerator, a washing machine, an air conditioner, a smart curtain, a router, a set-top box, a mobile phone, a personal computer (PC) smart TV, a laser projection device, a monitor, an electronic whiteboard, a wearable device, a vehicle-mounted device, an electronic table, etc.
[0048] In some embodiments, a user can operate the display device 300 through a smart device 200 or a control device 100. When the display device 300 receives an audio signal, it can communicate data with a server 400, wherein the display device 300 can be allowed to communicate with the server 400 through a local area network (LAN) or a wireless local area network (WLAN).
[0049] The server 400 may be a server that provides various services, such as a server that supports audio data collected by the display device 300. The server may analyze and process the received audio data, and feed back the processing results (such as endpoint information) to the terminal device. The server 400 may be a server cluster or multiple server clusters, and may include one or more types of servers.
[0050] The display device 300 can be hardware or software. When the display device 300 is hardware, it can be various electronic devices with sound collection functions, including but not limited to smart speakers, smart phones, TVs, tablet computers, e-book readers, smart watches, players, computers, AI devices, robots, smart vehicles, etc. When the display device 300 is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, for providing sound collection services), or it can be implemented as a single software or software module. No specific limitation is made here.
[0051] It should be noted that the live interactive method provided in the embodiment of the present application can be executed by the server 400, can also be executed by the display device 300, and can also be executed jointly by the server 400 and the display device 300, and the present application does not limit this.
[0052] Figure 2 FIG. 4 shows a hardware configuration block diagram of a display device 300 according to an exemplary embodiment. Figure 2 The display device 300 shown includes at least one of a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, an audio processor, a RAM, a ROM, and first to nth interfaces for input / output.
[0053] The communicator 220 is a component for communicating with an external device or server according to various communication protocol types. For example, the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or a near field communication protocol chip, and an infrared receiver. The display device 300 can establish transmission and reception of control signals and data signals with the server 400 through the communicator 220.
[0054] The user interface 280 may be used to receive external control signals.
[0055] The detector 230 is used to collect signals from the external environment or the external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.
[0056] The sound collector can be a microphone, also known as a "microphone" or "microphone", which can be used to receive the user's voice and convert the sound signal into an electrical signal. The display device 300 can be provided with at least one microphone. In other embodiments, the display device 300 can be provided with two microphones, which can not only collect sound signals but also realize the noise reduction function. In other embodiments, the display device 300 can also be provided with three, four or more microphones to realize the collection of sound signals, noise reduction, identification of the sound source, and realization of directional recording functions, etc.
[0057] In addition, the microphone may be built into the display device 300, or the microphone may be connected to the display device 300 by wire or wirelessly. Of course, the embodiment of the present application does not limit the position of the microphone on the display device 300. Alternatively, the display device 300 may not include a microphone, that is, the microphone is not provided in the display device 300. The display device 300 may be connected to an external microphone (also referred to as a microphone) through an interface (such as a USB interface 130). The external microphone may be fixed to the display device 300 by an external fixing (such as a camera bracket with a clip).
[0058] The controller 250 controls the operation of the display device 300 and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 300.
[0059] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a RAM Random Access Memory (RAM), a ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.
[0060] In some embodiments, the operating system of the smart device is an Android system, for example, Figure 3As shown, the display device 300 can be logically divided into an application layer (“application layer” for short) 31 , a kernel layer 32 and a hardware layer 33 .
[0061] Among them, Figure 3 As shown, the hardware layer may include Figure 2 The controller 250, the communicator 220, the detector 230, etc. are shown. The application layer 31 includes one or more applications. The application can be a system application or a third-party application. For example, the application layer 31 includes a speech recognition application, which can provide a live interactive interface and service for connecting the display device 300 with the server 400.
[0062] The kernel layer 32 serves as a software middleware between the hardware layer and the application layer 31 and is used to manage and control hardware and software resources.
[0063] In some embodiments, the kernel layer 32 includes a detector driver, which is used to send the voice data collected by the detector 230 to the voice recognition application. Exemplarily, when the voice recognition application in the display device 300 is started and the display device 300 establishes a communication connection with the server 400, the detector driver is used to send the voice data of the user input collected by the detector 230 to the voice recognition application. Afterwards, the voice recognition application sends the query information containing the voice data to the intent recognition module 102 in the server. The intent recognition module 102 is used to input the voice data sent by the display device 300 into the intent recognition model.
[0064] To clearly illustrate the embodiments of the present application, Figure 4 A speech recognition network architecture provided in an embodiment of the present application is described.
[0065] See also Figure 4 As shown, Figure 4 A schematic diagram of a live broadcast network architecture provided for an embodiment of the present application. Figure 4 In the embodiment, the display device is used to receive input information and output the processing result of the information. The speech recognition module is deployed with a speech recognition service for recognizing audio as text; the semantic understanding module is deployed with a semantic understanding service for semantically parsing text; the business management module is deployed with a business instruction management service for providing business instructions; the language generation module is deployed with a language generation service (NLG) for converting instructions for the display device to execute into text language; the speech synthesis module is deployed with a speech synthesis (TTS) service for processing the text language corresponding to the instruction and sending it to the speaker for broadcasting. In one embodiment, Figure 4 In the illustrated architecture, there may be multiple physical service devices deployed with different business services, or one or more functional services may be integrated into one or more physical service devices.
[0066] In some embodiments, the following Figure 4 The process of processing the information input to the display device in the illustrated architecture is described by way of example, taking the case where the information input to the display device is a voice command input by voice as an example:
[0067] [Speech Recognition]
[0068] After receiving a voice command inputted via voice, the display device may perform noise reduction processing and feature extraction on the audio of the voice command, wherein the noise reduction processing may include steps such as removing echoes and ambient noise.
[0069] [Semantic understanding]
[0070] Using acoustic models and language models, the identified candidate texts and associated context information are understood by natural language, and the texts are parsed into structured, machine-readable information, such as business domain, intent, word slots, etc. to express semantics, etc. The executable intent is obtained to determine the intent confidence score. The semantic understanding module selects one or more candidate executable intents based on the determined intent confidence score.
[0071] [Business Management]
[0072] The semantic understanding module sends execution instructions to the corresponding business management module based on the semantic analysis results of the text of the voice command to execute the operation corresponding to the voice command, complete the user's request for this operation, and provide feedback on the execution results of the operation corresponding to the voice command.
[0073] In some embodiments, the display device 300 receives a reply video sent by a server during a live video broadcast through the communicator 220, wherein the reply video is a video related to the interactive information in the live video broadcast; plays the reply video through the controller 250, and obtains a first duration and a target timestamp, wherein the first duration is the playback duration of the reply video, and the target timestamp is the timestamp for starting to play the reply video; obtains an acceleration multiple; and obtains a second duration based on the first duration and the acceleration multiple; and is configured through the display 260 to: after ending the playback of the reply video, play the live video for the second duration starting from the target timestamp at the acceleration multiple.
[0074] In some embodiments, the display device 300 obtains the acceleration multiple through the controller 250 according to the audio features of the live video broadcast and the live feature algorithm model; wherein the live feature algorithm model is a model obtained by training a preset machine learning model based on a live feature sample data set, and the live feature sample data includes: multiple live feature sample data, and the live feature sample data includes: sample audio features and the acceleration multiple corresponding to the sample audio features; the audio features include: at least one of tone features, speaking speed features, voice quality features, and voice frequency features.
[0075] In some embodiments, the display device 300 obtains the second duration through the controller 250 according to the acceleration multiple, the first duration, and the following formula:
[0076]
[0077] Wherein, T is the second duration, a is the acceleration multiple, and t is the first duration.
[0078] In some embodiments, the display device 300 receives the acceleration multiple sent by the server through the communicator 220 .
[0079] In some embodiments, the display device 300 plays the live video at a preset rate through the display 260 after playing the live video at the acceleration multiple for the second duration starting from the target timestamp.
[0080] In some embodiments, the server 400 receives interactive information sent by the display device through the communicator 410; the interactive information is: interactive information input by the user during the live video broadcast; the controller 420 is configured to: obtain a reply text corresponding to the interactive information, generate a reply audio based on the reply text, and obtain a reply video based on the reply audio; the communicator 410 is also configured to: send the reply video to the display device.
[0081] In some embodiments, the server 400 drives the corresponding facial key point coordinate sequence and the limb joint point coordinate sequence based on the driving model through the controller 420 to obtain the reply video; wherein the driving model is a model obtained by training a first preset machine learning model based on a driving sample data set, and the driving sample data set includes: multiple driving sample data; the driving sample data includes: a sample facial key point coordinate sequence, a sample limb joint point coordinate sequence, and a reply video corresponding to the sample facial key point coordinate sequence and the sample limb joint point coordinate sequence.
[0082] In some embodiments, the server 400 obtains a three-dimensional model corresponding to the host of the live video broadcast through the controller 420; obtains the facial key point coordinates and limb joint point coordinates of the three-dimensional model; obtains a facial key point coordinate sequence based on the reply audio and the facial key point coordinates; obtains a limb joint point coordinate sequence based on the reply audio and the limb joint point coordinates; obtains the reply video based on the facial key point coordinate sequence, the limb joint point coordinate sequence and the reply audio.
[0083] In some embodiments, the server 400 obtains the acceleration multiple through the controller 420 according to the audio features of the live video broadcast and the live feature algorithm model; wherein the live feature algorithm model is a model obtained by training a second preset machine learning model based on a live feature sample data set, and the live feature sample data includes: multiple live feature sample data, and the live feature sample data includes: sample audio features and the acceleration multiple corresponding to the sample audio features; the audio features include: at least one of tone features, speaking speed features, voice quality features, and voice frequency features; and the acceleration multiple is sent to the display device.
[0084] Figure 5 The following is a flow chart showing an exemplary method of live interactive broadcasting provided by an embodiment of the present application. Figure 5 As shown, the live interactive method provided in the embodiment of the present application includes the following steps S501 to S504:
[0085] S501. Receive a reply video sent by a server during a live video broadcast.
[0086] The reply video stream is a video related to the interactive information in the live video.
[0087] For example, during a live video broadcast, viewers in the live broadcast room may post interactive information in the comment area in the barrage area, where the interactive information may be questions asked by viewers in the barrage area; for example, the current live broadcast room is introducing snack A, and the interactive information may be questions such as: what are the raw materials of snack A, or: if you place an order now, when will it be shipped?
[0088] In some embodiments, the server may generate a corresponding reply video based on the interactive information, and send the reply video to a display device.
[0089] S502: Play the reply video and obtain a first duration and a target timestamp.
[0090] The first duration is the playing duration of the reply video, and the target timestamp is the timestamp of starting to play the reply video.
[0091] In some embodiments, the method for obtaining the playback duration of the reply video, that is, the first duration, may be: when the video stream of the reply video is obtained, when decoding the video stream, each video frame will have a presentation time stamp (PTS), and the display time stamp value (pts_last) of the last frame of the video stream of the reply video and the PTS value (pts_00) of the first frame are obtained, and the calculation formula is as follows:
[0092]
[0093] Wherein, t is the first duration, in seconds.
[0094] In some embodiments, when the timestamp of playing the reply video is i, the target timestamp is i. According to the target timestamp i, the timestamp of stopping playing the live video when starting to play the reply video can be clearly obtained, so as to continue playing the live video from the target timestamp i, thereby preventing the audience from missing the live video content.
[0095] S503, obtaining an acceleration multiple; and obtaining a second duration according to the first duration and the acceleration multiple.
[0096] In some embodiments, after the display device obtains the reply video, it is necessary to stop the live broadcast of the original live broadcast room and play the reply video immediately. Therefore, after the reply video is played, the live broadcast of the original live broadcast room will also play the live broadcast of the same length as the reply video. Therefore, in order to ensure that the audience in the live broadcast room does not miss the playback content of the live broadcast room, it is necessary to catch up with the live broadcast progress to the current progress of the live broadcast room at double speed after the reply video.
[0097] S504: After finishing playing the reply video, playing the live video for the second duration at the acceleration multiple starting from the target timestamp.
[0098] In some embodiments, when the target timestamp is i, after the reply video of the first duration is played, the live video of the second duration is played at a double speed from the target timestamp i.
[0099] In some embodiments, the video streams of the reply video and the live video are processed asynchronously to improve efficiency.
[0100] The display device, server and live interactive method provided by the embodiment of the present application receive a reply video sent by the server through a communicator during the live video broadcast, wherein the reply video is a video related to the interactive information in the live video broadcast; the reply video is played through a controller, and a first duration and a target timestamp are obtained, wherein the first duration is the playback duration of the reply video, and the target timestamp is the timestamp at which the reply video starts to be played; an acceleration multiple is obtained; and a second duration is obtained according to the first duration and the acceleration multiple; finally, after the playback of the reply video is ended, the live video is played at the acceleration multiple times from the target timestamp for the second duration. The embodiment of the present application can continue to play the live video of the second duration from the target timestamp at the acceleration multiple after the reply video is played from the target timestamp, thereby avoiding the situation in which the audience misses the live video content due to the playback of the reply video during the live video broadcast. Therefore, the embodiment of the present application can improve the audience experience during the live video broadcast.
[0101] As an extension and refinement of the above embodiment, Figure 6 The following is a flow chart showing an exemplary method of live interactive broadcasting provided by an embodiment of the present application. Figure 6 As shown, the live interactive method provided in the embodiment of the present application includes the following steps S601 to S604:
[0102] S601. Receive a reply video sent by a server during a live video broadcast.
[0103] The reply video stream is a video related to the interactive information in the live video.
[0104] S602: Play the reply video and obtain a first duration and a target timestamp.
[0105] The first duration is the playing duration of the reply video, and the target timestamp is the timestamp of starting to play the reply video.
[0106] S603. Obtain the acceleration multiple according to the audio features of the live video broadcast and the live broadcast feature algorithm model; and obtain the second duration according to the first duration and the acceleration multiple.
[0107] Among them, the live broadcast feature algorithm model is a model obtained by training a preset machine learning model based on a live broadcast feature sample data set, and the live broadcast feature sample data includes: multiple live broadcast feature sample data, and the live broadcast feature sample data includes: sample audio features and the acceleration multiples corresponding to the sample audio features; the audio features include: tone features, speaking speed features, voice quality features, and at least one of voice frequency features.
[0108] In some embodiments, the audio features of the live video may be: the voice features of the host in the live video. After the reply video is played, when catching up with the current live broadcast progress, it is necessary to play the live video of the second length at a reasonable acceleration multiple to ensure that the audience in the live broadcast room can play the live video at a double speed while clearly perceiving the audio text of the live video through hearing.
[0109] Specifically, the acceleration multiple can be obtained through the audio features of the live video broadcast and the live broadcast feature algorithm model, wherein the audio features include: at least one of: tone features, speech speed features, voice quality features, and voice frequency features.
[0110] The tone feature is that when the video is accelerated, the rate of accelerated video playback can be determined by analyzing the change of tone; the speech rate feature is similar to the tone. When the video is accelerated, the rate of accelerated video playback can be determined by analyzing the change of speech rate; the voice quality feature is to determine the rate of accelerated video playback by analyzing the change of voice quality; the voice frequency feature is that when the video is accelerated, the audio frequency will become higher. The rate of accelerated video playback can be determined by analyzing the change of audio frequency.
[0111] In some embodiments, the preset machine learning model may be a machine learning algorithm model such as a deep learning neural network model, a convolutional neural network model, etc. The embodiment of the present invention does not limit the specific type of the live feature algorithm model.
[0112] In some embodiments, in the above step S603, the method for obtaining the second duration may include the following steps:
[0113] The second duration is obtained according to the acceleration multiple, the first duration, and the following formula:
[0114]
[0115] Wherein, T is the second duration, a is the acceleration multiple, and t is the first duration.
[0116] Exemplarily, when the playback time of the reply video is 2 minutes, that is, the first time length is 2 minutes and the acceleration multiple is 2 times, a=2, t=2 are substituted into the above formula for calculation, and the second time length T obtained is 4 minutes. It can be seen that after the 2-minute reply video is played, it is necessary to play the 4-minute original live video from the target timestamp at 2 times the speed to catch up with the current live progress.
[0117] In some embodiments, the method for obtaining the acceleration multiple may also be: receiving the acceleration multiple sent by the server.
[0118] Specifically, the acceleration multiple can be obtained on the server side, and then the server sends the acceleration multiple to the display device. After receiving the acceleration multiple, the display device obtains the first duration, determines the second duration, and completes the speed playback.
[0119] S604: After finishing playing the reply video, playing the live video for the second duration at the acceleration multiple starting from the target timestamp.
[0120] S605: Play the live video at a preset rate.
[0121] In some embodiments, after the playback of the reply video ends, the live video is played for the second duration starting from the target timestamp at the acceleration multiple, indicating that the progress of the live video broadcast in the current live broadcast room has been caught up, and then the live video can be played at the preset rate.
[0122] Figure 7 The flowchart of another live interactive method provided by the embodiment of the present application is shown as an example. Figure 7 As shown, the live interactive method provided in the embodiment of the present application includes the following steps S701 to S705:
[0123] S701: Receive interactive information sent by a display device.
[0124] The interactive information is: the interactive information input by the user during the live video broadcast.
[0125] In some embodiments, before receiving the interactive information sent by the display device, the display device also needs to obtain the interactive information. The method for obtaining the interactive information may be: collecting barrage in the barrage area of the live broadcast room, obtaining barrage in the form of questions, and sending it to the server as interactive information.
[0126] S702: Obtain a reply text corresponding to the interactive information.
[0127] In some embodiments, the method for obtaining the reply text corresponding to the interactive information may be: first obtaining the semantics of the interactive information, matching the reply text corresponding to the semantics in a preset question and answer library through the semantics, and if there is no matching reply text in the preset question and answer library, retrieving through an online search engine to obtain the reply text.
[0128] Among them, the preset question and answer library is generated based on big data, and the data may not be comprehensive. Some semantically corresponding reply texts are not included or set, so it is necessary to search through an online engine to obtain the optimal reply text.
[0129] In some embodiments, the method for obtaining the semantics of the interactive information may be: obtaining it through natural language processing (NPL), and then obtaining the reply text through semantics.
[0130] S703: Generate a reply audio based on the reply text.
[0131] In some embodiments, the method for generating a reply audio based on the reply text may be: converting the reply text into a voice format, namely the reply audio, through text-to-speech (TTS) technology.
[0132] Specifically, the speech synthesis technology usually includes two modules, the front-end and the back-end. The front-end module mainly analyzes the input text and extracts the linguistic information required by the back-end module. The front-end module generally includes sub-modules such as text regularization, word segmentation, part-of-speech prediction, polyphone disambiguation, and rhythm prediction. The back-end module generates speech waveforms through certain methods based on the front-end analysis results; the back-end module is generally divided into two main technical lines: speech synthesis based on statistical parameter modeling (Statistical Parameter Speech Synthesis, SPSS), and speech synthesis based on unit selection and waveform splicing.
[0133] In some embodiments, the voice factors corresponding to the reply text can be obtained through the front end of the above-mentioned speech synthesis technology; and the reply audio after speech synthesis can be obtained through the back end of the above-mentioned speech synthesis technology.
[0134] In some embodiments, the above-mentioned step S703, while generating a reply audio based on the reply text, can also obtain the host's timbre characteristics through the audio in the current live video broadcast, and generate a reply audio with the same timbre as the host, so that when the reply video is played, the audience can hear the audio with the same timbre as the host, thereby improving the interactive effect and further improving the audience's viewing experience.
[0135] S704: Obtain a reply video based on the reply audio.
[0136] In some embodiments, before obtaining the reply video based on the reply audio, a picture associated with the reply text can also be obtained based on keywords in the reply text; and the background of the portrait in each frame of the reply video is replaced with the associated picture.
[0137] Among them, replacing the background of the portrait in each frame of the reply video with the associated picture can be achieved by cutting out the area where the host image in each frame of the reply video is located through cutout processing, and using the associated picture as the background of the host image. For example, if the current live video is introducing a coat, the background of the host image can be replaced with a display picture of the coat, so that the audience of the live video can obtain more abundant relevant information when watching the reply video, thereby improving the audience's live viewing experience.
[0138] S705: Send the reply video to the display device.
[0139] In some embodiments, before sending the reply video to the display device, an intermediate frame can be generated based on the last frame and the first frame of any segment of the historical video of the live video to fill the gap in the reply video stream and ensure the continuity of the overall video when it is played in a loop.
[0140] In some embodiments, the server may include a central control module, which is used to obtain the display device corresponding to the interactive information, so as to accurately send the reply video corresponding to the interactive information to the display device that issued the interactive information, thereby ensuring that the audience in the live broadcast room can accurately obtain the required reply. The central control module can also obtain other reply videos related to the interactive information based on the interactive information, and send the other reply videos together to the display device corresponding to the interactive information, so that the audience can obtain richer and more useful replies, thereby improving the audience's viewing experience.
[0141] As an extension and refinement of the above embodiment, the live interactive method provided in the embodiment of the present application further includes the following steps 1 and 2:
[0142] Step 1: Obtain an acceleration multiple based on the audio features of the live video and the live feature algorithm model.
[0143] Among them, the live broadcast feature algorithm model is a model obtained by training a second preset machine learning model based on a live broadcast feature sample data set, and the live broadcast feature sample data includes: multiple live broadcast feature sample data, and the live broadcast feature sample data includes: sample audio features and the acceleration multiples corresponding to the sample audio features; the audio features include: tone features, speaking speed features, voice quality features, and at least one of voice frequency features.
[0144] Step 2: Send the acceleration multiple to the display device.
[0145] In some embodiments, the live broadcast feature algorithm model can also be deployed on the server side. After obtaining the acceleration multiple on the server side, the server sends the acceleration multiple to the corresponding display device, so that the display device can play the live broadcast video of the second length according to the acceleration multiple to catch up with the current live broadcast progress.
[0146] As an extension and refinement of the above embodiment, the live interactive method further includes the following steps:
[0147] S801. Obtain a three-dimensional model corresponding to the host of the live video broadcast.
[0148] In some embodiments, the three-dimensional model corresponding to the host of the live video broadcast can be a three-dimensional model corresponding to a virtual digital human obtained through pre-training. For example, the three-dimensional model of the corresponding virtual digital human is obtained through pre-training based on the image of a cartoon character familiar to the audience; it can also be based on the live broadcast of a real host, and the stable image of the real host in the video can be extracted at intervals of preset time periods as the real host image and cached, and then the corresponding three-dimensional model is generated based on the real host image.
[0149] Specifically, when the host of the live video broadcast is a real host, the method for obtaining the image of the real host can be: first, obtain the stable image of the host in the video by periodically extracting the host, and then detect whether the stable image of the host in the video is qualified, that is, determine whether the portrait of the host in the stable image in the video continues to exist during the live video broadcast, and determine whether the quality of the stable image of the host in the video meets the standard; if the portrait of the host in the stable image in the video continues to exist during the live video broadcast, and the stable image of the host in the video meets the standard, then determine that the stable image of the host in the video is qualified, so as to ensure that the quality of the reply video generated based on the host image in the subsequent time is higher.
[0150] S802: Obtain the facial key point coordinates and limb joint point coordinates of the three-dimensional model.
[0151] In some embodiments, the host's image can be divided into two parts: face and limbs. The facial key point coordinates are multiple key point coordinates of the host's face, for example, they can be 68 key point coordinates of the host's face; the limb joint points can be multiple joint point coordinates of the host's limbs. For example, when only the host's upper body appears in the live video interface, the limb joint point coordinates can be 8 limb joint point coordinates of the host's left upper limb (left palm, left wrist, left elbow and left shoulder) and right upper limb (right palm, right wrist, right elbow and right shoulder).
[0152] S803. Obtain a facial key point coordinate sequence according to the reply audio and the facial key point coordinates.
[0153] In some embodiments, the facial key point coordinate sequence can be obtained by combining the key point prediction model with the response audio.
[0154] S804, obtaining a limb joint point coordinate sequence according to the reply audio and the limb joint point coordinates.
[0155] In some embodiments, the limb joint point coordinate sequence can be obtained by combining the limb joint point prediction model with the response audio.
[0156] S805. Obtain the reply video based on the facial key point coordinate sequence, the limb joint point coordinate sequence and the reply audio.
[0157] In some embodiments, the host in the live video broadcast can be a virtual host or a real host. When the host in the live broadcast room is a virtual host, the reply video can be directly obtained through the virtual digital human driving model and the reply text, and the reply video can be sent to the display device.
[0158] In some embodiments, in the above step S805, based on the facial key point coordinate sequence, the limb joint point coordinate sequence and the reply audio, the method for obtaining the reply video may include the following steps:
[0159] The facial key point coordinate sequence and the limb joint point coordinate sequence corresponding to each other are driven based on the driving model to obtain the reply video.
[0160] Among them, the driving model is a model obtained by training a first preset machine learning model based on a driving sample data set, and the driving sample data set includes: multiple driving sample data; the driving sample data includes: a sample facial key point coordinate sequence, a sample limb joint point coordinate sequence, and a response video corresponding to the sample facial key point coordinate sequence and the sample limb joint point coordinate sequence.
[0161] Fig. 9 A flowchart of a live interactive method provided in an embodiment of the present application is shown. Fig. 9 As shown, including;
[0162] The receiving module 901 is used to receive the interactive information sent by the display device.
[0163] The reply audio acquisition module 902 is used to generate a reply audio based on the reply text corresponding to the interactive information.
[0164] The facial key point sequence acquisition module 903 is used to acquire the facial key point coordinate sequence corresponding to the reply text based on the anchor image and the reply text.
[0165] The body movement choreography module 904 is used to obtain a body joint point coordinate sequence corresponding to the reply text based on the host image and the reply text.
[0166] The driving module 905 is used to obtain the response video according to the response audio, the facial key point coordinate sequence and the limb joint point coordinate sequence in combination with the driving model.
[0167] The sending module 906 is used to send the reply video to a display device.
[0168] Fig.10 The interactive flow chart of the live interactive method provided by the embodiment of the present application is shown as follows: Fig.10 As shown, the live interactive method provided in the embodiment of the present application includes the following steps:
[0169] S1001. The display device sends interactive information to the server.
[0170] The interactive information is: the interactive information input by the user during the live video broadcast.
[0171] S1002: The server obtains a reply text corresponding to the interactive information.
[0172] S1003. The server generates a reply audio based on the reply text.
[0173] S1004. The server obtains a response video based on the response audio.
[0174] S1005. The server sends the reply video to the display device.
[0175] S1006: The display device plays the reply video and obtains a first duration and a target timestamp.
[0176] The first duration is the playing duration of the reply video, and the target timestamp is the timestamp of starting to play the reply video.
[0177] S1007. The display device obtains the acceleration multiple according to the audio features of the live video broadcast and the live broadcast feature algorithm model; and obtains the second duration according to the first duration and the acceleration multiple.
[0178] S1008. After finishing playing the reply video, playing the live video for the second duration at the acceleration multiple starting from the target timestamp.
[0179] For example, refer to Fig.11 As shown, Fig.11A logic block diagram of a live interactive method is provided based on the above-mentioned multiple embodiments, wherein, during a live video broadcast, an audience posts a question in a comment area or a bullet screen area, after obtaining interactive information during the live video broadcast, a semantic analysis is performed on the interactive information to obtain the semantics of the interactive information, and then a request is initiated to a server central control based on the semantic analysis result, wherein the request is used to generate a corresponding reply video in response to the semantic analysis result to feed back to the audience; after receiving the semantic analysis result, the server central control first obtains a reply text according to the semantic analysis result, and then obtains the voice phonemes of the reply text based on the reply text and TTS-Frontend (text-to-speech model front end), synthesizes a reply audio based on the voice phonemes, obtains a key point coordinate sequence based on the reply audio and a three-dimensional model, drives the key point coordinate sequence based on a driving model, and synthesizes a final reply video stream by the server central control, sends the video stream to a display device, and controls the playback by a playback controller of the display device, and catches up with the progress of the live video broadcast at a double speed.
[0180] In some embodiments, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the live interactive method described in any of the above embodiments.
[0181] In some embodiments, an embodiment of the present application provides a computer program product. When the computer program product runs on a computer, the computer implements the live interactive method described in any embodiment of the third aspect or the fourth aspect.
[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
[0183] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.
Claims
1. A display device, It is characterized in that include: The communicator is configured to: receive a reply video sent by a server during a live video broadcast, wherein the reply video is a video related to the interactive information in the live video broadcast; The controller is configured to: play the reply video and obtain a first duration and a target timestamp, wherein the first duration is the play duration of the reply video and the target timestamp is the timestamp for starting to play the reply video; Obtaining an acceleration multiple; and obtaining a second duration according to the first duration and the acceleration multiple; The display is configured to: after finishing playing the reply video, play the live video for the second duration at the acceleration multiple starting from the target timestamp.
2. The display device according to claim 1, It is characterized in that The controller is specifically configured as follows: Acquire the acceleration multiple according to the audio features of the live video broadcast and the live broadcast feature algorithm model; Among them, the live broadcast feature algorithm model is a model obtained by training a preset machine learning model based on a live broadcast feature sample data set, and the live broadcast feature sample data set includes: multiple live broadcast feature sample data, and the live broadcast feature sample data includes: sample audio features and the acceleration multiples corresponding to the sample audio features; the audio features include: tone features, speaking speed features, voice quality features, and at least one of voice frequency features.
3. The display device according to claim 1, It is characterized in that The controller is specifically configured as follows: The second duration is obtained according to the acceleration multiple, the first duration, and the following formula: Wherein, T is the second duration, a is the acceleration multiple, and t is the first duration.
4. The display device according to claim 1, It is characterized in that The communicator is further configured to receive the acceleration multiple sent by the server.
5. The display device according to claim 1, It is characterized in that The display, after playing the live video for the second duration at the acceleration multiple from the target timestamp, is further configured to: The live video is played at a preset rate.
6. A server, It is characterized in that include: The communicator is configured to: receive interactive information sent by the display device; The interactive information is: interactive information input by the user during the live video broadcast; The controller is configured to: obtain a reply text corresponding to the interactive information, generate a reply audio based on the reply text, and obtain a reply video based on the reply audio; The communicator is further configured to: send the reply video to the display device.
7. The server according to claim 6, It is characterized in that The controller is specifically configured as follows: Get the 3D model corresponding to the host of the live video broadcast; Obtaining the facial key point coordinates and limb joint point coordinates of the three-dimensional model; Acquire a facial key point coordinate sequence according to the reply audio and the facial key point coordinates; Acquire a limb joint point coordinate sequence according to the reply audio and the limb joint point coordinates; The reply video is obtained based on the facial key point coordinate sequence, the limb joint point coordinate sequence and the reply audio.
8. The server according to claim 6, It is characterized in that The controller is also configured to: Drive the facial key point coordinate sequence and the limb joint point coordinate sequence corresponding to each other based on the driving model to obtain the reply video; Among them, the driving model is a model obtained by training a first preset machine learning model based on a driving sample data set, and the driving sample data set includes: multiple driving sample data; the driving sample data includes: a sample facial key point coordinate sequence, a sample limb joint point coordinate sequence, and a response video corresponding to the sample facial key point coordinate sequence and the sample limb joint point coordinate sequence.
9. The server according to claim 6, It is characterized in that The controller is also configured to: Obtaining an acceleration multiple according to the audio features of the live video broadcast and a live broadcast feature algorithm model; The live broadcast feature algorithm model is a model obtained by training a second preset machine learning model based on a live broadcast feature sample data set, and the live broadcast feature sample data set includes: a plurality of live broadcast feature sample data, and the live broadcast feature sample data includes: a sample audio feature and an acceleration multiple corresponding to the sample audio feature; the audio feature includes: at least one of a tone feature, a speech speed feature, a voice quality feature, and a voice frequency feature; The communicator is further configured to send the acceleration multiple to the display device.
10. A live interactive method, It is characterized in that include: During the live video broadcast, a reply video sent by a server is received, wherein the reply video is a video related to the interactive information in the live video broadcast; Play the reply video and obtain a first duration and a target timestamp, wherein the first duration is the playback duration of the reply video and the target timestamp is the timestamp for starting to play the reply video; Obtaining an acceleration multiple; and obtaining a second duration according to the first duration and the acceleration multiple; After finishing playing the reply video, playing the live video for the second duration at the acceleration multiple starting from the target timestamp.
11. A live interactive method, It is characterized in that include: Receive interactive information sent by the display device; The interactive information is: interactive information input by the user during the live video broadcast; Obtaining a reply text corresponding to the interactive information; Generate a reply audio based on the reply text; Acquire a response video based on the response audio; The reply video is sent to the display device.
Citation Information
Cited By
Interaction method and device based on virtual character, electronic equipment and storage medium
CN121531149A
