Method and apparatus for processing live-streaming video, and device and storage medium

Abstract

Provided in the present disclosure are a method and apparatus for processing a live-streaming video, and a device and a storage medium. The method comprises: in response to a stream-viewing request carrying a preset auxiliary stream-viewing identifier, acquiring an audio clip in a live-streaming video stream corresponding to the stream-viewing request; extracting a keywords from a text fragment corresponding to the audio clip, and acquiring an auxiliary stream viewing video material corresponding to the keywords; on the basis of the auxiliary stream viewing video material corresponding to the keywords, generating an auxiliary stream-viewing video clip corresponding to the audio clip; and returning the auxiliary stream-viewing video clip for the stream-viewing request.

Description

Live video processing method, device, equipment and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202311121758.6, filed on August 31, 2023, entitled “A method, device, equipment and storage medium for processing live video”. The entire contents of that application are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of data processing, and in particular to a method, apparatus, device, and storage medium for processing live video. Background Art

[0004] With the continuous development of online live broadcast technology, video live broadcast has gradually become one of the main ways for people to obtain information and social entertainment. However, for users who usually use sign language to communicate, there is a situation where they cannot efficiently obtain live broadcast content.

[0005] Therefore, how to better meet the needs of efficiently understanding live broadcast content in live broadcast scenarios, thereby improving user experience, is a technical problem that urgently needs to be solved.

[0006] Summary of the Invention

[0007] In order to solve the above technical problems, the embodiments of the present disclosure provide a method for processing live video.

[0008] In a first aspect, the present disclosure provides a method for processing live video, the method comprising:

[0009] In response to a viewing request carrying a preset auxiliary viewing identifier, obtaining an audio clip in a live video stream corresponding to the viewing request;

[0010] Extracting keywords from the text segment corresponding to the audio segment, and obtaining auxiliary viewing video materials corresponding to the keywords;

[0011] Based on the auxiliary viewing and broadcasting video material corresponding to the keyword, generating an auxiliary viewing and broadcasting video segment corresponding to the audio segment;

[0012] The auxiliary viewing video segment is returned in response to the viewing request.

[0013] In an optional implementation, obtaining the auxiliary viewing video material corresponding to the keyword includes:

[0014] The auxiliary viewing and broadcasting video material corresponding to the keyword is obtained from the pre-established mapping relationship between the vocabulary and the auxiliary viewing and broadcasting video material.

[0015] In an optional implementation, the auxiliary viewing video segment includes an auxiliary viewing action based on a virtual image display, and the auxiliary viewing video segment is used to express the semantic information of the audio segment through the auxiliary viewing action.

[0016] In an optional implementation, the auxiliary viewing video clip is used to be played synchronously with the live video stream on the live broadcast room page.

[0017] In an optional implementation, generating the auxiliary viewing video clip corresponding to the audio clip based on the auxiliary viewing video material corresponding to the keyword includes:

[0018] Generate a joint animation based on the auxiliary viewing video material corresponding to the keyword; wherein the joint animation is used to represent the characteristics of the auxiliary viewing action in the auxiliary viewing video material;

[0019] The joint animation is associated with a preset virtual image model so that the virtual image corresponding to the preset virtual image model is displayed following the joint animation, and an auxiliary viewing video segment corresponding to the audio segment is obtained.

[0020] In an optional embodiment, associating the joint animation with a preset avatar model so that the avatar corresponding to the preset avatar model is displayed following the joint animation, and obtaining the auxiliary viewing video segment corresponding to the audio segment includes:

[0021] Associating the joint animation with a preset avatar model;

[0022] Determining a key frame of the avatar corresponding to the preset avatar model in the joint animation;

[0023] An auxiliary video segment corresponding to the audio segment is generated based on the key frame.

[0024] In an optional implementation, before generating the auxiliary viewing video segment corresponding to the audio segment based on the key frame, the method further includes:

[0025] calibrating display position information of the virtual image between adjacent key frames to obtain calibrated key frames;

[0026] Accordingly, the step of generating the auxiliary video segment corresponding to the audio segment based on the key frame includes:

[0027] An auxiliary viewing video segment corresponding to the audio segment is generated based on the calibrated key frame; wherein the virtual image in the auxiliary viewing video segment follows the calibrated key frame to display the auxiliary viewing action.

[0028] In an optional embodiment, calibrating the display position information of the virtual image between adjacent key frames to obtain calibrated key frames includes:

[0029] The display position information of the virtual image between adjacent key frames is determined by an interpolation algorithm to obtain a calibrated key frame.

[0030] In an optional embodiment, before associating the joint animation with a preset avatar model so that the avatar corresponding to the preset avatar model is displayed following the joint animation and obtaining the auxiliary viewing video segment corresponding to the audio segment, the method further includes:

[0031] The preset virtual image model is used to generate a virtual image corresponding to the anchor of the live video stream.

[0032] In an optional implementation manner, before returning the auxiliary viewing video segment in response to the viewing request, the method further includes:

[0033] Determining a text segment corresponding to the audio segment as subtitle content of the audio segment;

[0034] The subtitle content corresponding to the audio segment is returned in response to the broadcast request; wherein the subtitle content is used to be displayed synchronously with the auxiliary broadcast video segment on the live broadcast room page.

[0035] In a second aspect, the present disclosure provides a live video processing device, the device comprising:

[0036] an acquisition module, configured to respond to a viewing request carrying a preset auxiliary viewing identifier and acquire an audio segment in a live video stream corresponding to the viewing request;

[0037] An extraction module, configured to extract keywords from the text segment corresponding to the audio segment, and obtain auxiliary viewing video materials corresponding to the keywords;

[0038] A generating module, configured to generate an auxiliary viewing and broadcasting video segment corresponding to the audio segment based on the auxiliary viewing and broadcasting video material corresponding to the keyword;

[0039] The returning module is configured to return the auxiliary viewing video segment in response to the viewing request.

[0040] In a third aspect, the present disclosure provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium. When the instructions are executed on a terminal device, the terminal device implements the above method.

[0041] In a fourth aspect, the present disclosure provides a live video processing device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method when executing the computer program.

[0042] In a fifth aspect, the present disclosure provides a computer program product, which includes a computer program / instructions, and the computer program / instructions implement the above method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0044] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0045] FIG1 is a flow chart of a method for processing live video provided by an embodiment of the present disclosure;

[0046] FIG2 is a flowchart of another method for processing live video provided by an embodiment of the present disclosure;

[0047] FIG3 is a schematic structural diagram of a live video processing device provided by an embodiment of the present disclosure;

[0048] FIG4 is a schematic structural diagram of a live video processing device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that, in the absence of conflict, the embodiments of the present disclosure and the features therein can be combined with each other.

[0050] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0051] With the continuous development of online live broadcast technology, video live broadcast has gradually become one of the main ways for people to obtain information and social entertainment. However, for target users who usually use sign language to communicate, there is a situation where they cannot efficiently obtain live broadcast content.

[0052] Therefore, how to meet the needs of target users to understand the live broadcast content in the live broadcast scenario and thus improve the user experience is a technical problem that needs to be solved urgently.

[0053] To this end, the embodiment of the present disclosure provides a method for processing live video, which responds to a viewing request carrying a preset auxiliary viewing identifier, obtains an audio segment in the live video stream corresponding to the viewing request; extracts keywords in the text segment corresponding to the audio segment, and obtains auxiliary viewing video materials corresponding to the keywords; generates an auxiliary viewing video segment corresponding to the audio segment based on the auxiliary viewing video materials corresponding to the keywords; and returns the auxiliary viewing video segment in response to the viewing request. It can be seen that the embodiment of the present disclosure can generate an auxiliary viewing video segment corresponding to the audio segment based on the keywords corresponding to the audio segment in the live video stream and the auxiliary viewing video materials, and return the auxiliary viewing video segment, which can facilitate users to understand the live content based on the auxiliary viewing video segment, thereby improving the user experience.

[0054] Based on this, an embodiment of the present disclosure provides a method for processing live video. Referring to FIG1 , which is a flow chart of a method for processing live video provided by an embodiment of the present disclosure, the method specifically includes:

[0055] S101: In response to a viewing request carrying a preset auxiliary viewing identifier, an audio segment in a live video stream corresponding to the viewing request is obtained.

[0056] The live video processing method provided by the embodiments of the present disclosure can be applied to a server, where the server may include a cloud server, a local server, etc.

[0057] In the embodiment of the present disclosure, if a client user triggers a viewing request for a live broadcast room while browsing media content, the client sends a viewing request carrying a preset auxiliary viewing identifier to the server, which is used to request an auxiliary viewing video clip corresponding to the viewing request from the server.

[0058] In the disclosed embodiments, the preset auxiliary viewing identifier carried in the viewing request can be used to indicate whether the client needs to enable the function of synchronously displaying auxiliary viewing actions (sign language) on the live broadcast room page corresponding to the target live broadcast room. By enabling the sign language auxiliary function, the client user can send a viewing request carrying the preset auxiliary viewing identifier to the server to request the audio clip of the live video stream corresponding to the viewing request.

[0059] Among them, sign language is a language that communicates through gestures, which uses gestures to simulate images or syllables to express certain meanings or words.

[0060] In the embodiment of the present disclosure, after receiving a viewing request carrying a preset auxiliary viewing identifier, the server searches for an audio segment in the live video stream corresponding to the viewing request so that keywords in a text segment corresponding to the audio segment can be subsequently extracted based on the audio segment.

[0061] In the embodiment of the present disclosure, the audio clip belongs to the audio information stream corresponding to the live video stream. Subsequently, the voice information in the audio clip can be analyzed to generate an auxiliary video clip that can be used to represent the voice information of the audio clip.

[0062] In an optional implementation, after receiving a viewing request carrying a preset auxiliary viewing identifier, the server can first pull the video clip corresponding to the live broadcast room from the live broadcast server, and then call a video processing tool to extract the audio clip corresponding to the video clip from the video clip.

[0063] In another optional implementation, after receiving a viewing request carrying a preset auxiliary viewing identifier, the server can also directly pull the audio information stream corresponding to the live broadcast room from the live broadcast server, and then determine the audio segment based on the audio information stream, so that the corresponding auxiliary viewing video segment can be generated based on the audio segment.

[0064] S102: extracting keywords from the text segment corresponding to the audio segment, and obtaining auxiliary viewing and broadcasting video materials corresponding to the keywords.

[0065] The text segment is obtained by performing speech recognition on the audio segment.

[0066] In practical applications, audio segments can be converted into text segments based on speech recognition technology, so as to subsequently extract keywords in the text segments corresponding to the audio segments. The embodiments of the present disclosure do not limit the implementation method of speech-to-text conversion.

[0067] In an optional implementation, before extracting keywords from the text segment corresponding to the audio segment, the text segment can also be segmented. In actual applications, for text segments in English style, spaces can be directly used for word segmentation, while for text segments in Chinese style, a third-party library can be used for word segmentation. After word segmentation, the text segment can more accurately extract keywords from the text segment.

[0068] For example, first perform speech recognition on the audio clip. Suppose the text clip obtained by speech recognition is "Nice to meet you", then perform word segmentation on the text clip to obtain the words "very, happy, to meet, you", and then extract keywords from the words "very, happy, to meet, you" to obtain the keywords "happy, to meet, you" corresponding to the text clip, which are used to subsequently determine the corresponding auxiliary viewing video materials based on the keywords.

[0069] In the embodiment of the present disclosure, after extracting the keywords in the text segment corresponding to the audio segment, it is also necessary to obtain the auxiliary viewing and broadcasting video materials corresponding to the keywords.

[0070] In an optional implementation, there is a mapping relationship between vocabulary and auxiliary viewing video materials, wherein the auxiliary viewing video materials may include auxiliary viewing actions displayed by the target person. In actual applications, the auxiliary viewing actions displayed in the auxiliary viewing video materials can be used to represent the meaning of the vocabulary.

[0071] In an optional implementation, a mapping relationship between the sign language vocabulary and the auxiliary viewing video material can be established in advance based on the sign language vocabulary and sign language videos in the preset sign language data set, so that when the keywords in the text segment corresponding to the audio segment are subsequently extracted, the auxiliary viewing video material corresponding to the keyword can be obtained based on the pre-established mapping relationship between the vocabulary and the auxiliary viewing video material.

[0072] S103: Based on the auxiliary viewing and broadcasting video material corresponding to the keyword, generate an auxiliary viewing and broadcasting video segment corresponding to the audio segment.

[0073] In an optional implementation, the auxiliary viewing video clip includes an auxiliary viewing action based on a virtual image display, and the auxiliary viewing action is used to represent the semantic information of the audio clip.

[0074] In an embodiment of the present disclosure, after obtaining the audio segment in the live video stream of the live broadcast room, the server can determine the corresponding auxiliary viewing video segment based on the audio segment, so that the user can understand the semantic information of the audio segment based on the auxiliary viewing video segment without having to hear the audio segment.

[0075] In the disclosed embodiments, the assisted viewing video clips include assisted viewing actions based on a virtual avatar. A virtual avatar refers to an image created on a physical display screen or in a virtual space using computer 2D rendering or 3D modeling technology. The image may include a virtual human, animal, or cartoon character. Specifically, the virtual avatar may be a two-dimensional or three-dimensional virtual image. The virtual avatar in the assisted viewing video clips may be the virtual avatar of the host in the live broadcast room.

[0076] In actual applications, the virtual image can be controlled to simulate human body movements or postures, etc., so as to realize the function of performing auxiliary viewing actions through the virtual image and generating corresponding auxiliary viewing video clips for interaction. For example, assuming that the audio clip in the live video stream of the live broadcast room obtained includes "Hello", the virtual image can be controlled to perform the auxiliary viewing action of "Hello" to generate an auxiliary viewing video clip corresponding to the audio clip "Hello", and play it synchronously on the live broadcast room page, thereby realizing interaction with the live broadcast viewing users.

[0077] In actual applications, audio clips can also be converted into auxiliary video clips based on a preset algorithm, so that users of the target client can understand the detailed live content based on the auxiliary video clips. The specific implementation process is introduced in the following embodiments and will not be repeated here.

[0078] S104: Return the auxiliary viewing video segment in response to the viewing request.

[0079] In the disclosed embodiment, the auxiliary viewing video clip is used to be played synchronously with the live video stream on the live broadcast room page.

[0080] In the embodiment of the present disclosure, after the server determines the auxiliary viewing video segment corresponding to the audio segment, it can return the auxiliary viewing video segment to the client; after receiving the auxiliary viewing video segment corresponding to the audio segment, the client plays the auxiliary viewing video segment synchronously with the live video stream on the live room page.

[0081] In the disclosed embodiment, since the auxiliary viewing video clip includes auxiliary viewing actions displayed based on a virtual image, and the auxiliary viewing actions can be used to represent the semantic information of the audio clip, therefore, during the process of synchronously playing the auxiliary viewing video clip corresponding to the audio clip on the live broadcast room page, the user of the target client does not need to hear the live broadcast content, but can understand the detailed live broadcast content based on the auxiliary viewing actions displayed by the virtual image in the auxiliary viewing video clip without hearing the live broadcast content.

[0082] In an optional implementation, in order to further enrich the functions related to live video and meet more user needs, not only the auxiliary viewing video clip can be returned for the viewing request, but also the subtitle content corresponding to the audio clip can be returned for the viewing request, so that the subtitle content can be displayed synchronously on the live broadcast page to assist users in understanding the live broadcast content.

[0083] Specifically, the server first determines the text segment corresponding to the audio segment as the subtitle content of the audio segment. In actual applications, the audio segment can be converted into a text segment by calling the speech-to-text interface, and then used as the subtitle content of the audio segment for subsequent synchronous display on the live broadcast room page. The specific method of converting the audio segment into a text segment is not limited in the embodiments of this disclosure.

[0084] After determining the subtitle content corresponding to the audio clip, the subtitle content corresponding to the audio clip is returned in response to the viewing request. After receiving the subtitle content, the client can simultaneously display the subtitle content and the auxiliary viewing video clip on the live broadcast room page, so that the user can understand the live broadcast content based on the subtitle content and the auxiliary viewing video clip.

[0085] In the embodiment of the present disclosure, the display style and display position of the subtitle content on the live broadcast room page can be set based on actual needs. For example, it can be displayed at a preset position below the live broadcast room page, etc. The embodiment of the present disclosure does not limit this.

[0086] In the live video processing method provided by the embodiment of the present disclosure, in response to a viewing request carrying a preset auxiliary viewing identifier, an audio segment in the live video stream corresponding to the viewing request is obtained; keywords in the text segment corresponding to the audio segment are extracted, and auxiliary viewing video materials corresponding to the keywords are obtained; based on the auxiliary viewing video materials corresponding to the keywords, an auxiliary viewing video segment corresponding to the audio segment is generated; and the auxiliary viewing video segment is returned in response to the viewing request. It can be seen that the embodiment of the present disclosure can generate an audio segment based on the keywords and auxiliary viewing video materials corresponding to the audio segment in the live video stream, convert it into a corresponding auxiliary viewing video segment, and return the auxiliary viewing video segment in response to the viewing request, which can facilitate users to understand the live content based on the auxiliary viewing video segment, thereby improving the user experience.

[0087] In actual applications, in order to enable the virtual image to more accurately complete the auxiliary viewing action, the embodiment of the present disclosure also provides a live video processing method, which can better complete the association between the virtual image and the precise action, so that the user can more accurately understand the live content based on the auxiliary viewing action displayed by the virtual image. In an optional implementation, known software can be used to implement the function of converting audio clips into auxiliary viewing video clips, thereby better completing the association between the virtual image and the precise action, so that the virtual image can more accurately complete the auxiliary viewing action.

[0088] In an embodiment of the present disclosure, after obtaining the auxiliary viewing video material corresponding to the keyword, a joint animation can be generated based on the auxiliary viewing video material corresponding to the keyword, wherein the joint animation can be used to characterize the characteristics of the auxiliary viewing action in the auxiliary viewing video material.

[0089] In actual applications, the specific implementation of generating joint animation based on the auxiliary viewing video material corresponding to the keyword can be to capture the motion of the auxiliary viewing video material corresponding to the keyword based on a known algorithm, generate the corresponding joint animation based on the captured motion, and then associate the joint animation with the preset virtual image model so that the virtual image corresponding to the preset virtual image model follows the joint animation for display.

[0090] In the disclosed embodiment, joint animation can be added to a preset virtual image model and the joint animation can be associated with the preset virtual image model so that the virtual image corresponding to the preset virtual image model can be associated with the joint animation, thereby generating an auxiliary viewing video clip including auxiliary viewing actions displayed by the virtual image.

[0091] In an optional implementation, the virtual image corresponding to the preset virtual image model can be converted into an auxiliary viewing video clip including auxiliary viewing actions displayed by the virtual image based on rendering technology in computer graphics, such as rasterization rendering.

[0092] In practical applications, in order to make the auxiliary viewing and broadcasting actions displayed by the virtual image in the auxiliary viewing and broadcasting video clips more coherent, after associating the joint animation with the preset virtual image model, the key frames of the virtual image corresponding to the preset virtual image model in the joint animation are first determined, and then the display position information of the virtual image between adjacent key frames is calibrated to obtain the calibrated key frames. Finally, the auxiliary viewing and broadcasting video clips are generated based on the calibrated key frames, wherein the virtual image in the auxiliary viewing and broadcasting video clips displays the auxiliary viewing and broadcasting actions following the calibrated key frames.

[0093] It can be seen that in the embodiment of the present disclosure, by calibrating the display position information of the virtual image between adjacent key frames, the auxiliary viewing actions displayed by the virtual image in the auxiliary viewing video clip are more coherent, which further facilitates the user to intuitively understand the live broadcast content based on the auxiliary viewing video clip, thereby improving the user experience.

[0094] In an optional implementation, before associating the joint animation with the preset virtual image model, a preset virtual image model can be pre-created based on known software, and by adjusting the points, lines, surfaces and other parameters of each part of the preset virtual image model, after the preset virtual image model is associated with the joint animation, it can more accurately follow the joint animation to display the auxiliary viewing action.

[0095] It can be seen that in the live video processing method provided by the embodiment of the present disclosure, by extracting keywords in the text segment corresponding to the audio segment, and obtaining the auxiliary viewing video material corresponding to the keyword from the pre-established mapping relationship between the vocabulary and the auxiliary viewing video material, and generating the auxiliary viewing video segment based on the auxiliary viewing video material corresponding to the keyword, it is possible to facilitate users to understand the live broadcast content based on the auxiliary viewing video segment, thereby further improving the user experience.

[0096] Based on the content of the above embodiments, the present disclosure provides a method for processing live video, which is applied to a target client. FIG2 is a flowchart of another method for processing live video provided by the present disclosure. The method includes:

[0097] S201: Sending a viewing request carrying a preset auxiliary viewing identifier to a server.

[0098] The viewing request is used to request generation of an auxiliary viewing video segment for an audio segment in a live video stream. The auxiliary viewing video segment includes an auxiliary viewing action based on a virtual image display. The auxiliary viewing action is used to represent semantic information of the audio segment.

[0099] The live video processing method provided by the embodiments of the present disclosure can be applied to a target client. For example, the target client may include a client deployed on a smart phone, a client deployed on a tablet computer, etc.

[0100] In the disclosed embodiment, when a user is browsing media content, if a live viewing request is triggered for a target live broadcast room, the target client where the user is located sends a viewing request to the server to request a live video stream from the server.

[0101] In the embodiment of the present disclosure, the preset auxiliary viewing identifier can be used to indicate whether the target client needs to enable the function of synchronously displaying the auxiliary viewing action on the live broadcast room page corresponding to the target live broadcast room. After the function of the preset auxiliary viewing identifier is enabled, the preset auxiliary viewing identifier can be carried when sending a viewing request to the server, and is used to request the server for the auxiliary viewing video clip corresponding to the live video stream. The user can learn about the live content of the live broadcast room through the auxiliary viewing video clip.

[0102] In the embodiment of the present disclosure, after receiving a viewing request from a target live broadcast room, the server can obtain an audio segment in the live video stream of the target live broadcast room based on a preset auxiliary viewing identifier carried in the viewing request, determine the auxiliary viewing video segment corresponding to the audio segment, and return the auxiliary viewing video segment to the target client.

[0103] S202: Receive the auxiliary viewing video clip, and play the auxiliary viewing video clip synchronously with the live video stream on the live broadcast room page corresponding to the target live broadcast room.

[0104] In the disclosed embodiment, after receiving the auxiliary viewing video clip returned by the server, the target service client plays the auxiliary viewing video clip synchronously with the live video stream on the live room page corresponding to the target live room.

[0105] In the disclosed embodiment, since the auxiliary viewing video clip includes auxiliary viewing actions displayed based on the virtual image, and the auxiliary viewing actions can be used to represent the semantic information of the audio clip, therefore, on the live broadcast room page corresponding to the target live broadcast room, during the process of playing the video clip synchronously with the live video stream, the user does not need to hear the live broadcast content, but can understand the specific live broadcast content based on the auxiliary viewing actions displayed by the virtual image in the auxiliary viewing video clip.

[0106] In an optional implementation, in order to further enrich the functions related to live video and meet more user needs, the server can not only return the auxiliary video clips to the target client, but also return the subtitle content corresponding to the audio clips to the target client, so as to synchronously display the subtitle content on the live broadcast page of the target client to assist users in understanding the live broadcast content.

[0107] In the embodiment of the present disclosure, after the target client receives the auxiliary viewing video clip and subtitle content corresponding to the audio clip, it can simultaneously display the subtitle content and the auxiliary viewing video clip on the live broadcast room page corresponding to the target live broadcast room, so that the user of the target client can understand the live broadcast content based on the subtitle content and the auxiliary viewing video clip.

[0108] In the live video processing method provided by the embodiment of the present disclosure, a viewing request carrying a preset auxiliary viewing identifier is first sent to the server, wherein the viewing request is used to request the generation of an auxiliary viewing video segment for the audio segment in the live video stream of the target live room; after receiving the auxiliary viewing video segment sent by the server, the auxiliary viewing video segment is played synchronously with the live video stream on the live room page corresponding to the target live room. It can be seen that the embodiment of the present disclosure can facilitate users to understand the live content based on the auxiliary viewing video segment by synchronously displaying the auxiliary viewing video segment corresponding to the received audio segment on the live room page corresponding to the target live room, thereby improving the user experience.

[0109] Based on the above method embodiment, the present disclosure further provides a live video processing device. Referring to FIG3 , which is a structural diagram of a live video processing device provided by an embodiment of the present disclosure, the device includes:

[0110] The acquisition module 301 is configured to respond to a viewing request carrying a preset auxiliary viewing identifier and acquire an audio segment in a live video stream corresponding to the viewing request;

[0111] Extraction module 302, for extracting keywords from the text segment corresponding to the audio segment, and obtaining auxiliary viewing and broadcasting video materials corresponding to the keywords;

[0112] A generating module 303 is configured to generate an auxiliary viewing and broadcasting video segment corresponding to the audio segment based on the auxiliary viewing and broadcasting video material corresponding to the keyword;

[0113] The returning module 304 is configured to return the auxiliary viewing video segment in response to the viewing request.

[0114] In an optional implementation, the acquisition module includes:

[0115] The acquisition submodule is used to acquire the auxiliary viewing and broadcasting video material corresponding to the keyword from the pre-established mapping relationship between the vocabulary and the auxiliary viewing and broadcasting video material.

[0116] In an optional implementation, the auxiliary viewing video segment includes an auxiliary viewing action based on a virtual image display, and the auxiliary viewing video segment is used to express the semantic information of the audio segment through the auxiliary viewing action.

[0117] In an optional implementation, the auxiliary viewing video clip is used to be played synchronously with the live video stream on the live broadcast room page.

[0118] In an optional implementation, the generating module includes:

[0119] The first generating submodule is configured to generate a joint animation based on the auxiliary viewing video material corresponding to the keyword; wherein the joint animation is used to characterize the characteristics of the auxiliary viewing action in the auxiliary viewing video material;

[0120] The display submodule is used to associate the joint animation with a preset virtual image model so that the virtual image corresponding to the preset virtual image model follows the joint animation and displays the auxiliary viewing video segment corresponding to the audio segment.

[0121] In an optional embodiment, the display submodule includes:

[0122] an association submodule, for associating the joint animation with a preset avatar model;

[0123] a determination submodule, configured to determine a key frame of the avatar corresponding to the preset avatar model in the joint animation;

[0124] The second generating submodule is configured to generate an auxiliary viewing video segment corresponding to the audio segment based on the key frame.

[0125] In an optional embodiment, the display submodule further includes:

[0126] a calibration submodule, configured to calibrate display position information of the virtual image between adjacent key frames to obtain calibrated key frames;

[0127] Accordingly, the second generation submodule is specifically used to:

[0128] An auxiliary viewing video segment corresponding to the audio segment is generated based on the calibrated key frame; wherein the virtual image in the auxiliary viewing video segment follows the calibrated key frame to display the auxiliary viewing action.

[0129] In an optional implementation, the calibration submodule includes:

[0130] The display submodule is used to determine the display position information of the virtual image between adjacent key frames through an interpolation algorithm to obtain a calibrated key frame.

[0131] In an optional embodiment, the device further includes:

[0132] The virtual image generation module is used to use the preset virtual image model to generate a virtual image corresponding to the anchor of the live video stream.

[0133] In an optional embodiment, the device further includes:

[0134] A text segment generating module, configured to determine a text segment corresponding to the audio segment as subtitle content of the audio segment;

[0135] The subtitle content returning module is used to return the subtitle content corresponding to the audio segment in response to the viewing request; wherein, the subtitle content is used to be displayed synchronously with the auxiliary viewing video segment on the live broadcast room page.

[0136] In the live video processing device provided by the embodiment of the present disclosure, in response to a viewing request carrying a preset auxiliary viewing identifier, an audio segment in the live video stream corresponding to the viewing request is obtained; keywords in the text segment corresponding to the audio segment are extracted, and auxiliary viewing video materials corresponding to the keywords are obtained; based on the auxiliary viewing video materials corresponding to the keywords, an auxiliary viewing video segment corresponding to the audio segment is generated; and the auxiliary viewing video segment is returned in response to the viewing request. It can be seen that the embodiment of the present disclosure can generate an auxiliary viewing video segment corresponding to the audio segment based on the keywords corresponding to the audio segment in the live video stream and the auxiliary viewing video materials, and return the auxiliary viewing video segment, which can facilitate users to understand the live content based on the auxiliary viewing video segment, thereby improving the user experience.

[0137] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure also provide a computer-readable storage medium, which stores instructions. When the instructions are executed on a terminal device, the terminal device implements the live video processing method described in the embodiments of the present disclosure.

[0138] The embodiments of the present disclosure further provide a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the live video processing method described in the embodiments of the present disclosure is implemented.

[0139] In addition, the embodiment of the present disclosure further provides a live video processing device, as shown in FIG4 , which may include:

[0140] Processor 401, memory 402, input device 403, and output device 404. The number of processors 401 in the live video processing device can be one or more, and Figure 4 uses one processor as an example. In some embodiments of the present disclosure, processor 401, memory 402, input device 403, and output device 404 can be connected via a bus or other means, wherein Figure 4 uses the bus connection as an example.

[0141] Memory 402 can be used to store software programs and modules. Processor 401 executes the various functional applications and data processing of the live video processing device by running the software programs and modules stored in memory 402. Memory 402 may primarily include a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application required for a function, and the like. Furthermore, memory 402 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Input device 403 may be used to receive input digital or character information and generate signal input related to user settings and function control of the live video processing device.

[0142] Specifically in this embodiment, the processor 401 will load the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 will run the applications stored in the memory 402, thereby realizing the various functions of the above-mentioned live video processing device.

[0143] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0144] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing live video, comprising: In response to a viewing request carrying a preset auxiliary viewing identifier, obtaining an audio segment in a live video stream corresponding to the viewing request; Extracting keywords from the text segment corresponding to the audio segment, and obtaining auxiliary viewing video materials corresponding to the keywords; Based on the auxiliary viewing and broadcasting video material corresponding to the keyword, generating an auxiliary viewing and broadcasting video segment corresponding to the audio segment; The auxiliary viewing video segment is returned in response to the viewing request.

2. The method according to claim 1, wherein obtaining the auxiliary video material corresponding to the keyword comprises: The auxiliary viewing and broadcasting video material corresponding to the keyword is obtained from the pre-established mapping relationship between the vocabulary and the auxiliary viewing and broadcasting video material.

3. The method according to claim 1, wherein the auxiliary viewing video segment includes an auxiliary viewing action based on a virtual image display, and the auxiliary viewing video segment is used to express the semantic information of the audio segment through the auxiliary viewing action.

4. The method according to claim 1, wherein the auxiliary viewing video clip is used to be played synchronously with the live video stream on the live broadcast room page.

5. The method according to claim 1, wherein generating the auxiliary viewing video clip corresponding to the audio clip based on the auxiliary viewing video material corresponding to the keyword comprises: Generate a joint animation based on the auxiliary viewing video material corresponding to the keyword; wherein the joint animation is used to characterize the characteristics of the auxiliary viewing action in the auxiliary viewing video material; The joint animation is associated with a preset virtual image model so that the virtual image corresponding to the preset virtual image model follows the joint animation for display, and an auxiliary viewing video segment corresponding to the audio segment is obtained.

6. The method according to claim 5, wherein the step of associating the joint animation with a preset virtual image model so that the virtual image corresponding to the preset virtual image model follows the joint animation for display, and obtaining the auxiliary viewing video segment corresponding to the audio segment comprises: Associating the joint animation with a preset avatar model; Determining a key frame of the virtual image corresponding to the preset virtual image model in the joint animation; An auxiliary video segment corresponding to the audio segment is generated based on the key frame.

7. The method according to claim 6, wherein before generating the auxiliary video segment corresponding to the audio segment based on the key frame, it also includes: Calibrate the display position information of the virtual image between adjacent key frames to obtain calibrated key frames; Correspondingly, the step of generating the auxiliary video segment corresponding to the audio segment based on the key frame includes: An auxiliary viewing video segment corresponding to the audio segment is generated based on the calibrated key frame; wherein the virtual image in the auxiliary viewing video segment follows the calibrated key frame to display the auxiliary viewing action.

8. The method according to claim 7, wherein calibrating the display position information of the virtual image between adjacent key frames to obtain the calibrated key frames comprises: The display position information of the virtual image between adjacent key frames is determined by an interpolation algorithm to obtain a calibrated key frame.

9. The method according to claim 5, wherein the step of associating the joint animation with a preset virtual image model so that the virtual image corresponding to the preset virtual image model follows the joint animation for display, and obtaining the auxiliary viewing video segment corresponding to the audio segment, further comprises: The preset virtual image model is used to generate a virtual image corresponding to the anchor of the live video stream.

10. The method according to claim 1, wherein before returning the auxiliary viewing video segment in response to the viewing request, the method further comprises: Determining a text segment corresponding to the audio segment as subtitle content of the audio segment; The subtitle content corresponding to the audio clip is returned in response to the viewing request; wherein the subtitle content is used to be displayed synchronously with the auxiliary viewing video clip on the live broadcast room page.

11. A live video processing device, comprising: An acquisition module, configured to respond to a viewing request carrying a preset auxiliary viewing identifier and acquire an audio segment in a live video stream corresponding to the viewing request; An extraction module, used to extract keywords from the text segment corresponding to the audio segment, and obtain auxiliary viewing and broadcasting video materials corresponding to the keywords; A generating module, configured to generate an auxiliary viewing and broadcasting video segment corresponding to the audio segment based on the auxiliary viewing and broadcasting video material corresponding to the keyword; The returning module is used to return the auxiliary viewing video segment in response to the viewing request.

12. A computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device implements the method according to any one of claims 1 to 10.

13. A live video processing device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 10 is implemented.

Images