Live video editing methods and devices

Abstract

This application provides a live video editing method, which includes: acquiring target modal data of the live video to be edited, wherein the target modal data includes at least audio data and image data; performing multi-dimensional analysis on the live video based on the target modal data, and determining highlights in the live video based on the analysis results; editing the highlights, and obtaining an edited video based on the editing results. The technical solution of this application can automatically edit highlights in live videos, improving the efficiency and accuracy of live video editing.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a live video editing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Live stream videos often contain some exciting segments, such as the streamer singing or dancing. If these exciting segments are to be shared further, they need to be manually edited; otherwise, they will lose attention once the live stream ends.

[0003] However, manually editing segments of live video is time-consuming and laborious, and is prone to problems such as omissions or labeling errors, resulting in low efficiency and accuracy.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention

[0005] This application provides a live video editing method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems mentioned above.

[0006] One aspect of this application provides a live video editing method, the method comprising:

[0007] Obtain the target modal data of the live video to be edited, wherein the target modal data includes at least audio data and image data;

[0008] The live video is analyzed from multiple dimensions based on the target modal data, and the highlights in the live video are determined based on the analysis results.

[0009] The aforementioned highlights are edited, and an edited video is obtained based on the editing results.

[0010] Optionally, the step of performing multi-dimensional analysis on the live video based on the target modal data, and determining the highlights of the live video based on the analysis results, includes:

[0011] The first target content of the live video is identified based on the audio data, and first identification information is generated based on the first target content. The first identification information is used to identify the segment corresponding to the first target content.

[0012] The second target content of the live video is identified based on the image data, and second identification information is generated based on the second target content. The second identification information is used to identify the segment corresponding to the second target content.

[0013] Time alignment is performed on the first and second identification information;

[0014] The start time and tag information of the highlight segment are determined based on the first and second identifier information after time alignment.

[0015] Optionally, identifying the first target content of the live video based on the audio data includes:

[0016] Audio features are extracted based on the audio data, and the audio features include at least Mel frequency cepstral coefficients and chromaticity features;

[0017] Based on the audio features, a deep learning model is used to identify the first target content of the live video.

[0018] Optionally, the step of identifying the second target content of the live video based on the image data includes:

[0019] Based on the image data, optical flow estimation, motion recognition, and / or human pose estimation are used to identify the second target content of the live video.

[0020] Optionally, the method further includes:

[0021] Obtain the keywords of the edited video content;

[0022] Generative artificial intelligence is used to generate the cover and title of the edited video based on the content keywords.

[0023] Optionally, after obtaining the edited video from the editing results, the method further includes:

[0024] Determine the category of the edited video;

[0025] A program list is generated based on the category, the program list including the program order;

[0026] The target video is synthesized based on the program list and then published.

[0027] Optionally, the method further includes:

[0028] When playing the target video, obtain the background of the broadcaster corresponding to the target video;

[0029] The program list is displayed in the anchor's background, showing information about the currently playing or upcoming programs.

[0030] Optionally, the method further includes:

[0031] Upon receiving a gift sent to the target video, a configured mapping table is obtained, which includes the mapping relationship between gifts and programs;

[0032] Based on the correspondence table, determine the target program corresponding to the gift from the target video;

[0033] Play the target program.

[0034] Optionally, the method further includes:

[0035] When multiple gifts are received for the target video, the programs corresponding to the multiple gifts are placed into a preset queue in the order in which the gifts were received.

[0036] Retrieve programs from the preset queue for playback;

[0037] When the program queue is empty, the programs are played in the order of the program schedule.

[0038] Another aspect of this application provides a live video editing apparatus, the apparatus comprising:

[0039] The acquisition module is used to acquire target modal data of the live video to be edited, wherein the target modal data includes at least audio data and image data;

[0040] The determination module is used to perform multi-dimensional analysis on the live video based on the target modal data, and determine the highlights in the live video based on the analysis results;

[0041] The editing module is used to edit the highlights and generate an edited video based on the editing results.

[0042] Another aspect of this application provides a computer device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein: the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0043] Another aspect of this application provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method described above.

[0044] Another aspect of this application provides a computer program product including a computer program that, when executed by a processor, implements the method described above.

[0045] The embodiments of this application employing the above-described technical solution may have the following advantages:

[0046] By acquiring the target modal data of the live video to be edited, performing multi-dimensional analysis based on the target modal data, identifying the highlights in the live video based on the analysis results, editing the highlights, and generating an edited video based on the editing results, automatic editing of highlights in live videos can be achieved, improving the efficiency and accuracy of live video editing. Attached Figure Description

[0047] The accompanying drawings exemplify embodiments and form part of the specification, serving to explain exemplary implementations of the embodiments together with the textual description. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0048] Figure 1 The diagram illustrates the operating environment of the live video editing method according to Embodiment 1 of this application.

[0049] Figure 2 A flowchart illustrating a live video editing method according to Embodiment 1 of this application is shown schematically;

[0050] Figure 3 Schematic illustration Figure 2 Flowchart of the sub-steps in step S102;

[0051] Figure 4 Schematic illustration Figure 3 Flowchart of the sub-steps in step S200;

[0052] Figure 5 The illustration schematically shows the new process of the live video editing method according to Embodiment 1 of this application;

[0053] Figure 6 This illustration schematically shows another additional process in the live video editing method according to Embodiment 1 of this application;

[0054] Figure 7 This illustration schematically shows another new step in the live video editing method according to Embodiment 1 of this application;

[0055] Figure 8 The diagram illustrates the interface for playing the target video.

[0056] Figure 9This illustration schematically shows another new step in the live video editing method according to Embodiment 1 of this application;

[0057] Figure 10 This illustration schematically shows another new step in the live video editing method according to Embodiment 1 of this application;

[0058] Figure 11 A block diagram of a live video editing apparatus according to Embodiment 2 of this application is schematically shown; and

[0059] Figure 12 A schematic diagram of the hardware architecture of a computer device according to Embodiment 3 of this application is shown. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0061] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0062] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.

[0063] First, a definition of the terminology used in this application is provided:

[0064] Mel-Frequency Cepstral Coefficients (MFCC) are a feature extraction technique widely used in speech recognition, speaker recognition, and other fields. They simulate the human ear's perception of sound, converting the speech signal from the time domain to the frequency domain and obtaining a set of coefficients that characterize speech features through a series of transformations.

[0065] Chroma features are a type of feature used in audio signal processing. They represent pitch information in an audio signal and can be used to identify harmony, melody, and pitch in music.

[0066] Optical flow estimation is a computer vision technique used to describe the motion of each pixel in an image sequence. Simply put, it infers the motion of objects or the camera in an image by analyzing pixel changes between consecutive video frames.

[0067] Action recognition is a computer vision technology that aims to enable computers to understand and recognize actions contained in video or image sequences, just like humans do.

[0068] Human pose estimation: This is the process by which computers can accurately identify the positions of various joints in a human body and depict the human pose using images or videos.

[0069] 3D Convolutional Neural Network (3D CNN): A deep learning model specifically designed for processing three-dimensional data. Unlike traditional 2D CNNs, which are mainly used for processing image data, 3D CNNs exhibit unique advantages when processing data with a time dimension, such as videos and medical images.

[0070] OpenPose is an open-source library for human pose estimation based on deep learning. It can detect and track the 2D poses of multiple people from images or videos in real time, including key points such as body, hands, and face.

[0071] Secondly, to facilitate understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the relevant technologies are described below:

[0072] Live video content usually includes some exciting segments, such as the streamer's singing or dancing talents. However, these segments lose attention after the live stream ends, and users cannot find them to watch again.

[0073] Currently, if these highlights are to be shared further, they require manual editing and annotation, which is time-consuming, labor-intensive, and prone to omissions or annotation errors, resulting in low efficiency and accuracy. Furthermore, the content shared in this way lacks user interaction, leading to a lack of audience engagement and difficulty in maintaining viewer activity.

[0074] Therefore, this application provides a live video editing technology solution. This solution allows for the automatic editing of highlight segments from live videos, improving the efficiency and accuracy of live video editing. Simultaneously, it enables interaction between the edited highlight segments and the user, enhancing viewer engagement and maintaining viewer activity. Details are provided below.

[0075] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0076] Figure 1 The illustration shows an environmental application diagram according to an embodiment of this application.

[0077] The environmental diagram may include a service platform 2, broadcaster terminals (4A, 4B, ..., 4M), and viewer terminals (6A, 6B, ..., 6N). In a live broadcast scenario, the broadcaster terminals (4A, 4B, ..., 4M) log in to the service platform 2 and push live broadcast data to the viewer terminals (6A, 6B, ..., 6N) in real time through the service platform 2.

[0078] Service platform 2 can provide live streaming services, which can be a single server, a server cluster, or a cloud computing service center.

[0079] The broadcast terminals (4A, 4B, ..., 4M) are used to generate live streaming data in real time and to push the live streaming data. The live streaming data may include audio data or video data. The broadcast terminals can be electronic devices such as smartphones or tablets. Alternatively, the broadcast terminals can be virtual computing instances within service platform 2.

[0080] Viewer terminals (6A, 6B, ..., 6N) can be configured to receive live data from the broadcaster terminal in real time. Viewer terminals (6A, 6B, ..., 6N) can be any type of computing device, such as smartphones, tablets, laptops, smart TVs, in-vehicle terminals, etc. Viewer terminals (6A, 6B, ..., 6N) can have a built-in browser or dedicated program to receive the live data and output content to the user. The content may include video, audio, comments, text data, and / or the like.

[0081] The audience terminals (6A, 6B, ..., 6N) may include a player. The player outputs (e.g., displays, presents) content to the user. This content may include video, audio, comments, text data, and / or the like. The audience terminals (6A, 6B, ..., 6N) may include an interface that may include an input element (touchscreen). For example, the input element may be configured to receive user instructions that cause the audience terminals (6A, 6B, ..., 6N) to perform various operations, such as sending bullet comments, entering comments, sending gifts, etc.

[0082] The broadcast terminals (4A, 4B, ..., 4M), viewer terminals (6A, 6B, ..., 6N), and service platform 2 can be connected via a network. The network may include various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, and / or proxy devices. The network may include physical links, such as coaxial cable links, twisted-pair cable links, fiber optic links, and combinations thereof and / or the like. The network may include wireless links, such as cellular links, satellite links, Wi-Fi links, and / or the like.

[0083] It should be noted that the number of broadcast terminals and viewer terminals shown in the diagram is merely illustrative and is not intended to limit the scope of patent protection of this application. In practice, any number of broadcast terminals and viewer terminals may be used.

[0084] The technical solutions of this application are described below through multiple embodiments, using a service platform as the implementing entity. It should be understood that these embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments described herein.

[0085] Example 1

[0086] Figure 2 A flowchart illustrating a live video editing method according to Embodiment 1 of this application is shown schematically.

[0087] like Figure 2 As shown, the live video editing method may include steps S100~S104, wherein:

[0088] Step S100: Obtain the target modal data of the live video to be edited, wherein the target modal data includes at least audio data and image data.

[0089] Step S102: Perform multi-dimensional analysis on the live video based on the target modal data, and determine the highlights in the live video based on the analysis results.

[0090] Step S104: Edit the exciting segments and obtain an edited video based on the editing results.

[0091] The live video editing method provided in this embodiment obtains the target modal data of the live video to be edited, performs multi-dimensional analysis on the live video based on the target modal data, identifies the highlights in the live video based on the analysis results, edits the highlights, and generates an edited video based on the editing results. This method can realize automatic editing of highlights in live videos, improving the efficiency and accuracy of live video editing.

[0092] The following combination Figure 2The steps in steps S100 to S104, as well as other optional steps, are described in detail.

[0093] Step S100 The target modal data of the live video to be edited is obtained, and the target modal data includes at least audio data and image data.

[0094] Specifically, a decoder can be used to extract the audio stream of a live video, obtaining the audio data of the live video; simultaneously, the live video can be decoded into frame-by-frame images, obtaining the image data of the live video. In practical applications, the target modal data can also include text data, specifically including text data such as subtitles, comments, and chat messages.

[0095] Step S102 Based on the target modal data, the live video is analyzed in multiple dimensions, and the highlights in the live video are determined based on the analysis results.

[0096] Specifically, audio and image data can be analyzed separately to obtain audio analysis results and image analysis results respectively. Then, the results of audio and image analysis can be combined to obtain a comprehensive analysis result. Finally, based on the comprehensive analysis result, the highlights included in the live video can be determined. Some exemplary solutions are provided below.

[0097] In optional embodiments, such as Figure 3 As shown, step S102 may include:

[0098] Step S200: Identify the first target content of the live video based on the audio data, and generate first identification information based on the first target content. The first identification information is used to identify the segment corresponding to the first target content.

[0099] Step S202: Identify the second target content of the live video based on the image data, and generate second identification information based on the second target content. The second identification information is used to identify the segment corresponding to the second target content.

[0100] Step S204: Time alignment is performed on the first identification information and the second identification information.

[0101] Step S206: Determine the start time information and tag information of the highlight segment based on the time-aligned first and second identifier information.

[0102] The first target content can be content that can be determined based on audio data, such as singing, dialogue, or instrumental performance. The first identification information can include content category information (e.g., singing), content information (e.g., the song title), and segment information (e.g., segment start time). The second target content can be content that can be determined based on image data, such as dancing, sports, or instrumental performance. The second identification information can include content category information (e.g., dancing), content information (e.g., the dance name), and segment information. Tag information can include content category and content information.

[0103] Specifically, feature extraction can be performed on audio data. Based on the extracted features, a model can be used to identify the first target content, and then specific first identifier information can be generated based on the identified first target content. For image data, a model can be used to identify information such as actions in the live video as the second target content, and then specific second identifier information can be generated based on the identified second target content. After obtaining the first and second identifier information, the first and second identifier information are aligned on the timeline, and the two parts of data are merged for synchronization processing. Based on the synchronized information, the start time information and tag information of the highlights in the live video are determined. Highlights can be one or more, which can be obtained from audio data analysis, image data analysis, or a combination of audio and image data analysis. For example, for highlights in dance, since the background music usually starts earlier and ends later than the dance movements, determining the highlights solely based on the dance movements may result in incomplete background music information. However, by combining the analysis of audio and image data, the complete and accurate start time of the highlights can be obtained.

[0104] In this embodiment, by identifying the first target content of the live video based on audio data and generating first identification information based on the first target content, and identifying the second target content of the live video based on image data and generating second identification information based on the second target content, the first and second identification information are time-aligned, and the start time information and tag information of the highlights are determined based on the time-aligned first and second identification information. This allows for multi-dimensional analysis of the live video, comprehensively considering multiple factors to determine the highlights in the live video, thereby improving the accuracy of highlight determination.

[0105] In optional embodiments, such as Figure 4 As shown, in step S200, identifying the first target content of the live video based on audio data may include:

[0106] Step S300: Extract audio features based on the audio data, wherein the audio features include at least Mel frequency cepstral coefficients and chromaticity features.

[0107] Step S302: Based on the audio features, a deep learning model is used to identify the first target content of the live video.

[0108] Specifically, audio features can be extracted from audio data using audio feature extraction techniques. These extracted features are then input into a pre-trained deep learning model, which identifies the first target content in the live video. Audio feature extraction techniques can include Mel-frequency cepstral coefficient extraction, chroma feature extraction, cepstral analysis, and zero-crossing rate analysis. Besides frequency-cepstral coefficients and chroma features, audio features can also include other audio features. The deep learning model can be pre-trained using a large amount of training data, enabling it to output the specific first target content based on the input audio features. The deep learning model can be a convolutional neural network, a recurrent neural network, or a long short-term memory network, among others.

[0109] In this embodiment, by extracting audio features based on audio data and using a deep learning model based on these audio features to identify the first target content of the live video, the first target content of the live video can be effectively identified through audio features such as Mel frequency cepstral coefficients and chromaticity features, thereby improving the accuracy of audio analysis of live videos.

[0110] In an optional embodiment, step S202, identifying the second target content of the live video based on image data, may include: identifying the second target content of the live video based on image data using optical flow estimation, motion recognition, and / or human pose estimation.

[0111] Specifically, at least some of the techniques from optical flow estimation, action recognition models, and human pose estimation methods can be used to identify the second target content (such as dynamic content) contained in the image data, thereby obtaining the second target content of the live video. The action recognition model can be, for example, a 3D convolutional neural network or a long short-term memory network. The action recognition model can be pre-trained with a large amount of training data to enable it to recognize the second target content contained in the input image. Additionally, the human pose estimation method can be, for example, OpenPose.

[0112] In this embodiment, by using optical flow estimation, action recognition, and / or human pose estimation based on image data to identify the second target content in the live video, the second target content contained in the live video can be effectively identified, thereby improving the accuracy of image analysis of the live video.

[0113] Step S104The selected highlights are edited, and an edited video is obtained based on the editing results.

[0114] Specifically, given the start time and tag information of the highlights, the live video can be edited based on the start time of the highlights to obtain the highlights. At the same time, the relevant tag information of the highlights can be obtained, and the title, content, name, and other information of the edited video can be obtained based on the relevant tag information.

[0115] In optional embodiments, such as Figure 5 As shown, the live video editing method in this application embodiment may further include:

[0116] Step S400: Obtain the content keywords of the edited video.

[0117] Step S402: Generative artificial intelligence is used to generate the cover and title of the edited video based on the content keywords.

[0118] Specifically, the content keywords of the edited video can be determined by obtaining the corresponding text information, such as subtitles or comments; alternatively, the content keywords can be determined by combining the first and second identifier information; or, the audio data corresponding to the edited video can be converted into text information through speech recognition, and then the content keywords can be determined based on the converted text information. Of course, a combination of two or more of the above methods can also be used to determine the content keywords. After obtaining the content keywords of the edited video, a generative artificial intelligence model is used to generate the cover and title of the edited video based on the content keyword prompts. The generative artificial intelligence model can include a text generation model and an image generation model, with the text generation model responsible for generating the title and the image generation model responsible for generating the cover. Alternatively, a large model can directly generate both the cover and title simultaneously based on the content keywords.

[0119] In this embodiment, by obtaining the content keywords of the edited video, generative artificial intelligence is used to generate the cover and title of the edited video based on the content keywords, which can improve the intelligence level of the cover and title generation of the edited video and improve the efficiency of video editing.

[0120] In optional embodiments, such as Figure 6 As shown, after step S104, that is, after obtaining the edited video based on the editing results, the live video editing method of this application embodiment may further include:

[0121] Step S500: Determine the category of the edited video.

[0122] Step S502: Generate a program list according to the category, the program list including the program order.

[0123] Step S504: Synthesize the target video based on the program list and publish the target video.

[0124] The category of a video clip can be obtained from the tag information corresponding to the video clip. For example, if the tag information of a highlight clip is determined to be "singing" based on the first and second identifier information after time alignment, then the category of the video clip corresponding to the highlight clip can be determined to be "singing".

[0125] When generating a program list based on categories, you can group video clips of the same category together, alternate video clips of different categories in a certain order, or combine the time sequence of the video clips themselves to obtain the program list order.

[0126] Specifically, the category of the edited video can be obtained based on the tag information of the highlights corresponding to the edited video. Then, the program order can be obtained according to the category of the edited video in a certain order, and a program list can be generated. Finally, the edited videos can be merged into a target video based on the program list for submission and publication.

[0127] In this embodiment, by determining the category of the edited video, generating a program list based on the category, synthesizing the target video based on the program list, and publishing the target video, a program list can be automatically generated and published based on the edited video. This enables the publication of edited videos, facilitates the subsequent playback of related edited videos based on the program list, and allows for corresponding interactive configurations around the program list.

[0128] In optional embodiments, such as Figure 7 As shown, the live video editing method in this application embodiment may further include:

[0129] Step S600: While playing the target video, obtain the background of the broadcaster corresponding to the target video.

[0130] Step S602: Display the program list in the anchor background and display the program information that is currently playing or about to be played.

[0131] The broadcaster background can include the broadcaster's avatar and the background image used. Specifically, the corresponding broadcaster information can be determined based on the live video corresponding to the target video, and then the broadcaster background can be obtained based on the corresponding broadcaster information. Then, while the target video is playing, the program list is displayed in the broadcaster background, showing the currently playing or upcoming program information.

[0132] Please refer to Figure 8This is a schematic diagram illustrating the playback of the target video. As shown in the figure, the display interface can show the program list and information about the currently playing program. When playing a dance-themed video clip, the video frame can only include the host's avatar; alternatively, it can include the host's avatar while using the host's background image, with the clip's background image playing on top of the clip's background image. When playing a singing-themed video clip, both the host's avatar and background image can be displayed simultaneously.

[0133] In this embodiment, by obtaining the background of the host corresponding to the target video while playing the target video, displaying the program list in the host background, and displaying the program information that is currently playing or about to be played, an environment and atmosphere similar to a live video can be created, thereby improving the viewer's experience.

[0134] In optional embodiments, such as Figure 9 As shown, the live video editing method in this application embodiment may further include:

[0135] Step S700: Upon receiving a gift sent to the target video, obtain a configured correspondence table, which includes the correspondence between gifts and programs.

[0136] Step S702: Determine the target program corresponding to the gift from the target video based on the correspondence table.

[0137] Step S704: Play the target program.

[0138] The mapping table can be configured by the broadcaster or uniformly configured by the service platform. To inform viewers of the correspondence between gifts and programs, the program list can include gifts corresponding to each program, or the correspondence can be displayed in a certain area of ​​the playback interface. The specific settings can be configured according to actual needs.

[0139] Specifically, when watching a target video, users can send a specific gift based on the correspondence between gifts and programs to play the desired program. Upon receiving a gift for the target video, the service platform can retrieve the correspondence table configured for the current target video, determine the target program corresponding to the current gift, and then play the target program. To create a pleasant atmosphere, after the user sends a gift and selects the target program, a notification message indicating that the user has sent a gift and that the target program is about to start can be played before the target program begins playing.

[0140] In this embodiment, upon receiving a gift sent to the target video, a configured mapping table containing the correspondence between gifts and programs is obtained. Based on the mapping table, the target program corresponding to the gift is determined from the target video and played. This facilitates user interaction with the target video via gifts, enabling interaction between the user and the edited video, enlivening the viewing atmosphere, and improving the user experience.

[0141] In optional embodiments, such as Figure 10 As shown, the live video editing method in this application embodiment may further include:

[0142] Step S800: When multiple gifts are received for the target video, the programs corresponding to the multiple gifts are placed into a preset queue in the order in which the gifts are received.

[0143] Step S802: Take the program from the preset queue and play it.

[0144] Step S804: When the programs in the preset queue are empty, the programs are played in the order of the program list.

[0145] Specifically, when multiple gifts are received for the target video, the program corresponding to each gift is determined, and the programs corresponding to multiple gifts are placed into a preset queue according to the order in which the gifts are received. Then, the programs are taken out from the preset queue in order and played. After one program is finished playing, the next program is taken out from the preset queue and played to ensure the integrity of the program playback. When the programs in the preset queue are empty, the playback of programs is resumed according to the program list order.

[0146] In this embodiment, when multiple gifts are received for a target video, the corresponding programs are placed into a preset queue in the order in which the gifts are received, and the programs are taken out from the preset queue for playback. When the preset queue is empty, the programs are played in the order of the program list. This can effectively manage user interaction and ensure the interactive experience and orderly playback of programs.

[0147] Example 2

[0148] Figure 11 The diagram schematically illustrates a live video editing apparatus according to Embodiment 2 of this application. This apparatus can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of this application. The program module referred to in this embodiment is a series of computer program instruction segments capable of performing a specific function. The following description will specifically introduce the function of each program module in this embodiment. For example... Figure 11As shown, the device 900 may include an acquisition module 910, a determination module 920, and a editing module 930, wherein:

[0149] The acquisition module 910 is used to acquire target modal data of the live video to be edited, wherein the target modal data includes at least audio data and image data;

[0150] The determination module 920 is used to perform multi-dimensional analysis on the live video based on the target modal data, and determine the highlights in the live video based on the analysis results;

[0151] The editing module 930 is used to edit the exciting segments and obtain an edited video based on the editing results.

[0152] In an optional embodiment, the determining module 920 is further configured to:

[0153] The first target content of the live video is identified based on the audio data, and first identification information is generated based on the first target content. The first identification information is used to identify the segment corresponding to the first target content.

[0154] The second target content of the live video is identified based on the image data, and second identification information is generated based on the second target content. The second identification information is used to identify the segment corresponding to the second target content.

[0155] Time alignment is performed on the first and second identification information;

[0156] The start time and tag information of the highlight segment are determined based on the first and second identifier information after time alignment.

[0157] In an optional embodiment, the determining module 920 is further configured to:

[0158] Audio features are extracted based on the audio data, and the audio features include at least Mel frequency cepstral coefficients and chromaticity features;

[0159] Based on the audio features, a deep learning model is used to identify the first target content of the live video.

[0160] In an optional embodiment, the determining module 920 is further configured to:

[0161] Based on the image data, optical flow estimation, motion recognition, and / or human pose estimation are used to identify the second target content of the live video.

[0162] In an optional embodiment, the device 900 is further used for:

[0163] Obtain the keywords of the edited video content;

[0164] Generative artificial intelligence is used to generate the cover and title of the edited video based on the content keywords.

[0165] In an optional embodiment, the device 900 is further used for:

[0166] Determine the category of the edited video;

[0167] A program list is generated based on the category, the program list including the program order;

[0168] The target video is synthesized based on the program list and then published.

[0169] In an optional embodiment, the device 900 is further used for:

[0170] When playing the target video, obtain the background of the broadcaster corresponding to the target video;

[0171] The program list is displayed in the anchor's background, showing information about the currently playing or upcoming programs.

[0172] In an optional embodiment, the device 900 is further used for:

[0173] Upon receiving a gift sent to the target video, a configured mapping table is obtained, which includes the mapping relationship between gifts and programs;

[0174] Based on the correspondence table, determine the target program corresponding to the gift from the target video;

[0175] Play the target program.

[0176] In an optional embodiment, the device 900 is further used for:

[0177] When multiple gifts are received for the target video, the programs corresponding to the multiple gifts are placed into a preset queue in the order in which the gifts were received.

[0178] Retrieve programs from the preset queue for playback;

[0179] When the program queue is empty, the programs are played in the order of the program schedule.

[0180] Example 3

[0181] Figure 12This illustration schematically depicts the hardware architecture of a computer device 10000 suitable for implementing a live video editing method according to Embodiment 3 of this application. In some embodiments, the computer device 10000 may be a terminal device such as a smartphone, wearable device, tablet computer, personal computer, in-vehicle terminal, game console, virtual device, workbench, digital assistant, set-top box, or robot. In other embodiments, the computer device 10000 may be a rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers), etc. Figure 12 As shown, the computer device 10000 includes, but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate and be linked with each other via a system bus. Wherein:

[0182] The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of a computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as program code for live video editing methods. In addition, the memory 10010 can also be used to temporarily store various types of data that have been output or will be output.

[0183] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.

[0184] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 10000 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0185] It should be pointed out that, Figure 12 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0186] In this embodiment, the live video editing method stored in memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of this application.

[0187] Example 4

[0188] This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the live video editing method in the embodiments.

[0189] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium may include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the live video editing method in this embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0190] Example 5

[0191] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in the above embodiments.

[0192] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computer devices. They can be centralized on a single computer device or distributed across a network of multiple computer devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computer device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0193] It should be noted that the above are merely preferred embodiments of this application and do not limit the scope of patent protection of this application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A method for editing live video, characterized in that, The method includes: Obtain the target modal data of the live video to be edited, wherein the target modal data includes at least audio data and image data; The live video is analyzed from multiple dimensions based on the target modal data, and the highlights in the live video are determined based on the analysis results. The aforementioned highlights are edited, and an edited video is obtained based on the editing results; Determine the category of the edited video and generate a program list; Synthesize the target video based on the program list; When playing the target video, obtain the background of the broadcaster corresponding to the target video; The program list is displayed in the anchor's background, showing information about the currently playing or upcoming programs.

2. The method according to claim 1, characterized in that, The step of performing multi-dimensional analysis on the live video based on the target modal data, and determining the highlights of the live video based on the analysis results, includes: The first target content of the live video is identified based on the audio data, and first identification information is generated based on the first target content. The first identification information is used to identify the segment corresponding to the first target content. The second target content of the live video is identified based on the image data, and second identification information is generated based on the second target content. The second identification information is used to identify the segment corresponding to the second target content. Time alignment is performed on the first and second identification information; The start time and tag information of the highlight segment are determined based on the first and second identifier information after time alignment.

3. The method according to claim 2, characterized in that, The step of identifying the first target content of the live video based on the audio data includes: Audio features are extracted based on the audio data, and the audio features include at least Mel frequency cepstral coefficients and chromaticity features; Based on the audio features, a deep learning model is used to identify the first target content of the live video.

4. The method according to claim 2, characterized in that, The second target content identification based on the image data of the live video includes: Based on the image data, optical flow estimation, motion recognition, and / or human pose estimation are used to identify the second target content of the live video.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Obtain the keywords of the edited video content; Generative artificial intelligence is used to generate the cover and title of the edited video based on the content keywords.

6. The method according to any one of claims 1-4, characterized in that, The program list includes the program order.

7. The method according to claim 6, characterized in that, The method further includes: Upon receiving a gift sent to the target video, a configured mapping table is obtained, which includes the mapping relationship between gifts and programs; Based on the correspondence table, determine the target program corresponding to the gift from the target video; Play the target program.

8. The method according to claim 7, characterized in that, The method further includes: When multiple gifts are received for the target video, the programs corresponding to the multiple gifts are placed into a preset queue in the order in which the gifts were received. Retrieve programs from the preset queue for playback; When the program queue is empty, the programs are played in the order of the program schedule.

9. A live video editing device, characterized in that, The device includes: The acquisition module is used to acquire target modal data of the live video to be edited, wherein the target modal data includes at least audio data and image data; The determination module is used to perform multi-dimensional analysis on the live video based on the target modal data, and determine the highlights in the live video based on the analysis results; The editing module is used to edit the highlights and generate an edited video based on the editing results. The device is also used for: Determine the category of the edited video and generate a program list; Synthesize the target video based on the program list; When playing the target video, obtain the background of the broadcaster corresponding to the target video; The program list is displayed in the anchor's background, showing information about the currently playing or upcoming programs.

10. A computer device, characterized in that, include: At least one processor; and A memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8.

Images