Live video editing method and device

Abstract

The embodiment of the invention provides a live video editing method, and the method comprises the steps: obtaining the target modal data of a to-be-edited live video, and the target modal data at least comprises audio data and image data; performing multi-dimensional analysis on the live video based on the target modal data, and determining a wonderful fragment in the live video based on an analysis result; and editing the wonderful clip, and obtaining an edited video based on an editing result. According to the technical scheme provided by the embodiment of the invention, the highlights in the live video can be automatically edited, and the efficiency and accuracy of editing the live video are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a live video editing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] There are usually some exciting clips in live videos, such as the host's singing, dancing and other talents. If you want to spread these exciting clips for the second time, you need to manually edit them, otherwise these exciting clips will lose attention when the live broadcast is closed.

[0003] However, manually editing clips from live videos is time-consuming and laborious, and is prone to problems such as omissions in editing or labeling errors. It is inefficient and inaccurate.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0005] The embodiments of the present application provide a live video editing method, apparatus, computer equipment, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0006] One aspect of an embodiment of the present application provides a live video editing method, the method comprising: Acquire target modality data of the live video to be edited, wherein the target modality data includes at least audio data and image data; Performing a multi-dimensional analysis on the live video based on the target modal data, and determining a highlight segment in the live video based on the analysis result; The wonderful clips are edited, and an edited video is obtained based on the editing result.

[0007] Optionally, performing a multi-dimensional analysis on the live video based on the target modality data, and determining a highlight segment in the live video based on a result of the analysis, includes: Identifying a first target content of the live video based on the audio data, and generating first identification information based on the first target content, wherein the first identification information is used to identify a segment corresponding to the first target content; identifying a second target content of the live video based on the image data, and generating second identification information based on the second target content, wherein the second identification information is used to identify a segment corresponding to the second target content; Performing time alignment on the first identification information and the second identification information; The start time information and label information of the highlight segment are determined based on the time-aligned first identification information and the second identification information.

[0008] Optionally, the identifying the first target content of the live video based on the audio data includes: Extracting audio features based on the audio data, the audio features at least including Mel frequency cepstral coefficients and chromaticity features; A deep learning model is used based on the audio features to identify the first target content of the live video.

[0009] Optionally, the identifying the second target content of the live video based on the image data includes: Based on the image data, optical flow estimation, action recognition and / or human posture estimation are used to identify the second target content of the live video.

[0010] Optionally, the method further comprises: Obtaining content keywords of the clipped video; Generative artificial intelligence is used to generate the cover and title of the edited video based on the content keywords.

[0011] Optionally, after obtaining the edited video based on the result of the editing, the method further includes: Determining the category of the clipped video; generating a program list according to the category, wherein the program list includes a program list sequence; A target video is synthesized based on the program list and the target video is released.

[0012] Optionally, the method further comprises: When playing the target video, obtaining the anchor background corresponding to the target video; The program list is displayed in the anchor background, and information about programs currently being played or to be played is displayed.

[0013] Optionally, the method further comprises: In case of receiving a gift sent to the target video, obtaining a configured correspondence table, the correspondence table including a correspondence relationship between the gift and the program; Determining a target program corresponding to the gift from the target video based on the correspondence table; The target program is played.

[0014] Optionally, the method further comprises: In the case of receiving multiple gifts sent to the target video, placing the programs corresponding to the multiple gifts into a preset queue in the order in which the gifts are received; Taking out a program from the preset queue for playing; When the programs in the preset queue are empty, the programs are played in the order of the program list.

[0015] Another aspect of an embodiment of the present application provides a live video editing device, the device comprising: An acquisition module, used to acquire target modality data of the live video to be edited, wherein the target modality data includes at least audio data and image data; A determination module, configured to perform a multi-dimensional analysis on the live video based on the target modal data, and determine a highlight segment in the live video based on the analysis result; The editing module is used to edit the wonderful clips and obtain an edited video based on the editing results.

[0016] Another aspect of an embodiment of the present application provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein: the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0017] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0018] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0019] The above technical solution adopted in the embodiment of the present application may have the following advantages: By obtaining target modal data of the live video to be edited, performing a multi-dimensional analysis of the live video based on the target modal data, determining the highlight clips in the live video based on the analysis results, editing the highlight clips, and generating an edited video based on the editing results, automatic editing of highlight clips in the live video can be achieved, thereby improving the efficiency and accuracy of live video editing. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0021] Figure 1 The operating environment diagram of the live video editing method according to the first embodiment of the present application is schematically shown; Figure 2 The flowchart of the live video editing method according to the first embodiment of the present application is schematically shown; Figure 3 Schematically shows Figure 2 Flow chart of sub-steps of step S102; Figure 4 Schematically shows Figure 3 Flow chart of sub-steps of step S200; Figure 5 The newly added process of the live video editing method according to the first embodiment of the present application is schematically shown; Figure 6 Another newly added process of the live video editing method according to the first embodiment of the present application is schematically shown; Figure 7 Schematically illustrates another newly added process of the live video editing method according to the first embodiment of the present application; Figure 8 The interface diagram of the target video playback is schematically shown; Fig. 9 Schematically illustrates another newly added process of the live video editing method according to the first embodiment of the present application; Fig.10 Schematically illustrates another newly added process of the live video editing method according to the first embodiment of the present application; Fig.11 A block diagram of a live video editing device according to Embodiment 2 of the present application is schematically shown; and Fig.12 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0023] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0024] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0025] First, the following terms are explained: Mel-Frequency Cepstral Coefficients (MFCC): is a feature extraction technology widely used in speech recognition, speaker recognition and other fields. It simulates the human ear's perception of sound, converts speech signals from the time domain to the frequency domain, and obtains a set of coefficients that can characterize speech features through a series of transformations.

[0026] Chroma features: A feature in audio signal processing. It represents the pitch information in the audio signal and can be used to identify harmony, melody, and tone in music.

[0027] Optical flow estimation: is a computer vision technique used to describe the motion of each pixel in an image sequence. Simply put, it is to infer the motion of objects or cameras in the image by analyzing the pixel changes between consecutive video frames.

[0028] Action recognition: It is a computer vision technology that aims to enable computers to understand and recognize the actions contained in videos or image sequences like humans.

[0029] Human posture estimation: It enables computers to accurately identify the positions of various joints of the human body and depict the human posture through images or videos.

[0030] 3D Convolutional Neural Network (3D CNN): is a deep learning model specifically designed for processing three-dimensional data. Unlike traditional 2D CNN, which is mainly used to process image data, 3D CNN has unique advantages in processing data with a time dimension, such as videos and medical images.

[0031] OpenPose: is an open source library for human pose estimation based on deep learning. It can detect and track the two-dimensional poses of multiple people from images or videos in real time, including key points such as body, hands, and face.

[0032] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: In the content of live videos, there are usually some exciting clips, such as the host's singing, dancing and other talents. After the live broadcast is closed, they lose attention and users cannot find these exciting clips to watch again.

[0033] At present, if these wonderful clips are to be disseminated for a second time, they need to be manually edited and annotated, which is time-consuming and laborious, and prone to problems such as missed clips or incorrect annotations. It is inefficient and inaccurate. In addition, the content of secondary dissemination cannot achieve interaction with users, and the audience lacks a sense of participation, making it difficult to maintain the audience's activity.

[0034] To this end, the embodiment of the present application provides a live video editing technology solution. In this technical solution, the wonderful clips in the live video can be automatically edited to improve the efficiency and accuracy of live video editing; at the same time, the interaction between the edited wonderful clips and the user can be realized to improve the audience's sense of participation and maintain the audience's activity. See below for details.

[0035] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0036] Figure 1 The following schematically shows an environmental application diagram according to an embodiment of the present application.

[0037] The environment diagram may include a service platform 2, anchor terminals (4A, 4B, ..., 4M), and audience terminals (6A, 6B, ..., 6N). In a live broadcast scenario, the anchor terminal (4A, 4B, ..., 4M) logs in to the service platform 2, and the service platform 2 pushes the live broadcast data to the audience terminal (6A, 6B, ..., 6N) in real time.

[0038] Service platform 2 can provide live broadcast service, which can be a single server, a server cluster or a cloud computing service center.

[0039] The anchor terminal (4A, 4B, ..., 4M) is used to generate live broadcast data in real time and perform streaming operations on the live broadcast data. The live broadcast data may include audio data or video data. The anchor terminal may be an electronic device such as a smart phone or a tablet computer. Of course, the anchor terminal may be a virtual computing instance in the service platform 2.

[0040] The audience terminal (6A, 6B, ..., 6N) may be configured to receive the live broadcast data of the anchor terminal in real time. The audience terminal (6A, 6B, ..., 6N) may be any type of computing device, such as a smart phone, a tablet device, a laptop computer, a smart TV, a vehicle-mounted terminal, etc. The audience terminal (6A, 6B, ..., 6N) may have a built-in browser or a special program, and receive the live broadcast data through the browser or the special program to output content to the user. The content may include video, audio, comments, text data and / or the like.

[0041] The audience terminal (6A, 6B, ..., 6N) may include a player. The player outputs (e.g., displays, presents) content to the user. The content may include video, audio, comments, text data and / or the like. The audience terminal (6A, 6B, ..., 6N) may include an interface, which may include an input element (touch screen). For example, the input element may be configured to receive user instructions, and the user instructions may enable the audience terminal (6A, 6B, ..., 6N) to perform various operations, such as sending a bullet screen, entering a comment, giving a gift, etc.

[0042] The anchor terminals (4A, 4B, ..., 4M), the audience terminals (6A, 6B, ..., 6N) and the service platform 2 may be connected via a network. The network may include various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls and / or proxy devices, etc. The network may include physical links, such as coaxial cable links, twisted pair cable links, optical fiber links, combinations thereof and / or the like. The network may include wireless links, such as cellular links, satellite links, Wi-Fi links and / or the like.

[0043] It should be noted that the number of the host terminals and the audience terminals in the figure is only for illustration and is not used to limit the scope of patent protection of this application. According to actual conditions, there can be any number of host terminals and audience terminals.

[0044] The following uses the service platform as the execution subject to introduce the technical solution of the present application through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be interpreted as being limited to the embodiments described here.

[0045] Embodiment 1 Figure 2 The flowchart of the live video editing method according to the first embodiment of the present application is schematically shown.

[0046] like Figure 2 As shown, the live video editing method may include steps S100 to S104, wherein: Step S100, obtaining target modality data of the live video to be edited, wherein the target modality data at least includes audio data and image data.

[0047] Step S102: performing a multi-dimensional analysis on the live video based on the target modal data, and determining a highlight segment in the live video based on the analysis result.

[0048] Step S104, editing the wonderful clips and obtaining an edited video based on the editing result.

[0049] The live video editing method provided in the present embodiment obtains target modal data of the live video to be edited, performs multi-dimensional analysis on the live video based on the target modal data, determines wonderful clips in the live video based on the analysis results, edits the wonderful clips, and generates an edited video based on the editing results. This can realize automatic editing of wonderful clips in the live video and improve the efficiency and accuracy of live video editing.

[0050] The following combination Figure 2 , each step in steps S100~S104 and other optional steps are explained in detail.

[0051] Step S100 , obtain target modality data of the live video to be edited, and the target modality data at least includes audio data and image data.

[0052] Specifically, a decoder can be used to extract the audio stream of a live video to obtain the audio data of the live video; at the same time, the live video can be decoded into frame images to obtain the image data of the live video. In practical applications, the target modality data can also include text data, specifically, text data such as subtitles, comments, and chat information.

[0053] Step S102 , performing a multi-dimensional analysis on the live video based on the target modal data, and determining the wonderful segments in the live video based on the analysis results.

[0054] Specifically, the audio data and the image data can be analyzed separately to obtain the audio analysis result and the image analysis result; then, the audio analysis result and the image analysis result can be integrated to obtain the integrated analysis result, and finally, the highlights included in the live video can be determined according to the integrated analysis result. Some exemplary solutions are provided below.

[0055] In an optional embodiment, if Figure 3 As shown, step S102 may include: Step S200: identifying a first target content of the live video based on the audio data, and generating first identification information based on the first target content, wherein the first identification information is used to identify a segment corresponding to the first target content.

[0056] Step S202: identifying a second target content of the live video based on the image data, and generating second identification information based on the second target content, wherein the second identification information is used to identify a segment corresponding to the second target content.

[0057] Step S204: time-align the first identification information and the second identification information.

[0058] Step S206: determining the start time information and label information of the highlight segment based on the time-aligned first identification information and the second identification information.

[0059] The first target content may be content that can be determined based on audio data, such as singing, dialogue, musical instrument playing, etc. The first identification information may include content category information (such as singing), content information (such as the song name corresponding to singing), and content segment information (such as the start time of the segment). The second target content may be content that can be determined based on image data, such as dancing, sports, or musical instrument playing. The second identification information may include content category information (such as dancing), content information (such as the name corresponding to the dance), and content segment information. The tag information may include information such as the content category and content information tags.

[0060] Specifically, feature extraction can be performed on audio data, and a model can be used to identify the first target content based on the extracted features, and then specific first identification information can be generated based on the identified first target content; for image data, a model can be used to identify information such as actions in the live video as the second target content, and then specific second identification information can be generated based on the identified second target content; after obtaining the first identification information and the second identification information, the first identification information and the second identification information are aligned on the time axis, so that the two parts of data are merged for synchronization processing, and the start time information and label information of the highlight clip in the live video are determined based on the synchronized information. Among them, the highlight clip can be one or more, which can be obtained by analyzing audio data, or by analyzing image data, or by comprehensive analysis of audio data and image data. For example, for the highlight clip of a dance, since the background music of the dance usually starts earlier and ends later than the dance movements, if the highlight clip is determined based solely on the dance movements, the background music part may be incomplete, and combined with the analysis of audio data and image data, the complete and accurate start time of the highlight clip of the dance can be obtained.

[0061] In this embodiment, by identifying the first target content of the live video based on audio data and generating first identification information based on the first target content, identifying the second target content of the live video based on image data and generating second identification information based on the second target content, time-aligning the first identification information and the second identification information, and determining the start time information and label information of the highlight segment based on the time-aligned first identification information and the second identification information, the live video can be analyzed in multiple dimensions, and the highlight segments in the live video can be determined by comprehensively considering various factors, thereby improving the accuracy of determining the highlight segments.

[0062] In an optional embodiment, if Figure 4 As shown, in step S200, identifying the first target content of the live video based on the audio data may include: Step S300: extracting audio features based on the audio data, wherein the audio features at least include Mel-frequency cepstral coefficients and chromaticity features.

[0063] Step S302: using a deep learning model based on the audio features to identify the first target content of the live video.

[0064] Specifically, the audio features of the audio data can be extracted based on the audio feature extraction technology; then, the extracted audio features are input into a pre-trained deep learning model, and the first target content in the live video is identified using the deep learning model. Among them, the audio feature extraction technology may include Mel frequency cepstral coefficient extraction, chroma feature features, cepstral analysis and zero crossing rate, etc.; in addition to frequency cepstral coefficients and chroma features, audio features may also include other audio features. The deep learning model can be pre-trained with a large amount of training data so that it can output specific first target content based on the input audio features. The deep learning model can specifically be a convolutional neural network, a recurrent neural network or a long short-term memory network, etc.

[0065] In this embodiment, by extracting audio features based on audio data and using a deep learning model based on the audio features to identify the first target content of the live video, the first target content of the live video can be effectively identified through audio features such as Mel-frequency cepstral coefficients and chromaticity features, thereby improving the accuracy of audio analysis of the live video.

[0066] In an optional embodiment, in step S202, identifying the second target content of the live video based on the image data may include: identifying the second target content of the live video based on the image data using optical flow estimation, motion recognition and / or human posture estimation.

[0067] Specifically, at least some of the technical means in the optical flow estimation method, the action recognition model and the human posture estimation method can be used to identify the second target content (such as dynamic content) contained in the image data to obtain the second target content of the live video. Among them, the action recognition model can be, for example, a 3D convolutional neural network or a long short-term memory network. A large amount of training data can be used in advance to train the action recognition model so that it can recognize the second target content contained in the input image. In addition, the human posture estimation method can be, for example, OpenPose.

[0068] In this embodiment, by identifying the second target content of the live video based on image data using optical flow estimation, motion recognition and / or human posture estimation, the second target content contained in the live video can be effectively identified, thereby improving the accuracy of image analysis of the live video.

[0069] Step S104 , the wonderful clips are edited, and an edited video is obtained based on the editing result.

[0070] Specifically, when the start time and label information of the highlight clip are obtained, the live video can be edited according to the start time of the highlight clip to obtain the highlight clip. At the same time, the relevant label information of the highlight clip can be obtained, and the title, content, name and other information of the edited video can be obtained according to the relevant label information.

[0071] In an optional embodiment, if Figure 5 As shown, the live video editing method of the embodiment of the present application may also include: Step S400, obtaining content keywords of the clipped video.

[0072] Step S402: Generate a cover and title of the edited video using generative artificial intelligence based on the content keywords.

[0073] Specifically, the text information corresponding to the clipped video, such as subtitles or comments, etc., can be obtained, and the content keywords of the clipped video can be determined based on the corresponding text information; or, the content keywords of the clipped video can be determined in combination with the first identification information and the second identification information; or, the audio data corresponding to the clipped video can be converted into text information through speech recognition, and then the content keywords of the clipped video can be determined based on the converted text information. Of course, the content keywords of the clipped video can also be determined by combining the contents of the above two or more parts. After obtaining the content keywords of the clipped video, a generative artificial intelligence model is used to generate the cover and title of the clipped video according to the prompts of the content keywords. Among them, the generative artificial intelligence model can include a text generation model and an image generation model, respectively, the text generation model is responsible for generating the title, and the image generation model is responsible for generating the cover. It is also possible for a large model to directly generate the cover and title at the same time based on the content keywords.

[0074] In this embodiment, by obtaining content keywords of the edited video and using generative artificial intelligence to generate the cover and title of the edited video based on the content keywords, the intelligence level of the cover and title generation of the edited video can be improved, thereby improving the efficiency of video editing.

[0075] In an optional embodiment, if Figure 6 As shown, after step S104, that is, after obtaining the edited video based on the editing result, the live video editing method of the embodiment of the present application may further include: Step S500, determining the category of the clipped video.

[0076] Step S502: Generate a program list according to the category, wherein the program list includes a program list sequence.

[0077] Step S504: synthesize a target video based on the program list and publish the target video.

[0078] The category of the clipped video can be obtained from the tag information corresponding to the clipped video. For example, when the tag information of the highlight segment is determined to be "singing" based on the time-aligned first identification information and second identification information, the category of the clipped video corresponding to the highlight segment can be determined to be "singing".

[0079] When generating a program list according to categories, video clips of the same category may be put together, or video clips of different categories may be alternated in a certain order, or the program list sequence may be obtained in combination with the time sequence of the video clips themselves.

[0080] Specifically, the category of the clipped video can be obtained according to the tag information of the highlight clips corresponding to the clipped video, and then the program sequence can be obtained in a certain order according to the category of the clipped video, and a program list can be generated; then, according to the program list, the clipped videos can be merged into a target video for submission and release.

[0081] In this embodiment, by determining the category of the clipped video, generating a program list according to the category, synthesizing the target video based on the program list and publishing the target video, a program list can be automatically generated and published based on the clipped video, the publishing of the clipped video can be realized, and the subsequent playback of related clipped videos according to the program list can be facilitated, and corresponding interactive configuration can be performed around the program list.

[0082] In an optional embodiment, if Figure 7 As shown, the live video editing method of the embodiment of the present application may also include: Step S600, when playing the target video, obtaining the host background corresponding to the target video.

[0083] Step S602: Display the program list in the anchor background and display the information of the program currently being played or to be played.

[0084] The anchor background may include an anchor portrait and a background image. Specifically, the corresponding anchor information may be determined according to the live video corresponding to the target video, and then the anchor background may be obtained according to the corresponding anchor information. Then, when the target video is played, the program list is displayed in the anchor background, and the program information currently being played or to be played is displayed.

[0085] Please refer to Figure 8 , which is a schematic diagram of the target video playback. As shown in the figure, the display interface can display the program list and the currently playing program information. When playing a dance-type clip video, the video screen can only include the host's head portrait; it can also include the host's head portrait and use the host's background image, and the clip video screen is played on the background image; when playing a singing-type clip video, the host's head portrait and the host's background image can be displayed at the same time.

[0086] In this embodiment, by obtaining the host background corresponding to the target video while playing the target video, displaying the program list in the host background, and displaying the information of the program currently being played or about to be played, an environment and atmosphere similar to that of a live video can be created, thereby improving the audience experience.

[0087] In an optional embodiment, if Fig. 9 As shown, the live video editing method of the embodiment of the present application may also include: Step S700, when a gift sent to the target video is received, a configured correspondence table is obtained, wherein the correspondence table includes a correspondence between the gift and the program.

[0088] Step S702: determining a target program corresponding to the gift from the target video based on the correspondence table.

[0089] Step S704: play the target program.

[0090] The corresponding table can be configured by the anchor or uniformly configured by the service platform. In order to let the audience know the corresponding relationship between gifts and programs, the corresponding gifts of the programs can be included in the program list, or the corresponding relationship between gifts and programs can be displayed in a certain area of ​​the playback interface. The specific settings can be made according to actual needs.

[0091] Specifically, when watching a target video, the user can send a designated gift according to the correspondence between the gift and the program to play the desired program. When receiving a gift sent to a target video, the service platform can obtain the corresponding table of the current target video configuration, determine the target program corresponding to the current gift according to the corresponding table, and then play the target program. In order to create a good atmosphere, after the user sends a gift and determines the target program, the prompt information that the user has sent a gift and the target program is about to be played can be played first, and then the target program can be played.

[0092] In this embodiment, when a gift is received for a target video, a corresponding table including the correspondence between gifts and programs is obtained, and a target program corresponding to the gift is determined from the target video based on the corresponding table for playback. This facilitates user interaction with the target video through gifts, enables user interaction with the clipped video, enlivens the viewing atmosphere, and improves the user experience.

[0093] In an optional embodiment, if Fig.10 As shown, the live video editing method of the embodiment of the present application may also include: Step S800, when multiple gifts are received for the target video, the programs corresponding to the multiple gifts are placed into a preset queue in the order in which the gifts are received.

[0094] Step S802: taking out a program from the preset queue for playing.

[0095] Step S804: When the programs in the preset queue are empty, the programs are played in the order of the program list.

[0096] Specifically, when multiple gifts are received for the target video, the program corresponding to each gift is determined, and the programs corresponding to the multiple gifts are placed in a preset queue according to the order in which the gifts are received; then, the programs are taken out from the preset queue in order for playback; after one program is played, the next program is taken out from the preset queue for playback to ensure the integrity of the program playback; when the program in the preset queue is empty, the playback of the program is resumed in the order of the program list.

[0097] In this embodiment, when multiple gifts are received for a target video, the programs corresponding to the multiple gifts are placed into a preset queue in the order in which the gifts are received, and the programs are taken out from the preset queue for playing. When the programs in the preset queue are empty, the programs are played in the order of the program list. This can effectively manage user interactions and ensure interactive experience and orderly playback of programs.

[0098] Embodiment 2 Fig.11 The block diagram of the live video editing device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Fig.11 As shown, the device 900 may include an acquisition module 910, a determination module 920 and a clipping module 930, wherein: An acquisition module 910 is used to acquire target modality data of the live video to be edited, wherein the target modality data includes at least audio data and image data; A determination module 920, configured to perform a multi-dimensional analysis on the live video based on the target modality data, and determine a highlight segment in the live video based on the analysis result; The editing module 930 is used to edit the wonderful clips and obtain an edited video based on the editing results.

[0099] In an optional embodiment, the determination module 920 is further configured to: Identifying a first target content of the live video based on the audio data, and generating first identification information based on the first target content, wherein the first identification information is used to identify a segment corresponding to the first target content; identifying a second target content of the live video based on the image data, and generating second identification information based on the second target content, wherein the second identification information is used to identify a segment corresponding to the second target content; Performing time alignment on the first identification information and the second identification information; The start time information and label information of the highlight segment are determined based on the time-aligned first identification information and the second identification information.

[0100] In an optional embodiment, the determination module 920 is further configured to: Extracting audio features based on the audio data, the audio features at least including Mel frequency cepstral coefficients and chromaticity features; A deep learning model is used based on the audio features to identify the first target content of the live video.

[0101] In an optional embodiment, the determination module 920 is further configured to: Based on the image data, optical flow estimation, action recognition and / or human posture estimation are used to identify the second target content of the live video.

[0102] In an optional embodiment, the device 900 is further used for: Obtaining content keywords of the clipped video; Generative artificial intelligence is used to generate the cover and title of the edited video based on the content keywords.

[0103] In an optional embodiment, the device 900 is further used for: Determining the category of the clipped video; generating a program list according to the category, wherein the program list includes a program list sequence; A target video is synthesized based on the program list and the target video is released.

[0104] In an optional embodiment, the device 900 is further used for: When playing the target video, obtaining the anchor background corresponding to the target video; The program list is displayed in the anchor background, and information about programs currently being played or to be played is displayed.

[0105] In an optional embodiment, the device 900 is further used for: In case of receiving a gift sent to the target video, obtaining a configured correspondence table, the correspondence table including a correspondence relationship between the gift and the program; Determining a target program corresponding to the gift from the target video based on the correspondence table; The target program is played.

[0106] In an optional embodiment, the device 900 is further used for: In the case of receiving multiple gifts sent to the target video, placing the programs corresponding to the multiple gifts into a preset queue in the order in which the gifts are received; Taking out a program from the preset queue for playing; When the programs in the preset queue are empty, the programs are played in the order of the program list.

[0107] Embodiment 3 Fig.12 The hardware architecture diagram of a computer device 10000 suitable for implementing a live video editing method according to the third embodiment of the present application is schematically shown. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Fig.12 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as a hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 can also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed in the computer device 10000, such as the program code of the live video editing method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.

[0108] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0109] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0110] It should be pointed out that Fig.12 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0111] In this embodiment, the live video editing method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0112] Embodiment 4 An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the live video editing method in the embodiment are implemented.

[0113] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Of course, the computer-readable storage medium may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the live video editing method in the embodiment. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.

[0114] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0115] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0116] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A live video editing method, characterized in that: The method comprises: Acquire target modality data of the live video to be edited, wherein the target modality data includes at least audio data and image data; Performing a multi-dimensional analysis on the live video based on the target modal data, and determining a highlight segment in the live video based on the analysis result; The wonderful clips are edited, and an edited video is obtained based on the editing result.

2. The method according to claim 1, characterized in that The performing a multi-dimensional analysis on the live video based on the target modality data, and determining a highlight segment in the live video based on the analysis result, includes: Identifying a first target content of the live video based on the audio data, and generating first identification information based on the first target content, wherein the first identification information is used to identify a segment corresponding to the first target content; identifying a second target content of the live video based on the image data, and generating second identification information based on the second target content, wherein the second identification information is used to identify a segment corresponding to the second target content; Performing time alignment on the first identification information and the second identification information; The start time information and label information of the highlight segment are determined based on the time-aligned first identification information and the second identification information.

3. The method according to claim 2, characterized in that The identifying the first target content of the live video based on the audio data includes: Extracting audio features based on the audio data, the audio features at least including Mel frequency cepstral coefficients and chromaticity features; A deep learning model is used based on the audio features to identify the first target content of the live video.

4. The method according to claim 2, characterized in that: The identifying the second target content of the live video based on the image data includes: Based on the image data, optical flow estimation, action recognition and / or human posture estimation are used to identify the second target content of the live video.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Obtaining content keywords of the clipped video; Generative artificial intelligence is used to generate the cover and title of the edited video based on the content keywords.

6. The method according to any one of claims 1 to 4, characterized in that: After obtaining the edited video based on the result of the editing, the method further includes: Determining the category of the clipped video; generating a program list according to the category, wherein the program list includes a program list sequence; A target video is synthesized based on the program list and the target video is released.

7. The method according to claim 6, characterized in that The method further comprises: When playing the target video, obtaining the anchor background corresponding to the target video; The program list is displayed in the anchor background, and information about programs currently being played or to be played is displayed.

8. The method according to claim 6, characterized in that The method further comprises: In case of receiving a gift sent to the target video, obtaining a configured correspondence table, the correspondence table including a correspondence relationship between the gift and the program; Determining a target program corresponding to the gift from the target video based on the correspondence table; The target program is played.

9. The method according to claim 8, characterized in that The method further comprises: In the case of receiving multiple gifts sent to the target video, placing the programs corresponding to the multiple gifts into a preset queue in the order in which the gifts are received; Taking out a program from the preset queue for playing; When the programs in the preset queue are empty, the programs are played in the order of the program list.

10. A live video editing device, characterized in that: The device comprises: An acquisition module, used to acquire target modality data of the live video to be edited, wherein the target modality data includes at least audio data and image data; A determination module, configured to perform a multi-dimensional analysis on the live video based on the target modal data, and determine a highlight segment in the live video based on the analysis result; The editing module is used to edit the wonderful clips and obtain an edited video based on the editing results.

11. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images