Live video processing method, device, electronic device and readable storage medium
By extracting the audio data segments and corresponding video clips in the live video, the target video clips containing performance music are filtered out, which solves the problem of low return-viewing of live videos, realizes the function of users to quickly locate exciting clips, and improves live broadcast The experience of the platform.
Patent Information
- Application Number
- CN202211143372.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-20
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-09-20
AI Technical Summary
In the prior art, the live video viewing efficiency is low, and users need to manually drag the progress bar to watch the exciting clips in the live video.
By obtaining the audio data of the live video, extracting the audio data segment containing music as the target audio segment, thereby extracting the corresponding video segment, filtering out the target video segment containing the performance music, and adding it to the live broadcast room, the user can Play these exciting clips directly.
This improves the efficiency of live video review, and users can quickly locate the exciting parts of the live video, enriching the functions of the live broadcast platform.
Smart Images

Figure CN115866279B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video technology, and in particular to a live video processing method, device, electronic equipment and readable storage medium. Background Art
[0002] With the development of various live broadcast platforms, more and more users are using live broadcast platforms, and the requirements for live broadcast platforms are getting higher and higher. For example, in many cases, when users want to repeatedly play the highlights of a live video, they need to replay the live video.
[0003] In the prior art, when a user replays a live video that has already been generated, he can only replay it according to the video progress, or the user needs to manually drag the progress bar to control the replay progress of the live video. Therefore, the efficiency of replaying the live video is low. Summary of the invention
[0004] The purpose of the embodiments of the present invention is to provide a live video processing method, device, electronic device and readable storage medium to improve the efficiency of live video playback. The specific technical solution is as follows:
[0005] In a first aspect of the present invention, a live video processing method is provided, which may include:
[0006] Based on the live video to be processed, obtaining audio data corresponding to the live video to be processed;
[0007] Extracting an audio data segment containing music from the audio data as a target audio segment;
[0008] Extracting a video segment corresponding to the target audio segment from the live video to be processed to obtain an alternative video segment; the music contained in the alternative video segment is the same as the music contained in the target audio segment;
[0009] Selecting a target video segment from the candidate video segments, and adding the target video segment to the live broadcast room corresponding to the live broadcast video to be processed; the music contained in the target video segment is the music used for the performance;
[0010] In response to a triggering operation of the user on the target video segment, the target video segment is played for the user.
[0011] In a second aspect of the present invention, a live video processing device is provided, the device comprising:
[0012] An audio extraction module is used to obtain audio data corresponding to the live video to be processed based on the live video to be processed;
[0013] An audio segment analysis module, used for extracting an audio data segment containing music from the audio data as a target audio segment;
[0014] A video extraction module is used to extract a video segment corresponding to the target audio segment from the live video to be processed to obtain an alternative video segment; the music contained in the alternative video segment is the same as the music contained in the target audio segment;
[0015] A video clip screening module is used to screen a target video clip from the candidate video clips and add the target video clip to the live broadcast room corresponding to the live broadcast video to be processed; the music contained in the target video clip is the music used for the performance;
[0016] The video playing module is used to play the target video segment for the user in response to the user's triggering operation on the target video segment.
[0017] In a third aspect of the present invention, there is also provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0018] Memory, used to store computer programs;
[0019] The processor is used to implement the live video processing method as described in the first aspect when executing the program stored in the memory.
[0020] In a fourth aspect of the implementation of the present invention, a computer-readable storage medium is further provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the live video processing method described in the first aspect above.
[0021] The live video processing method provided by the embodiment of the present invention obtains the audio data corresponding to the live video to be processed based on the live video to be processed; extracts the audio data segment containing music from the audio data as the target audio segment; extracts the video clip corresponding to the target audio segment from the live video to be processed to obtain the candidate video clip; the music contained in the candidate video clip is the same as the music contained in the target audio segment; selects the target video clip from the candidate video clip, and adds the target video clip to the live room corresponding to the live video to be processed; the music contained in the target video clip is the music used for the performance; in response to the user's triggering operation on the target video clip, plays the target video clip for the user. In this way, the video clip containing music is automatically extracted from the audio data of the live video to be processed, and the target video clip is selected based on the video clip containing music. Since the target video clip contains the music used for the performance, it can be considered that the target video clip is a more exciting clip in the live video to be processed. In this way, by providing the target video clip to the user, the user can directly watch the more exciting parts of the live video according to the target video clip, thereby improving the efficiency of live video playback to a certain extent and enriching the functions of the live broadcast platform. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.
[0023] Figure 1 A flowchart of the steps of the live video processing method provided by an embodiment of the present invention;
[0024] Figure 2 A structural diagram of a live video processing device provided by an embodiment of the present invention;
[0025] Figure 3 A schematic diagram of an application of a live video processing device provided by an embodiment of the present invention;
[0026] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0027] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0028] Reference Figure 1 , which shows a flowchart of the steps of the live video processing method provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:
[0029] Step 101: Based on the live video to be processed, obtain audio data corresponding to the live video to be processed.
[0030] In the embodiment of the present invention, the above-mentioned live video to be processed refers to a video that needs to be watched back, which can be a live video generated by a live broadcast platform based on the live broadcast, or it can also be a video uploaded by the anchor himself. The embodiment of the present invention does not impose any specific restrictions on the source of the live video to be processed.
[0031] Among them, the audio data corresponding to the live video to be processed refers to the audio stream contained in the video stream. Specifically, the video source file corresponding to the video containing sound usually contains files in video format and audio format. Therefore, according to the format of the live video to be processed and based on the video format protocol description, the corresponding audio format file can be extracted from it, which is the audio data corresponding to the live video to be processed. For example, when the live video to be processed is in MP4 format, the corresponding audio data can be obtained by obtaining the MP3 format file from the storage file of the video. Specifically, the acquisition of audio data can be achieved in the form of software.
[0032] Step 102: extract an audio data segment containing music from the audio data to serve as a target audio segment.
[0033] Among them, since the clips containing music are often the more exciting clips in the live video, users have a high demand for replaying this clip, so the audio data obtained above can be detected to identify the clips containing music.
[0034] Specifically, the audio data may be segmented, and the data feature vectors of each segment of the audio data may be extracted to match the segment with a preset music library. When there is at least one matching music in the music library, the corresponding segment of the audio data may be used as the target audio segment. Alternatively, a music recognition model may be pre-trained, and the target audio segment may be obtained through the model.
[0035] Step 103: extract a video segment corresponding to the target audio segment from the live video to be processed to obtain an alternative video segment; the music contained in the alternative video segment is the same as the music contained in the target audio segment.
[0036] Among them, after obtaining the target audio segment through the above step 102, the start time and the end time of the target audio segment can be obtained based on the position of the target audio segment in the original audio data, so that based on the start time and the end time of the target audio segment, the video clips corresponding to the above start time and the end time can be cut out from the live video to be processed as alternative video clips.
[0037] Furthermore, since the live video to be processed corresponds to the initial audio data, the music contained in the alternative video segment obtained based on the start and end times of the target audio segment is the same as that in the target audio segment, that is, the alternative video segment and the target audio segment are one-to-one corresponding.
[0038] Step 104: Filter a target video segment from the candidate video segments, and add the target video segment to the live broadcast room corresponding to the live broadcast video to be processed; the music contained in the target video segment is the music used for the performance.
[0039] Step 105: In response to a triggering operation of the user on the target video segment, play the target video segment for the user.
[0040] Among them, since there may be a situation where the alternative video clips contain music, but the video content corresponding to the music clip is not the wonderful clip that the user needs (for example, the music is just background music played by the host), it is necessary to further filter out the wonderful clips that the user needs from the above alternative video clips, that is, the clips containing performances.
[0041] Specifically, the selected target video clips can be added to the corresponding live broadcast room. When the user enters the live broadcast room, he can select the target video clip he wants to watch to trigger the playback of the corresponding target video clip. Optionally, the user can also share, collect, and perform other operations on the above target video clips.
[0042] Optionally, the user may also download the target video clip to set the audio of the downloaded target video clip as a ringtone.
[0043] Optionally, all target video clips generated by different live broadcast rooms can be uniformly displayed in the display area of the live broadcast platform. Furthermore, it is possible to receive users' likes operations to generate a like list, and generate a music list, a dance list, etc. for target video clips of different categories according to the number of likes, so that more users can understand the characteristics of different live broadcast rooms and then choose the live broadcast room they are interested in to watch.
[0044] In summary, in the embodiment of the present invention, based on the live video to be processed, the audio data corresponding to the live video to be processed is obtained; the audio data segment containing music is extracted from the audio data as the target audio segment; the video clip corresponding to the target audio segment is extracted from the live video to be processed to obtain the candidate video clip; the music contained in the candidate video clip is the same as the music contained in the target audio segment; the target video clip is selected from the candidate video clips, and the target video clip is added to the live room corresponding to the live video to be processed; the music contained in the target video clip is the music used for the performance; in response to the user's triggering operation on the target video clip, the target video clip is played for the user. In this way, the video clip containing music is automatically extracted from the audio data of the live video to be processed, and the target video clip is selected based on the video clip containing music. Since the target video clip contains the music used for the performance, it can be considered that the target video clip is a more exciting clip in the live video to be processed. In this way, by providing the target video clip to the user, the user can directly watch the more exciting parts of the live video according to the target video clip, thereby improving the efficiency of live video playback to a certain extent and enriching the functions of the live broadcast platform.
[0045] Optionally, the operation of selecting the target video segment from the candidate video segments may include the following steps in this embodiment of the present invention:
[0046] Step 201: For any candidate video segment, perform person detection on video frames in the candidate video segment to obtain a person detection result.
[0047] Step 202: When the person detection result indicates that a person exists in the candidate video segment, behavior recognition is performed on the person in the candidate video segment to obtain a behavior recognition result.
[0048] Step 203: When the behavior recognition result indicates that the character performs a designated performance behavior, the candidate video segment is determined as the target video segment.
[0049] The image content in the target video clip may be adapted to the music included. For example, the music included may be generated by the video subject singing, or may be the accompaniment music for the video subject dancing, or may be the music produced by the video subject playing a musical instrument. The video subject refers to the person (e.g., anchor) included in the video.
[0050] Specifically, each candidate video clip can be detected frame by frame to detect whether there is a video subject, that is, whether there is a person. Specifically, the detection of the person can be achieved through a target detection algorithm (for example, Yolo v3 algorithm, etc.) to determine whether there is a person in each candidate video clip. Optionally, a quantity threshold can be set, and the video frame with the person is used as the target video frame. When the number of target video frames contained in an alternative video clip exceeds the above quantity threshold, it can be considered that there is a person in the alternative video clip, wherein the specific value of the quantity threshold can be determined according to the number of frames contained in the alternative video clip.
[0051] After obtaining the candidate video clips containing the characters, the characters' behaviors can be further identified. Specifically, the character's behavior identification can include lip shape detection, body detection, etc. Through lip shape detection, it can be determined whether the character is singing, and through body detection, it can be determined whether the character is dancing or playing a musical instrument. The above behavior identification results may include speaking, singing, dancing, playing a musical instrument, etc. Specifically, the above lip shape detection can determine whether the character is singing by detecting whether the lip shape of the character matches the current music lyrics.
[0052] The above-mentioned designated performance behaviors may include singing, dancing, playing musical instruments, etc. Of course, other types of performance behaviors may also be included, which are not limited in the embodiment of the present invention. Specifically, when the character in the candidate video clip performs the above-mentioned designated performance behaviors through behavior detection, it indicates that the candidate video clip is a wonderful video clip required by the user, so the candidate video clip can be used as the target video clip.
[0053] In the embodiment of the present invention, for any of the alternative video clips, person detection is performed on the video frames in the alternative video clip to obtain a person detection result; when the person detection result indicates that there is a person in the alternative video clip, behavior recognition is performed on the person in the alternative video clip to obtain a behavior recognition result; when the behavior recognition result indicates that the person performs a specified performance behavior, the alternative video clip is determined as the target video clip. In this way, by using the alternative video clips in which there are people and the people perform specified performance behaviors as the target video clips, video clips that do not contain performance behaviors can be eliminated from all the alternative video clips that contain music, avoiding the generation of video clips that contain background music but whose video content does not meet user needs, thereby further improving the quality of generating wonderful video clips and improving user experience.
[0054] Optionally, the operation of selecting the target video segment from the candidate video segments may include the following steps in this embodiment of the present invention:
[0055] Step 301: compress the candidate video clip according to a preset format.
[0056] The preset format refers to the resolution and frame number of the video. By compressing the candidate video clips according to the preset format, all the candidate video clips can be kept in a unified format, thereby facilitating subsequent processing operations. For example, the preset format can be a resolution of 50*50 and a frame number of 400. Of course, the preset format can also be set according to the original format of the video clip. Just select the appropriate format without affecting the video recognition rate.
[0057] Step 302: Use the compressed candidate video clip as an input of a preset video recognition model, and obtain the category to which the candidate video clip output by the preset video recognition model belongs.
[0058] Step 303: When the category characterizes that the candidate video segment contains a specified performance behavior, the candidate video segment is determined as the target video segment. The preset video recognition model is trained using sample video segments compressed in the preset format and sample labels of the sample video segments as training data, and the sample labels are used to characterize the specified performance behavior contained in the sample video segment.
[0059] The preset video recognition model may be pre-trained and used to classify candidate video clips.
[0060] Specifically, a deep learning model can be pre-established as an initial video recognition model, and a large amount of video data containing music can be collected as sample video clips. According to the content contained in the sample video, it can be divided into different categories and sample labels can be added. The sample labels correspond to the categories one by one. For example, singing, dancing, musical instruments and other categories can be included. After the above sample videos are uniformly compressed into the above preset format, a sample set is generated with the above sample labels. The above initial video recognition model is trained with the sample set to obtain a trained video recognition model. Therefore, the compressed alternative video clip can be used as input, and the category to which the alternative video clip belongs can be obtained based on the video recognition model, that is, the performance content contained in the alternative video clip. When the category is obtained as the above singing, dancing, and musical instrument category, it is considered that the alternative video clip contains the specified performance behavior, and the alternative video clip can be used as the target video clip.
[0061] In an embodiment of the present invention, the candidate video clip is compressed according to a preset format; the compressed candidate video clip is used as the input of a preset video recognition model to obtain the category of the candidate video clip output by the preset video recognition model; when the category represents that the candidate video clip contains a specified performance behavior, the candidate video clip is determined as the target video clip; wherein the preset video recognition model is trained using sample video clips compressed according to the preset format and sample labels of the sample video clips as training data, and the sample labels are used to represent the specified performance behavior contained in the sample video clips. In this way, by using sample video clips compressed according to the preset format as training samples, recognition errors caused by different formats can be avoided, thereby improving the training efficiency of the video recognition model. At the same time, compressing the candidate video clips to the preset format can be more suitable for the trained video recognition model, further improving the recognition accuracy.
[0062] Optionally, after selecting the target video segment from the candidate video segments, the embodiment of the present invention may further include the following steps:
[0063] Step 401: Add a type tag to the target video segment according to the specified performance behavior included in the target video segment; the type tag is used to characterize the type of the specified performance behavior included in the target video segment.
[0064] The above type tags can be used to distinguish the types of content contained in each target video clip, for example, they can include wonderful video types such as dancing, singing, and musical instruments. Specifically, after the categories of the candidate video clips are obtained based on the above steps 202 and 302, corresponding tags can be directly added based on the categories, that is, the above type tags correspond to the content contained in the target video clip.
[0065] In the embodiment of the present invention, a type tag is added to the target video segment according to the specified performance behavior included in the target video segment; the type tag is used to characterize the type of the specified performance behavior included in the target video segment. In this way, by adding corresponding type tags to each target video segment, the user can determine whether the video segment is the target type video that the user wants to watch through the video tag without playing the video, thereby improving the user's convenience in viewing the favorite video segment and further improving the efficiency of watching the live video back.
[0066] Optionally, the operation of extracting the audio data segment containing music from the audio data as the target audio segment may specifically include the following steps in this embodiment of the present invention:
[0067] Step 501: Segment the audio data into multiple audio data segments according to a preset duration, and identify audio data segments containing music from the multiple audio data segments to obtain candidate audio segments.
[0068] Among them, after obtaining the above-mentioned audio data, it can be divided according to the preset duration, or the live audio stream can be sampled at intervals according to the preset duration, and the above-mentioned preset duration can be set according to actual needs. For example, the preset duration can be the maximum duration required for a performance. Specifically, since the time for performing a song or a dance is usually kept within 5 minutes, the above-mentioned preset duration can be set to 5 minutes, and the above-mentioned audio data can be divided into multiple audio data segments with a duration of 5 minutes. Furthermore, after obtaining multiple audio data segments, they can be identified in segments, and the audio data segments containing music can be detected as alternative audio segments.
[0069] Step 502: For any candidate audio segment, if the music contained in the candidate audio segment and the music contained in an adjacent candidate audio segment belong to the same music, merge the candidate audio segment and the adjacent candidate audio segment to obtain one target audio segment.
[0070] Step 503: When the music included in the candidate audio segment and the music included in the adjacent candidate audio segment do not belong to the same music, determine the candidate audio segment as one of the target audio segments.
[0071] After the above step 501, there may be a situation where the time period of the same music is divided into two or more adjacent candidate audio segments, so it is necessary to further identify the divided candidate audio segments.
[0072] Among them, the adjacent alternative audio segments can be the audio segments before the current alternative audio segment, or the audio segments after the current alternative audio segment. Specifically, the alternative audio segments can be sorted in chronological order, and starting from the first alternative audio segment, the music contained in its end position and the music contained in the start position of the adjacent alternative audio segment are detected. When the music contained in the end position of a certain alternative audio segment and the music contained in the start position of the next alternative audio segment belong to the same music, it is considered that the wonderful clip lasts for the time of two clips. Therefore, it is necessary to merge the alternative audio segment with the next alternative audio segment as a complete target audio segment. Among them, the end time of the next alternative audio segment and the start time of the current alternative audio segment can be obtained as the start and end time of the complete target audio segment.
[0073] Optionally, after obtaining the above-mentioned complete target audio segment, the alternative audio segments adjacent to the complete target audio segment may continue to be detected. That is to say, when the duration of the music exceeds two or more segments, the start time of the first segment and the end time of the last segment may be used as the start and end time of the final target audio segment, until the music in any target audio segment does not belong to the same piece of music as the music contained in the adjacent target audio segment.
[0074] In an embodiment of the present invention, the audio data is divided into multiple audio data segments according to a preset time length, and an audio data segment containing music is identified from the multiple audio data segments to obtain an alternative audio segment; for any of the alternative audio segments, if the music contained in the alternative audio segment and the music contained in the adjacent alternative audio segment belong to the same music, the alternative audio segment and the adjacent alternative audio segment are merged to obtain one target audio segment; if the music contained in the alternative audio segment and the music contained in the adjacent alternative audio segment do not belong to the same music, the alternative audio segment is determined as one target audio segment. In this way, by merging the alternative audio segments belonging to the same music into one target audio segment, it is possible to avoid splitting the audio containing the same wonderful clip, so that the target video segment finally generated maintains coherence, thereby improving the quality of the generated target video clip.
[0075] Optionally, the operation of identifying the audio data segment containing music from the multiple audio data segments to obtain the candidate audio segment may specifically include the following steps in this embodiment of the present invention:
[0076] Step 601: For any of the audio data segments, use the audio data segment as an input of a preset audio recognition model, and obtain an audio recognition result output by the preset audio recognition model.
[0077] Step 602: If the audio recognition result indicates music contained in the audio data segment, the audio data segment is determined as the candidate audio segment. The preset audio recognition model is trained by using sample audio segments containing music as positive training samples and sample audio segments not containing music as negative training samples.
[0078] The preset audio recognition model may be pre-trained and used to identify audio data segments containing music from original audio segments.
[0079] Specifically, an initial deep learning model can be established in advance, and a large amount of audio data can be collected to train the model. Optionally, historical live audio data can be obtained from the live broadcast platform, and divided into positive samples and negative samples according to whether the audio data contains music. For example, the audio data segment in which the host is speaking and there is no background music can be used as a negative sample, and the audio data segment with background music can be used as a positive sample. Alternatively, a large amount of music data can be directly downloaded from the Internet and used as a positive sample.
[0080] Optionally, the above samples may also be segmented according to the above preset time length, thereby improving the recognition accuracy of the trained model for the audio data of the preset time length.
[0081] In an embodiment of the present invention, for any of the audio data segments, the audio data segment is used as the input of a preset audio recognition model, and an audio recognition result output by the preset audio recognition model is obtained; when the audio recognition result represents the music contained in the audio data segment, the audio data segment is determined as the candidate audio segment; wherein the preset audio recognition model is trained using sample audio segments containing music as positive training samples and sample audio segments not containing music as negative training samples. In this way, the efficiency of determining the candidate audio segments can be improved by directly recognizing the audio data through a pre-trained audio recognition model. At the same time, the accuracy of determining the candidate audio segments can be further improved by pre-training the audio recognition model through positive and negative samples.
[0082] Optionally, the embodiment of the present invention may further include the following steps:
[0083] Step 701: For any candidate audio segment, detect whether the end part of the music in the candidate audio segment is continuous with the start part of the music in the adjacent candidate audio segment.
[0084] Step 702: In the case of continuity, determine whether the music included in the candidate audio segment and the music included in the adjacent candidate audio segment belong to the same music.
[0085] It is understandable that when the music in adjacent alternative audio segments does not belong to the same music, there will often be an interruption in the audio data, that is, there will be a blank area in the process of switching from one piece of music to the next, resulting in discontinuity of the audio data. Therefore, it is possible to determine whether the music contained in the alternative audio segment and the adjacent alternative audio segment is continuous by detecting whether the music in the alternative audio segment and the adjacent alternative audio segment is continuous. Among them, the above-mentioned end part and start part can be a time period. For example, the last 5s of the data of the alternative audio segment can be used as the end part, and the first 5s of the data of the adjacent alternative audio segment can be used as the start part, so as to improve the accuracy of the detection.
[0086] Specifically, after obtaining the audio data of the ending part of the above-mentioned candidate audio segment and the starting part of the adjacent candidate audio segment, they can be combined to obtain a piece of test data, and the time domain graph integral is obtained based on the test data. When the time domain graph integral value of the test data is less than a preset reference value, it is considered that there is a blank area in the test data, indicating that the music of the ending part of the candidate audio segment is discontinuous with the music of the starting part of the adjacent candidate audio segment, so that it can be determined that the music contained in the two does not belong to the same music. Among them, the above-mentioned reference value can be obtained by integrating the time domain graph of a continuous audio data in advance.
[0087] Optionally, the candidate audio segment and the adjacent candidate audio segment may be output, and the user's input may be received, and it may be determined based on the input result whether the music contained in the two belongs to the same music.
[0088] In the embodiment of the present invention, for any of the candidate audio segments, it is detected whether the end part of the music in the candidate audio segment is continuous with the start part of the music in the adjacent candidate audio segment; if they are continuous, it is determined that the music contained in the candidate audio segment and the music contained in the adjacent candidate audio segment belong to the same music. In this way, by detecting whether the music in two adjacent candidate audio segments is continuous, it is possible to accurately determine whether the music contained in the two belongs to the same music, thereby further improving the accuracy of the subsequent generation of the target video segment.
[0089] Reference Figure 2 , which shows a structural diagram of a live video processing device provided by an embodiment of the present invention, such as Figure 2 As shown, the device may include:
[0090] The audio extraction module 201 is used to obtain audio data corresponding to the live video to be processed based on the live video to be processed;
[0091] An audio segment analysis module 202, configured to extract an audio data segment containing music from the audio data as a target audio segment;
[0092] The video extraction module 203 is used to extract the video segment corresponding to the target audio segment from the live video to be processed to obtain an alternative video segment; the music contained in the alternative video segment is the same as the music contained in the target audio segment;
[0093] The video clip screening module 204 is used to screen a target video clip from the candidate video clips and add the target video clip to the live broadcast room corresponding to the live broadcast video to be processed; the music contained in the target video clip is the music used for the performance;
[0094] The video playing module 205 is used to play the target video segment for the user in response to the user's triggering operation on the target video segment.
[0095] Optionally, the video clip screening module 204 includes:
[0096] A detection submodule, configured to perform person detection on video frames in any of the candidate video segments to obtain a person detection result;
[0097] A behavior recognition submodule, configured to, when the person detection result indicates that a person exists in the candidate video segment, perform behavior recognition on the person in the candidate video segment to obtain a behavior recognition result;
[0098] The first determination submodule is configured to determine the candidate video segment as the target video segment when the behavior recognition result indicates that the character performs a designated performance behavior.
[0099] Optionally, the video clip screening module 204 includes:
[0100] A compression submodule, used for compressing the candidate video clip according to a preset format;
[0101] A first acquisition submodule is used to use the compressed candidate video segment as an input of a preset video recognition model to obtain the category to which the candidate video segment output by the preset video recognition model belongs;
[0102] The second determination submodule is used to determine the alternative video segment as the target video segment when the category to which it belongs characterizes that the alternative video segment contains a specified performance behavior; wherein the preset video recognition model is trained using sample video segments compressed according to the preset format and sample labels of the sample video segments as training data, and the sample labels are used to characterize the specified performance behavior contained in the sample video segment.
[0103] Optionally, the device 20 further includes:
[0104] The label adding module is used to add a type label to the target video segment after screening the target video segment from the candidate video segments according to the specified performance behavior contained in the target video segment; the type label is used to characterize the type of the specified performance behavior contained in the target video segment.
[0105] Optionally, the audio segment analysis module 202 includes:
[0106] a segmentation submodule, configured to segment the audio data into a plurality of audio data segments according to a preset duration, and identify an audio data segment containing music from the plurality of audio data segments to obtain a candidate audio segment;
[0107] a merging submodule, configured to, for any candidate audio segment, merge the candidate audio segment with the adjacent candidate audio segment to obtain the target audio segment if the music contained in the candidate audio segment and the music contained in the adjacent candidate audio segment are the same music;
[0108] The third determination submodule is configured to determine the candidate audio segment as one of the target audio segments if the music included in the candidate audio segment and the music included in the adjacent candidate audio segment do not belong to the same music.
[0109] Optionally, the video extraction module 203 includes:
[0110] A second acquisition submodule is used to use any of the audio data segments as an input of a preset audio recognition model, and obtain an audio recognition result output by the preset audio recognition model;
[0111] The fourth determination submodule is used to determine the audio data segment as the alternative audio segment when the audio recognition result represents the music contained in the audio data segment; wherein the preset audio recognition model is trained using sample audio segments containing music as positive training samples and sample audio segments not containing music as negative training samples.
[0112] Optionally, the device 20 further includes:
[0113] A continuity detection module, used for detecting, for any of the candidate audio segments, whether the end portion of the music in the candidate audio segment is continuous with the start portion of the music in the adjacent candidate audio segment;
[0114] The fifth determination module is used to determine, in a continuous case, whether the music included in the candidate audio segment and the music included in the adjacent candidate audio segment belong to the same music.
[0115] Figure 3 FIG. 4 shows an application schematic diagram of a live video processing device provided by an embodiment of the present invention. Figure 3 As shown, the application of the live video processing device may include the following steps:
[0116] The live video stream to be processed is input into the audio extraction module; after the audio extraction module obtains the corresponding audio data, it can be input into the initial music recognition model as a training sample for training. Of course, the audio data can also be obtained from the network to train the model.
[0117] After the initial music recognition model is trained to obtain a deep learning model, the model can be uploaded to the audio segment analysis module, and the target audio segment can be obtained through the model. After the video extraction module obtains the corresponding candidate video segment through the target audio segment, it can be input into the initial video classification model as a training sample for training. Of course, the model can also be trained by obtaining video data from the Internet. After the initial video classification model is trained to obtain a deep learning model, the model can be uploaded to the video segment screening module, and the target video segment can be obtained through the model.
[0118] After obtaining the target video segments, labels can be added to each segment based on the category of the target video segment, and the labeled target video segments are output, so that the user can select a video of interest to watch.
[0119] In summary, the live video processing device provided in the embodiment of the present invention obtains the audio data corresponding to the live video to be processed based on the live video to be processed; extracts the audio data segment containing music from the audio data as the target audio segment; extracts the video clip corresponding to the target audio segment from the live video to be processed to obtain the candidate video clip; the music contained in the candidate video clip is the same as the music contained in the target audio segment; selects the target video clip from the candidate video clip, and adds the target video clip to the live room corresponding to the live video to be processed; the music contained in the target video clip is the music used for the performance; in response to the user's triggering operation on the target video clip, plays the target video clip for the user. In this way, the video clip containing music is automatically extracted from the audio data of the live video to be processed, and the target video clip is selected based on the video clip containing music. Since the target video clip contains the music used for the performance, it can be considered that the target video clip is a more exciting clip in the live video to be processed. In this way, by providing the target video clip to the user, the user can directly watch the more exciting parts of the live video according to the target video clip, thereby improving the efficiency of live video playback to a certain extent and enriching the functions of the live broadcast platform.
[0120] The embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004.
[0121] Memory 1003, used for storing computer programs;
[0122] The processor 1001 is used to execute the program stored in the memory 1003, and implements the following steps:
[0123] Based on the live video to be processed, obtaining audio data corresponding to the live video to be processed;
[0124] Extracting an audio data segment containing music from the audio data as a target audio segment;
[0125] Extracting a video segment corresponding to the target audio segment from the live video to be processed to obtain an alternative video segment; the music contained in the alternative video segment is the same as the music contained in the target audio segment;
[0126] Selecting a target video segment from the candidate video segments, and adding the target video segment to the live broadcast room corresponding to the live broadcast video to be processed; the music contained in the target video segment is the music used for the performance;
[0127] In response to a triggering operation of the user on the target video segment, the target video segment is played for the user.
[0128] Optionally, the selecting a target video segment from the candidate video segments includes:
[0129] For any of the candidate video segments, performing person detection on video frames in the candidate video segment to obtain a person detection result;
[0130] When the person detection result indicates that a person exists in the candidate video segment, performing behavior recognition on the person in the candidate video segment to obtain a behavior recognition result;
[0131] In a case where the behavior recognition result indicates that the character performs a designated performance behavior, the candidate video segment is determined as the target video segment.
[0132] Optionally, the selecting a target video segment from the candidate video segments includes:
[0133] compressing the candidate video clip according to a preset format;
[0134] Using the compressed candidate video clip as an input of a preset video recognition model, and obtaining the category to which the candidate video clip output by the preset video recognition model belongs;
[0135] In a case where the category represents that the candidate video segment includes a specified performance behavior, determining the candidate video segment as the target video segment;
[0136] The preset video recognition model is trained using sample video clips compressed according to the preset format and sample labels of the sample video clips as training data, and the sample labels are used to characterize specified performance behaviors contained in the sample video clips.
[0137] Optionally, after selecting the target video segment from the candidate video segments, the method further includes:
[0138] According to the designated performance behavior included in the target video segment, a type tag is added to the target video segment; the type tag is used to characterize the type of the designated performance behavior included in the target video segment.
[0139] Optionally, extracting an audio data segment containing music from the audio data as a target audio segment includes:
[0140] The audio data is divided into a plurality of audio data segments according to a preset time length, and an audio data segment containing music is identified from the plurality of audio data segments to obtain a candidate audio segment;
[0141] For any candidate audio segment, if the music contained in the candidate audio segment and the music contained in an adjacent candidate audio segment are the same music, merging the candidate audio segment with the adjacent candidate audio segment to obtain one target audio segment;
[0142] In the case that the music included in the candidate audio segment and the music included in the adjacent candidate audio segment do not belong to the same music, the candidate audio segment is determined as one of the target audio segments.
[0143] Optionally, the step of identifying an audio data segment containing music from the plurality of audio data segments to obtain a candidate audio segment comprises:
[0144] For any of the audio data segments, use the audio data segment as an input of a preset audio recognition model, and obtain an audio recognition result output by the preset audio recognition model;
[0145] In a case where the audio recognition result represents music contained in the audio data segment, determining the audio data segment as the candidate audio segment;
[0146] The preset audio recognition model is trained by using sample audio segments containing music as positive training samples and sample audio segments not containing music as negative training samples.
[0147] Optionally, also include:
[0148] For any of the candidate audio segments, detecting whether the end portion of the music in the candidate audio segment is continuous with the start portion of the music in the adjacent candidate audio segment;
[0149] In the case of continuity, it is determined that the music contained in the candidate audio segment and the music contained in the adjacent candidate audio segment belong to the same music.
[0150] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0151] The communication interface is used for communication between the above terminal and other devices.
[0152] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0153] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0154] In another embodiment of the present invention, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the live video processing method described in any one of the above embodiments.
[0155] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute the live video processing method described in any one of the above embodiments.
[0156] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.
[0157] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0158] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims
1. A live video processing method, characterized in that: The method comprises: Based on the live video to be processed, obtaining audio data corresponding to the live video to be processed; The audio data is divided into a plurality of audio data segments according to a preset time length, and an audio data segment containing music is identified from the plurality of audio data segments to obtain a candidate audio segment; For any of the candidate audio segments, merging the audio data of the ending part of the music in the candidate audio segment with the audio data of the starting part of the music in the adjacent candidate audio segment to obtain test data; Based on the time domain graph integral value of the test data, detecting whether there is a blank area between the end part of the music in the candidate audio segment and the start part of the music in the adjacent candidate audio segment, so as to determine whether the candidate audio segment is continuous with the adjacent candidate audio segment; In the case of continuity, determining that the music included in the candidate audio segment and the music included in the adjacent candidate audio segment belong to the same music; When the music included in the candidate audio segment and the music included in the adjacent candidate audio segment belong to the same music, merging the candidate audio segment and the adjacent candidate audio segment to obtain a target audio segment; Extracting a video segment corresponding to the target audio segment from the live video to be processed to obtain an alternative video segment; the music contained in the alternative video segment is the same as the music contained in the target audio segment; Selecting a target video segment from the candidate video segments, and adding the target video segment to the live broadcast room corresponding to the live broadcast video to be processed; the music contained in the target video segment is the music used for the performance; In response to a triggering operation of the user on the target video segment, the target video segment is played for the user.
2. The method according to claim 1, characterized in that The step of selecting a target video segment from the candidate video segments includes: For any of the candidate video segments, performing person detection on video frames in the candidate video segment to obtain a person detection result; When the person detection result indicates that a person exists in the candidate video segment, performing behavior recognition on the person in the candidate video segment to obtain a behavior recognition result; In a case where the behavior recognition result indicates that the character performs a designated performance behavior, the candidate video segment is determined as the target video segment.
3. The method according to claim 1, characterized in that The step of selecting a target video segment from the candidate video segments includes: compressing the candidate video clip according to a preset format; Using the compressed candidate video clip as an input of a preset video recognition model, and obtaining the category to which the candidate video clip output by the preset video recognition model belongs; In a case where the category represents that the candidate video segment includes a specified performance behavior, determining the candidate video segment as the target video segment; The preset video recognition model is trained using sample video clips compressed according to the preset format and sample labels of the sample video clips as training data, and the sample labels are used to characterize specified performance behaviors contained in the sample video clips.
4. The method according to claim 2 or 3, characterized in that: After selecting the target video segment from the candidate video segments, the method further includes: According to the designated performance behavior included in the target video segment, a type tag is added to the target video segment; the type tag is used to characterize the type of the designated performance behavior included in the target video segment.
5. The method according to claim 1, characterized in that The method further comprises: In the case that the music included in the candidate audio segment and the music included in the adjacent candidate audio segment do not belong to the same music, the candidate audio segment is determined as one of the target audio segments.
6. The method according to claim 5, characterized in that The step of identifying an audio data segment containing music from the plurality of audio data segments to obtain a candidate audio segment comprises: For any of the audio data segments, use the audio data segment as an input of a preset audio recognition model, and obtain an audio recognition result output by the preset audio recognition model; In a case where the audio recognition result represents music contained in the audio data segment, determining the audio data segment as the candidate audio segment; The preset audio recognition model is trained by using sample audio segments containing music as positive training samples and sample audio segments not containing music as negative training samples.
7. A live video processing device, characterized in that: The device comprises: An audio extraction module is used to obtain audio data corresponding to the live video to be processed based on the live video to be processed; An audio segment analysis module, used for extracting an audio data segment containing music from the audio data as a target audio segment; A video extraction module is used to extract a video segment corresponding to the target audio segment from the live video to be processed to obtain an alternative video segment; the music contained in the alternative video segment is the same as the music contained in the target audio segment; A video clip screening module is used to screen a target video clip from the candidate video clips and add the target video clip to the live broadcast room corresponding to the live broadcast video to be processed; the music contained in the target video clip is the music used for the performance; A video playing module, configured to play the target video segment for the user in response to a triggering operation of the user on the target video segment; The audio segment analysis module comprises: a segmentation submodule, configured to segment the audio data into a plurality of audio data segments according to a preset duration, and identify an audio data segment containing music from the plurality of audio data segments to obtain a candidate audio segment; a merging submodule, configured to merge, for any candidate audio segment, the audio data of the end part of the music in the candidate audio segment and the audio data of the start part of the music in the adjacent candidate audio segment to obtain test data; and based on the time domain graph integral value of the test data, if the music contained in the candidate audio segment and the music contained in the adjacent candidate audio segment belong to the same music, merge the candidate audio segment with the adjacent candidate audio segment to obtain one target audio segment; a continuity detection module, configured to detect, for any of the candidate audio segments, whether there is a blank area between the end of the music in the candidate audio segment and the start of the music in the adjacent candidate audio segment, so as to determine whether the candidate audio segment is continuous with the adjacent candidate audio segment; The fifth determination module is used to determine, in a continuous case, whether the music included in the candidate audio segment and the music included in the adjacent candidate audio segment belong to the same music.
8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 6 when executing a program stored in a memory.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Method, device and storage medium for obtaining video segments
CN109218746A
Dance segment identification method, dance segment identification device and storage medium
CN112100436A
Audio detection method and device, storage medium and electronic equipment
CN114596878A
