Video processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video processing method and device, electronic equipment and a storage medium. The video processing method comprises the following steps: separating an original video into an image frame sequence and an audio stream; detecting invalid image frames from the image frame sequence, and detecting invalid audio clips from the audio stream; determining an image invalid period according to the position of the invalid image frame in the original video, and determining an audio invalid period according to the position of the invalid audio clip in the original video; integrating the image invalid time period and the audio invalid time period to obtain a video invalid time period, and determining the video of the video invalid time period in the original video as an invalid video clip; and filtering invalid video clips in the original video to obtain a target video. According to the invention, the video is automatically detected from the two modes of image and sound, the invalid content can be accurately and comprehensively identified, the labor cost is saved, the invalid fragments in the video are filtered based on the detection results of the two modes, and the efficiency and accuracy of video clipping are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a video processing method, device, electronic device and storage medium. Background Art

[0002] In live broadcast scenarios, a large number of live videos will be generated. Considering the time limit of the video in the actual delivery process, it is impossible to let users watch the entire live video directly. Instead, we hope to edit and aggregate the meaningful content in the live video and provide it to users. Therefore, it is necessary to trim the meaningless or redundant clips in the live video.

[0003] In the process of realizing the present invention, the inventors found that the current cropping of live video mainly relies on manual operation, which has the problems of high cost, low cropping efficiency and poor cropping quality. Summary of the invention

[0004] The embodiments of the present invention provide a video processing method, device, electronic device and storage medium, which can save labor costs and improve the efficiency and accuracy of video cutting.

[0005] In a first aspect, an embodiment of the present invention provides a video processing method, including:

[0006] Separate the original video into image frame sequence and audio stream;

[0007] Detecting invalid image frames from the image frame sequence, and detecting invalid audio segments from the audio stream;

[0008] Determine an invalid image period according to the position of the invalid image frame in the original video, and determine an invalid audio period according to the position of the invalid audio segment in the original video;

[0009] Integrate the image invalid period and the audio invalid period to obtain a video invalid period, and determine the video in the video invalid period in the original video as an invalid video segment;

[0010] The invalid video segments in the original video are filtered out to obtain the target video.

[0011] Integrating the invalid image time periods and the invalid audio time periods can be done by taking the intersection of the invalid image time periods and the invalid audio time periods, or by taking the union of the invalid image time periods and the invalid audio time periods. Taking the intersection time period can reduce the possibility of accidental deletion of valid content, thereby obtaining a video with higher audiovisual requirements and higher content integrity; taking the union time period can quickly identify most invalid video clips, saving time and cost for post-production video editing.

[0012] The above-mentioned video processing method separates the original video into image and audio, which is convenient for subsequent separate and parallel processing of the image and audio, and is convenient for processing different modal data in different ways according to their different characteristics; the video is automatically detected from the two modalities of image and sound, so that the detection algorithm can be more targeted and can accurately and comprehensively identify invalid content; by locating the position of invalid image frames and invalid audio clips in the video, it is ensured that the image and audio can be processed synchronously on the timeline to maintain the consistency of the image and audio; integrating the invalid time periods of the image and audio can avoid omission or repeated processing, making the invalid time period marking of the video more accurate; combining the invalid detection results of the image and audio to determine the invalid video clip can ensure that the determination of the invalid video clip is more reliable; based on the detection results of the two modalities, the invalid video clips in the video are filtered out, thereby improving the efficiency and accuracy of video cropping; from the detection to the filtering of invalid content, end-to-end intelligent cropping is realized, without the need for manual operation, saving labor costs. The design of "divide first and combine later" is adopted. The first division is to divide the video into an image frame sequence and an audio stream, and detect invalid periods separately. The second combination is to merge the detected invalid periods of the image and audio, and perform synchronous processing and cropping to ensure the synchronization and consistency of the image and audio after the overall processing, and improve the quality of video cropping. Generally speaking, in live broadcast scenes, there are high requirements for both image quality and sound. Therefore, the method of performing invalid detection on the video and filtering out invalid content from the two modes of image and sound proposed in the embodiment of the present invention is particularly suitable for processing live videos generated in live broadcast scenes, and solves the problems of high cost, low cropping efficiency, and poor cropping quality caused by manual cropping of live videos.

[0013] Optionally, detecting an invalid image frame from the image frame sequence includes:

[0014] Using at least two image detection models to detect invalid image frames in the image frame sequence to obtain at least two types of invalid image frames, wherein each of the at least two image detection models is used to detect one type of invalid image frames;

[0015] The at least two types of invalid image frames are determined as the invalid image frames.

[0016] By using at least two image detection models to detect invalid image frames, different types of invalid image frames can be detected, which can more comprehensively cover various possible invalid frame scenarios and reduce missed detections of invalid frames.

[0017] Optionally, determining the image invalid period according to the position of the invalid image frame in the original video includes:

[0018] Determine, according to the position of each type of invalid image frame in the original video, an invalid time period corresponding to each type of invalid image frame, and obtain at least two types of image invalid time periods;

[0019] The union of the at least two types of image invalid time periods is calculated to obtain the image invalid time period.

[0020] Taking the union of at least two types of image invalid periods and determining the union period as the image invalid period can more comprehensively and accurately identify invalid image frames in the video, thereby improving the quality and efficiency of video processing and handling.

[0021] Optionally, detecting an invalid audio segment from the audio stream includes:

[0022] Using at least two audio detection models to detect invalid audio segments in the audio stream to obtain at least two types of invalid audio segments, wherein each of the at least two audio detection models is used to detect one type of invalid audio segment;

[0023] The at least two types of invalid audio segments are determined as the invalid audio segments.

[0024] Using at least two audio detection models to detect invalid audio clips can realize the detection of different types of invalid audio clips, can more comprehensively cover various possible invalid audio clip scenarios, and reduce the missed detection of invalid audio clips.

[0025] Optionally, the determining the audio invalid period according to the position of the invalid audio segment in the original video includes:

[0026] Determine, according to the position of each type of invalid audio segment in the original video, an invalid time period corresponding to each type of invalid audio segment, to obtain at least two types of audio invalid time periods;

[0027] The union of the at least two types of audio invalid time periods is calculated to obtain the audio invalid time period.

[0028] Taking the union of at least two types of audio invalid time periods and determining the union period as the audio invalid time period can more comprehensively and accurately identify invalid audio segments in the video, thereby improving the quality and efficiency of video processing and handling.

[0029] Optionally, filtering out the invalid video segments in the original video to obtain the target video includes:

[0030] Filtering out the invalid video segments in the original video to obtain valid video segments;

[0031] The video segments that meet the preset filtering conditions in the valid video segments are filtered out to obtain the target video.

[0032] By filtering out valid video segments that meet preset filtering conditions, the image quality can be further improved.

[0033] Optionally, separating the original video into an image frame sequence and an audio stream comprises:

[0034] Separating the original video into images and sounds;

[0035] Downsampling the image to obtain the image frame sequence;

[0036] The sound is segmented using a sliding window to obtain the audio stream.

[0037] Downsampling images can reduce the amount of data processing and storage space requirements. Sliding window segmentation of sounds can improve the efficiency and accuracy of audio processing.

[0038] In a second aspect, an embodiment of the present invention provides a video processing device, including:

[0039] A separation module, used for separating the original video into an image frame sequence and an audio stream;

[0040] A detection module, used to detect invalid image frames from the image frame sequence and invalid audio segments from the audio stream;

[0041] a determination module, configured to determine an invalid image period according to a position of the invalid image frame in the original video, and determine an invalid audio period according to a position of the invalid audio segment in the original video;

[0042] An integration module, configured to integrate the image invalid period and the audio invalid period to obtain a video invalid period, and determine the video in the video invalid period in the original video as an invalid video segment;

[0043] The filtering module is used to filter out the invalid video segments in the original video to obtain the target video.

[0044] In a third aspect, an electronic device provided by an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, a video processing method as in any embodiment of the present invention is implemented.

[0045] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, a video processing method as in any embodiment of the present invention is implemented.

[0046] The description of the second, third and fourth aspects of the embodiments of the present invention can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second, third and fourth aspects can refer to the beneficial effects analysis of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 is a flow chart of a video processing method provided by an embodiment of the present invention;

[0049] Figure 2 is another flow chart of the video processing method provided by an embodiment of the present invention;

[0050] Figure 3 is an example diagram of a video processing method provided by an embodiment of the present invention;

[0051] Figure 4 is a structural schematic diagram of a video processing device provided by an embodiment of the present invention;

[0052] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0054] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] Figure 1 The video processing method provided by the embodiment of the present invention is a flowchart of the video processing method provided by the embodiment of the present invention. The video processing method provided by the embodiment of the present invention can be applied to the scene where invalid content in the live video needs to be trimmed. The video processing method can be executed by the video processing device provided by the embodiment of the present invention, and the device can be implemented in software and / or hardware. In a specific embodiment, the device can be integrated into an electronic device, and the electronic device can be a personal computer, a computer, a server, etc. The following embodiment is described by taking the example of integrating the video processing device into an electronic device, refer to Figure 1 The video processing method of this embodiment may include the following steps:

[0056] Step 110, separating the original video into an image frame sequence and an audio stream.

[0057] The original video is a video that needs to be trimmed to remove redundant or invalid content. It can be a live video, speech video, conference video, event video, etc. The original video consists of images and sounds. The image frame sequence can be a static image extracted from the original video. These static images are arranged in chronological order and can form video content when played continuously. Each frame image in the image frame sequence has attribute information, which can include the position information of the frame image in the original video. The position information can be represented by information such as position coding and timestamp; of course, the attribute information can also include other information such as image resolution and format. The audio stream refers to the continuous audio signal extracted from the original video. The audio stream can include all sound data, such as speech, dialogue, background music, environmental sound effects, etc.

[0058] Some multimedia processing tools can be used to separate the original video into an image frame sequence and an audio stream. For example, multimedia processing tools can be used to split the original video into image frames and save them as a series of image files, and the formats of these image files are, for example, .jpg or .png formats. For example, if there is a 30-second original video and 2 images are extracted per second, an image frame sequence containing 60 frames of images can be obtained. Multimedia processing tools can be used to extract the audio track in the original video and save it as a separate audio file, and the format of the audio file is, for example, .wav or .mp3 format. For example, if there is a 30-second original video, the audio sampling rate is 44.1kHz, mono, and the bit depth is 16 bits, then the audio stream can be saved as a .wav file, which contains 10 seconds of audio data with a sampling rate of 44.1kHz.

[0059] Image frame sequences and audio streams represent the visual and auditory data of the original video, respectively. By extracting and separating them, it is easier to analyze and process the original video in different modes. Visual data and auditory data have different characteristics, which can be processed in different ways according to the characteristics of different modal data. In addition, separating the original video into image frame sequences and audio streams can also facilitate the subsequent synchronous and parallel processing of visual data and auditory data, thereby improving data processing efficiency.

[0060] Step 120: Detect invalid image frames from the image frame sequence, and detect invalid audio segments from the audio stream.

[0061] Invalid image frames and invalid audio clips may be content that does not contribute positively to the audience experience and content quality. Images have features such as brightness, color, resolution, and the shape of the presented object. Appropriate image detection algorithms or models can be designed or selected based on these features to detect invalid image frames. For example, a brightness detection model can be used to detect image brightness. Images that are too bright or too dark are usually invalid; a blurred image detection model can be used to detect blurred images. Blurred images are usually invalid. Invalid image frames may include, for example, black screen images, images without signals, images with highly repetitive content, blurred images, distorted images, etc. Audio has features such as frequency, waveform, volume, and loudness. Appropriate audio detection algorithms or models can be designed or selected based on these features to detect invalid audio clips. For example, a volume detection model can be used to detect volume. Audio clips with too low volume are usually invalid; a noise detection model can be used to detect noise. Noise clips are usually invalid. Invalid audio clips may include, for example, silent segments, noise segments, abnormal speech speed segments, etc. Abnormal speech speed segments may include segments that speak too fast or too slow.

[0062] Since the original video is separated into an image frame sequence and an audio stream, the data of the two modes are independent. Therefore, the detection of invalid image frames from the image frame sequence and the detection of invalid audio segments from the audio stream can be performed in parallel to shorten the detection time and improve the detection efficiency. The invalid image frames detected can have one or more frames, and each frame image has position information, and the position information indicates the position of the corresponding frame image in the original video. The invalid audio segment can also have one or more segments, and each audio segment also has position information, and the position information indicates the position of the corresponding audio segment in the original video.

[0063] Furthermore, after the image and audio are separated, the most appropriate and optimized detection algorithms can be used respectively. Each modality and type of data can be analyzed independently according to its characteristics, reducing misjudgments and missed judgments and improving detection accuracy. The quality of image and audio processing will not affect each other. An error in one part will not directly affect the processing results of the other part. During the detection process, if a detection model (such as an image detection model) fails or errors, it can be corrected separately without affecting another detection model (such as an audio detection model), and the overall system fault tolerance is enhanced. In addition, by separating images and audio, the complex video processing task is split into two parts: image processing and audio processing, which simplifies the processing difficulty of each step and facilitates debugging and optimization.

[0064] Step 130, determining an invalid image period according to the position of the invalid image frame in the original video, and determining an invalid audio period according to the position of the invalid audio segment in the original video.

[0065] The image invalid period is a time period consisting of the corresponding time of invalid image frames, and can be a time period consisting of multiple consecutive invalid image frames, that is, these invalid image frames appear continuously on the time axis, and an invalid period can be defined according to the start time and end time of the consecutive invalid image frames. For example, if 10 invalid image frames appear continuously on the time axis, the timestamp of the start frame is 00:01:05, and the timestamp of the end frame is 00:01:10, then the corresponding image invalid period is 00:01:05 to 00:01:10. The determined image invalid period can have one or more segments.

[0066] The audio invalid period is a time period consisting of the time corresponding to the invalid audio segment. An audio invalid period can be defined according to the start time and end time of an invalid audio segment. For example, if the start time of an invalid audio segment is 00:02:18 and the end time is 00:02:27, the corresponding audio invalid period is 00:02:18 to 00:02:27. The determined audio invalid period can also have one or more segments.

[0067] Invalid image frames and invalid audio clips are detected separately, and the invalid image periods and audio invalid periods are determined in the original video according to the detection results, providing data support for subsequent data integration and data filtering. By clarifying the invalid image periods and invalid audio periods, invalid clips can be quickly located and deleted during post-editing, simplifying operations. Furthermore, by determining the two types of invalid periods, it is also convenient to re-associate the data of the two modalities in the future, align the image and audio on the timeline, and make the association between multimodal data closer and more accurate, ensuring that the content after editing is more coherent and consistent, and improving the overall processing effect. With reference to the two types of invalid periods, when deleting invalid clips later, the image and audio can be updated synchronously to ensure data consistency and avoid situations where the image and audio cannot correspond.

[0068] Step 140 , integrating the image invalid period and the audio invalid period to obtain the video invalid period, and determining the video in the video invalid period in the original video as an invalid video segment.

[0069] The specific integration operation may be to find the intersection of the image invalid period and the audio invalid period, or to find the union of the image invalid period and the audio invalid period, and the intersection period or the union period may be determined as the video invalid period. The video invalid period refers to the invalid period determined by combining the image and the audio in the original video. There may be multiple video invalid periods, and each video invalid period corresponds to a segment in the original video, that is, an invalid video segment. One video invalid period corresponds to one invalid video segment, and there may be multiple invalid video segments.

[0070] The intersection period refers to the period corresponding to when both the image frame and the audio segment are judged to be invalid, that is, only when a segment is determined to be invalid in both the image and audio dimensions, its corresponding period is determined to be an invalid period. The union period refers to the period corresponding to when either the image frame or the audio segment is judged to be invalid, that is, as long as one of the image or audio dimensions of a segment is determined to be invalid, its corresponding period is determined to be an invalid period.

[0071] Determining the intersection period as the invalid video period, with stricter screening conditions, can effectively avoid false alarms caused by small changes or interference in a single dimension, thereby reducing the possibility of mistaken deletion of valid content, and can be applied to videos with relatively high requirements for visual and auditory quality. In addition, taking the intersection period can also help retain more useful information, and can also be applied to videos with high content integrity requirements. Determining the union period as the invalid video period, with looser screening conditions, can help to more comprehensively clean up invalid clips, and is suitable for videos with relatively low requirements for visual and / or auditory quality. Taking the union period can quickly identify most invalid clips, saving time and cost for post-video editing. In practical applications, these two methods can be used according to actual scenarios or needs.

[0072] Step 150, filtering out invalid video segments in the original video to obtain the target video.

[0073] Specifically, after filtering out invalid video clips in the original video, some auxiliary processing can be done to improve the quality of the target video. Auxiliary processing can include smooth transition, audio smoothing, re-encoding, etc. Smooth transition For example, if there is an obvious clipping node, a fade-in and fade-out effect can be added to make the transition more natural. Audio smoothing For example, editing may cause audio incoherence, and audio smoothing processing such as fade-in and fade-out effects or transition sound effects processing can be used to make the audio coherent and natural. Re-encoding For example, the video can be re-encoded to ensure that the format and quality of the final video meet the requirements.

[0074] In this embodiment, the original video is separated into image and audio, which is convenient for subsequent separate and parallel processing of the image and audio, and for processing data of different modes in different ways according to their different characteristics; the video is automatically detected from the two modes of image and sound, so that the detection algorithm can be more targeted and can accurately and comprehensively identify invalid content; by locating the position of invalid image frames and invalid audio clips in the video, synchronous processing on the timeline is ensured to maintain the consistency of image and audio; the invalid time periods of the image and audio are integrated to avoid omission or repeated processing, making the invalid time period marking of the video more accurate; the invalid video clip is determined in combination with the invalid detection results of the image and audio, which can ensure that the determination of the invalid video clip is more reliable; the invalid clips in the video are filtered out based on the detection results of the two modes, thereby improving the efficiency and accuracy of video cropping; from the detection to the filtering of invalid content, end-to-end intelligent cropping is realized, without the need for manual operation, saving labor costs. The design of "divide first and combine later" is adopted. The first division is to divide the video into an image frame sequence and an audio stream, and detect invalid periods separately. The second combination is to merge the detected invalid periods of the image and audio, and perform synchronous processing and cropping to ensure the synchronization and consistency of the image and audio after the overall processing, and improve the quality of video cropping. Generally speaking, in live broadcast scenes, there are high requirements for both image quality and sound. Therefore, the method of performing invalid detection on the video and filtering out invalid content from the two modes of image and sound proposed in this embodiment is particularly suitable for processing live videos generated in live broadcast scenes, and solves the problems of high cost, low cropping efficiency, and poor cropping quality caused by manual cropping of live videos.

[0075] The video processing method provided by the embodiment of the present invention is further described below. Figure 2 As shown, the method of this embodiment includes:

[0076] Step 210, separating the original video into images and sounds.

[0077] That is, the original video is separated into visual data and auditory data. Some common multimedia processing tools can be used to separate the original video.

[0078] Step 220, downsampling the image to obtain an image frame sequence.

[0079] Since when the current frame image in the video is invalid, the previous and subsequent frames often also contain invalid content, downsampling can be used to sample the image to reduce the amount of data processing and reduce storage space requirements. Downsampling is a sampling method that reduces the number of image frames. Through downsampling, one frame can be extracted from the original image every few frames to form a new image frame sequence. Specifically, a downsampling factor can be set to uniformly downsample the original image according to the downsampling factor.

[0080] For example, suppose that the images separated from the original video are represented by a set I, that is, I = {I1, I2, ..., I n}, that is, after separation, we get a set of n consecutive images, I n represents the nth image in set I. The frame reduction factor is λ f It means that after downsampling the image, the image frame sequence obtained is represented by set I ′ Indicates that I ′ = {I ′ 1,I ′ 2,…,I ′ m}, I ′ m Represents the set I ′ The mth image in , m, n are both integers greater than 2, and m is less than n, then the downsampling process can be expressed as:

[0081] I ′ = Downsample(I,λ f )

[0082] Step 221: Use at least two image detection models to detect invalid image frames in the image frame sequence to obtain at least two types of invalid image frames.

[0083] The image detection model can be obtained by training the training image data in advance, and can be a detection model implemented by artificial intelligence (AI). There can be at least two image detection models, each of which is used to detect a type of invalid image frame. The types of invalid image frames include black screen images, no signal images, repeated images, blurred images, distorted images, etc. For example, for black screen images, the image detection model can be used to detect the distribution of pixel colors in the frame. For example, if most of the pixels are pure black or a specific color, it can be identified as a black screen image. For example, for repeated images, the image detection model can be used to compare the similarity between frames. If the similarity between frames is too high, it can be considered as a repeated image. For repeated images with high similarity between frames, one frame can be retained. For example, in multiple frames of repeated images, the first frame or the last frame can be retained for subsequent processing. For blurred images, the image detection model can be used to detect the clarity of the picture based on the variance of the Laplace transform, and low variance values ​​usually correspond to blurred images. At least two image detection models can detect the image frame sequence in parallel to improve detection efficiency.

[0084] By using at least two image detection models to detect invalid image frames, different types of invalid image frames can be detected, which can more comprehensively cover various possible invalid frame scenarios and reduce missed detections of invalid frames.

[0085] Step 222, determining the invalid time period corresponding to each type of invalid image frame according to the position of each type of invalid image frame in the original video, and obtaining at least two types of image invalid time periods.

[0086] All types of invalid image frames can be determined as invalid image frames, that is, invalid image frames cover all types of invalid image frames. The invalid period corresponding to each type of invalid image frame can be determined according to the start time and end time of the consecutive occurrence of such invalid image frames. At least two types of image invalid periods may include black screen image period, no signal image period, highly repetitive content image period, blurred image period, distorted image period, etc. By determining the invalid period corresponding to each type of invalid image frame, it is convenient to integrate and filter different types of invalid image frames in the future.

[0087] Step 223, finding the union of at least two types of image invalid time periods to obtain the image invalid time period.

[0088] The image invalid period is the invalid period in the original video identified from the image dimension. For example, there are two types of image invalid periods. Assuming that the black screen image invalid period is 00:01:05-00:01:10 and the blurred image invalid period is 00:01:10-00:01:18, the final image invalid period can be 00:01:05-00:01:18. Taking the union of at least two types of image invalid periods as the image invalid period, the invalid image frames in the video can be identified more comprehensively and accurately, thereby improving the quality and efficiency of video processing and handling.

[0089] Step 230, perform sliding window segmentation on the sound to obtain an audio stream.

[0090] The basic principle of sliding window segmentation is to divide the audio signal into multiple overlapping or non-overlapping windows, each window representing the audio data of a time period. This segmentation method helps to improve the efficiency and accuracy of audio processing because it allows the audio data in each window to be analyzed independently while retaining the temporal continuity of the audio signal. Specifically in this embodiment, a sliding window with a smaller time range can be set, and the continuous audio signal can be segmented into multiple smaller, relatively independent audio segments without overlapping using this sliding window. These segments are called audio streams.

[0091] Step 231: Use at least two audio detection models to detect invalid audio segments in the audio stream to obtain at least two types of invalid audio segments.

[0092] The audio detection model can be obtained by training audio data in advance, and can be a detection model implemented by artificial intelligence. There can be at least two audio detection models, each of which is used to detect a type of invalid audio segment. The types of invalid audio segments include silent segments, noise segments, abnormal speech speed segments, etc. For example, for silent segments, the audio detection model can be used to detect the amplitude or sound intensity of the audio stream corresponding to each sliding window. If the amplitude or sound intensity of the audio segment is lower than a certain threshold, it can be identified as a silent segment. For noise segments, the audio detection model can be used to use a spectrum analysis method to analyze the spectrum characteristics of the audio signal, and identify whether it is a noise segment based on the spectrum characteristics. The spectrum characteristics of the noise segment include line spectrum and continuous spectrum. For abnormal speech speed segments, the audio detection model can be used to use a speech feature extraction method to determine the speech speed by analyzing the time characteristics of the audio. Too fast or too slow speech speed may cause the user to be unable to hear or recognize the content, which belongs to abnormal segments. At least two audio detection models can detect the audio stream in parallel to improve detection efficiency.

[0093] Step 232: Determine the invalid time period corresponding to each type of invalid audio segment according to the position of each type of invalid audio segment in the original video, and obtain at least two types of audio invalid time periods.

[0094] All types of invalid audio segments can be determined as invalid audio segments, that is, invalid audio segments cover various types of invalid audio segments. At least two types of audio invalid time periods can include a silent time period, a noise time period, an abnormal speech speed time period, etc. By determining the invalid time periods corresponding to various types of invalid audio segments, it is convenient to integrate and filter different types of invalid audio segments in the future.

[0095] Step 233, finding the union of at least two types of audio invalid time periods to obtain the audio invalid time period.

[0096] The audio invalid period is the invalid period in the original video identified from the audio dimension. For example, there are two types of audio invalid periods. Assuming that the silent period is 00:02:18-00:02:27 and the noise period is 00:02:27-00:02:32, the audio invalid period can be 00:02:18-00:02:32. Taking the union of at least two types of audio invalid periods as the audio invalid period, the invalid audio segments in the video can be identified more comprehensively and accurately, thereby improving the quality and efficiency of video processing and handling.

[0097] In practical applications, steps 220 to 223 and steps 230 to 233 may be performed in parallel, that is, image detection and sound detection may be performed in parallel to improve detection efficiency.

[0098] Step 240 , integrating the image invalid period and the audio invalid period to obtain the video invalid period, and determining the video in the video invalid period in the original video as an invalid video segment.

[0099] The intersection of the image invalid period and the audio invalid period can be calculated, or the union of the image invalid period and the audio invalid period can be calculated, and the intersection period or the union period can be determined as the video invalid period. The video invalid period refers to the invalid period determined by combining the image and the audio in the original video. There can be multiple video invalid periods, and each video invalid period corresponds to a segment in the original video, which is an invalid video segment. One video invalid period corresponds to one invalid video segment, and there can also be multiple invalid video segments.

[0100] Determining the intersection period as the video invalid period can reduce the possibility of mistakenly deleting valid content, thereby obtaining videos with higher audio-visual requirements and higher content integrity. Determining the union period as the video invalid period can quickly identify most invalid video clips, saving time and cost for post-video editing. In practical applications, one can be used according to the scene or actual needs.

[0101] Step 250: filter out invalid video segments in the original video to obtain valid video segments.

[0102] There may be multiple valid video clips, and each valid video clip may have different attributes, which may include the duration, content, click rate, interaction rate, etc. of the valid video clip. The click rate and interaction rate may be obtained by counting user feedback when the original video is generated.

[0103] Step 260: Filter out the video segments that meet the preset filtering conditions from the valid video segments to obtain the target video.

[0104] The preset filtering conditions are pre-set conditions for filtering out valid video clips, and can be set according to actual needs based on the properties of the valid video clips. For example, the preset filtering conditions can be filtering out shorter clips, filtering out clips that are not related to the live broadcast theme, filtering out clips with highly repetitive content, filtering out clips with low click-through rates and interaction rates, etc., to improve video quality. In live broadcast scenarios, live videos often contain the host's interjections (short duration), mechanical repetition of a certain voice content, chat content that is not related to the theme, content that users are not interested in (low click-through rate and interaction rate), etc. By further filtering the valid video clips with preset filtering conditions, the final video quality can be higher.

[0105] The video processing method provided by the present invention is described below by taking the processing of live video as an example. Figure 3 , specifically as follows:

[0106] First, the live video to be trimmed can be obtained, and the live video can be separated into an image frame sequence and an audio stream; for the image frame sequence, multiple image detection models can be used to detect multiple types of invalid image frames, which can include black screen images, blurred images, repeated images, distorted images, etc.; for the audio stream, multiple audio detection models can be used to detect multiple types of invalid audio segments, which can include silent segments, noise segments, abnormal speech speed segments, etc. Next, the invalid time period corresponding to the invalid image frame can be determined according to the starting position and ending position of the invalid image frame in the live video to obtain the image invalid time period; the invalid time period corresponding to the invalid audio segment can be determined according to the starting position and ending position of the invalid audio segment in the live video to obtain the audio invalid time period, and the intersection or union of the image invalid time period and the audio invalid time period is taken, and the intersection time period or the union time period is determined as the video invalid time period, and the video corresponding to the video invalid time period in the live video is determined as the invalid video segment. Filter out invalid video clips from live videos to obtain valid video clips, filter out short video clips, irrelevant to the live broadcast theme, highly repetitive content, and low click-through rate and / or interaction rate from valid video clips, and aggregate, filter, smooth, and process the remaining clips after filtering to obtain new videos. The new videos can be put on relevant platforms for users to play.

[0107] In this embodiment, the original video is separated into image and audio, which is convenient for subsequent separate and parallel processing of the image and audio, and for processing data of different modes in different ways according to their different characteristics; the video is automatically detected from the two modes of image and sound, so that the detection algorithm can be more targeted and can accurately and comprehensively identify invalid content; by locating the position of invalid image frames and invalid audio clips in the video, synchronous processing on the timeline is ensured to maintain the consistency of image and audio; the invalid time periods of the image and audio are integrated to avoid omission or repeated processing, making the invalid time period marking of the video more accurate; the invalid video clip is determined in combination with the invalid detection results of the image and audio, which can ensure that the determination of the invalid video clip is more reliable; the invalid clips in the video are filtered out based on the detection results of the two modes, thereby improving the efficiency and accuracy of video cropping; from the detection to the filtering of invalid content, end-to-end intelligent cropping is realized, without the need for manual operation, saving labor costs. The design of "divide first and combine later" is adopted. The first division is to divide the video into an image frame sequence and an audio stream, and detect invalid periods separately. The second combination is to merge the detected invalid periods of the image and audio, and perform synchronous processing and cropping to ensure the synchronization and consistency of the image and audio after the overall processing, and improve the quality of video cropping. Generally speaking, in live broadcast scenes, there are high requirements for both image quality and sound. Therefore, the method of performing invalid detection on the video and filtering out invalid content from the two modes of image and sound proposed in this embodiment is particularly suitable for processing live videos generated in live broadcast scenes, and solves the problems of high cost, low cropping efficiency, and poor cropping quality caused by manual cropping of live videos.

[0108] Figure 4 is a schematic diagram of the structure of a video processing device provided by an embodiment of the present invention, and the device is suitable for executing the video processing method provided by an embodiment of the present invention, such as Figure 4 As shown, the device may specifically include:

[0109] A separation module 401 is used to separate the original video into an image frame sequence and an audio stream;

[0110] A detection module 402, configured to detect invalid image frames from the image frame sequence and invalid audio segments from the audio stream;

[0111] A determination module 403, configured to determine an invalid image period according to a position of the invalid image frame in the original video, and determine an invalid audio period according to a position of the invalid audio segment in the original video;

[0112] An integration module 404 is used to integrate the image invalid period and the audio invalid period to obtain a video invalid period, and determine the video in the video invalid period in the original video as an invalid video segment;

[0113] The filtering module 405 is used to filter out the invalid video segments in the original video to obtain the target video.

[0114] In one embodiment, the detection module 402 detects invalid image frames from the image frame sequence, including:

[0115] Using at least two image detection models to detect invalid image frames in the image frame sequence to obtain at least two types of invalid image frames, wherein each of the at least two image detection models is used to detect one type of invalid image frames;

[0116] The at least two types of invalid image frames are determined as the invalid image frames.

[0117] In one embodiment, the determination module 403 determines the image invalid period according to the position of the invalid image frame in the original video, including:

[0118] Determine, according to the position of each type of invalid image frame in the original video, an invalid time period corresponding to each type of invalid image frame, and obtain at least two types of image invalid time periods;

[0119] The union of the at least two types of image invalid time periods is calculated to obtain the image invalid time period.

[0120] In one embodiment, the detection module 402 detects invalid audio segments from the audio stream, including:

[0121] Using at least two audio detection models to detect invalid audio segments in the audio stream to obtain at least two types of invalid audio segments, wherein each of the at least two audio detection models is used to detect one type of invalid audio segment;

[0122] The at least two types of invalid audio segments are determined as the invalid audio segments.

[0123] In one embodiment, the determining module 403 determines the audio invalid period according to the position of the invalid audio segment in the original video, including:

[0124] Determine, according to the position of each type of invalid audio segment in the original video, an invalid time period corresponding to each type of invalid audio segment, to obtain at least two types of audio invalid time periods;

[0125] The union of the at least two types of audio invalid time periods is calculated to obtain the audio invalid time period.

[0126] In one embodiment, the filtering module 405 is specifically used for:

[0127] Filtering out the invalid video segments in the original video to obtain valid video segments;

[0128] The video segments that meet the preset filtering conditions in the valid video segments are filtered out to obtain the target video.

[0129] In one embodiment, the separation module 401 is specifically used for:

[0130] Separating the original video into images and sounds;

[0131] Downsampling the image to obtain the image frame sequence;

[0132] The sound is segmented using a sliding window to obtain the audio stream.

[0133] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0134] The device of the embodiment of the present invention separates the original video into image and audio, which is convenient for subsequent separate and parallel processing of the image and audio, and is convenient for processing data of different modes in different ways according to their different characteristics; the video is automatically detected from the two modes of image and sound, so that the detection algorithm can be more targeted and can accurately and comprehensively identify invalid content; by locating the position of invalid image frames and invalid audio clips in the video, synchronous processing on the timeline is ensured to maintain the consistency of the image and audio; the invalid time periods of the image and audio are integrated to avoid omission or repeated processing, so that the invalid time period marking of the video is more accurate; the invalid detection results of the image and audio are combined Determining invalid video segments can ensure that the determination of invalid video segments is more reliable; filtering out invalid video segments in the video based on the detection results of the two modes improves the efficiency and accuracy of video cropping; from detecting to filtering out invalid content, end-to-end intelligent cropping is achieved, without the need for manual operation, saving labor costs; generally speaking, in live broadcast scenes, there are high requirements for image quality and sound. Therefore, the method for performing invalid detection on videos and filtering out invalid content from the two modes of image and sound proposed in the embodiment of the present invention is particularly suitable for processing live videos generated in live broadcast scenes, and solves the problems of high cost, low cropping efficiency, poor cropping quality, etc. caused by manual cropping of live videos.

[0135] An embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the video processing method provided in any of the above embodiments when executing the program.

[0136] An embodiment of the present invention further provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the video processing method provided by any of the above embodiments is implemented.

[0137] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of an electronic device suitable for implementing an embodiment of the present invention. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0138] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the computer system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0139] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0140] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are executed.

[0141] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0142] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0143] The modules and / or units involved in the embodiments of the present invention may be implemented in software or hardware. The modules and / or units described may also be arranged in a processor. For example, it may be described as follows: a processor includes a separation module, a detection module, a determination module, an integration module, and a filtering module. The names of these modules do not, in some cases, constitute limitations on the modules themselves.

[0144] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0145] The original video is separated into an image frame sequence and an audio stream; invalid image frames are detected from the image frame sequence, and invalid audio segments are detected from the audio stream; the image invalid period is determined according to the position of the invalid image frame in the original video, and the audio invalid period is determined according to the position of the invalid audio segment in the original video; the image invalid period and the audio invalid period are integrated to obtain the video invalid period, and the video in the video invalid period in the original video is determined as an invalid video segment; the invalid video segment in the original video is filtered out to obtain the target video.

[0146] According to the technical solution of the embodiment of the present invention, the original video is separated into image and audio, which is convenient for subsequent separate and parallel processing of the image and audio, and is convenient for processing different modal data in different ways according to their different characteristics; the video is automatically detected from the two modalities of image and sound, so that the detection algorithm can be more targeted and can accurately and comprehensively identify invalid content; by locating the position of invalid image frames and invalid audio clips in the video, synchronous processing on the timeline is ensured to maintain the consistency of image and audio; the invalid time periods of the image and audio are integrated to avoid omission or repeated processing, so that the invalid time period marking of the video is more accurate; the invalid detection of image and audio is combined The invalid video clips are determined based on the test results, which can ensure that the determination of invalid video clips is more reliable; invalid clips in the video are filtered out based on the detection results of the two modes, thereby improving the efficiency and accuracy of video cropping; from the detection to the filtering of invalid content, end-to-end intelligent cropping is achieved, and no manual operation is required, thus saving labor costs; generally speaking, in live broadcast scenes, there are high requirements for image quality and sound. Therefore, the method for performing invalid detection on videos and filtering out invalid content from the two modes of image and sound proposed in the embodiment of the present invention is particularly suitable for processing live videos generated in live broadcast scenes, and solves the problems of high cost, low cropping efficiency, poor cropping quality, etc. caused by manual cropping of live videos.

[0147] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0148] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0149] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A video processing method, characterized in that: include: Separate the original video into image frame sequence and audio stream; Detecting invalid image frames from the image frame sequence, and detecting invalid audio segments from the audio stream; Determine an invalid image period according to the position of the invalid image frame in the original video, and determine an invalid audio period according to the position of the invalid audio segment in the original video; Integrate the image invalid period and the audio invalid period to obtain a video invalid period, and determine the video in the video invalid period in the original video as an invalid video segment; The invalid video segments in the original video are filtered out to obtain the target video.

2. The video processing method according to claim 1, characterized in that: The detecting an invalid image frame from the image frame sequence comprises: Using at least two image detection models to detect invalid image frames in the image frame sequence to obtain at least two types of invalid image frames, wherein each of the at least two image detection models is used to detect one type of invalid image frames; The at least two types of invalid image frames are determined as the invalid image frames.

3. The video processing method according to claim 2, characterized in that: The step of determining the image invalid period according to the position of the invalid image frame in the original video comprises: Determine, according to the position of each type of invalid image frame in the original video, an invalid time period corresponding to each type of invalid image frame, and obtain at least two types of image invalid time periods; The union of the at least two types of image invalid time periods is calculated to obtain the image invalid time period.

4. The video processing method according to claim 1, characterized in that: The detecting an invalid audio segment from the audio stream comprises: Using at least two audio detection models to detect invalid audio segments in the audio stream to obtain at least two types of invalid audio segments, wherein each of the at least two audio detection models is used to detect one type of invalid audio segment; The at least two types of invalid audio segments are determined as the invalid audio segments.

5. The video processing method according to claim 4, characterized in that: The step of determining the audio invalid period according to the position of the invalid audio segment in the original video includes: Determine, according to the position of each type of invalid audio segment in the original video, an invalid time period corresponding to each type of invalid audio segment, and obtain at least two types of audio invalid time periods; The union of the at least two types of audio invalid time periods is calculated to obtain the audio invalid time period.

6. The video processing method according to any one of claims 1 to 5, characterized in that: The filtering out the invalid video segments in the original video to obtain the target video includes: Filtering out the invalid video segments in the original video to obtain valid video segments; The video segments that meet the preset filtering conditions in the valid video segments are filtered out to obtain the target video.

7. The video processing method according to claim 1, characterized in that: The method of separating the original video into an image frame sequence and an audio stream comprises: Separating the original video into images and sounds; Downsampling the image to obtain the image frame sequence; The sound is segmented using a sliding window to obtain the audio stream.

8. A video processing device, characterized in that: include: A separation module, used for separating the original video into an image frame sequence and an audio stream; A detection module, used to detect invalid image frames from the image frame sequence and invalid audio segments from the audio stream; a determination module, configured to determine an invalid image period according to a position of the invalid image frame in the original video, and determine an invalid audio period according to a position of the invalid audio segment in the original video; An integration module, configured to integrate the image invalid period and the audio invalid period to obtain a video invalid period, and determine the video in the video invalid period in the original video as an invalid video segment; The filtering module is used to filter out the invalid video segments in the original video to obtain the target video.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the video processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the video processing method according to any one of claims 1 to 7 is implemented.

Images