Video processing method and device, equipment and storage medium

Abstract

This application discloses a video processing method, apparatus, device, and storage medium, applicable to fields such as computers and cloud computing. The method includes: determining a cropping time interval for the video to be processed; determining a target image frame sequence based on the cropping time interval and the timestamps of each image frame in the image frame sequence of the video to be processed; determining a target audio frame sequence based on the cropping time interval and the timestamps of each audio frame in the audio frame sequence of the video to be processed; determining the playback position of each image frame in the target image frame sequence and each audio frame in the target audio sequence within the corresponding cropped video of the video to be processed, based on the timestamps of each image frame in the target image frame sequence and the timestamps of each audio frame in the target audio sequence; and obtaining the cropped video based on the target audio frame sequence, the target image frame sequence, and the playback positions. Using this application embodiment, a cropped video with synchronized audio and video can be obtained by cropping the video to be processed, exhibiting high applicability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a video processing method and device, equipment and storage medium. BACKGROUND

[0002] With the continuous development of computer technology and multimedia technology, people also tend to watch videos on computer, mobile phone or tablet computer and other terminal devices and further edit the videos. The video editing includes cutting the video, cutting the long video into short video with shorter time, so as to achieve the purpose of sharing short video, video highlight segment interception and the like.

[0003] In the existing video cutting scheme, the starting time point and the ending time point of cutting the video image are determined first, and then the video is directly cut based on the starting time point and the ending time point. Based on the existing video cutting scheme, since the original video may have the phenomenon of audio and video out of synchronization, the cut video obtained still has the problem of audio and video out of synchronization, and the applicability is poor. SUMMARY

[0004] The embodiments of the present application provide a video processing method, device, equipment and storage medium, which can cut the video to obtain a cut video with matched audio and image, and can cut the to-be-processed video to obtain a cut video with synchronized audio and video, and the applicability is high.

[0005] In one aspect, the embodiments of the present application provide a video processing method, which comprises:

[0006] determining a cutting time interval of a to-be-processed video;

[0007] determining a target image frame sequence according to the cutting time interval and the time stamp of each image frame in the image frame sequence of the to-be-processed video, and determining a target audio frame sequence according to the cutting time interval and the time stamp of each audio frame in the audio frame sequence of the to-be-processed video;

[0008] determining the playing position of each image frame in the target image frame sequence and each audio frame in the target audio sequence in the cut video corresponding to the to-be-processed video according to the time stamp of each image frame in the target image frame sequence and the time stamp of each audio frame in the target audio sequence;

[0009] obtaining the cut video according to the target audio frame sequence and the target image frame sequence, and the playing position of each image frame in the target image frame sequence and each audio frame in the target audio sequence.

[0010] In another aspect, the embodiments of the present application provide a video processing device, which comprises:

[0011] The cutting time interval determination module is configured to determine a cutting time interval of the to-be-processed video.

[0012] The frame sequence determination module is configured to determine a target image frame sequence according to the cutting time interval and time stamps of each image frame in an image frame sequence of the to-be-processed video, and determine a target audio frame sequence according to the cutting time interval and time stamps of each audio frame in an audio frame sequence of the to-be-processed video.

[0013] The playing position determination module is configured to determine playing positions of each image frame in the target image frame sequence and each audio frame in the target audio sequence in a cutting video corresponding to the to-be-processed video according to the time stamps of each image frame in the target image frame sequence and the time stamps of each audio frame in the target audio sequence.

[0014] The cutting video determination module is configured to obtain the cutting video according to the target audio frame sequence and the target image frame sequence, and the playing positions of each image frame in the target image frame sequence and each audio frame in the target audio sequence.

[0015] In another aspect, an electronic device is provided, which includes a processor and a memory, the processor and the memory being connected to each other.

[0016] The memory is configured to store a computer program.

[0017] The processor is configured to execute the video processing method provided by the embodiments of the present application when the computer program is invoked.

[0018] In another aspect, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the video processing method provided by the embodiments of the present application.

[0019] In another aspect, a computer program product or a computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer instructions from the computer readable storage medium. The processor executes the computer instructions, so that the computer device implements the video processing method provided by the embodiments of the present application.

[0020] In the embodiment of the present application, the target image frame sequence is determined by clipping the time interval and the time stamp of each image frame in the image frame sequence of the to-be-processed video, and the target audio frame sequence is determined by clipping the time interval and the time stamp of each audio frame in the audio frame sequence of the to-be-processed video, so that the target image frame sequence and the target audio frame sequence correspond to the clipping time interval at the same time, avoiding the case that the target audio frame sequence corresponds to a sound segment and the target image frame sequence corresponds to a video picture. Further, after determining the playing positions of each image frame in the target image frame sequence and each audio frame in the target audio sequence in the to-be-processed video corresponding to the clipping video, each image frame in the target image frame sequence and each audio frame in the target audio sequence are located at the corresponding positions in the clipping video, further avoiding the case that the audio and the picture in the clipping video are out of synchronization, improving the video quality of the clipping video, and having high applicability. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0022] Figure 1 is a scene diagram of a video processing method provided by an embodiment of the present application;

[0023] Figure 2 is a flowchart of a video processing method provided by an embodiment of the present application;

[0024] Figure 3a is a scene diagram of determining a target image frame sequence provided by an embodiment of the present application;

[0025] Figure 3b is another scene diagram of determining a target image frame sequence provided by an embodiment of the present application;

[0026] Figure 4a is a scene diagram of the playing order of image frames and audio frames provided by an embodiment of the present application;

[0027] Figure 4b is another scene diagram of the playing order of image frames and audio frames provided by an embodiment of the present application;

[0028] Figure 4c is still another scene diagram of the playing order of image frames and audio frames provided by an embodiment of the present application;

[0029] Figure 5a is a scene diagram of time stamp synchronization provided by an embodiment of the present application;

[0030] Figure 5b is another scenario diagram of timestamp synchronization provided by an embodiment of the present application.

[0031] Figure 6a is a scenario diagram of processing an audio frame provided by an embodiment of the present application.

[0032] Figure 6b is another scenario diagram of processing an audio frame provided by an embodiment of the present application.

[0033] Figure 7 is a scenario diagram of determining a playing position provided by an embodiment of the present application.

[0034] Figure 8a is an application scenario diagram of a video processing method provided by an embodiment of the present application.

[0035] Figure 8b is another application scenario diagram of a video processing method provided by an embodiment of the present application.

[0036] Figure 9 is a structural diagram of a video processing apparatus provided by an embodiment of the present application.

[0037] Figure 10 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0039] Referring to Figure 1 , Figure 1 is a scenario diagram of a video processing method provided by an embodiment of the present application. Figure 1 A video to be processed 100 with a length of 3 minutes is shown in FIG. 1, and a clipping time interval 200 of the video to be processed 100 is from 1 minute 20 seconds to 2 minutes 20 seconds. For the video to be processed 100, the video content mainly consists of images and audio, for example, the video to be processed 100 consists of video pictures and video sound content.

[0040] The length of the clipping time interval 200 is the playing length of a clipping video 500 obtained after processing the video to be processed 100.

[0041] The video picture of the to-be-processed video 100 is formed by continuously playing each image frame in the image frame sequence 301 of the to-be-processed video 100, each image frame is arranged in the image frame sequence 301 according to the chronological order of the time stamp, and the time interval between each adjacent two image frames in the image frame sequence 301 is the playing interval of the image content corresponding to the two image frames.

[0042] The sound of the to-be-processed video 100 is formed by continuously playing each audio frame in the audio frame sequence 302 of the to-be-processed video 100, each audio frame is arranged in the audio frame sequence 401 according to the chronological order of the time stamp, and the time interval between each adjacent two audio frames in the audio frame sequence 302 is the playing interval of the sound content corresponding to the two audio frames.

[0043] Further, the target image frame sequence 401 is determined according to the clipping time interval 200 of the to-be-processed video 100 and the time stamp of each image frame in the image frame sequence 301 of the to-be-processed video 100. The video picture of the clipping video 500 corresponding to the to-be-processed video 100 is formed by continuously playing each image frame in the target image frame sequence 401. The image frames in the target image frame sequence 401 are part of the image frames in the image frame sequence 301 of the to-be-processed video.

[0044] Further, the target audio frame sequence 402 is determined according to the clipping time interval 200 of the to-be-processed video 100 and the time stamp of each audio frame in the audio frame sequence 302 of the to-be-processed video 100. The sound of the clipping video 500 corresponding to the to-be-processed video 100 is formed by continuously playing each audio frame in the target audio frame sequence 402. The audio frames in the target audio frame sequence 402 are part of the audio frames in the audio frame sequence 302 of the to-be-processed video.

[0045] Based on the time stamp of each image frame in the target image frame sequence 401 and the time stamp of each audio frame in the target audio frame sequence 402, the playing positions of each image frame in the target image frame sequence 401 and each audio frame in the target audio frame sequence 402 in the clipping video 500 are determined, and then based on the playing positions of each audio frame and each audio frame, the target image frame sequence 401 and the target audio frame sequence 402, the clipping video 500 corresponding to the to-be-processed video 100 can be obtained. The clipping video 500 obtained based on the target image frame sequence 401 and the target audio frame sequence 402 is part of the segments in the to-be-processed video 100.

[0046] Referring to Figure 2 , Figure 2 is a flowchart of a video processing method provided by an embodiment of the present application. As shown in Figure 2 , the video processing method provided by the embodiment of the present application can include the following steps:

[0047] Step S21, determining a clipping time interval of the video to be processed.

[0048] In some possible implementation manners, the video to be processed can be a real-time recorded video, such as a game screen recorded by screen recording software, a video recorded by a camera, a video downloaded through a network, or a video imported from a fixed storage space, and the like. The specific video to be processed can be determined based on actual application scenarios, and is not limited herein.

[0049] The fixed storage space can be a hard disk, a device memory, a cloud storage space, a database, a blockchain, and the like. The specific fixed storage space can be determined based on actual application scenarios, and is not limited herein.

[0050] The blockchain is a new application mode of distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. The blockchain is essentially a decentralized database, and is a series of data blocks associated using a cryptographic method. Each data block is used to store videos of various formats in various fields.

[0051] In some possible implementation manners, the clipping time interval of the video to be processed can be determined based on a clipping operation of a user.

[0052] Specifically, in response to a clipping operation of a user on the video to be processed, a start time point and an end time point corresponding to the clipping operation are determined, and a time interval corresponding to the start time point and the end time point is determined as the clipping time interval of the video to be processed.

[0053] The clipping operation can be a clipping operation triggered after the user inputs the start time point and the end time point, can be a selection operation for selecting the start time point and the end time point triggered by the user on the video to be clipped, or can be a clipping operation triggered based on a voice instruction input by the user and including the start time point and the end time point. The specific clipping operation can be determined based on actual application scenarios, and is not limited herein.

[0054] For example, if it is detected that the user performs a clipping operation on the video to be processed, the start time point is determined to be 1 minute and 1 second, and the end time point is determined to be 1 minute and 50 seconds based on the clipping operation of the user. The clipping time interval of the video to be processed is determined to be a time interval from 1 minute and 1 second to 1 minute and 50 seconds.

[0055] In some possible implementation manners, the clipping time interval of the video to be processed can be determined based on a video picture.

[0056] Specifically, if the to-be-processed video needs to be cropped to obtain a specific video segment in the to-be-processed video, a starting time point and an ending time point corresponding to the specific video segment corresponding to the to-be-processed video can be determined, and then a time interval corresponding to the starting time point and the ending time point corresponding to the specific video segment is determined as a cropping time interval of the to-be-processed video.

[0057] The starting time point of the specific video segment can be a time point at which a video frame including a specific keyword in a first played subtitle appears for the first time, and the ending time point of the specific video segment can be a time point at which a video frame including the specific keyword in a last played subtitle appears for the last time.

[0058] The starting time point of the specific video segment can be a time point at which a video frame including a specific keyword in a first played subtitle appears for the first time, and the ending time point of the specific video segment can be a time point at which a video frame including the specific keyword in a last played subtitle appears for the last time.

[0059] The starting time point of the specific video segment can be a time point at which a video frame including a specific keyword in a first played subtitle appears for the first time, and the ending time point of the specific video segment can be a time point at which a video frame including the specific keyword in a last played subtitle appears for the last time.

[0060] The number of occurrences of the video frame in the to-be-processed video, the subtitle included in the video frame, and the similarity of the video frame to the specific image can be determined based on cloud computing and artificial intelligence machine learning, and can be determined based on actual application scene requirements, which is not limited herein. Cloud computing is a product of the integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage, virtualization, load balancing, and the like. Based on cloud computing, the computing efficiency of determining the number of occurrences of the video frame, the subtitle included in the video frame, and the similarity of the video frame to the specific image in the embodiments of the present application can be improved.

[0061] In step S22, the target image frame sequence is determined according to the cropping time interval and the time stamps of the image frames in the image frame sequence of the to-be-processed video, and the target audio frame sequence is determined according to the cropping time interval and the time stamps of the audio frames in the audio frame sequence of the to-be-processed video.

[0062] In some possible embodiments, the image frame sequence of the to-be-processed video is a frame sequence formed by each image frame constituting a video picture of the to-be-processed video. In the image frame sequence of the to-be-processed video, each image frame is arranged in a time stamp order, and the time stamp of each image frame is a corresponding play time of each image frame when each image frame is written into a video file. For example, when an image frame is written into a video file, if the image frame needs to be played at the third second, the time stamp of the image frame is 3 seconds.

[0063] In some possible embodiments, the target image frame sequence corresponding to the to-be-processed video is a frame sequence corresponding to each image frame of the cropped video of the to-be-processed video, and each image frame in the target image frame sequence is part of the image frame sequence corresponding to the to-be-processed video. When the target image frame sequence is determined, the start time point and the end time point of the target image frame sequence can be determined based on the cropping time interval of the to-be-processed video and the time stamp of each image frame in the image frame sequence, and the target image frame sequence can be determined from the image frame sequence corresponding to the to-be-processed video according to the start time point and the end time point of the target image frame sequence, that is, the frame sequence in the image frame sequence of the to-be-processed video corresponding to the time interval of the start time point and the end time point of the target image frame sequence is determined as the target image frame sequence.

[0064] Specifically, the time stamp in the time stamp of each image frame in the image frame sequence of the to-be-processed video that matches the start time point of the cropping time interval is determined as the start time point of the target image frame sequence. The time stamp in the time stamp of each image frame in the image frame sequence of the to-be-processed video that matches the end time point of the cropping time interval is determined as the end time point of the target image frame sequence.

[0065] As an example, if the cropping time interval of the to-be-processed video is 10 seconds to 20 seconds, that is, the video picture of the to-be-processed video from 10 seconds to 20 seconds needs to be cropped, the time stamp that matches the tenth second can be determined from the time stamp of each image frame in the image frame sequence of the to-be-processed video, and the time stamp is determined as the start time point of the target image frame sequence. At the same time, the time stamp that matches the twentieth second is determined from the time stamp of each image frame in the image frame sequence of the to-be-processed video, and the time stamp is determined as the end time point of the target image frame sequence.

[0066] The timestamp matched with the start time point of the clipping time interval can be the timestamp of the image frame corresponding to the start time point of the clipping time interval, and the timestamp matched with the end time point of the clipping time interval can be the timestamp of the image frame corresponding to the end time point of the clipping time interval. For example, the timestamp of the image frame corresponding to the 10th second in the image frame sequence of the to-be-processed video is determined as the start time point of the target image frame sequence (i.e., the 10th second), and the timestamp of the image frame corresponding to the 20th second in the image frame sequence of the to-be-processed video is determined as the end time point of the target image frame sequence (i.e., the 20th second).

[0067] If the start time point of the clipping time interval is located between the timestamps of two image frames in the image frame sequence of the to-be-processed image, the timestamp matched with the start time point of the clipping time interval can be the closest time to the start time point of the clipping time interval; and if the end time point of the clipping time interval is located between the timestamps of two image frames in the image frame sequence of the to-be-processed image, the timestamp matched with the end time point of the clipping time interval can be the closest timestamp to the end time point of the clipping time interval.

[0068] Referring to Figure 3a , Figure 3a is a scene diagram for determining a target image frame sequence provided by an embodiment of the present application. Figure 3a The image frame sequence of the to-be-processed image is shown in FIG. 1, and the image frames in the image frame sequence are, for example, the image frames with timestamps of 6.0365s, 7.092042s, 7.134708s, 11.937376s and 12.234322s. In the case that the start time point of the clipping time interval of the to-be-processed video is 6.5s, since the start time point of the clipping time interval is located between the timestamp 6.0365s and the timestamp 7.092042s, and the time distance between the timestamp 6.0365s and the start time point 6.5s of the clipping time interval is less than the time distance between the timestamp 7.092042s and the start time point 6.5s of the clipping time interval, the timestamp 6.0365s is determined as the start time point of the target image frame sequence.

[0069] Similarly, in the case that the end time point of the clipping time interval of the to-be-processed video is 12s, since the end time point of the clipping time interval is located between the timestamp 11.937376s and the timestamp 12.234322s, and the time distance between the timestamp 11.937376s and the end time point 12s of the clipping time interval is less than the time distance between the timestamp 12.234322s and the end time point 12s of the clipping time interval, the timestamp 11.937376s is determined as the end time point of the target image frame sequence.

[0070] Based on the starting time point and the ending time point of the target image frame sequence, the image frame sequence corresponding to a time interval of 6.0365s to 11.937376s in the image frame sequence can be determined, and the image frame sequence in the time interval is determined as the target image frame sequence.

[0071] Optionally, for most videos, there are often image key frames and image non-key frames in each image frame in the image frame sequence of the video. Among them, the image key frame contains a complete image, which can be directly played when it needs to be played. The image non-key frame does not contain a complete image, which needs to restore a complete image through an algorithm with the adjacent image frames before and after it, and then play it. Based on this, in order to make the beginning and end of the final obtained cropped video have complete video content, the image key frames in each image frame in the image frame sequence of the video to be processed can be determined, and then the target image frame sequence is determined based on the cropping time interval and the time stamp of each image key frame in the video to be processed.

[0072] Among them, the time stamp in the time stamp of each image frame in the image frame sequence of the video to be processed that matches the starting time point of the cropping time interval is determined as the starting time point of the target image frame sequence, including:

[0073] The time stamp in the time stamp of each image key frame in the video to be processed that matches the starting time point of the cropping time interval is determined as the starting time point of the target image frame sequence.

[0074] That is, the time stamp that matches the starting time point of the cropping time interval can be the time stamp in the time stamp of each image key frame in the image frame sequence that matches the starting time point of the cropping time interval. For example, the time stamp in the time stamp of each image key frame in the image frame sequence that corresponds to the starting time point of the cropping time interval, or the time stamp in the time stamp of each image key frame that is closest to the starting time point of the cropping time interval.

[0075] Among them, the time stamp in the time stamp of each image frame that matches the ending time point of the cropping time interval is determined as the ending time point of the target image frame sequence, including:

[0076] The time stamp in the time stamp of each image key frame that matches the ending time point of the cropping time interval is determined as the ending time point of the target image frame sequence.

[0077] The time stamp matching the end time point of the clipping time interval can be a time stamp in the time stamps of the image key frames in the image frame sequence that matches the end time point of the clipping time interval. For example, a time stamp in the time stamps of the image key frames in the image frame sequence that corresponds to the end time point of the clipping time interval, or a time stamp in the time stamps of the image key frames that is closest to the end time point of the clipping time interval.

[0078] Referring to Figure 3b , Figure 3b is another scene schematic diagram for determining a target image frame sequence provided by an embodiment of the present application. Figure 3b The image frame sequence of the image to be processed is shown, such as the image non-key frames with time stamps of 6.0365s, 7.134708s and 11.937376s, and the image key frames with time stamps of 7.092042s and 12.234322s. In the case that the start time point of the clipping time interval of the image to be processed is 6.5s, the start time point of the clipping time interval is between the time stamp 6.0365s and the time stamp 7.092042s, and if the time stamp 7.092042s is closest to the start time point 6.5s of the clipping time interval, the time stamp 7.092042s is determined as the start time point of the target image frame sequence.

[0079] Similarly, in the case that the end time point of the clipping time interval of the image to be processed is 12s, the end time point of the clipping time interval is between the time stamp 11.937376s and the time stamp 12.234322s, and if the time stamp 12.234322s is closest to the end time point 12s of the clipping time interval, the time stamp 12.234322s is determined as the end time point of the target image frame sequence.

[0080] Based on the start time point and the end time point of the target image frame sequence, the image frame sequence corresponding to the time interval of 7.092042s to 12.234322s in the image frame sequence can be determined, and the image frame sequence of the time interval is determined as the target image frame sequence.

[0081] In some possible embodiments, the audio frame sequence of the to-be-processed video is a frame sequence formed by audio frames constituting the sound of the to-be-processed video. The target audio frame sequence corresponding to the to-be-processed video is a frame sequence formed by audio frames corresponding to the cropped video of the to-be-processed video, and each audio frame in the target audio frame sequence is part of the audio frame sequence corresponding to the to-be-processed video. In the audio frame sequence of the to-be-processed video, each audio frame is arranged in the order of timestamps, and the timestamp of each audio frame is the corresponding playback time of each audio frame when each audio frame is written into a video file. For example, when an audio frame is written into a video file, if the audio frame needs to be played at the 10th second, the timestamp of the audio frame is 10 seconds.

[0082] When the to-be-processed video is generated, each image frame of the to-be-processed video is written into a video file in the order of playback to obtain an image frame sequence, and each audio frame of the to-be-processed video is written into the video file in the order of playback to obtain an audio frame sequence, and the timestamp of each image frame and each audio frame corresponds to the same time axis. In the case that each image frame and each audio frame is normally written into the video file, each image frame in the image frame sequence of the to-be-processed video and each audio frame in the audio frame sequence are played in the order of timestamps, for example, each image frame in the image frame sequence is played through a video track, and each audio frame in the audio frame sequence is played through an audio track at the same time, so that the effect of playing audio and video synchronously is achieved.

[0083] Because the sensitivity of human eyes and human ears is not high, the picture seen by a person and the sound heard by the person will not disappear in a period of time, so even if a certain audio frame and a corresponding image frame are not played strictly in the order of timestamps, and there is a certain time difference (for example, less than 0.1s) between the timestamp of the audio frame and the image frame, the effect of playing audio and video synchronously can also be achieved. See Figure 4a , Figure 4a is a scene diagram of the playback order of image frames and audio frames provided by an embodiment of the present application. Figure 4a shows the playback order of each image frame and each audio frame of a certain video relative to the same time axis when the video is played. Each image frame of the video is played in the order of timestamps relative to the time axis, and each audio frame of the video is played in the order of timestamps relative to the time axis. Although the audio frame with the timestamp of 0.075s is played after the image frame with the timestamp of 0.1s, because the time difference between the two is very small, the effect of playing audio and video synchronously is not affected in actual playback.

[0084] In the process of actually writing the image frames and the audio frames of the to-be-processed video into the video file, the audio frames can be written into the video file earlier than the image frames, so that the sound is played earlier than the video picture when the to-be-processed video is played. For example, the audio of the to-be-processed video is normally played, and the image frame that should be played at the 0th second is played at the 3rd second due to a writing error. Referring to Figure 4b , Figure 4b is another scene schematic diagram of the playing order of the image frames and the audio frames provided by the embodiment of the present application. In the case that the audio frames of the to-be-processed video are written into the video file earlier than the image frames, although the audio frames of the to-be-processed video are arranged in the order of the timestamps to obtain an audio frame sequence, and the image frames are arranged in the order of the timestamps to obtain an image frame sequence, the actual playing time of each image frame in the image frame sequence is later than the playing time corresponding to the timestamp of the image frame, so that the playing effect of audio and video out of synchronization occurs when the to-be-processed video is played.

[0085] In the process of actually writing the image frames and the audio frames of the to-be-processed video into the video file, the image frames can be written into the video file earlier than the audio frames, so that the video picture is played earlier than the sound when the to-be-processed video is played. For example, the video picture of the to-be-processed video is normally played, and the audio frame that should be played at the 0th second is played at the 3rd second due to a writing error. Referring to Figure 4c , Figure 4c is another scene schematic diagram of the playing order of the image frames and the audio frames provided by the embodiment of the present application. In the case that the image frames of the to-be-processed video are written into the video file earlier than the audio frames, although the audio frames of the to-be-processed video are arranged in the order of the timestamps to obtain an audio frame sequence, and the image frames are arranged in the order of the timestamps to obtain an image frame sequence, the actual playing time of each audio frame in the image frame sequence is later than the playing time corresponding to the timestamp of the audio frame, so that the playing effect of audio and video out of synchronization occurs when the to-be-processed video is played.

[0086] Based on the above problems, when determining the target audio frame sequence, if the starting time point and the ending time point corresponding to the target image frame sequence of the to-be-processed video are respectively taken as the starting time point and the ending time point corresponding to the target audio frame sequence of the to-be-processed video, the video picture corresponding to the obtained target audio frame sequence and the target image frame sequence may not correspond. Therefore, when determining the target audio frame sequence, the starting time point and the ending time point of the target audio frame sequence can be determined based on the clipping time interval of the to-be-processed video and the time stamps of the audio frames in the audio frame sequence, and the target audio frame sequence can be determined from the audio frame sequence corresponding to the to-be-processed video according to the starting time point and the ending time point of the target audio frame sequence, that is, the frame sequence in the audio frame sequence of the to-be-processed video corresponding to the time interval of the starting time point and the ending time point of the target audio frame sequence is determined as the target audio frame sequence.

[0087] Specifically, the time stamp in the time stamps of the audio frames in the audio frame sequence of the to-be-processed video that matches the starting time point of the clipping time interval is determined as the starting time point of the target audio frame sequence. The time stamp in the time stamps of the audio frames in the audio frame sequence of the to-be-processed video that matches the ending time point of the clipping time interval is determined as the ending time point of the target audio frame sequence.

[0088] As an example, if the clipping time interval of the to-be-processed video is 0 seconds to 20 seconds, that is, the sound segment of 0 seconds to 20 seconds of the to-be-processed video needs to be clipped, the time stamp matching the 0th second can be determined from the time stamps of the audio frames in the audio frame sequence of the to-be-processed video, and the time stamp is determined as the starting time point of the target audio frame sequence. At the same time, the time stamp matching the 20th second is determined from the time stamps of the audio frames in the audio frame sequence of the to-be-processed video, and the time stamp is determined as the ending time point of the target audio frame sequence.

[0089] Among them, the time stamp matching the starting time point of the clipping time interval can be the time stamp of the audio frame corresponding to the starting time point of the clipping time interval, and the time stamp matching the ending time point of the clipping time interval can be the time stamp of the audio frame corresponding to the ending time point of the clipping time interval. For example, the time stamp of the audio frame corresponding to the 0th second in the above audio frame sequence of the to-be-processed video is determined as the starting time point of the target audio frame sequence (i.e. the 0th second), and the time stamp of the audio frame corresponding to the 20th second in the above audio frame sequence of the to-be-processed video is determined as the ending time point of the target audio frame sequence (i.e. the 20th second).

[0090] If the start time point of the clipping time interval is located between the time stamps of two audio frames in the audio frame sequence of the to-be-processed audio, the time stamp matched with the start time point of the clipping time interval can be the time closest to the start time point of the clipping time interval. If the end time point of the clipping time interval is located between the time stamps of two audio frames in the audio frame sequence of the to-be-processed audio, the time stamp matched with the end time point of the clipping time interval can be the time stamp closest to the end time point of the clipping time interval.

[0091] Based on this, since the time stamp of the audio frame of the to-be-processed video can represent that the corresponding audio frame actually corresponds to the playing time of the to-be-processed video, even if the actual playing time of the audio frame of the to-be-processed video is earlier than the playing time corresponding to the time stamp thereof, or the actual playing time thereof is later than the playing time corresponding to the time stamp thereof, the sound segment corresponding to the target audio frame sequence determined based on the time stamp of each audio frame is actually the sound segment corresponding to the clipping time interval of the to-be-processed video.

[0092] Optionally, for most videos, like the image frames of a video, there are often audio key frames and audio non-key frames in each audio frame in the audio frame sequence of the video. The audio key frame contains complete audio, and the audio key frame can be directly played when it needs to be played. The audio non-key frame does not contain complete audio, and the complete audio needs to be restored by means of the audio frames adjacent to the audio non-key frame before and after the audio non-key frame through an algorithm, and then the complete audio is played. Based on this, in order to make the beginning and the end of the finally obtained clipping video have complete sound segments, the audio key frames in each audio frame in the audio frame sequence of the to-be-processed video can be determined, and then the target audio frame sequence can be determined based on the clipping time interval and the time stamps of the audio key frames in the to-be-processed video.

[0093] The time stamp matched with the start time point of the clipping time interval in the time stamps of each audio frame in the audio frame sequence of the to-be-processed video is determined as the start time point of the target audio frame sequence, including:

[0094] The time stamp matched with the start time point of the clipping time interval in the time stamps of each audio key frame in the to-be-processed video is determined as the start time point of the target audio frame sequence.

[0095] That is, the time stamp matched with the start time point of the clipping time interval can be the time stamp matched with the start time point of the clipping time interval in the time stamps of each audio key frame in the audio frame sequence. For example, the time stamp corresponding to the start time point of the clipping time interval in the time stamps of each audio key frame in the audio frame sequence, or the time stamp closest to the start time point of the clipping time interval in the time stamps of each audio key frame.

[0096] The timestamp in each audio frame that matches the end time point of the clipping time interval is determined as the end time point of the target audio frame sequence, including:

[0097] The timestamp in each audio key frame that matches the end time point of the clipping time interval is determined as the end time point of the target audio frame sequence.

[0098] That is, the timestamp that matches the end time point of the clipping time interval can be the timestamp in each audio key frame in the audio frame sequence that matches the end time point of the clipping time interval. For example, the timestamp in each audio key frame in the audio frame sequence that corresponds to the end time point of the clipping time interval, or the timestamp in each audio key frame that is closest to the end time point of the clipping time interval.

[0099] In step S23, the time stamps of each image frame in the target image frame sequence and the time stamps of each audio frame in the target audio sequence are determined to determine the playing positions of each image frame in the target image frame sequence and each audio frame in the target audio sequence in the clipping video corresponding to the to-be-processed video.

[0100] In actual applications, after the target image frame sequence of the clipping video is obtained by clipping the to-be-processed video, the audio frames corresponding to each image frame in the target image frame sequence are subjected to time stamp synchronization processing, so that the sound and the video picture in the final obtained clipping video are as consistent as possible.

[0101] Referring to Figure 5a , Figure 5a is a scene diagram of time stamp synchronization provided by an embodiment of the present application. In Figure 5a It is shown that each image frame and the corresponding audio frame in the determined target image frame sequence correspond to the playing order of the to-be-processed video, that is, the corresponding audio frame or image frame is played at the time indicated by the time stamp of each frame when the to-be-processed video is played. If the time stamps of the audio frames corresponding to the target image frame sequence are synchronized after the target image frame sequence is determined, such as synchronizing the time stamp of the audio frame with the time stamp of the image frame in the previous playing order 6.160000s, and synchronizing the time stamp of the audio frame with the time stamp of the image frame in the previous playing order 6.126667s, when the target image frame sequence is played based on the video track and the corresponding audio frames are played based on the audio track, the audio frame with the time stamp of 7.092042s and the audio frame with the time stamp of 7.134708s will be played in advance, resulting in the playing time of the sound segment corresponding to the target audio sequence being advanced.

[0102] Referring to Figure 5b ,Figure 5b is another scenario of timestamp synchronization provided by an embodiment of the present application. In Figure 5b shows that each image frame in the determined target image frame sequence and the corresponding audio frame correspond to the playing order of the to-be-processed video, that is, the corresponding audio frame or image frame is played at the time indicated by the timestamp of each frame when the to-be-processed video is played. If the corresponding audio frames are not timestamp-synchronized after the target image frame sequence is determined, for example, the timestamp of the audio frame with the timestamp of 8.662333 s is synchronized with the timestamp of the image frame with the timestamp of 12.076667 s in the playing order, the audio frame with the timestamp corresponding to the earlier playing time will be played late.

[0103] In actual application, if the audio frames corresponding to each image frame in the target image frame sequence of the cropped video obtained by cropping the to-be-processed video are not timestamp-synchronized after the target image frame sequence is determined, in the case that the to-be-processed video has audio-visual asynchronization, the cropped video will have no sound at one end when played or have sound loss in the latter half of the video. That is, the cropped video obtained still has audio-visual asynchronization when played.

[0104] Referring to Figure 6a , Figure 6a is a scenario of processing of audio frames provided by an embodiment of the present application. If Figure 6a the target image frame sequence obtained is the image frame corresponding to 6 s to 12 s of the to-be-processed video, if the corresponding audio frames are not timestamp-synchronized after the target image frame sequence is obtained, there is no other audio frame before the audio frame with the timestamp of 7.092042 s, resulting in sound loss when the final cropped video is played.

[0105] Referring to Figure 6b , Figure 6b is another scenario of processing of audio frames provided by an embodiment of the present application. If Figure 6b the target image frame sequence obtained is the image frame corresponding to 12 s to 18 s of the to-be-processed video, if the corresponding audio frames are not timestamp-synchronized after the target image frame sequence is obtained, when the cropped video is obtained, the audio frames with the timestamp less than 12 s in the audio frames corresponding to the target image frame sequence will be discarded, but it is impossible to determine other audio frames after the audio frame with the timestamp of 14.678333 s and before 18 s, resulting in no sound playing when the final cropped video is played for a period of time.

[0106] Based on this, even if the target audio frame sequence obtained based on step S23 is determined based on the timestamps of the audio frames in the to-be-processed video and the clipping time interval, in the case where the to-be-processed video is out of sync, whether or not the audio frames in the target audio frame sequence are timestamped, the final clipping video may still have a sound loss or out-of-sync situation. Therefore, after the target audio frame sequence and the target image frame sequence are obtained, it is necessary to re-determine the corresponding playback positions of the audio frames in the target audio frame sequence and the image frames in the target image frame sequence in the clipping video, so that the final clipping video can play the sound segment corresponding to the target audio frame sequence, and the sound and video of the clipping video are played synchronously.

[0107] In some possible implementations, the corresponding playback positions of the image frames and the audio frames in the to-be-processed video corresponding to the clipping video can be determined according to the timestamps of the image frames in the target image frame sequence and the timestamps of the audio frames in the target audio frame sequence. The playback positions of the image frames and the audio frames in the to-be-processed video corresponding to the clipping video are the playback times corresponding to the image frames and the audio frames when the clipping video is played.

[0108] Since the timestamps of the image frames in the target image frame sequence can represent the playback times of the corresponding image frames in the to-be-processed video, and the timestamps of the audio frames in the target audio frame sequence can represent the actual playback times of the corresponding audio frames relative to the to-be-processed video (for example, a certain image frame should be played at the 10th second of the to-be-processed video), the actual corresponding playback positions of the image frames and the audio frames in the to-be-processed video corresponding to the clipping video can be determined based on the timestamps of the image frames in the target image frame sequence and the timestamps of the audio frames in the target audio frame sequence.

[0109] Specifically, the order of the timestamps of the image frames in the target image frame sequence and the audio frames in the target audio frame sequence can be determined as the playback positions of the image frames and the audio frames in the to-be-processed video corresponding to the clipping video. That is, the order of the timestamps of the image frames and the audio frames is the playback positions of the audio frames and the image frames corresponding to the clipping video.

[0110] In some possible implementation manners, when determining the corresponding playing positions of each image frame in the target image frame sequence and each audio frame in the target audio frame sequence to the cropped video corresponding to the to-be-processed video, one image frame can be read from the target image frame sequence and one audio frame can be read from the target audio frame sequence, the time stamps of the image frame and the audio frame are compared to arrange the frame with the smaller time stamp in front to obtain a frame sequence. Further, one image frame or audio frame is read from the target image frame sequence again, and the time stamps of the frame are compared with the time stamps of each frame in the frame sequence obtained previously, and the frame is placed in the frame sequence according to the time stamp sequence. This is repeated until each image frame in the target image frame sequence and each audio frame in the target audio frame sequence are read and sorted, and the corresponding playing positions of each image frame in the target image frame sequence and each audio frame in the target audio frame sequence to the cropped video corresponding to the to-be-processed video are determined based on the final frame sequence.

[0111] In step S24, the cropped video is obtained according to the target audio frame sequence and the target image frame sequence, and the corresponding playing positions of each image frame in the target image frame sequence and each audio frame in the target audio frame sequence.

[0112] In some possible implementation manners, the cropped video is obtained by synchronously writing each image frame and each audio frame into a video file according to the target audio frame sequence and the target image frame sequence, and the corresponding playing positions of each image frame and each audio frame. That is, each audio frame in the target audio frame sequence and each image frame in the target image frame sequence are synchronously written into a video file according to the corresponding playing positions of each image frame and each audio frame to the cropped video, to obtain the cropped video. Therefore, when the cropped video is played, the corresponding playing positions of each image frame and each audio frame can be determined based on the video file of the cropped video, and then the audio frame and the image frame are synchronously played according to the corresponding playing positions of each audio frame and each image frame.

[0113] Referring to Figure 7 , Figure 7 is a scene diagram for determining a playing position provided by an embodiment of the present application. As shown in Figure 7 , if the cropping time interval is 6s to 12s, the target image frame sequence determined based on the cropping time interval and the time stamps of each image frame in the image frame sequence of the to-be-processed video is as shown in Figure 7 , the target audio frame sequence determined based on the cropping time interval and the time stamps of each audio frame in the audio frame sequence of the to-be-processed video is as shown in Figure 7 . Based on the time stamp sequence of each image frame and each audio frame in Figure 7 , each image frame and each audio frame can be sorted to obtain a frame sequence as shown in Figure 7Each image frame and each audio frame corresponds to a play position of the cropped video. That is, when playing the video to be cropped, the image frame with the timestamp of 6.0s, the image frame with the timestamp of 6.0365s, the image frame with the timestamp of 6.093333s, the audio frame with the timestamp of 7.092042s, and the like are played in sequence until the image frame with the timestamp of 12.030000s is played, and the cropped video is played completely.

[0114] In some possible implementation manners, after determining that each image frame in the target image frame sequence and each audio frame in the target audio frame sequence correspond to the play positions of the cropped video corresponding to the video to be processed, the time difference between the timestamps of each audio frame and each image frame can be increased on the basis of keeping the relative size and arrangement order of the timestamps of each image frame and each audio frame unchanged, so that the final cropped video has the effect of slow speed (such as 0.5 times speed) playing. Alternatively, the time difference between the timestamps of each audio frame and each image frame can be reduced, so that the final cropped video has the effect of fast speed (such as 2 times speed) playing.

[0115] In some possible implementation manners, after obtaining the cropped video according to the target audio frame sequence and the target image frame sequence, the cropped video can be played in response to a play operation on the cropped video, based on the fact that each image frame and each audio frame corresponding to the cropped video correspond to the play order of the cropped video.

[0116] Based on the video processing method provided in the embodiments of the present application, a video (such as a game picture) recorded in real time can be cropped to obtain a cropped video, and a video obtained based on multiple acquisition manners (such as a movie or a TV series) can be intercepted, thereby further improving user experience. Figure 8a As shown in FIG. 1, Figure 8a is a schematic diagram of an application scenario of the video processing method provided in the embodiments of the present application. Figure 8a As shown in FIG. 2, an interface diagram after a game match is ended is shown. When a user plays a game match, a game background records a game picture of the user in real time. After the user ends the match, the recorded game picture can be cropped based on the video processing method provided in the embodiments of the present application to obtain a highlight moment of the user in the game match. A play interface of the cropped video of the highlight moment of the user is included in an interface after the game match and the cropped video is played to the user. As shown in FIG. 3, Figure 8b As shown in FIG. 4, Figure 8b is another schematic diagram of an application scenario of the video processing method provided in the embodiments of the present application. In Figure 8b , the video processing method provided in the embodiments of the present application can be used to crop a game match video, so that a player can more specifically review tactics based on the cropped video.

[0117] In the embodiment of the present application, the target image frame sequence is determined by clipping the time interval and the time stamp of each image frame in the image frame sequence of the to-be-processed video, and the target audio frame sequence is determined by clipping the time interval and the time stamp of each audio frame in the audio frame sequence of the to-be-processed video, so that the target image frame sequence and the target audio frame sequence correspond to the clipping time interval at the same time, avoiding the case that the video picture corresponding to the target audio frame sequence does not correspond to the sound segment corresponding to the target image frame sequence. Further, after determining the playing position of each image frame in the target image frame sequence and each audio frame in the target audio sequence in the to-be-processed video corresponding to the clipping video, each image frame in the target image frame sequence and each audio frame in the target audio sequence are located at the corresponding position in the clipping video, further avoiding the case that the audio and picture in the clipping video are out of synchronization, improving the video quality of the clipping video, and having high applicability. Further, based on the method provided in the embodiment of the present application, the playing speed of the clipping video can be changed while the video is being clipped, improving the user experience and having high applicability.

[0118] Referring to Figure 9 , Figure 9 is a structural schematic diagram of a video processing device provided in the embodiment of the present application. The video processing device 1 provided in the embodiment of the present application comprises:

[0119] a clipping time interval determination module 11, configured to determine a clipping time interval of a to-be-processed video;

[0120] a frame sequence determination module 12, configured to determine a target image frame sequence according to the clipping time interval and the time stamp of each image frame in the image frame sequence of the to-be-processed video, and determine a target audio frame sequence according to the clipping time interval and the time stamp of each audio frame in the audio frame sequence of the to-be-processed video;

[0121] a playing position determination module 13, configured to determine the playing position of each image frame in the target image frame sequence and each audio frame in the target audio sequence in a clipping video corresponding to the to-be-processed video according to the time stamp of each image frame in the target image frame sequence and the time stamp of each audio frame in the target audio sequence;

[0122] a clipping video determination module 14, configured to obtain the clipping video according to the target audio frame sequence and the target image frame sequence, and the playing position of each image frame in the target image frame sequence and each audio frame in the target audio sequence.

[0123] In some possible implementation manners, the frame sequence determination module 12 is configured to:

[0124] determining, as a start time point of the target image frame sequence, a time stamp in the time stamps of the image frames in the to-be-processed video that matches the start time point of the clipping time interval;

[0125] determining, as an end time point of the target image frame sequence, a time stamp in the time stamps of the image frames that matches the end time point of the clipping time interval.

[0126] determining the target image frame sequence from the image frame sequence according to the start time point and the end time point of the target image frame sequence.

[0127] In some possible implementation manners, the frame sequence determining module 12 is configured to:

[0128] determining an image key frame in each image frame in the image frame sequence;

[0129] determining, as a start time point of the target image frame sequence, a time stamp in the time stamps of the image key frames in the to-be-processed video that matches the start time point of the clipping time interval;

[0130] determining, as an end time point of the target image frame sequence, a time stamp in the time stamps of the image key frames that matches the end time point of the clipping time interval.

[0131] In some possible implementation manners, the frame sequence determining module 13 is configured to:

[0132] determining, as a start time point of the target audio frame sequence, a time stamp in the time stamps of the audio frames in the audio frame sequence of the to-be-processed video that matches the start time point of the clipping time interval;

[0133] determining, as an end time point of the target audio frame sequence, a time stamp in the time stamps of the audio frames that matches the end time point of the clipping time interval.

[0134] determining the target audio frame sequence from the audio frame sequence according to the start time point and the end time point of the target audio frame sequence.

[0135] In some possible implementation manners, the frame sequence determining module 13 is configured to:

[0136] determining an audio key frame in each audio frame in the audio frame sequence;

[0137] determining, as a start time point of the target audio frame sequence, a time stamp in the time stamps of the audio key frames in the to-be-processed video that matches the start time point of the clipping time interval;

[0138] The timestamp in the timestamp of each of the audio key frames matching the end time point of the clipping time interval is determined as an end time point of the target audio frame sequence.

[0139] In some possible implementation manners, the playing position determining module 13 is configured to:

[0140] The order of the timestamps of each image frame in the target image frame sequence and each audio frame in the target audio sequence is determined as the playing position of each image frame in the target image frame sequence and each audio frame in the target audio sequence in the clipping video corresponding to the to-be-processed video.

[0141] In some possible implementation manners, the clipping video determining module 14 is further configured to:

[0142] In response to a playing operation on the clipping video, the clipping video is played according to the playing positions of the image frames and the audio frames in the clipping video.

[0143] The video processing apparatus can be a computer program (including program code) running in a computer device. For example, the video processing apparatus is an application software, which can be used to perform the implementation manners provided in each of the steps described above. For details, refer to the implementation manners provided in each of the steps described above, which will not be described here again. Figure 2

[0144] In some possible implementation manners, the video processing apparatus provided in the embodiments of the present application can be implemented in a combination of software and hardware. For example, the video processing apparatus provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can be one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic elements.

[0145] In some possible implementation manners, the video processing apparatus provided in the embodiments of the present application can be implemented in a software manner, Figure 9 ​The video processing apparatus shown can be software in the form of programs and plug-ins, and includes a series of modules, including a clip time interval determination module 11, a frame sequence determination module 12, a play position determination module 13, and a clip video determination module 14. The clip time interval determination module 11, the frame sequence determination module 12, the play position determination module 13, and the clip video determination module 14 are used to implement the video processing method provided in the embodiments of the present application.

[0146] In the embodiments of the present application, the target image frame sequence is determined by the clip time interval and the timestamps of the image frames in the image frame sequence of the video to be processed, and the target audio frame sequence is determined by the clip time interval and the timestamps of the audio frames in the audio frame sequence of the video to be processed, so that the target image frame sequence and the target audio frame sequence correspond to the clip time interval at the same time, avoiding the situation that the sound segment corresponding to the target audio frame sequence and the video picture corresponding to the target image frame sequence do not correspond. Further, after the play positions of the image frames of the target image frame sequence and the audio frames of the target audio sequence in the clip video corresponding to the video to be processed are determined, the image frames of the target image frame sequence and the audio frames of the target audio sequence are all located at the corresponding positions in the clip video, further avoiding the situation that the audio and video of the clip video are out of sync, improving the video quality of the clip video, and having high applicability.

[0147] Referring to Figure 10 , Figure 10 is a structural schematic diagram of an electronic device provided in the embodiments of the present application. As Figure 10 shown, the electronic device 1000 in the embodiments can include a processor 1001, a network interface 1004, and a memory 1005, in addition, the electronic device 1000 can further include a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between the components. The user interface 1003 can include a display, a keyboard, and optionally the user interface 1003 can further include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 1005 can be a high-speed RAM memory, or a non-volatile memory, for example, at least one disk memory. The memory 1005 can optionally be at least one storage device located away from the aforementioned processor 1001. As Figure 10 shown, the memory 1005 as a computer readable storage medium can include an operating system, a network communication module, a user interface module, and a device control application program.

[0148] In Figure 10In the electronic device 1000 shown, the network interface 1004 can provide network communication functions; the user interface 1003 is mainly used to provide an interface for user input; and the processor 1001 can be used to invoke a device control application stored in the memory 1005 to implement the video processing method provided in the embodiments of the present application.

[0149] It should be understood that, in some possible implementations, the processor 1001 described above can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The memory can include a read-only memory and a random access memory, and provide the processor with instructions and data. Part of the memory can also include a non-volatile random access memory. For example, the memory can also store device type information.

[0150] In specific implementations, the electronic device 1000 described above can execute the implementation manners provided in each step of the above method through various functional modules built therein, and specific implementation manners can be referred to the implementation manners provided in each step described above, which will not be described herein. Figure 2

[0151] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method provided in each step of the above method. Specific implementation manners can be referred to the implementation manners provided in each step described above, which will not be described herein. Figure 2

[0152] ​​The computer readable storage medium can be an internal storage unit of the aforementioned video processing apparatus or electronic device, such as a hard disk or memory of the electronic device. The computer readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. The computer readable storage medium can also include a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), and the like. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of the electronic device. The computer readable storage medium is used to store the computer program and other programs and data required by the electronic device. The computer readable storage medium can also be used to temporarily store data that has been output or will be output.

[0153] The embodiments of the present application provide a computer program product or computer program, which includes computer instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device perform the method provided by each step of the embodiments. Figure 2

[0154] The terms "first", "second", and the like in the claims and specification and drawings of the present application are used to distinguish different objects, and are not used to describe a particular order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or electronic device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product, or electronic device. In this document, the term "embodiment" means that the specific features, structures, or characteristics described in connection with the embodiment can be included in at least one embodiment of the present application. The phrase is shown at various places in the specification does not necessarily refer to the same embodiment, nor is it independent or alternative to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. The term "and / or" used in the specification and claims of the present application refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0155] ​Those skilled in the art can realize the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be realized in electronic hardware, computer software, or a combination of both. The above described embodiments have been described in connection with the particular examples, and it will be evident to those skilled in the art that various modifications can be made thereto without parting from the spirit and scope of the application, and the application is not to be limited to the disclosed embodiments.

[0156] The above disclosure is merely preferred embodiments of the application and is not to be used to limit the scope of the application. Equivalent variations of the disclosed embodiments can be constructed without departing from the spirit and scope of the application, which is defined by the following claims.

Claims

1. A method of video processing, the method comprising: The method comprises: determining a specific video segment in a to-be-processed video, determining a time interval corresponding to a starting time point and an ending time point of the specific video segment as a clipping time interval of the to-be-processed video; wherein the starting time point and the ending time point of the specific video segment are respectively a time point at which a video frame with the highest occurrence frequency in the to-be-processed video is played for the first time and for the last time, or the starting time point of the specific video segment is a time point corresponding to a video frame including a specific keyword in subtitles played for the first time and for the last time in the to-be-processed video, or the starting time point and the ending time point of the specific video segment are respectively time points corresponding to video frames with a similarity higher than a similarity threshold to a specific image played for the first time and for the last time; the to-be-processed video is obtained by real-time recording of game screens when a user plays a game session; the occurrence frequency of a video frame in the to-be-processed video, subtitles included in the video frame, and the similarity of the video frame to a specific image are determined based on a machine learning manner; after the user ends the session, determining a target image frame sequence according to the clipping time interval and time stamps of each image frame in an image frame sequence of the to-be-processed video, determining a starting time point of a target audio frame sequence from a time stamp in each audio frame of an audio frame sequence of the to-be-processed video that matches the starting time point of the clipping time interval or is closest to the starting time point of the clipping time interval, determining an ending time point of the target audio frame sequence from a time stamp in each audio frame that matches the ending time point of the clipping time interval or is closest to the ending time point of the clipping time interval, and determining the target audio frame sequence from the audio frame sequence according to the starting time point and the ending time point of the target audio frame sequence; wherein the time stamp of each image frame is a play time in the to-be-processed video at which the corresponding image frame should originally be played when the image frame is written into a video file of the to-be-processed video, and the time stamp of each audio frame is a play time in the to-be-processed video at which the corresponding audio frame should originally be played when the audio frame is written into the video file of the to-be-processed video; the play time corresponding to the time stamp of the image frame is different from an actual play time of the image frame in the to-be-processed video, or the play time corresponding to the time stamp of the audio frame is different from an actual play time of the audio frame in the to-be-processed video; reading one image frame from the target image frame sequence and one audio frame from the target audio frame sequence, arranging the image frame and the audio frame according to the timestamps in ascending order to obtain a frame sequence; reading another image frame from the target image frame sequence or reading another audio frame from the target audio frame sequence, comparing the time stamp of the image frame or the audio frame read again with the time stamp of each frame in the frame sequence obtained previously and adding the image frame or the audio frame to the frame sequence according to the order of the time stamp, and continuously reading the image frame or the audio frame and sorting until all the image frames in the target image frame sequence and all the audio frames in the target audio frame sequence are read and sorted; determining the playing position of each image frame in the target image frame sequence and each audio frame in the target audio frame sequence in the cropped video corresponding to the to-be-processed video according to the frame sequence; writing the image frames and the audio frames into a video file according to the target audio frame sequence and the target image frame sequence and the playing position of each image frame in the target image frame sequence and each audio frame in the target audio frame sequence, obtaining the cropped video and displaying the cropped video to the user.

2. The method of claim 1, wherein, The method according to the cropped time interval and the time stamp of each image frame in the image frame sequence of the to-be-processed video to determine a target image frame sequence comprises: determining the time stamp matching the starting time point of the cropped time interval in the time stamp of each image frame in the image frame sequence of the to-be-processed video as the starting time point of the target image frame sequence; determining the time stamp matching the ending time point of the cropped time interval in the time stamp of each image frame as the ending time point of the target image frame sequence; determining the target image frame sequence from the image frame sequence according to the starting time point and the ending time point of the target image frame sequence.

3. The method of claim 2, wherein, Further comprising: determining the image key frame in each image frame in the image frame sequence; the method according to the cropped time interval and the time stamp of each image frame in the image frame sequence of the to-be-processed video to determine a target image frame sequence comprises: determining the time stamp matching the starting time point of the cropped time interval in the time stamp of each image key frame in the to-be-processed video as the starting time point of the target image frame sequence; the method according to the cropped time interval and the time stamp of each image frame in the image frame sequence of the to-be-processed video to determine a target image frame sequence comprises: determining the time stamp matching the ending time point of the cropped time interval in the time stamp of each image key frame as the ending time point of the target image frame sequence.

4. The method of claim 1, wherein, Further comprising: determining the audio key frame in each audio frame in the audio frame sequence; the method according to the cropped time interval and the time stamp of each image frame in the image frame sequence of the to-be-processed video to determine a target image frame sequence comprises: determining the time stamp matching the starting time point of the cropped time interval in the time stamp of each audio frame in the audio frame sequence of the to-be-processed video as the starting time point of the target audio frame sequence; determining, as a starting time point of the target audio frame sequence, a time stamp matched with the starting time point of the clipping time interval in time stamps of each audio key frame in the video to be processed; the determining, as the ending time point of the target audio frame sequence, the time stamp matched with the ending time point of the clipping time interval in the time stamp of each audio frame includes: determining, as the ending time point of the target audio frame sequence, a time stamp matched with the ending time point of the clipping time interval in time stamps of each audio key frame.

5. The method of claim 1, wherein, The method further includes: in response to a playing operation for the clipped video, playing the clipped video according to corresponding playing positions of each image frame and each audio frame in the clipped video.

6. A video processing apparatus, comprising: The device includes: a clipping time interval determination module configured to determine a specific video segment in a video to be processed, and determine, as a clipping time interval of the video to be processed, a time interval corresponding to a starting time point and an ending time point of the specific video segment; wherein the starting time point and the ending time point of the specific video segment are respectively a time point at which a video frame appears for the first time and a time point at which the video frame appears for the last time in the video to be processed, or the starting time point of the specific video segment is a time point corresponding to a video frame including a specific keyword in subtitles played for the first time and for the last time in the video to be processed, or the starting time point and the ending time point of the specific video segment are respectively time points corresponding to video frames with a similarity higher than a similarity threshold to a specific image played for the first time and for the last time; the video to be processed is obtained by real-time recording of game screens during a game session of a user; the appearance frequency of a video frame in the video to be processed, subtitles included in the video frame, and the similarity of the video frame to a specific image are determined based on a machine learning manner. a frame sequence determination module configured to determine a target image frame sequence according to the clipping time interval and the timestamps of the image frames in the image frame sequence of the to-be-processed video after the user ends the game, determine a timestamp in the timestamps of the audio frames in the audio frame sequence of the to-be-processed video that matches the starting time point of the clipping time interval or is closest to the starting time point of the clipping time interval as a starting time point of a target audio frame sequence, determine a timestamp in the timestamps of the audio frames that matches the ending time point of the clipping time interval or is closest to the ending time point of the clipping time interval as an ending time point of the target audio frame sequence, and determine the target audio frame sequence from the audio frame sequence according to the starting time point and the ending time point of the target audio frame sequence; wherein the timestamp of each image frame is the playback time of the corresponding image frame in the to-be-processed video when the image frame is originally written into a video file of the to-be-processed video, and the timestamp of each audio frame is the playback time of the corresponding audio frame in the to-be-processed video when the audio frame is originally written into the video file of the to-be-processed video; the playback time corresponding to the timestamp of the image frame is different from the actual playback time of the image frame in the to-be-processed video, or the playback time corresponding to the timestamp of the audio frame is different from the actual playback time of the audio frame in the to-be-processed video; a playback position determination module configured to read one image frame from the target image frame sequence and one audio frame from the target audio frame sequence, arrange the image frame and the audio frame in the order of timestamps from small to large to obtain a frame sequence, read another image frame from the target image frame sequence or read another audio frame from the target audio frame sequence, compare the timestamp of the read image frame or audio frame with the timestamps of the frames in the frame sequence obtained previously and add the read image frame or audio frame to the frame sequence in the order of timestamps, continuously read image frames or audio frames and arrange them until all the image frames in the target image frame sequence and all the audio frames in the target audio frame sequence are read and arranged, and determine the playback positions of the image frames in the target image frame sequence and the audio frames in the target audio frame sequence in the clipping video corresponding to the to-be-processed video according to the frame sequence; a clipping video determination module configured to write the image frames and the audio frames into a video file synchronously according to the target audio frame sequence, the target image frame sequence, and the playback positions of the image frames in the target image frame sequence and the audio frames in the target audio frame sequence, obtain the clipping video, and display the clipping video to the user.

7. The apparatus of claim 6, wherein, The frame sequence determination module is configured to determine the target image frame sequence according to the clipping time interval and the timestamps of the image frames in the image frame sequence of the to-be-processed video. determine, as a start time point of the target image frame sequence, a time stamp in each image frame of the image frame sequence that matches the start time point of the clipping time interval; determine, as an end time point of the target image frame sequence, a time stamp in each image frame of the image frame sequence that matches the end time point of the clipping time interval; determine the target image frame sequence from the image frame sequence according to the start time point and the end time point of the target image frame sequence.

8. The apparatus of claim 7, wherein, The frame sequence determination module is further configured to: determine image key frames in each image frame of the image frame sequence; determine, as a start time point of the target image frame sequence, a time stamp in each image key frame of the image frame sequence that matches the start time point of the clipping time interval; determine, as an end time point of the target image frame sequence, a time stamp in each image key frame of the image frame sequence that matches the end time point of the clipping time interval.

9. The apparatus of claim 6, wherein, The frame sequence determination module is further configured to: determine audio key frames in each audio frame of the audio frame sequence; determine, as a start time point of the target audio frame sequence, a time stamp in each audio key frame of the image frame sequence that matches the start time point of the clipping time interval; determine, as an end time point of the target audio frame sequence, a time stamp in each audio key frame of the image frame sequence that matches the end time point of the clipping time interval.

10. The apparatus of claim 6, wherein, The clipped video determination module is further configured to: in response to a play operation on the clipped video, play the clipped video according to the corresponding play positions of each image frame and each audio frame in the clipped video.

11. An electronic device, comprising: include a processor and a memory, the processor and the memory being connected to each other; the memory is configured to store a computer program; the processor is configured to execute the method of any one of claims 1 to 5 when the computer program is invoked.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method of any one of claims 1 to 5.

13. A computer program product, the computer program product comprising computer instructions stored in a computer readable storage medium, a processor of an electronic device reading the computer instructions from the computer readable storage medium, the processor executing the computer instructions to implement the method of any one of claims 1 to 5.

Images