Method, apparatus, medium and program product for complex frame processing

Abstract

The invention provides a frame supplementing processing method and device, electronic equipment, a computer readable medium and a computer program product. The method comprises the following steps: determining frame supplementing position information of a target audio / video, a video frame number or an audio sample number needing to be supplemented, and coding mode information; after a corresponding number of video frames or audio samples are obtained, frame compensation processing is carried out on the target audio and video based on the determined frame compensation position information; and coding and packaging the target audio and video after the frame compensation processing. According to the method and the device, the frame filling processing is performed by determining the frame filling position information corresponding to the frame filling processing, the video frame number or the audio sample number needing to be filled and the coding mode information, so that the problems of audio and video desynchrony and data discontinuity are effectively solved at a WEB end, and the video playing quality and the user experience are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, computer-readable medium and computer program product for performing frame interpolation processing. Background Art

[0002] When playing audio and video, you may encounter audio and video asynchrony or data gaps (GAPs), which usually cause playback freezes or interruptions, affecting the user experience. To solve these problems, you can fill the video frame or mute the audio, or use the video automatic alignment tool to handle them.

[0003] However, due to the lack of direct video encoding and decoding capabilities, existing solutions on the web usually skip or crop the audio and video when the audio and video are not aligned or there is a gap between the audio and video. Although skipping can quickly solve the problem, it may cause content loss, especially when the unsynchronized part contains important information. Although cropping can avoid content loss, it may leave unnatural breakpoints in the audio and video stream, affecting the overall fluency and coherence. Summary of the invention

[0004] Multiple aspects of the present application provide a method, an apparatus, an electronic device, a computer-readable medium, and a computer program product for performing frame interpolation processing.

[0005] In one aspect of the present application, a method for performing frame interpolation processing is provided, wherein the method comprises:

[0006] Determine the target audio and video frame complement position information, the number of video frames or audio samples to be complemented, and the encoding method information;

[0007] After obtaining a corresponding number of video frames or audio samples, performing frame interpolation processing on the target audio and video based on the determined interpolation frame position information;

[0008] The target audio and video after the frame supplementation processing is encoded and packaged.

[0009] In one aspect of the present application, a device for performing frame interpolation processing is provided, wherein the device comprises:

[0010] A device for determining the target audio and video frame supplement position information, the number of video frames or audio samples to be supplemented, and the encoding method information;

[0011] A device for performing frame interpolation processing on the target audio and video based on the determined interpolation frame position information after acquiring a corresponding number of video frames or audio samples;

[0012] A device for encoding and packaging target audio and video after frame interpolation processing.

[0013] Another aspect of the present application provides an electronic device, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the embodiment of the application.

[0017] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method of the embodiment of the application.

[0018] Another aspect of the present application provides a computer program product, including a computer program, which implements the method of the embodiment of the present application when executed by a processor.

[0019] In the solution provided in the embodiment of the present application, frame supplementation processing is performed by determining the frame supplementation position information corresponding to the frame supplementation processing, the number of video frames or audio samples to be supplemented, and the encoding method information, thereby effectively handling the audio and video asynchrony and data discontinuity problems on the WEB side, and improving the quality of video playback and user experience; in the case of using MSE for video playback, the embodiment of the present application adopts the FMP4 encapsulation method, which realizes encapsulation based on any time point without the need to encapsulate from the beginning of the video, thereby improving processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0022] Figure 1 A schematic flow chart of a method for performing frame interpolation processing provided in an embodiment of the present application is shown;

[0023] Figure 2A schematic diagram of the structure of a device for performing frame interpolation processing provided in an embodiment of the present application is shown;

[0024] Figure 3 A schematic diagram of the structure of a device suitable for implementing the solution in the embodiment of the present application is shown.

[0025] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0028] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0029] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0030] Figure 1 A schematic flow chart of a method for performing frame interpolation processing provided in an embodiment of the present application is shown. The method at least includes step S101, step S102 and step S103.

[0031] The execution subject of the method may be a client device or an application running on the client device. The client device includes but is not limited to computers, mobile phones, tablet computers, smart watches, bracelets and other terminal devices.

[0032] Preferably, the method is implemented based on a browser on the WEB side, or based on an application of a browser kernel.

[0033] According to one embodiment, before step S101, the method further includes step S104.

[0034] In step S104, it is determined whether the target audio or video to be played needs to be subjected to frame interpolation processing.

[0035] Specifically, whether to perform frame interpolation processing on the target audio or video to be played is determined based on a predetermined frame interpolation processing condition.

[0036] Optionally, when at least any one of the following is met, it is determined that the frame interpolation processing condition is met:

[0037] 1) The difference between the audio duration and the video duration of the target audio and video to be played is greater than a predetermined threshold;

[0038] 2) The existence of a decoding gap (GAP) in the middle of the video is detected through the MSE source buffer.

[0039] Refer to the following Figure 1 To illustrate, in step S101, the target audio and video frame supplementation position information, the number of video frames or audio samples to be supplemented, and the encoding method information are determined.

[0040] The target audio and video is the audio and video to be played in the client device and needs to be processed by frame interpolation.

[0041] The frame interpolation position information includes information for indicating the starting position and the ending position of the frame to be interpolated in the target audio or video, for example, the starting time point and the ending time point of the frame to be interpolated, or the time point range of the frame to be interpolated.

[0042] If the video part of the target audio and video needs to be supplemented, the number of supplemented video frames needs to be determined; if the audio part of the target audio and video needs to be supplemented, the number of supplemented audio samples needs to be determined.

[0043] According to one embodiment, the method determines the frame supplement position information by executing step S1011.

[0044] Specifically, the method of determining the interpolation frame position information includes but is not limited to:

[0045] 1) According to the audio duration information and video duration information of the target audio and video, determine the starting position and ending position of the interpolation frame in the target audio and video as the interpolation frame position information.

[0046] Specifically, by comparing the audio duration and video duration of the target audio and video, the starting position of the interpolated frame is determined to be the end time of the shorter duration file. If the video duration is shorter, the end time of the video is used as the starting position of the interpolated frame; if the audio duration is shorter, the end time of the audio is used as the starting position of the interpolated frame.

[0047] For example, if the video duration is 30.123 seconds and the audio duration is 40.323 seconds, the starting position of the interpolated frame is determined to be the end time of the shorter one, that is, 30.123 seconds.

[0048] 2) Obtaining the discontinuity interval of the target audio and video, so as to obtain the interpolation frame position information based on the starting position of the discontinuity interval in the target audio and video.

[0049] Specifically, the discontinuity points of the audio and video data are obtained through the MSE (Media Source Extensions) SourceBuffer. Under normal circumstances, a decoded audio and video buffer is continuous. When an exception occurs, a discontinuity will occur in the middle of the buffer, forming a discontinuous interval. For example, if there are intervals [0,3.123) and (4.256,100], it can be determined that there is a buffer missing interval from 3.123 seconds to 4.256 seconds. Based on the discontinuity interval [3.123,4.256], the range of the interpolation frame is determined to be the discontinuity interval, and then the starting position of the frame interpolation process is 3.123 seconds and the ending position is 4.256 seconds.

[0050] According to one embodiment, the method determines the number of video frames or audio samples that need to be supplemented by executing step S1012.

[0051] In step S1012, the number of video frames to be supplemented is calculated based on the frame rate and missing time of the video part in the target audio and video; or, the number of audio samples to be supplemented is calculated based on the sampling rate, number of channels and missing time of the audio part in the target audio and video.

[0052] The step of calculating the number of video frames to be supplemented in step S1012 includes:

[0053] By parsing the audio and video stream of the target audio and video, the media metadata of the video part is obtained. The media metadata includes the frame rate (Frames Per Second, FPS), resolution, encoding format, etc. of the video.

[0054] Next, the time difference between the video portion and the audio portion is calculated by comparing the time stamps of the video portion and the audio portion. The time difference may be positive or negative, depending on whether the video is faster or slower than the audio.

[0055] Next, the number of video frames that need to be supplemented is calculated based on the calculated time difference and the frame rate of the video.

[0056] For example, based on the calculated time difference and the frame rate of the video, the number of video frames N that need to be supplemented can be calculated according to the following formula:

[0057] N = Math.Floor(T * F) (1)

[0058] Where T represents the time difference, and F represents the frame rate. The number of video frames that need to be supplemented N is obtained by calculating the product of the frame rate and the time difference and then rounding down. Rounding down is used here because in the synchronization process, the audio is usually used as the time baseline, and the video duration can be slightly lower than the audio to ensure that the audio is not delayed due to the extension of the video.

[0059] The step of calculating the number of audio samples to be supplemented in step S1012 includes:

[0060] By parsing the audio and video stream of the target audio and video, media metadata of the audio part is obtained. The metadata includes the audio sampling rate (Samples Per Second, SPS), the number of channels, and the audio arrangement.

[0061] Next, similar to video frame number supplementation, the time difference between the video part and the audio part is calculated by comparing the time stamps of the video part and the audio part.

[0062] Next, the number of samples that need to be supplemented is calculated according to the sampling rate, the number of channels and the audio arrangement of the audio part.

[0063] Specifically, if the arrangement mode is interleaved, that is, the multi-channel data in the audio stream is interleaved, and the data dimension (array dimension) is 1, the number of audio samples N that need to be supplemented can be calculated by the following formula:

[0064] N = Math.Ceil(S * C * T) (2)

[0065] Among them, S represents the sampling rate, C represents the number of channels, and T represents the time difference.

[0066] If the arrangement mode is independent, that is, the data of each channel is stored independently, the data dimension increases to C. At this time, the number of single-dimensional data samples required for each channel is: S*T, and the total data volume is still calculated by the above formula (2), that is, the total data volume N=Math.Ceil(S*C*T).

[0067] The encoding method information includes various information for indicating the adopted audio and video encoding methods.

[0068] According to one embodiment, the method determines the encoding mode by executing step S1013 and step S1014.

[0069] In step S1013, it is determined whether the client device supports the encoding type of the target audio and video.

[0070] Specifically, the encoding capability information of the client device is obtained. The encoding capability information includes but is not limited to the encoders and CPU logical core numbers supported by the client device. Then, based on the encoding capability information, it is determined whether the client device supports the encoding type of the target audio and video to be processed.

[0071] For example, the acquired encoding capability information indicates that the current client device supports VideoEncoder or AudioEncoder, and the number of CPU logical cores exceeds 16. Next, based on the encoding capability information, it is determined whether the current audio and video encoding type, such as avc1.640028, is supported.

[0072] In step S1014, if the client device supports the encoding type of the target audio and video, the encoding type is used as the encoding mode information; if the client device does not support the encoding type of the target audio and video, the process ends.

[0073] Continue to refer to Figure 1 To illustrate, in step S102, after obtaining a corresponding number of video frames or audio samples, frame interpolation processing is performed on the target audio and video based on the determined interpolation frame position information.

[0074] Specifically, based on the number of video frames or audio samples to be supplemented determined in step S101, a corresponding number of video frames and audio samples are obtained. Optionally, the obtained video frames are blank frames, and the obtained audio samples are silent audio samples. Then, based on the supplemented frame position information determined in step S101, the obtained video frames or audio samples are inserted into the target audio and video according to the supplemented frame start position and end position indicated by the supplemented frame position.

[0075] In step S103, the target audio and video after the frame supplementation processing is encoded and packaged.

[0076] Specifically, step S103 includes step S1031 and step S1032.

[0077] In step S1031, the target audio and video after the frame supplementation processing is encoded according to the encoding method information.

[0078] The start time of encoding corresponds to the supplementary frame position information determined in step S101.

[0079] In step S1032, the encoded target audio and video is encapsulated using the FMP4 encapsulation algorithm.

[0080] The FMP4 encapsulation algorithm supports the ability to encapsulate and fragment media frames with non-zero start times.

[0081] Specifically, through FMP4 encapsulation, the target audio and video after the frame supplementation processing is divided into segments of predetermined lengths, and these segments can be transmitted and decoded independently of other segments.

[0082] According to one embodiment, the method further includes step S105.

[0083] In step S105, the encapsulated segments are added to the MSE source buffer for decoding and playback.

[0084] For example, through FMP4 encapsulation, the target audio and video after frame supplementation processing is divided into multiple segments with a duration of 5 seconds, and the obtained segments are dynamically inserted into the MSE source buffer for decoding and playback.

[0085] Among them, the method according to this embodiment decodes the video fragments based on Media Source Extensions (MSE). MSE defines the MediaSource object as the media data source of HTMLMediaElement in HTML 5. The MediaSource object can have one or more source buffer (SourceBuffer) objects. The application can dynamically add data fragments to the SourceBuffer object, and can adaptively adjust the data quality of the added media data according to system performance and other factors. The data from the SourceBuffer object can be decoded into audio, video or text data.

[0086] The method of the present application is described below with reference to an example.

[0087] When a user watches live broadcasts through a web browser, the client of the user requests and downloads audio and video segments from the corresponding server. After the download is completed, the client player interpolates the target audio and video to be played and decodes and plays them after interpolation. The method of this example uses MSE to dynamically add corresponding segments to the SourceBuffer object defined in MSE for decoding.

[0088] After downloading a video segment, it is first determined whether to perform frame interpolation based on the predetermined frame interpolation processing conditions. The frame interpolation processing conditions include: the difference between the audio duration and the video duration is greater than 0.8 seconds (based on the audio clock capability and online data statistics) or the presence of a decoding gap (GAP) in the middle of the video is detected through the MSE SourceBuffer.

[0089] If the frame interpolation condition is met, the frame interpolation process is performed by executing the above steps S101 to S103 accordingly, and then decoding and packaging are performed. If the frame interpolation condition is not met, the newly downloaded video segment is added to the SourceBuffer and waits for decoding and playback.

[0090] According to the method of the embodiment of the present application, frame supplementation processing is performed by determining the frame supplementation position information corresponding to the frame supplementation processing, the number of video frames or audio samples to be supplemented, and the encoding method information, thereby effectively processing the audio and video asynchrony and data discontinuity problems on the WEB side, and improving the quality of video playback and user experience; for the case of using MSE for video playback, the embodiment of the present application adopts the FMP4 encapsulation method, which realizes encapsulation based on any time point without the need to encapsulate from the beginning of the video, thereby improving processing efficiency.

[0091] In addition, the embodiment of the present application also provides a device for performing frame interpolation processing, the structure of which is as follows: Figure 2 shown.

[0092] The device includes: a device for determining the interpolation frame position information of the target audio and video, the number of video frames or audio samples to be supplemented, and the encoding method information (hereinafter referred to as "information determination device 101"), a device for performing frame interpolation processing on the target audio and video based on the determined interpolation frame position information after obtaining a corresponding number of video frames or audio samples (hereinafter referred to as "frame interpolation processing device 102"), and a device for encoding and packaging the target audio and video after the frame interpolation processing (hereinafter referred to as "encoding packaging device 103").

[0093] Wherein, the device is included in the client equipment.

[0094] Preferably, the device is implemented based on a browser on the WEB side, or based on an application of a browser kernel.

[0095] According to one embodiment, the device further includes a frame supplement determination device.

[0096] The frame interpolation judgment device judges whether the target audio or video to be played needs to be processed by frame interpolation.

[0097] Specifically, the frame interpolation determination device determines whether to perform frame interpolation processing on the target audio or video to be played based on a predetermined frame interpolation processing condition.

[0098] Optionally, when at least any one of the following is met, it is determined that the frame interpolation processing condition is met:

[0099] 1) The difference between the audio duration and the video duration of the target audio and video to be played is greater than a predetermined threshold;

[0100] 2) The existence of a decoding gap (GAP) in the middle of the video is detected through the MSE source buffer.

[0101] Refer to the following Figure 2 To explain, the information determining device 101 determines the target audio and video frame supplementation position information, the number of video frames or audio samples to be supplemented, and the encoding method information.

[0102] The target audio and video is the audio and video to be played in the client device and needs to be processed by frame interpolation.

[0103] The frame interpolation position information includes information for indicating the starting position and the ending position of the frame to be interpolated in the target audio or video, for example, the starting time point and the ending time point of the frame to be interpolated, or the time point range of the frame to be interpolated.

[0104] If the video part of the target audio and video needs to be supplemented, the information determination device 101 needs to determine the number of supplemented video frames; if the audio part of the target audio and video needs to be supplemented, the information determination device 101 needs to determine the number of supplemented audio samples.

[0105] According to an embodiment, the information determining device 101 determines the frame interpolation position information corresponding to the frame interpolation process.

[0106] Specifically, the information determining device 101 determines the frame interpolation position information corresponding to the frame interpolation process in the following manners, but is not limited to:

[0107] 1) According to the audio duration information and video duration information of the target audio and video, determine the starting position and ending position of the interpolation frame in the target audio and video as the interpolation frame position information.

[0108] Specifically, the information determination device 101 determines the starting position of the supplementary frame as the end time of the shorter duration file by comparing the audio duration and the video duration of the target audio and video. If the video duration is shorter, the end time of the video is used as the starting position of the supplementary frame; if the audio duration is shorter, the end time of the audio is used as the starting position of the supplementary frame.

[0109] For example, if the video duration is 30.123 seconds and the audio duration is 40.323 seconds, the starting position of the interpolated frame is determined to be the end time of the shorter one, that is, 30.123 seconds.

[0110] 2) Obtaining the discontinuity interval of the target audio and video, so as to obtain the interpolation frame position information based on the starting position of the discontinuity interval in the target audio and video.

[0111] Specifically, the information determination device 101 obtains the discontinuity point of the audio and video data through the MSE (Media Source Extensions) SourceBuffer. Under normal circumstances, a decoded audio and video buffer is continuous. When an exception occurs, a discontinuity will occur in the middle of the buffer, forming a discontinuous interval. For example, if there are intervals [0,3.123) and (4.256,100], it can be determined that there is a buffer missing interval from 3.123 seconds to 4.256 seconds. Based on the discontinuous interval [3.123,4.256], the range of the frame supplement is determined to be the discontinuous interval, and then the starting position of 3.123 seconds and the ending position of 4.256 seconds of the frame supplement processing are obtained.

[0112] According to one embodiment, the information determining device 101 comprises a supplementary calculation device, and the supplementary calculation device is used to determine the number of video frames or audio samples that need to be supplemented.

[0113] The supplementary calculation device calculates the number of video frames that need to be supplemented according to the frame rate and missing time of the video part in the target audio and video; or, the supplementary calculation device calculates the number of audio samples that need to be supplemented according to the sampling rate, number of channels and missing time of the audio part in the target audio and video.

[0114] The step of the supplementary calculation device calculating the number of video frames to be supplemented includes:

[0115] The supplementary computing device parses the audio and video stream of the target audio and video to obtain the media metadata of the video portion, wherein the media metadata includes the frame rate (Frames Per Second, FPS), resolution, encoding format, etc. of the video.

[0116] Next, the complementary calculation device calculates the time difference between the video portion and the audio portion by comparing the time stamps of the video portion and the audio portion. The time difference may be positive or negative, depending on whether the video is faster or slower than the audio.

[0117] Next, the supplement calculation device calculates the number of video frames that need to be supplemented according to the calculated time difference and the frame rate of the video.

[0118] For example, based on the calculated time difference and the frame rate of the video, the number of video frames N that need to be supplemented is calculated according to the following formula (1).

[0119] The step of calculating the number of audio samples to be supplemented by the supplement calculation device comprises:

[0120] The supplementary computing device parses the audio and video stream of the target audio and video to obtain the media metadata of the audio part, wherein the metadata includes the audio sampling rate (Samples Per Second, SPS), the number of channels, and the audio arrangement.

[0121] Next, similar to video frame number supplementation, the supplement calculation device calculates the time difference between the video part and the audio part by comparing the time stamps of the video part and the audio part.

[0122] Next, the supplement calculation device calculates the number of samples that need to be supplemented according to the sampling rate, the number of channels and the audio arrangement of the audio part.

[0123] Specifically, if the arrangement mode is interlaced, that is, the multi-channel data in the audio stream is interlaced, the data dimension (array dimension) is 1, and the number of audio samples N that need to be supplemented is calculated by formula (2).

[0124] If the arrangement mode is independent, that is, the data of each channel is stored independently, the data dimension increases to C. At this time, the number of single-dimensional data samples required for each channel is: S*T, and the total data volume is still calculated by the above formula (2), that is, the total data volume N=Math.Ceil(S*C*T).

[0125] The encoding method information includes various information for indicating the adopted audio and video encoding methods.

[0126] According to one embodiment, the device includes a coding type determination device and a coding mode determination device.

[0127] The encoding type determination device determines whether the client device supports the encoding type of the target audio and video.

[0128] Specifically, the encoding type determination device obtains encoding capability information of the client device. The encoding capability information includes but is not limited to the encoders and CPU logical core numbers supported by the client device. Then, based on the encoding capability information, it is determined whether the client device supports the encoding type of the audio and video corresponding to the target audio and video to be processed.

[0129] For example, the acquired encoding capability information indicates that the current client device supports VideoEncoder or AudioEncoder, and the number of CPU logical cores exceeds 16. Next, the encoding type determination device determines whether the current audio and video encoding type, such as avc1.640028, is supported based on the encoding capability information.

[0130] If the client device supports the encoding type of the target audio and video, the encoding mode determination device uses the encoding type as the encoding mode information; if the client device does not support the encoding type of the target audio and video, the processing is terminated.

[0131] Continue to refer to Figure 2 To illustrate, after acquiring a corresponding number of video frames or audio samples, the frame interpolation processing device 102 performs frame interpolation processing on the target audio and video based on the determined frame interpolation position information.

[0132] Specifically, the frame supplementation processing device 102 obtains a corresponding number of video frames and audio samples based on the number of video frames or audio samples to be supplemented determined by the information determination device 101. Optionally, the video frames obtained by the frame supplementation processing device 102 are blank frames, and the audio samples obtained are silent audio samples. Then, based on the frame supplementation position information determined by the information determination device 101, the obtained video frames or audio samples are inserted into the target audio and video according to the frame supplementation start position and end position indicated by the frame supplementation position.

[0133] The encoding and packaging device 103 encodes and packages the target audio and video after the frame supplementation processing.

[0134] The encoding and packaging device 103 further includes an encoding processing device and a packaging processing device.

[0135] The encoding processing device encodes the target audio and video after the frame supplementation processing according to the encoding method information.

[0136] The start time of the encoding corresponds to the supplementary frame position information determined by the information determination device 101 .

[0137] The encapsulation processing device uses the FMP4 encapsulation algorithm to encapsulate the encoded target audio and video.

[0138] The FMP4 encapsulation algorithm supports the ability to encapsulate and fragment media frames with non-zero start times.

[0139] Specifically, the encapsulation processing device divides the target audio and video after the frame supplementation processing into segments of predetermined lengths through FMP4 encapsulation, and these segments can be transmitted and decoded independently of other segments.

[0140] According to one embodiment, the device further comprises a slice adding device.

[0141] The fragment adding device adds the fragments obtained after encapsulation to the MSE source buffer for decoding and playback.

[0142] For example, through FMP4 encapsulation, the target audio and video after frame supplementation processing is divided into multiple slices with a duration of 5 seconds, and the slice adding device dynamically inserts the obtained slices into the MSE source buffer for decoding and playback.

[0143] Among them, the device according to this embodiment decodes the video fragments based on Media Source Extensions (MSE). MSE defines the MediaSource object as the media data source of HTMLMediaElement in HTML 5. The MediaSource object can have one or more source buffer (SourceBuffer) objects. The application can dynamically add data fragments to the SourceBuffer object, and can adaptively adjust the data quality of the added media data according to system performance and other factors. The data from the SourceBuffer object can be decoded into audio, video or text data.

[0144] The device of the present application is described below with reference to an example.

[0145] The device is included in a browser on the WEB side. When a user watches a live broadcast through a browser on the WEB side, the client of the user requests and downloads audio and video segments from the corresponding server. After the download is completed, the client player performs frame interpolation processing on the target audio and video to be played and decodes and plays the target audio and video after the frame interpolation processing. The method of this example uses MSE to dynamically add corresponding segments to the SourceBuffer object defined in MSE for decoding.

[0146] After downloading a video segment, the frame interpolation judgment device first determines whether to perform frame interpolation based on a predetermined frame interpolation processing condition. The frame interpolation processing condition includes: the difference between the audio duration and the video duration is greater than 0.8 seconds (based on the audio clock capability and online data statistics) or the MSE SourceBuffer detects that there is a decoding gap (GAP) in the middle of the video.

[0147] If the frame interpolation condition is met, frame interpolation processing is performed accordingly by executing the operations of the information determination device 101, the frame interpolation processing device 102 and the encoding and packaging device 103, and then decoding and packaging are performed. If the frame interpolation condition is not met, the newly downloaded video segment is added to the SourceBuffer and waits for decoding and playback.

[0148] According to the device of the embodiment of the present application, frame supplementation processing is performed by determining the frame supplementation position information corresponding to the frame supplementation processing, the number of video frames or audio samples that need to be supplemented, and the encoding method information, thereby effectively processing the audio and video asynchrony and data discontinuity problems on the WEB side, and improving the quality of video playback and user experience; for the case of using MSE for video playback, the embodiment of the present application adopts the FMP4 encapsulation method, which realizes encapsulation based on any time point without the need to encapsulate from the beginning of the video, thereby improving processing efficiency.

[0149] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the method for performing frame interpolation processing in the aforementioned embodiment, and the principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0150] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0151] Figure 3The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In RAM1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through bus 1204. Input / output (I / O, Input / Output) interface 1205 is also connected to bus 1204.

[0152] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0153] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above functions defined in the method of the present application are executed.

[0154] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0155] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0156] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0157] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0158] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0159] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0160] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0161] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0162] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0164] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0166] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for performing frame interpolation processing, wherein: The method comprises: Determine the target audio and video frame complement position information, the number of video frames or audio samples to be complemented, and the encoding method information; After obtaining a corresponding number of video frames or audio samples, performing frame interpolation processing on the target audio and video based on the determined interpolation frame position information; The target audio and video after the frame supplementation processing is encoded and packaged.

2. The method according to claim 1, wherein: The method comprises the following steps of determining the number of video frames or audio samples to be supplemented: The number of video frames to be supplemented is calculated based on the frame rate and missing time of the video part in the target audio and video; or the number of audio samples to be supplemented is calculated based on the sampling rate, number of channels and missing time of the audio part in the target audio and video.

3. The method according to claim 2, wherein: The calculating the number of video frames to be supplemented according to the frame rate and missing time of the video part in the target audio and video includes: By parsing the audio and video stream of the target audio and video, media metadata of the video portion is obtained; Calculating a time difference between the video portion and the audio portion by comparing time stamps of the video portion and the audio portion; The number of video frames that need to be supplemented is calculated based on the calculated time difference and the frame rate of the video.

4. The method according to claim 2, wherein: The method of calculating the number of audio samples to be supplemented according to the sampling rate, the number of channels and the missing time of the audio part in the target audio and video includes: By parsing the audio and video stream of the target audio and video, media metadata of the audio part is obtained; Calculating a time difference between the video portion and the audio portion by comparing time stamps of the video portion and the audio portion; The number of samples that need to be supplemented is calculated based on the sampling rate, the number of channels and the audio arrangement of the audio part.

5. The method according to claim 1, wherein: The method comprises the following steps of determining the position information of the supplemented frame: Determine the start position and the end position of the interpolation frame in the target audio and video according to the audio duration information and the video duration information of the target audio and video as the interpolation frame position information; or The discontinuity interval of the target audio and video is obtained to obtain the interpolation frame position information based on the starting position of the discontinuity interval in the target audio and video.

6. The method according to claim 1, wherein: The step of encoding and packaging the target audio and video after the frame supplementation process comprises: Determine whether the client device supports the encoding type of the target audio and video; If the client device supports the encoding type of the target audio and video, the encoding type is used as the encoding mode information; if the client device does not support the encoding type of the target audio and video, the processing is terminated.

7. The method according to claim 6, wherein: The determining whether the client device supports the encoding type of the target audio and video includes: Obtain encoding capability information of client devices; Based on the encoding capability information, determine whether the client device supports the encoding type of the target audio and video.

8. The method according to claim 6, wherein: The encoding and packaging of the target audio and video after the frame supplementation process includes: Encoding the target audio and video after the frame supplementation processing according to the encoding method information; Use the FMP4 encapsulation algorithm to encapsulate the encoded target audio and video.

9. The method according to any one of claims 1 to 8, wherein: The method further comprises: The encapsulated segments are added to the MSE source buffer for decoding and playback.

10. A device for performing frame interpolation processing, wherein: The device comprises: A device for determining the target audio and video frame supplement position information, the number of video frames or audio samples to be supplemented, and the encoding method information; A device for performing frame interpolation processing on the target audio and video based on the determined interpolation frame position information after acquiring a corresponding number of video frames or audio samples; A device for encoding and packaging target audio and video after frame interpolation processing.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

12. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images