Complementary frame processing method, device and equipment for audio and video coding and medium

Abstract

The invention discloses a supplementary frame processing method, device and equipment for audio and video coding and a medium, and belongs to the technical field of audio and video data processing. The method comprises the following steps of: inputting acquired audio and video data frames into a coding module frame by frame, and caching the audio and video data frames frame by frame; if it is identified that the frame compensation trigger event exists, providing a frame compensation data frame for the coding module based on the cached audio and video data frame; reading an output result output to a cache queue by the coding module, and determining whether continuous frame complement data frames exceed an interruption threshold value or not; and if so, generating a null frame data frame, and providing the null frame data frame to the coding module. According to the technical scheme, the timeliness of frame supplementation in the audio and video coding process can be improved, the reliability of judging whether interruption occurs or not can be improved, and timely processing can be carried out under the condition that acquisition interruption is recognized.

Description

Technical Field

[0001] The present application belongs to the technical field of audio and video data processing, and specifically relates to a method, device, equipment and medium for processing interpolation frames of audio and video coding. Background Art

[0002] With the rapid development of Internet technology, it has become very common to use real-time video recording to conduct video surveillance and course recording.

[0003] The audio and video data collection methods of the real-time collection system may include recording by a camera, data transmission by a USB (Universal Serial Bus) interface, and high-definition video access by an HDMI (High Definition Multimedia Interface), etc. Taking the video collection by a camera as an example, after the camera collects image signals and audio signals through collection devices such as a camera and a microphone, the collected video signals are pre-processed to convert them into a unified format, and then the above audio and video signals are encoded by an encoder.

[0004] The existing related technologies only collect and encode audio and video signals, but do not take into account the jitter or frame loss of audio and video data frames caused by signal instability and signal interruption during the process of collecting audio and video signals, and the encoder does not have the function of automatic frame supplement. As a result, the audio and video data output by the encoder also has jitter or frame loss. Therefore, the existing related technologies have the problem of untimely frame supplement of audio and video data frames and unstable video frame rate. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method, device, equipment and medium for frame interpolation processing of audio and video encoding, which can solve the problems of untimely frame interpolation of audio and video data frames and unstable video frame rate. By caching the collected audio and video data frame by frame, and providing frame interpolation data frames to the encoding module based on the cached audio and video data frames when a frame interpolation trigger event is identified, the timeliness of audio and video frame interpolation can be improved. By reading the output results of the encoding module output to the cache queue, it is determined whether the continuous frame interpolation data frames exceed the interruption threshold, and generating empty frame data frames when the interruption threshold is exceeded. This can improve the reliability of interruption judgment and the timeliness of processing interruption phenomena.

[0006] In a first aspect, an embodiment of the present application provides a method for processing frame interpolation in audio and video coding, the method comprising:

[0007] The collected audio and video data frames are input into the encoding module frame by frame, and cached frame by frame;

[0008] If a frame supplementation trigger event is identified, providing a frame supplementation data frame to the encoding module based on the cached audio and video data frame;

[0009] Reading the output result output by the encoding module to the cache queue to determine whether the continuous frame supplementation data frames exceed the interruption threshold;

[0010] If it exceeds, an empty frame data frame is generated and provided to the encoding module.

[0011] Furthermore, after reading the output result output by the encoding module to the cache queue, the method further includes:

[0012] If it is identified that there is a preset redundant frame supplement event, a target frame is determined in the cache queue and a deletion operation is performed on the target frame.

[0013] Further, if a preset redundant frame supplement event is identified, a target frame is determined in the cache queue, and a deletion operation is performed on the target frame, including:

[0014] If it is identified that the actual number of output frames in the cache queue within the preset monitoring time exceeds the theoretical number of frames, it is determined that a preset redundant frame supplement event occurs;

[0015] The frame-filling data frame is determined as a target frame within the preset monitoring time, and a deletion operation is performed on the target frame.

[0016] Further, determining the frame-filling data frame as the target frame within the preset monitoring time period includes:

[0017] The frame-complementing data frames within a preset time period before the audio and video data frames collected within the preset monitoring time period are determined as target frames.

[0018] Furthermore, after reading the output result output by the encoding module to the cache queue, the method further includes:

[0019] If it is identified that the continuous interpolation frame data frames exceed the jamming threshold, a re-interpolation instruction is issued to provide the interpolation frame data frames to the encoding module based on the currently cached audio and video data frames;

[0020] A target frame is determined in the continuous frame-complementing data frames, and a deletion operation is performed on the target frame.

[0021] Further, providing the encoding module with a supplementary frame data frame based on the cached audio and video data frame includes:

[0022] Copy the latest audio and video data frame in the cache to obtain a frame-filling data frame;

[0023] The frame complementation data frame is provided to the encoding module.

[0024] Further, after providing the frame supplementation data frame to the encoding module based on the cached audio and video data frame, the method further includes:

[0025] Marking the interpolated frame data frame with an interpolated frame label;

[0026] Accordingly, reading the output result output by the encoding module to the cache queue to determine whether the continuous frame supplementation data frames exceed the interruption threshold includes:

[0027] Identify whether the output result output by the encoding module to the cache queue carries a frame supplement tag;

[0028] It is determined whether the continuous frame-filling data frames exceed the interruption threshold according to the recognition result.

[0029] In a second aspect, an embodiment of the present application provides a frame interpolation processing device for audio and video coding, the device comprising:

[0030] The buffer unit is used to input the collected audio and video data frames into the encoding module frame by frame, and buffer them frame by frame;

[0031] A frame interpolation unit, configured to provide an interpolation frame data frame to the encoding module based on the cached audio and video data frame when a frame interpolation triggering event is identified;

[0032] An interruption determination unit, configured to read the output result output by the encoding module to the cache queue, and determine whether the continuous frame supplementation data frames exceed an interruption threshold;

[0033] The empty frame generating unit is used to generate an empty frame data frame when the continuous frame supplementation data frames exceed the interruption threshold, and provide the empty frame data frame to the encoding module.

[0034] Furthermore, the device further comprises a redundant frame interpolation processing unit; the redundant frame interpolation processing unit is used to:

[0035] If it is identified that there is a preset redundant frame supplement event, a target frame is determined in the cache queue and a deletion operation is performed on the target frame.

[0036] Furthermore, the redundant frame supplementation processing unit is specifically used for:

[0037] If it is identified that the actual number of output frames in the cache queue within the preset monitoring time exceeds the theoretical number of frames, it is determined that a preset redundant frame supplement event occurs;

[0038] The frame-filling data frame is determined as a target frame within the preset monitoring time, and a deletion operation is performed on the target frame.

[0039] Furthermore, the redundant frame supplementation processing unit is specifically used for:

[0040] The frame-complementing data frames within a preset time period before the audio and video data frames collected within the preset monitoring time period are determined as target frames.

[0041] Furthermore, the device further includes a jamming processing unit; the jamming processing unit is used to:

[0042] If it is identified that the continuous interpolation frame data frames exceed the jamming threshold, a re-interpolation instruction is issued to provide the interpolation frame data frames to the encoding module based on the currently cached audio and video data frames;

[0043] A target frame is determined in the continuous frame-complementing data frames, and a deletion operation is performed on the target frame.

[0044] Furthermore, the frame interpolation unit is specifically used for:

[0045] Copy the latest audio and video data frame in the cache to obtain a frame-filling data frame;

[0046] The frame complementation data frame is provided to the encoding module.

[0047] Furthermore, the frame supplement unit is further used for:

[0048] Marking the interpolated frame data frame with an interpolated frame label;

[0049] Accordingly, the interrupt determination unit is specifically used for:

[0050] Identify whether the output result output by the encoding module to the cache queue carries a frame supplement tag;

[0051] It is determined whether the continuous frame-filling data frames exceed the interruption threshold according to the recognition result.

[0052] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0053] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0054] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0055] In an embodiment of the present application, the collected audio and video data frames are input into the encoding module frame by frame, and are cached frame by frame; if a frame supplement trigger event is identified, a frame supplement data frame is provided to the encoding module based on the cached audio and video data frame; the output result output by the encoding module to the cache queue is read to determine whether the continuous frame supplement data frames exceed the interruption threshold; if exceeded, an empty frame data frame is generated and provided to the encoding module. The above-mentioned audio and video coding frame supplement processing method can solve the problems of untimely video data frame supplement and unstable video frame rate. By caching the collected audio and video data frame by frame, and providing the encoding module with a frame supplement data frame based on the cached audio and video data frame when a frame supplement trigger event is identified, the timeliness of video frame supplement can be improved. By reading the output result output by the encoding module to the cache queue, it is determined whether the continuous frame supplement data frames exceed the interruption threshold, and an empty frame data frame is generated when the interruption threshold is exceeded, which can improve the reliability of interruption judgment and the timeliness of processing interruption phenomena. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flowchart of a frame supplementation processing method for audio and video coding provided by an embodiment of the present application;

[0057] Figure 2 It is a structural diagram of the connection between the encoding module and the cache queue provided in an embodiment of the present application;

[0058] Figure 3 A method for identifying a preset redundant frame supplementation event provided in an embodiment of the present application;

[0059] Figure 4 It is a flowchart of a target frame deletion method provided in an embodiment of the present application;

[0060] Figure 5 It is a flowchart of another target frame deletion method provided in an embodiment of the present application;

[0061] Figure 6 It is a flowchart of a frame supplementation processing method for audio and video coding provided by an embodiment of the present application;

[0062] Figure 7 It is a structural diagram of a frame interpolation processing device for audio and video coding provided in an embodiment of the present application;

[0063] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0065] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0066] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0067] In conjunction with the accompanying drawings, the following detailed description is given of the frame supplementation processing method, device, equipment and medium for audio and video coding provided by the embodiments of the present application through specific embodiments and their application scenarios.

[0068] Figure 1 FIG. 1 is a flow chart of a frame supplementation processing method for audio and video coding provided by an embodiment of the present application. Figure 1 As shown, the specific steps include:

[0069] S101, input the collected audio and video data frames into the encoding module frame by frame, and cache them frame by frame;

[0070] First of all, the application scenario of this solution can be the scenario of audio and video encoding, recording and transmission, especially the scenario of real-time acquisition of audio and video.

[0071] Based on the above usage scenarios, it can be understood that the executor of this solution can be smart terminals such as mobile phones, tablets and computers for receiving audio and video data, or cameras, microphones and other terminal devices with recording functions for recording audio and video, without specific limitation here.

[0072] Among them, the audio and video data frame can be the basic component unit of a section of audio and video, or it can be the smallest processing unit of the codec, including the video image data frame and the audio data frame obtained by real-time acquisition. The frame rate of the video data frame and the audio data frame can be the same. The encoding module can be a module for encoding the audio and video data frame to compress the audio and video data or perform other processing methods. Take the image frame encoding and compression in audio and video as an example: the encoder encodes and compresses multiple images to produce GOP (Group of Pictures) in sections, and the decoder reads the GOP in sections during playback, decodes it, reads the picture, and then renders it for display. GOP (Group of Pictures) is the compressed image data, including an I frame (key frame) and several B frames (bidirectional reference frames) / P frames (forward reference frames). Among them, the I frame is the first frame in the video sequence, which contains all the information of the first frame image and can be decoded independently without referring to other images; the P frame is also called the inter-frame prediction coding frame, which represents the difference between the current frame and the previous frame (the previous frame may be an I frame or a P frame), and requires reference to the previous frame for encoding; the B frame is also called the bidirectional prediction coding frame, which represents the difference between the current frame and the previous and next frames. When decoding, not only the previous cached picture needs to be obtained, but also the subsequent picture needs to be decoded, and the final picture is obtained by superimposing the previous and next pictures with the current frame data.

[0073] In one embodiment, audio and video data frames can be collected in real time, and the collected audio and video data frames are input into the encoding module frame by frame to be encoded by the encoding module. While the collected audio and video data frames are input into the encoding module frame by frame, the audio and video data frames are cached frame by frame to avoid the problem that the lost frame data frames cannot be encoded due to data jitter and signal attenuation. The frame-by-frame cache of the audio and video data frames can be a collection cache queue preset in the memory, and while the collected audio and video data frames are input into the encoding module frame by frame, they are also input into the collection cache queue frame by frame. The number of data frames in the collection cache queue is related to the video acquisition system memory and the audio and video frame rate, and generally does not exceed 5 frames. That is, when 5 frames are reached, when a new frame of audio and video data frames is transmitted to the collection cache queue, the data frame with the earliest cache time in the collection cache queue can be deleted, and the other data frames in the collection cache queue are moved forward one position in sequence according to the cache order, so that the latest frame of data frame is cached to the last position of the cache queue.

[0074] S102, if a frame supplementation triggering event is identified, providing a frame supplementation data frame to the encoding module based on the cached audio and video data frame;

[0075] In one embodiment, the frame interval duration of the data frame can be calculated according to the frame rate of the audio and video data frame, and by real-time monitoring whether the audio and video data frame is received by the encoding module within the frame interval duration, it is identified whether there is a frame supplement trigger event. For example: if the frame rate of the audio and video data frame is 25fps, then the frame interval time of the audio and video data frame is 40ms. When it is detected that the time for the encoding module to receive the previous frame reaches the frame interval time, that is, 40ms, and a new frame of data frame is not received, it indicates that data jitter and other problems may have occurred, and data supplementation is required. At this time, the identification result is that there is a frame supplement trigger event. If it is identified that there is a frame supplement trigger event, the latest cached frame of audio and video data frame can be obtained from the cached audio and video data frame, and the latest cached frame of audio and video data frame can be provided to the encoding module as a frame supplement data frame.

[0076] In this embodiment, optionally, providing a frame supplementation data frame to the encoding module based on the cached audio and video data frame includes:

[0077] Copy the latest audio and video data frame in the cache to obtain a frame-filling data frame;

[0078] The frame complementation data frame is provided to the encoding module.

[0079] In one embodiment, since the collected audio and video data are cached frame by frame while being input into the encoding module frame by frame, the latest cached audio and video data frame can be used as a frame-filling data frame. Specifically, the cached latest audio and video data frame can be copied to obtain a frame-filling data frame, and the frame-filling data frame is provided to the encoding module.

[0080] In the embodiment of the present application, by copying the latest cached audio and video data frame to obtain a frame-padded data frame, and providing the frame-padded data frame to the encoding module, the error between the frame-padded data frame and the actual data frame of the frame can be reduced. At the same time, by copying the cached data frame, signal attenuation caused by multiple transmission and reception of the cached audio and video data frame can be avoided, thereby improving the reliability of the cached audio and video data frame.

[0081] S103, reading the output result output by the encoding module to the cache queue, and determining whether the continuous frame supplementation data frames exceed the interruption threshold;

[0082] The output result may be a result obtained by the encoding module encoding the audio and video data frames in sequence according to the input order of the audio and video data frames. The encoding module may store the output result in the cache queue in sequence according to the encoding order.

[0083] Figure 2 is a schematic diagram of the structure of the connection between the encoding module and the cache queue provided in the embodiment of the present application, such as Figure 2 As shown, to avoid the problem of cache lag, the cache queue can be directly connected to the encoding module to receive the output result of the encoding module in real time. To avoid occupying too much memory, a fixed number of data frames can be set in the cache queue, and the data frames cached in the queue can be updated in real time according to the output result of the encoding module. In this solution, the interruption threshold can be the maximum duration that can be achieved by continuous frame supplementation under the premise that the acquisition signal is not interrupted.

[0084] In one embodiment, the encoding module can encode the collected audio and video data frames and the data frames obtained by frame supplementation. Therefore, the output result output by the encoding module to the cache queue may include the result of encoding the collected audio and video data frames, and also include the result of encoding the data frames obtained by frame supplementation. The data frames obtained by frame supplementation have frame supplementation tags, so the audio and video acquisition system can identify the encoding result of the frame supplementation data frame by reading the output result output by the encoding module to the cache queue, calculate the duration of continuous frame supplementation data frames according to the timestamp carried in the frame supplementation data frame, and determine whether the continuous frame supplementation data frames exceed the interruption threshold according to the duration. Since the acquisition signal is interrupted or there is no signal, it is impossible to provide new acquisition data frames to the encoding module. At this time, it is necessary to provide the encoding module with frame supplementation data frames based on the cached audio and video data frames. Therefore, when the duration of continuous frame supplementation data frames exceeds the interruption threshold, it means that the current acquisition signal has been interrupted. For example: assuming that the frame rate of the audio and video data frame is 25fps, the interruption threshold can be set to 1s. By real-time monitoring of the interpolation data frames in the cache queue and the timestamps of the interpolation data frames, when all the data frames from the first frame to the twenty-fifth frame obtained at the monitoring moment are interpolation data frames with continuous timestamps, it means that there are too many interpolation data frames. At this time, it can be considered that the current acquisition signal is abnormal and is in a no-signal state or a signal interruption state.

[0085] In a feasible embodiment, optionally, after reading the output result output by the encoding module to the cache queue, the method further includes:

[0086] If it is identified that there is a preset redundant frame supplement event, a target frame is determined in the cache queue and a deletion operation is performed on the target frame.

[0087] Among them, the preset redundant interpolation frame event may be an event in which the actual number of interpolation frames is higher than the required number of interpolation frames. Since the audio and video data frames may have data jitter and other problems during the data acquisition process, the collected audio and video data frames will not be able to be transmitted to the encoding module at a fixed frame rate. At this time, in order to avoid frame loss, interpolation frame data frames can be provided to the encoding module based on the cached audio and video data frames. When the jittered audio and video data frames are delayed in arriving at the encoding module, an interpolation frame redundant event will appear in the cache queue.

[0088] Figure 3 The method for identifying a preset redundant frame supplement event provided in the embodiment of the present application is as follows: Figure 3 As shown, the solid rectangular digital frame represents the frame number of the audio and video data frame, the dotted frame represents the preset time frame, and the solid rectangular digital frame in the dotted frame represents the data frame contained in the preset time period.

[0089] In one embodiment, the frame rate of the audio and video data frame is set to 25fps, the preset time period is 140ms, and redundant frame supplement events are identified in real time for the cached data frames in the cache queue through the preset time frame. Figure 3 As shown, when there is no redundant frame supplement event, since the frame interval is 40ms, there should be only 4 data frames in the dotted frame; when there are more than 4 data frames in the dotted frame, it indicates that there is a redundant frame supplement event. Figure 3 The situation shown is the situation where a redundant frame supplementation event exists.

[0090] In one embodiment, if it is identified that there is a preset redundant frame supplement event in the cache queue, the frame supplement data frame in the cache queue may be determined as a target frame, and a deletion operation may be performed on the target frame.

[0091] In the embodiment of the present application, after reading the output result output by the encoding module to the cache queue, by identifying whether there is a preset redundant frame supplement event and deleting the redundant frame supplement, the problem of redundant frame supplement can be avoided and the stability of the frame rate of the video file can be improved.

[0092] S104: If exceeded, generate an empty frame data frame and provide the empty frame data frame to the encoding module.

[0093] The empty frame data frame may be a data frame used to indicate that the current acquisition signal is interrupted or there is no acquisition signal. The display content in the empty frame data frame may be a black screen or "no signal" or the like.

[0094] In one embodiment, if the continuous frame supplementation data frames exceed the interruption threshold, it means that the acquisition signal of the audio and video data frame is interrupted. At this time, an empty frame data frame is generated and provided to the encoding module to feedback the signal interruption to the user.

[0095] The technical solution provided by the embodiment of the present application is to input the collected audio and video data frames into the encoding module frame by frame, and cache them frame by frame; if a frame supplement trigger event is identified, the frame supplement data frame is provided to the encoding module based on the cached audio and video data frame; the output result output by the encoding module to the cache queue is read to determine whether the continuous frame supplement data frames exceed the interruption threshold; if exceeded, an empty frame data frame is generated and provided to the encoding module. The problem of untimely video data frame supplement and unstable video frame rate can be solved by the above-mentioned audio and video coding frame supplement processing method. By caching the collected audio and video data frame by frame, and providing the frame supplement data frame to the encoding module based on the cached audio and video data frame when the frame supplement trigger event is identified, the timeliness of video frame supplement can be improved. By reading the output result output by the encoding module to the cache queue, it is determined whether the continuous frame supplement data frames exceed the interruption threshold, and an empty frame data frame is generated when the interruption threshold is exceeded, which can improve the reliability of interrupt judgment and the timeliness of interrupt feedback.

[0096] Figure 4 is a flow chart of the target frame deletion method provided by the embodiment of the present application. Figure 4 As shown, the specific steps include:

[0097] S401, if it is identified that the actual number of output frames in the cache queue within the preset monitoring time exceeds the theoretical number of frames, it is determined that a preset redundant frame supplement event exists;

[0098] The preset monitoring duration may be a time period for monitoring the data frames in the cache queue. It is understandable that the preset monitoring duration should be greater than at least one frame interval duration, and for ease of identification, the preset monitoring duration may be set to an integer multiple of the adjacent frame interval durations. The theoretical number of frames may be the number of data frames output by the encoding module to the cache queue at the frame rate of the audio and video data frames within the preset monitoring duration.

[0099] In one embodiment, the theoretical number of frames can be calculated based on the preset monitoring time and frame rate, and the actual number of output frames in the cache queue can be obtained, and the actual number of output frames can be compared with the theoretical number of frames. If it is identified that the actual number of output frames within the preset monitoring time in the cache queue exceeds the theoretical number of frames, it is determined that a preset redundant frame supplement event exists.

[0100] S402: Determine the frame of the supplementary frame data as a target frame within the preset monitoring time period, and delete the target frame.

[0101] In one embodiment, when a frame-complementing data frame appears within the preset monitoring time period, the data frame may be used as a target frame and the target frame may be deleted.

[0102] In another embodiment, it can be understood that if multiple interpolation data frames appear within the preset monitoring time, and the number of actual output frames also exceeds the number of theoretical frames, then the interval time between the interpolation data frame and the acquisition data frame can be calculated according to the timestamp carried by the data frame to classify the interpolation data frame. When the interval time between the interpolation data frame and the acquisition data frame is less than the frame interval time of the acquisition data frame, and the previous frame data frame is the interpolation data frame, the interpolation data frame is used as the target frame and the target frame is deleted.

[0103] Optionally, determining the frame-filling data frame as the target frame within the preset monitoring time period includes:

[0104] The frame-complementing data frames within a preset time period before the audio and video data frames collected within the preset monitoring time period are determined as target frames.

[0105] In one embodiment, the frame-filling data frame within the preset time length before the audio and video data frame collected within the preset monitoring time length can be determined as the target frame. For example: if the preset frame rate is 25fps, the frame interval length is 40ms. When the encoding module receives the previous frame for 40ms without receiving a new frame of data, the frame-filling data frame is provided to the encoding module based on the cached audio and video data frame. However, the encoding module receives the new frame of data when the time for receiving the previous frame reaches 50ms. In this case, the frame of data filled in the 40ms above can be used as the target frame.

[0106] In the embodiment of the present application, by determining the supplementary frame data frame within the previous preset time length of the audio and video data frame collected within the preset monitoring time length as the target frame, the accuracy of determining the target frame can be improved.

[0107] The technical solution provided in the embodiment of the present application can avoid the problem of redundant frame supplementation by identifying that the actual number of output frames within a preset monitoring time in the cache queue exceeds the theoretical number of frames, determining that there is a preset redundant frame supplementation event, determining the frame supplementation data frame as the target frame within the preset monitoring time, and deleting the target frame, thereby improving the reliability of the judgment of interruption of video information acquisition.

[0108] Figure 5 FIG. 1 is a flow chart of another target frame deletion method provided by an embodiment of the present application. Figure 5 As shown, the specific steps include:

[0109] S501, if it is identified that the continuous interpolation frame data frames exceed the jamming threshold, a re-interpolation instruction is issued to provide the interpolation frame data frames to the encoding module based on the currently cached audio and video data frames;

[0110] The freeze threshold may be the maximum number of frames that can be continuously filled with the same frame filling data frame without causing the user to perceive freezes in the video when watching the video.

[0111] In one embodiment, when the same interpolation frame data frame is interpolated too many times in succession, the interpolation frame stays in the final generated video for too long, causing the user to perceive the video image as stuck. Therefore, in order to improve the smoothness of the video image, the interpolation frame data frames that cause the stuck can be partially deleted. If it is identified that the continuous interpolation frame data frames exceed the stuck threshold, a re-interpolation instruction is issued to provide the latest audio and video data frame currently cached as the interpolation frame data frame to the encoding module.

[0112] S502: Determine a target frame in the continuous frame-complementing data frames, and perform a deletion operation on the target frame.

[0113] In one embodiment, the continuous interpolation data frame exceeding the jam threshold can be determined as a target frame, and the target frame can be deleted. For example: assuming that the jam threshold is 3, the continuous interpolation data frame is the same data frame as the first frame, and the sum of the number of frames of the first frame and the continuous interpolation data frame is greater than 3, exceeding the jam threshold, then a frame interpolation instruction can be issued to interpolate the latest frame of audio and video data frame received in the cache queue. At this time, a redundant interpolation data frame can be obtained, and one of the continuous interpolation data frames that is repeated with the first frame is used as the target frame, and the target frame is deleted.

[0114] The technical solution provided in the embodiment of the present application, by issuing a re-frame instruction when identifying that continuous frame-filling data frames exceed a freeze threshold, providing frame-filling data frames to the encoding module based on the current cached audio and video data frames, determining the target frame in the continuous frame-filling data frames, and deleting the target frame, can avoid the user's experience of freeze when watching the video and improve the smoothness of the video picture.

[0115] Figure 6 FIG. 1 is a flow chart of a frame supplementation processing method for audio and video coding provided by an embodiment of the present application. Figure 6 As shown, the specific steps include:

[0116] S601, input the collected audio and video data frames into the encoding module frame by frame, and cache them frame by frame;

[0117] S602: If a frame supplementation triggering event is identified, providing a frame supplementation data frame to the encoding module based on the cached audio and video data frame;

[0118] S603, marking the frame interpolation data frame with a frame interpolation label;

[0119] S604, identifying whether the output result output by the encoding module to the cache queue carries a frame supplement tag;

[0120] The frame interpolation label may be a label used to identify the frame interpolation data frame. The frame interpolation label may be used to distinguish whether the current data frame is a frame interpolation data frame.

[0121] In one embodiment, after providing the interpolation frame data frame to the encoding module, the interpolation frame tag can be marked on the interpolation frame data frame through code, and by identifying whether the output result output by the encoding module to the cache queue has the tag ID of the interpolation frame tag, it is identified whether the output result carries the interpolation frame tag.

[0122] S605, determining whether the continuous frame-filling data frames exceed the interruption threshold according to the recognition result;

[0123] In one embodiment, the number of consecutive interpolation frame data frames in the cache queue is determined based on an identification result of whether the output result in the cache queue carries an interpolation frame tag, and further, it is determined whether the number of consecutive interpolation frame data frames exceeds an interruption threshold.

[0124] S606: If exceeded, generate an empty frame data frame and provide the empty frame data frame to the encoding module.

[0125] The technical solution provided in the embodiment of the present application can improve the accuracy of identifying the interpolation data frames, thereby improving the reliability of judging the interruption of the acquisition signal, by marking the interpolation data frames with interpolation tags, identifying whether the output results output by the encoding module to the cache queue carry the interpolation tags, and determining whether the continuous interpolation data frames exceed the interruption threshold according to the identification results.

[0126] Figure 7 Schematic diagram of the structure of the frame supplementation processing device for audio and video coding provided by the embodiment of the present application. Figure 7 As shown, specifically including the following:

[0127] The cache unit 701 is used to input the collected audio and video data frames into the encoding module frame by frame, and cache them frame by frame;

[0128] A frame interpolation unit 702 is configured to provide an interpolation frame data frame to the encoding module based on the cached audio and video data frame when a frame interpolation triggering event is identified;

[0129] An interruption determination unit 703 is used to read the output result output by the encoding module to the cache queue, and determine whether the continuous frame supplementation data frames exceed the interruption threshold;

[0130] The empty frame generating unit 704 is used to generate an empty frame data frame when the continuous frame supplementation data frames exceed the interruption threshold, and provide the empty frame data frame to the encoding module.

[0131] Furthermore, the device further comprises a redundant frame interpolation processing unit; the redundant frame interpolation processing unit is used to:

[0132] If it is identified that there is a preset redundant frame supplement event, a target frame is determined in the cache queue and a deletion operation is performed on the target frame.

[0133] Furthermore, the redundant frame supplementation processing unit is specifically used for:

[0134] If it is identified that the actual number of output frames in the cache queue within the preset monitoring time exceeds the theoretical number of frames, it is determined that a preset redundant frame supplement event occurs;

[0135] The frame-filling data frame is determined as a target frame within the preset monitoring time, and a deletion operation is performed on the target frame.

[0136] Furthermore, the redundant frame supplementation processing unit is specifically used for:

[0137] The frame-complementing data frames within a preset time period before the audio and video data frames collected within the preset monitoring time period are determined as target frames.

[0138] Furthermore, the device further includes a jamming processing unit; the jamming processing unit is used to:

[0139] If it is identified that the continuous interpolation frame data frames exceed the jamming threshold, a re-interpolation instruction is issued to provide the interpolation frame data frames to the encoding module based on the currently cached audio and video data frames;

[0140] A target frame is determined in the continuous frame-complementing data frames, and a deletion operation is performed on the target frame.

[0141] Furthermore, the frame interpolation unit 702 is specifically configured to:

[0142] Copy the latest audio and video data frame in the cache to obtain a frame-filling data frame;

[0143] The frame complementation data frame is provided to the encoding module.

[0144] Furthermore, the frame interpolation unit 702 is further configured to:

[0145] Marking the interpolated frame data frame with an interpolated frame label;

[0146] Accordingly, the interruption determination unit 703 is specifically configured to:

[0147] Identify whether the output result output by the encoding module to the cache queue carries a frame supplement tag;

[0148] It is determined whether the continuous frame-filling data frames exceed the interruption threshold according to the recognition result.

[0149] The technical solution provided in the embodiment of the present application includes a cache unit, which is used to input the collected audio and video data frames into the encoding module frame by frame, and cache them frame by frame; a frame supplement unit, which is used to provide the encoding module with frame supplement data frames based on the cached audio and video data frames when identifying the existence of a frame supplement trigger event; an interrupt determination unit, which is used to read the output result of the encoding module output to the cache queue, and determine whether the continuous frame supplement data frames exceed the interrupt threshold; and an empty frame generation unit, which is used to generate empty frame data frames when the continuous frame supplement data frames exceed the interrupt threshold, and provide the empty frame data frames to the encoding module. The above-mentioned frame interpolation processing device for audio and video encoding can solve the problems of untimely frame interpolation of video data frames and unstable video frame rate. By caching the collected audio and video data frame by frame, and providing frame interpolation data frames to the encoding module based on the cached audio and video data frames when a frame interpolation triggering event is identified, the timeliness of video frame interpolation can be improved. By reading the output results of the encoding module output to the cache queue, it is determined whether the continuous frame interpolation data frames exceed the interruption threshold, and generating empty frame data frames when the interruption threshold is exceeded. This can improve the reliability of interruption judgment and the timeliness of interruption feedback.

[0150] The frame supplementation processing device for audio and video coding in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0151] The frame interpolation processing device for audio and video coding in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0152] The frame supplement processing device for audio and video coding provided in the embodiment of the present application can realize Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0153] Figure 8 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 8 As shown, the embodiment of the present application also provides an electronic device 800, including a processor 801, a memory 802, and a program or instruction stored in the memory 802 and executable on the processor 801. When the program or instruction is executed by the processor 801, each process of the embodiment of the above-mentioned audio and video encoding frame supplementation processing method is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0154] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0155] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned audio and video encoding frame supplementation processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0156] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0157] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned audio and video encoding frame supplementation processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0158] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0159] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0160] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0161] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0162] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A frame interpolation processing method for audio and video coding, characterized in that: The method comprises: The collected audio and video data frames are input into the encoding module frame by frame, and cached frame by frame; If a frame supplementation trigger event is identified, providing a frame supplementation data frame to the encoding module based on the cached audio and video data frame; Reading the output result output by the encoding module to the cache queue to determine whether the continuous frame supplementation data frames exceed the interruption threshold; If it exceeds, an empty frame data frame is generated and provided to the encoding module.

2. The method according to claim 1, characterized in that: After reading the output result output by the encoding module to the cache queue, the method further includes: If it is identified that there is a preset redundant frame supplement event, a target frame is determined in the cache queue and a deletion operation is performed on the target frame.

3. The method according to claim 2, characterized in that If a preset redundant frame supplement event is identified, a target frame is determined in the cache queue, and a deletion operation is performed on the target frame, including: If it is identified that the actual number of output frames in the cache queue within the preset monitoring time exceeds the theoretical number of frames, it is determined that a preset redundant frame supplement event occurs; The frame-filling data frame is determined as a target frame within the preset monitoring time, and a deletion operation is performed on the target frame.

4. The method according to claim 3, characterized in that Determining the frame-filling data frame as a target frame within the preset monitoring time period includes: The frame-complementing data frames within a preset time period before the audio and video data frames collected within the preset monitoring time period are determined as target frames.

5. The method according to claim 1, characterized in that After reading the output result output by the encoding module to the cache queue, the method further includes: If it is identified that the continuous interpolation frame data frames exceed the jamming threshold, a re-interpolation instruction is issued to provide the interpolation frame data frames to the encoding module based on the currently cached audio and video data frames; A target frame is determined in the continuous frame-complementing data frames, and a deletion operation is performed on the target frame.

6. The method according to claim 1, characterized in that Providing a frame supplementation data frame to the encoding module based on the cached audio and video data frame, including: Copy the latest audio and video data frame in the cache to obtain a frame-filling data frame; The frame complementation data frame is provided to the encoding module.

7. The method according to claim 1, characterized in that After providing the frame supplementation data frame to the encoding module based on the cached audio and video data frame, the method further includes: Marking the interpolation frame data frame with an interpolation frame label; Accordingly, reading the output result output by the encoding module to the cache queue to determine whether the continuous frame supplementation data frames exceed the interruption threshold includes: Identify whether the output result output by the encoding module to the cache queue carries a frame supplement tag; It is determined whether the continuous frame-filling data frames exceed the interruption threshold according to the recognition result.

8. A frame interpolation processing device for audio and video coding, characterized in that: The device comprises: The buffer unit is used to input the collected audio and video data frames into the encoding module frame by frame, and buffer them frame by frame; A frame interpolation unit, configured to provide an interpolation frame data frame to the encoding module based on the cached audio and video data frame when a frame interpolation triggering event is identified; An interruption determination unit, configured to read the output result output by the encoding module to the cache queue, and determine whether the continuous frame supplementation data frames exceed an interruption threshold; The empty frame generating unit is used to generate an empty frame data frame when the continuous frame supplementation data frames exceed the interruption threshold, and provide the empty frame data frame to the encoding module.

9. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the frame supplementation processing method for audio and video coding as described in any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the frame supplementation processing method for audio and video encoding as described in any one of claims 1-7 are implemented.

Images