Video frame filtering method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a video frame filtering method and device, electronic equipment and storage medium, belonging to the technical field of computer. The method comprises: for any video frame in a video frame queue, determining a plurality of reference video frames of the video frame from the video frame queue; in the case that the number of idle memory in the idle memory pool is not less than the first number, covering the data in the first memory in the idle memory pool with the video frame and at least one reference video frame, the idle memory in the idle memory pool being used to store the encoded video frames in the video frame queue in a pyramid structure; filtering the video frame based on the video frame in the first memory and the video frame in the second memory, the second memory being used to store at least one reference video frame in a pyramid structure. The above scheme can reduce the additional memory overhead by multiplexing the idle memory in the idle memory pool, avoiding frequent memory allocation, and thus improving the encoding efficiency of the video.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a video frame filtering method, apparatus, electronic device, and storage medium. Background Technology

[0002] Video consists of consecutive video frames, and these frames share similarities. Therefore, to facilitate video storage and transmission, video needs to be encoded to reduce its storage space. Improving video encoding efficiency is a technical problem that needs to be solved.

[0003] In video coding techniques, Motion Compensated Temporal Filtering (MCTF) utilizes the temporal correlation of video to perform temporal filtering on video frames, thereby reducing temporal redundancy information generated during video frame referencing and improving video coding efficiency.

[0004] However, this scheme requires allocating a fixed amount of memory to read a fixed number of video frames and create a pyramid structure corresponding to that number of video frames when performing temporal filtering on any given video frame, thus incurring additional memory overhead. Excessive memory overhead can not only affect encoding stability but may also cause online crashes due to insufficient memory, reducing video encoding efficiency. Summary of the Invention

[0005] This disclosure provides a video frame filtering method, apparatus, electronic device, and storage medium. By reusing free memory in a free memory pool, it can reduce additional memory overhead, avoid frequent memory allocation, and thus improve video encoding efficiency. The technical solution of this disclosure is as follows:

[0006] According to one aspect of the embodiments of this disclosure, a video frame filtering method is provided, comprising:

[0007] For any video frame in the video frame queue, a plurality of reference video frames are determined from the video frame queue, the plurality of reference video frames being used to provide reference information during the filtering process of the video frame;

[0008] When the number of free memory in the free memory pool is not less than a first number, the data in the first memory in the free memory pool is overwritten with the video frame and at least one reference video frame. The free memory in the free memory pool is used to store the encoded video frames in the video frame queue in a pyramid structure. The first number is the number of video frames plus the number of reference video frames to be constructed into the pyramid structure. The number of the first memory is the first number.

[0009] Based on the video frames in the first memory and the video frames in the second memory, the video frames are filtered, and the second memory is used to store at least one reference video frame in a pyramid structure.

[0010] According to another aspect of the embodiments of this disclosure, a video frame filtering apparatus is provided, comprising:

[0011] The determining unit is configured to, for any video frame in the video frame queue, determine a plurality of reference video frames from the video frame queue, the plurality of reference video frames being used to provide reference information during the filtering process of the video frame;

[0012] The overlay unit is configured to overlay the data in the first memory of the free memory pool with the video frame and at least one reference video frame when the number of free memory in the free memory pool is not less than a first number. The free memory in the free memory pool is used to store the encoded video frames in the video frame queue in a pyramid structure. The first number is the number of video frames plus the number of reference video frames to be constructed into the pyramid structure. The number of the first memory is the first number.

[0013] The filtering unit is configured to filter the video frames based on video frames in the first memory and video frames in the second memory, the second memory being used to store at least one reference video frame in a pyramid structure.

[0014] In some embodiments, the determining unit includes:

[0015] The storage subunit is configured to place a copy of the video frame into the free memory pool for any video frame in the video frame queue.

[0016] A determination subunit is configured to determine the frame identifiers of the plurality of reference video frames from the video frame queue based on the frame identifiers of the video frames;

[0017] The first acquisition subunit is configured to, for any reference video frame, acquire a copy of the reference video frame from the free memory pool based on the frame identifier of the reference video frame, when the reference video frame is being filtered or has been filtered.

[0018] The second acquisition subunit is configured to acquire the reference video frame from the video frame queue when the reference video frame is not filtered.

[0019] In some embodiments, the apparatus further includes:

[0020] The acquisition unit is configured to acquire at least one second memory from the free memory pool based on the frame identifiers of the plurality of reference video frames;

[0021] The quantity determination unit is configured to determine the first quantity based on the quantity of the at least one second memory and the quantity of the plurality of reference video frames.

[0022] In some embodiments, the overlay unit is further configured to, when the number of free memory in the free memory pool is less than the first number, overlay the data in the free memory in the free memory pool with the video frame and at least one reference video frame, wherein the number of free memory is a second number; create a third number of third memory, wherein the third number is the difference between the first number and the second number, the third memory being used to store at least one reference video frame in a pyramid structure, and filter the video frame based on the video frame in the first memory, the video frame in the second memory, and the video frame in the third memory.

[0023] In some embodiments, the apparatus further includes:

[0024] An encoding unit is configured to encode the video frame in response to the completion of video frame filtering;

[0025] A first storage unit is configured to place the video frame into an encoding completion queue in response to the completion of video frame encoding, the encoding completion queue being used to store video frames that have been encoded in the video frame queue.

[0026] The second storage unit is configured to, in response to the encoding completion queue not including all video frames in the video frame queue and the video frame not needing to be referenced by unencoded video frames in the video frame queue, place the memory storing the video frame as free memory into the free memory pool.

[0027] In some embodiments, the apparatus further includes:

[0028] The memory release unit is configured to release free memory in the free memory pool in response to the encoding completion queue including all video frames in the video frame queue.

[0029] In some embodiments, the apparatus further includes:

[0030] The pre-analysis unit is configured to pre-analyze the video frame to obtain the frame type and quantization parameters of the video frame, wherein the quantization parameters are used to reflect the spatial details of the video frame.

[0031] The judgment unit is configured to determine whether the video frame needs filtering based on the frame type and the quantization parameters.

[0032] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0033] One or more processors;

[0034] Memory used to store the executable program code of the processor;

[0035] The processor is configured to execute the program code to implement the aforementioned video frame filtering method.

[0036] According to another aspect of the present disclosure, a computer-readable storage medium is provided that, when the program code in the computer-readable storage medium is executed by a processor of an electronic device, enables the electronic device to perform the video frame filtering method described above.

[0037] According to another aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described video frame filtering method.

[0038] This disclosure provides a video frame filtering method. Since video frames need to reference other adjacent video frames during filtering, multiple reference video frames can be determined from the video frame queue before filtering. When the amount of free memory in the free memory pool is not less than the total number of video frames and reference video frames to be used to construct the pyramid structure, the data in the first memory space of the free memory pool can be overwritten with the video frames and at least one reference video frame; that is, the free memory in the free memory pool is reused. Then, the video frames are filtered based on the video frames in the first memory and the reference video frames stored in the second memory. By reusing the free memory in the free memory pool before filtering the video frames, additional memory overhead can be reduced, frequent memory allocation can be avoided, and thus the video encoding efficiency can be improved.

[0039] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0041] Figure 1 This is a schematic diagram illustrating the implementation environment of a video frame filtering method according to an exemplary embodiment.

[0042] Figure 2 This is a flowchart illustrating a video frame filtering method according to an exemplary embodiment.

[0043] Figure 3 This is a flowchart illustrating another video frame filtering method according to an exemplary embodiment.

[0044] Figure 4 This is a schematic diagram illustrating a free memory according to an exemplary embodiment.

[0045] Figure 5 This is a schematic diagram illustrating a free memory pool according to an exemplary embodiment.

[0046] Figure 6 This is a flowchart illustrating a video frame filtering method according to an exemplary embodiment.

[0047] Figure 7 This is a block diagram illustrating a video frame filtering device according to an exemplary embodiment.

[0048] Figure 8 This is a block diagram illustrating another video frame filtering device according to an exemplary embodiment.

[0049] Figure 9 This is a block diagram illustrating a terminal according to an exemplary embodiment.

[0050] Figure 10 This is a block diagram illustrating a server according to an exemplary embodiment. Detailed Implementation

[0051] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0052] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0053] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the video frames and reference video frames involved in this disclosure were obtained with full authorization.

[0054] Figure 1 This is a schematic diagram illustrating an implementation environment for a video frame filtering method according to an exemplary embodiment. See also... Figure 1 The implementation environment specifically includes: terminal 101 and server 102.

[0055] Terminal 101 can be at least one of the following devices: smartphone, smartwatch, desktop computer, laptop, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), and laptop computer. An application can be installed and run on terminal 101, and users can log in to the application through terminal 101 to access the services provided by the application. Terminal 101 can be connected to server 102 via a wireless network or a wired network.

[0056] Terminal 101 can refer to one of a plurality of terminals, and this embodiment uses terminal 101 as an example. Those skilled in the art will know that the number of terminals can be more or less. For example, there can be several terminals, or dozens or hundreds of terminals, or more. This embodiment does not limit the number of terminals or the type of devices.

[0057] Server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some embodiments, server 102 undertakes the main computing work, and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, and terminal 101 undertakes the main computing work; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture. Server 102 can connect to terminal 101 and other terminals via a wireless network or wired network. Optionally, the number of servers can be more or less, and this disclosure does not limit this. Of course, server 102 may also include other functional servers to provide more comprehensive and diversified services.

[0058] Figure 2 This is a flowchart illustrating a video frame filtering method according to an exemplary embodiment, such as... Figure 2 As shown, the method is performed by an electronic device and includes the following steps:

[0059] In step S201, for any video frame in the video frame queue, the electronic device determines multiple reference video frames from the video frame queue. The multiple reference video frames are used to provide reference information during the video frame filtering process.

[0060] In this embodiment, the video frame queue can be a segment of the video to be encoded or a complete video. The video frame queue includes multiple consecutive video frames. For any video frame in the video frame queue, the electronic device can determine multiple video frames from the video frame queue and use the determined multiple video frames as multiple reference video frames for that video frame. These multiple reference video frames are used to provide reference information during the video frame filtering process. Optionally, the multiple reference video frames are adjacent video frames of that video frame, such as two video frames before and after the video frame; some reference video frames in the multiple reference video frames of different video frames may be the same or different. The number of multiple reference video frames can be 4, 6, 8, etc., and this embodiment does not limit this.

[0061] For example, for the 8th video frame in the video frame queue, if the number of multiple reference video frames is 8, then the multiple reference video frames for this video frame are the 4th, 5th, 6th, and 7th video frames and the 9th, 10th, 11th, and 12th video frames in the video frame queue.

[0062] In step S202, if the number of free memory in the free memory pool is not less than a first number, the electronic device overwrites the data in the first memory in the free memory pool with video frames and at least one reference video frame. The free memory in the free memory pool is used to store the encoded video frames in the video frame queue in a pyramid structure. The first number is the number of video frames plus the number of reference video frames to be constructed into the pyramid structure. The number of the first memory is the first number.

[0063] In this embodiment of the disclosure, before filtering the video frame, the server needs to construct a pyramid structure of the video frame and multiple reference video frames. This pyramid structure includes a quarter of the downsampled video frame, a half of the downsampled video frame, and the original-size video frame. Since a reference video frame for a video frame may be a reference video frame for other video frames in the video frame queue, the pyramid structure of the reference video frames for that video frame can be used by other video frames after the video frame is filtered. When filtering other video frames in the video frame queue, the electronic device may have already created a pyramid structure of partial reference video frames for the current video frame. Therefore, the electronic device only needs to create a pyramid structure for the video frame and at least one reference video frame for which the pyramid structure is to be constructed. The number of video frames and at least one reference video frame for which the pyramid structure is to be constructed is a first quantity.

[0064] The free memory pool comprises multiple free memory locations, which are used to store encoded video frames from the video frame queue in a pyramid structure. The electronic device can obtain a pyramid structure of video frames and at least one reference video frame by reusing the multiple free memory locations in the free memory pool. That is, when the number of free memory locations in the free memory pool is not less than a first quantity, the electronic device can obtain a pyramid structure of video frames and at least one reference video frame by overwriting the data in the first memory location of the free memory pool with video frames and at least one reference video frame. Here, the number of first memory locations is the first quantity, and at this time, the first quantity of first memory locations stores the pyramid structure of video frames and at least one reference video frame.

[0065] In step S203, the electronic device filters video frames based on video frames in the first memory and video frames in the second memory. The second memory is used to store at least one reference video frame in a pyramid structure.

[0066] In this embodiment, the first memory stores a pyramid structure of the current video frame and at least one reference video frame, and the second memory stores pyramid structures of the other reference video frames (excluding the at least one reference video frame). Based on the video frames in the first and second memory, the electronic device can obtain information about multiple reference video frames according to the pyramid structure of the current video frame and the pyramid structure of the multiple reference video frames, and filter the video frame based on the information of the multiple reference video frames. The pyramid structure of the other reference video frames is created during the filtering process of other video frames in the video frame queue; when filtering the current video frame, the electronic device can directly obtain the previously created pyramid structure of other reference video frames.

[0067] This disclosure provides a video frame filtering method. Since video frames need to reference other adjacent video frames during filtering, multiple reference video frames can be determined from the video frame queue before filtering. When the amount of free memory in the free memory pool is not less than the total number of video frames and reference video frames to be used to construct the pyramid structure, the data in the first memory space of the free memory pool can be overwritten with the video frames and at least one reference video frame; that is, the free memory in the free memory pool is reused. Then, the video frames are filtered based on the video frames in the first memory and the reference video frames stored in the second memory. By reusing the free memory in the free memory pool before filtering the video frames, additional memory overhead can be reduced, frequent memory allocation can be avoided, and thus the video encoding efficiency can be improved.

[0068] In some embodiments, for any video frame in the video frame queue, determining a plurality of reference video frames from the video frame queue includes:

[0069] For any video frame in the video frame queue, place a copy of the video frame into the free memory pool;

[0070] Based on the frame identifier of the video frame, determine the frame identifier of multiple reference video frames from the video frame queue;

[0071] For any reference video frame, if the reference video frame is being filtered or has already been filtered, a copy of the reference video frame is obtained from the free memory pool based on the frame identifier of the reference video frame.

[0072] If the reference video frame is not filtered, the reference video frame is obtained from the video frame queue.

[0073] In this embodiment, when filtering a video frame, the electronic device needs the original information of multiple reference video frames. During the acquisition of reference video frames, if a reference video frame is being filtered or has already been filtered, it indicates that the original information of the reference video frame has changed. Therefore, to ensure that the reference information provided by the reference video frames remains unchanged, the electronic device can obtain a copy of the reference video frame corresponding to the frame identifier from the free memory pool based on the frame identifier of the reference video frame. This copy contains the reference video frame before filtering, retaining its original information. If the reference video frame is not filtered, it indicates that the original information of the reference video frame has not changed, and the electronic device can directly obtain the reference video frame from the video frame queue. This avoids inconsistencies that may occur during parallel processing, improving the stability and efficiency of the encoding process.

[0074] In some embodiments, the method further includes:

[0075] Based on the frame identifiers of multiple reference video frames, at least one second memory is obtained from the free memory pool;

[0076] The first quantity is determined based on the number of at least one second memory and the number of multiple reference video frames.

[0077] In this embodiment of the disclosure, some reference video frames among multiple reference video frames of different video frames in the video frame queue may be the same or different. When filtering other video frames in the video frame queue, the electronic device may have already created a pyramid structure corresponding to some reference video frames of the current video frame. Therefore, after determining multiple reference video frames of a video frame from the video frame queue, the electronic device can obtain at least one second memory from a free memory pool based on the frame identifiers of the multiple reference video frames. This second memory stores the pyramid structure of the reference video frames corresponding to the frame identifiers. The number of at least one second memory is the number of reference video frames for which a pyramid structure has already been built. Based on the number of at least one second memory and the number of multiple reference video frames of the video frame, the electronic device can determine the number of reference video frames to be used to build a pyramid structure, and thus determine a first quantity.

[0078] In some embodiments, the method further includes:

[0079] If the amount of free memory in the free memory pool is less than the first amount, the data in the free memory in the free memory pool is overwritten with video frames and at least one reference video frame, and the amount of free memory is the second amount.

[0080] Create a third amount of third memory, the third amount being the difference between the first and second amounts, and use the third memory to store at least one reference video frame in a pyramid structure;

[0081] The video frames are filtered based on the video frames in the first memory, the second memory, and the third memory.

[0082] In this embodiment of the disclosure, the electronic device can obtain a pyramid structure of video frames and some reference video frames by reusing multiple free memories in the free memory pool. When the amount of free memory is insufficient, the number of remaining reference video frame pyramid structures to be created can be determined according to the amount of free memory, which can reduce memory overhead and improve the encoding efficiency of video frames to a certain extent.

[0083] In some embodiments, the method further includes:

[0084] In response to the completion of video frame filtering, the video frame is encoded;

[0085] In response to the completion of video frame encoding, the video frame is placed into the encoding completion queue, which is used to store the video frames that have been encoded in the video frame queue.

[0086] In response to the encoding completion queue not including all video frames in the video frame queue and the video frames not needing to be referenced by unencoded video frames in the video frame queue, the memory storing the video frames is placed into the free memory pool as free memory.

[0087] In this embodiment, when an encoded video frame needs to be referenced by an unencoded video frame in the video frame queue, the memory of the video frame is kept unchanged, and the system waits for the next video frame in the video frame queue to be encoded before determining again whether the video frame needs to be referenced by an unencoded video frame in the video frame queue. When an encoded video frame does not need to be referenced by an unencoded video frame in the video frame queue, the electronic device can place the memory of the video frame as free memory into a free memory pool for reuse by subsequent video frames that need filtering and multiple reference video frames. By making reasonable use of the free memory in the free memory pool, memory overhead can be reduced and the encoding efficiency of video frames can be improved.

[0088] In some embodiments, the method further includes:

[0089] In response to the encoding completion queue including all video frames in the video frame queue, free memory in the free memory pool is released.

[0090] In this embodiment of the disclosure, when the encoding completion queue includes all video frames in the video frame queue, it indicates that all video frames in the video frame queue have been encoded, and therefore the electronic device can release free memory in the free memory pool. By releasing the memory space of the free memory pool, the memory overhead of the electronic device can be reduced.

[0091] In some embodiments, the method further includes:

[0092] Pre-analysis of video frames yields the frame type and quantization parameters, which are used to reflect the spatial details of the video frames.

[0093] Based on frame type and quantization parameters, determine whether a video frame needs filtering.

[0094] In this embodiment of the disclosure, the electronic device can pre-analyze the video frames in the video frame queue to determine the video frames that need to be filtered in the video frame queue according to the frame type and quantization parameters of the video frames. This eliminates the need for the electronic device to filter every video in the video frame queue, thereby improving the encoding efficiency of the video frames.

[0095] The above Figure 2 The diagram illustrates the basic process of this disclosure. The following section will further elaborate on the solution provided in this disclosure based on one implementation method. Figure 3 This is a flowchart illustrating another video frame filtering method according to an exemplary embodiment. The method is performed by an electronic device; see [link to relevant documentation]. Figure 3 The method includes:

[0096] In step S301, the electronic device performs pre-analysis on the video frame to obtain the frame type and quantization parameters of the video frame. The quantization parameters are used to reflect the spatial details of the video frame.

[0097] In this embodiment, the video frame types include I (intra-prediction), B (bidirectional prediction), and GPB (inter-prediction) frames. An I-frame is an independent frame containing all its information; it can be simply understood as a static image, representing a video frame in the video frame queue that does not depend on other video frames, such as the first video frame. A GPB frame is a video frame obtained by compressing the current frame data based on the differences between the current frame and the adjacent previous frame, representing the difference between the current video frame and the previous video frame. A B-frame is a video frame obtained by compressing the current frame data based on the differences between the current frame and the adjacent previous and next frames, representing the difference between the current video frame and the preceding and following video frames. The quantization parameter (QP) is used to reflect the spatial detail of the video frame. When the quantization parameter of a video frame is small, most of the details of the video frame are preserved; when the quantization parameter of a video frame is large, some details of the video frame are lost, and the quality of the video frame degrades. Based on the frame type and quantization parameter of the video frame, the electronic device can determine whether the video frame needs filtering.

[0098] In step S302, the electronic device determines whether a video frame needs filtering based on the frame type and quantization parameters.

[0099] In this embodiment, regarding frame type, when the video frame type is I-frame or GPB frame, filtering is required. When the video frame type is B-frame, filtering is generally not required; however, B-frames referenced by a large number of video frames in the video frame queue also require filtering. Secondly, regarding quantization parameters, after roughly determining the video frames in the video frame queue that need filtering based on the frame type, the quantization parameters of the video frames can be used to further determine whether filtering is necessary. For example, if the quantization parameter of a video frame is greater than 17, it is determined that the video frame needs filtering; otherwise, filtering is not required. By pre-analyzing the video frames in the video frame queue, the electronic device can determine the video frames in the queue that need filtering based on the frame type and quantization parameters, thus eliminating the need for filtering every video in the queue and improving the encoding efficiency of the video frames.

[0100] In step S303, for any video frame in the video frame queue, the electronic device determines multiple reference video frames from the video frame queue. These multiple reference video frames are used to provide reference information during the video frame filtering process.

[0101] In this embodiment, the video frame queue includes multiple consecutive video frames. For any video frame in the video frame queue, if the video frame needs filtering, the electronic device can determine multiple video frames from the video frame queue as multiple reference video frames for that video frame. These multiple reference video frames provide reference information during the video frame filtering process. Optionally, the multiple reference video frames are adjacent video frames of the video frame, such as two video frames before and after the video frame; some reference video frames in the multiple reference video frames of different video frames may be the same or different. The number of multiple reference video frames can be 4, 6, 8, etc., and this embodiment does not limit this.

[0102] In some embodiments, before filtering any video frame in the video frame queue, the electronic device can place a copy of the video frame into an idle memory pool, and retrieve a copy of the reference video frame from the idle memory pool or directly from the video frame queue based on the filtering status of the reference video frame. Accordingly, for any video frame in the video frame queue, the electronic device places a copy of the video frame into the idle memory pool; the electronic device determines the frame identifiers of multiple reference video frames from the video frame queue based on the frame identifier of the video frame; for any reference video frame, if the reference video frame is being filtered or has already been filtered, the electronic device retrieves a copy of the reference video frame from the idle memory pool based on the frame identifier of the reference video frame; if the reference video frame is not filtered, the electronic device retrieves the reference video frame from the video frame queue. The idle memory pool stores a copy of the video frame, which is the video frame before filtering and retains the original information of the video frame. Multiple reference video frames are adjacent video frames of the video frame in the video frame queue; the electronic device can determine the frame identifiers of multiple adjacent reference video frames from the video frame queue based on the frame identifier of the video frame. In this embodiment, when filtering video frames, the electronic device needs to refer to the original information of multiple reference video frames. If a reference video frame is being filtered or has already been filtered, it indicates that the original information of the reference video frame has changed. Therefore, to ensure that the reference information provided by the reference video frames remains unchanged, the electronic device can obtain a copy of the reference video frame corresponding to the frame identifier from the free memory pool based on the frame identifier of the reference video frame. If the reference video frame is not filtered, it indicates that the original information of the reference video frame has not changed, and the electronic device can directly obtain the reference video frame from the video frame queue, avoiding inconsistencies that may occur during parallel processing and improving the stability and efficiency of the encoding process.

[0103] In step S304, the electronic device obtains at least one second memory from the free memory pool based on the frame identifiers of multiple reference video frames. The second memory is used to store at least one reference video frame in a pyramid structure.

[0104] In this embodiment of the disclosure, a reference video frame for a video frame may be a reference video frame for other video frames in the video frame queue. Therefore, after the filtering of a video frame is completed, the pyramid structure of the reference video frames can be used by other video frames. When filtering other video frames in the video frame queue, the electronic device may have already created a pyramid structure corresponding to some of the reference video frames of the current video frame. Therefore, after determining multiple reference video frames for a video frame from the video frame queue, the electronic device can obtain at least one second memory from a free memory pool based on the frame identifiers of the multiple reference video frames. This second memory stores the pyramid structure of the reference video frames corresponding to the frame identifiers.

[0105] In step S305, the electronic device determines a first quantity based on the number of at least one second memory and the number of multiple reference video frames. The first quantity is the number of video frames plus the number of reference video frames to be constructed into the pyramid structure.

[0106] In this embodiment of the disclosure, the number of at least one second memory is the number of reference video frames for which a pyramid structure has already been constructed. Based on the number of at least one second memory and the number of multiple reference video frames of the video frame, the electronic device can determine the number of reference video frames for which a pyramid structure is to be constructed, and thus determine the first number.

[0107] In some embodiments, when the current video frame is a reference video frame for other video frames, it indicates that the pyramid structure of the current video frame already exists and does not need to be recreated; that is, it does not need to overwrite the data in the free memory in the free memory pool. The electronic device can overwrite the data in the free memory in the free memory pool with at least one reference video frame of the current video frame. The electronic device filters the current video frame based on the current video frame in memory and the reference video frames in free memory.

[0108] It should be noted that the number of free memory in the free memory pool is limited. When the number of free memory is not less than the first number, the electronic device executes the following steps S306-S307; when the number of free memory is less than the first number, the electronic device executes the following steps S308-S310.

[0109] In step S306, if the number of free memory in the free memory pool is not less than a first number, the electronic device overwrites the data in the first memory in the free memory pool with video frames and at least one reference video frame. The free memory is used to store the encoded video frames in the video frame queue in a pyramid structure, and the number of the first memory is the first number.

[0110] In this embodiment, before filtering the video frames, the server needs to construct a pyramid structure of the video frames and multiple reference video frames. This pyramid structure includes a quarter-sized downsampled video frame, a half-sized downsampled video frame, and the original-size video frame. An idle memory pool includes multiple idle memory locations used to store the encoded video frames in the video frame queue in a pyramid structure. If the number of idle memory locations in the idle memory pool is not less than a first quantity, it indicates that the idle memory in the idle memory pool is sufficient to store the video frames and at least one reference frame to be used to construct the pyramid structure. The electronic device can obtain the pyramid structure of the video frames and at least one reference video frame by overwriting the data in the first memory location of the idle memory pool with the video frames and at least one reference video frame. The first memory location is the number of video frames plus the number of reference video frames to be used to construct the pyramid structure, i.e., the first quantity. At this time, the first memory location stores the pyramid structure of the video frames and at least one reference video frame.

[0111] For example, Figure 4 This is a schematic diagram illustrating a free memory according to an exemplary embodiment, such as... Figure 4 As shown, the free memory is used to store the encoded video frames in the video frame queue in a pyramid structure. The pyramid structure includes downsampled video frames at 1 / 4 resolution, downsampled video frames at 1 / 2 resolution, full-resolution video frames at their original size, and a pointer to the next pyramid structure, PyramidPic*next. Figure 5 This is a schematic diagram illustrating a free memory pool according to an exemplary embodiment, such as... Figure 5 As shown, the free memory pool includes multiple free memory blocks, which can be associated with other free memory blocks based on pointers to the next pyramid structure, forming a memory queue.

[0112] In step S307, the electronic device filters the video frames based on the video frames in the first memory and the video frames in the second memory.

[0113] In this embodiment, the first memory stores a pyramid structure of the current video frame and at least one reference video frame, and the second memory stores pyramid structures of other reference video frames (excluding the at least one reference video frame). Based on a first number of video frames in the first memory, at least one reference video frame, and other reference video frames in the second memory, the electronic device can obtain information about multiple reference video frames according to the video frame and the pyramid structure of the multiple reference video frames. Based on the information of the multiple reference video frames, the electronic device filters the video frame. The pyramid structure of the other reference video frames is created during the filtering process of other video frames in the video frame queue. When filtering the current video frame, the electronic device can directly obtain the previously created pyramid structure of other reference video frames.

[0114] In step S308, if the number of free memory in the free memory pool is less than a first number, the electronic device overwrites the data in the free memory in the free memory pool with video frames and at least one reference video frame, and the number of free memory is a second number.

[0115] In this embodiment of the disclosure, if the amount of free memory in the free memory pool is less than a first quantity, it indicates that the free memory in the free memory pool is insufficient to store the video frame and at least one reference frame of the pyramid structure to be constructed. The electronic device can obtain a pyramid structure of the video frame and a portion of the reference video frame by overwriting the data in the free memory of the free memory pool with the video frame and at least one reference video frame. The number of pyramid structures is the amount of free memory, i.e., the second quantity.

[0116] In step S309, the electronic device creates a third number of third memories, the third number being the difference between the first number and the second number, and the third memories are used to store at least one reference video frame in a pyramid structure.

[0117] In this embodiment, since the number of some reference video frames is less than the number of reference video frames in the multiple reference video frames to which the pyramid structure is to be built, the electronic device also needs to create some third memory for storing the reference video frames in a pyramid structure. The amount of this third memory is the difference between the number of video frames and the number of reference video frames to which the pyramid structure is to be built, and the second amount. By reusing multiple free memories in the free memory pool, the electronic device can obtain the pyramid structure of video frames and some reference video frames. When the amount of free memory is insufficient, it can determine the number of remaining reference video frames to be created based on the amount of free memory, which can reduce memory overhead to a certain extent and improve the encoding efficiency of video frames.

[0118] In step S310, the electronic device filters the video frames based on the video frames in the first memory, the video frames in the second memory, and the video frames in the third memory.

[0119] In this embodiment, the first memory stores a pyramid structure of the current video frame and at least one reference video frame; the second memory stores a pyramid structure of reference video frames for other video frames, which are also reference video frames for the current video frame; and the third memory stores a pyramid structure of reference video frames from among the multiple reference video frames for which a pyramid structure to be created. Based on a first number of video frames and reference video frames in the first memory, the second memory, and the third memory, the electronic device can obtain information about multiple reference video frames according to the pyramid structure of the video frame and the multiple reference video frames. And based on the information from the multiple reference video frames, the electronic device can filter the video frame.

[0120] In some embodiments, for any video frame in the video frame queue, the electronic device can encode the video frame after filtering is completed, and after encoding, the memory storing the video frame is placed into the free memory pool as free memory for reuse by other video frames. Accordingly, in response to video frame filtering completion, the electronic device encodes the video frame; in response to video frame encoding completion, the electronic device places the video frame into an encoding completion queue, which stores the encoded video frames in the video frame queue; in response to the encoding completion queue not including all video frames in the video frame queue and the video frame not needing to be referenced by unencoded video frames in the video frame queue, the electronic device places the memory storing the video frame into the free memory pool as free memory. When the encoding completion queue does not include all video frames in the video frame queue, it indicates that not all video frames in the video frame queue have been encoded, and when filtering the unencoded video frames in the video frame queue, it may be necessary to refer to the encoded video frames in the encoding completion queue. Therefore, when a video frame needs to be referenced by unencoded video frames in the video frame queue, the electronic device needs to retain the memory for storing the video frame unchanged; when a video frame does not need to be referenced by unencoded video frames in the video frame queue, the electronic device can put the memory for storing the video frame into the free memory pool as free memory, which can be reused by subsequent video frames that need to be filtered and multiple reference video frames of the video frame. By making reasonable use of the free memory in the free memory pool, memory overhead can be reduced and the encoding efficiency of video frames can be improved.

[0121] In some embodiments, the electronic device can release free memory in the free memory pool when all video frames have been encoded. Accordingly, in response to the encoding completion queue including all video frames in the video frame queue, the electronic device releases free memory in the free memory pool. Since the inclusion of all video frames in the video frame queue in the encoding completion queue indicates that all video frames in the video frame queue have been encoded, the electronic device can release free memory in the free memory pool. By releasing memory space in the free memory pool, the memory overhead of the electronic device can be reduced.

[0122] For example, Figure 6 This is a flowchart illustrating a video frame filtering method according to an exemplary embodiment, such as... Figure 6As shown, the input frame queue includes multiple video frames. For any video frame in the input frame queue, the pre-analysis module performs pre-analysis to obtain pre-analysis information, including frame type and quantization parameters. The electronic device places the video frames output by the pre-analysis model into the output frame queue. For any video frame in the output frame queue, based on the frame type and quantization parameters, it can be determined whether the video frame needs filtering. If the video frame does not need filtering, it is directly placed into the actual encoding queue for the formal encoding process. If the video frame needs filtering, multiple reference video frames are determined from the input frame queue, and memory is requested from the free memory pool based on the video frame to be filtered and the multiple reference video frames. If free memory exists in the memory pool, it can be reused directly, and then the filtered video frame is placed into the actual encoding queue for the formal encoding process. If no free memory exists in the memory pool, new memory needs to be created to store the video frame and multiple reference video frames in a pyramid structure, and then the filtered video frame is placed into the actual encoding queue. After video frame encoding is complete, the electronic device places the encoded video frames into the encoding completion queue. When all video frames have been encoded, the memory pool is traversed, and free memory in the memory pool is released, ending the encoding process. If encoding is not complete, and a video frame needs to be referenced by subsequent unencoded video frames, the memory for that video frame is retained, and the system waits for the next video frame in the queue to be encoded before re-evaluating whether that video frame needs to be referenced by unencoded video frames in the queue. If the video frame does not need to be referenced by subsequent unencoded video frames, the memory storing the video frame is pushed into the memory pool as free memory for use in subsequent video frames requiring filtering and for multiplexing of video frames that serve as references to other video frames.

[0123] This disclosure provides a video frame filtering method. Since video frames need to reference other adjacent video frames during filtering, multiple reference video frames can be determined from the video frame queue before filtering. When the amount of free memory in the free memory pool is not less than the total number of video frames and reference video frames to be used to construct the pyramid structure, the data in the first memory space of the free memory pool can be overwritten with the video frames and at least one reference video frame; that is, the free memory in the free memory pool is reused. Then, the video frames are filtered based on the video frames in the first memory and the reference video frames stored in the second memory. By reusing the free memory in the free memory pool before filtering the video frames, additional memory overhead can be reduced, frequent memory allocation can be avoided, and thus the video encoding efficiency can be improved.

[0124] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0125] Figure 7 This is a block diagram illustrating a video frame filtering apparatus according to an exemplary embodiment. (Refer to...) Figure 8 The device includes: a determination unit 701, a coverage unit 702, and a filtering unit 703.

[0126] The determining unit 701 is configured to determine multiple reference video frames from the video frame queue for any video frame in the video frame queue. The multiple reference video frames are used to provide reference information during the filtering process of the video frame.

[0127] The overlay unit 702 is configured to overlay the data in the first memory in the free memory pool with video frames and at least one reference video frame when the number of free memory in the free memory pool is not less than a first number. The free memory in the free memory pool is used to store the encoded video frames in the video frame queue in a pyramid structure. The first number is the number of video frames plus the number of reference video frames to be constructed into the pyramid structure. The number of the first memory is the first number.

[0128] The filtering unit 703 is configured to filter video frames based on video frames in a first memory and video frames in a second memory, wherein the second memory is used to store at least one reference video frame in a pyramid structure.

[0129] In some embodiments, Figure 8 This is a block diagram illustrating another video frame filtering apparatus according to an exemplary embodiment. See also Figure 8 The unit 701 is defined as follows:

[0130] Storage subunit 801 is configured to place a copy of the video frame into the free memory pool for any video frame in the video frame queue.

[0131] The determination subunit 802 is configured to determine the frame identifiers of multiple reference video frames from the video frame queue based on the frame identifiers of the video frames.

[0132] The first acquisition subunit 803 is configured to, for any reference video frame, acquire a copy of the reference video frame from the free memory pool based on the frame identifier of the reference video frame, when the reference video frame is being filtered or has been filtered.

[0133] The second acquisition subunit 804 is configured to acquire a reference video frame from the video frame queue when the reference video frame is not filtered.

[0134] In some embodiments, see continue to see Figure 8 The device also includes:

[0135] The acquisition unit 704 is configured to acquire at least one second memory from a free memory pool based on the frame identifiers of multiple reference video frames;

[0136] The quantity determination unit 705 is configured to determine a first quantity based on the quantity of at least one second memory and the quantity of multiple reference video frames.

[0137] In some embodiments, the overlay unit 702 is further configured to, when the number of free memory in the free memory pool is less than a first number, overlay the data in the free memory in the free memory pool with video frames and at least one reference video frame, the number of free memory being a second number; create a third number of third memory, the third number being the difference between the first number and the second number, the third memory being used to store at least one reference video frame in a pyramid structure, and filter the video frames based on the video frames in the first memory, the video frames in the second memory, and the video frames in the third memory.

[0138] In some embodiments, see continue to see Figure 8 The device also includes:

[0139] The encoding unit 706 is configured to encode a video frame in response to the completion of video frame filtering;

[0140] The first storage unit 707 is configured to place a video frame into an encoding completion queue in response to the completion of video frame encoding. The encoding completion queue is used to store video frames that have been encoded in the video frame queue.

[0141] The second storage unit 708 is configured to, in response to the encoding completion queue not including all video frames in the video frame queue and the video frames not needing to be referenced by unencoded video frames in the video frame queue, place the memory storing the video frames as free memory into the free memory pool.

[0142] In some embodiments, see continue to see Figure 8 The device also includes:

[0143] The memory release unit 709 is configured to release free memory in the free memory pool in response to all video frames in the encoding completion queue, including the video frame queue.

[0144] In some embodiments, see continue to see Figure 8 The device also includes:

[0145] The pre-analysis unit 710 is configured to pre-analyze video frames to obtain the frame type and quantization parameters of the video frames. The quantization parameters are used to reflect the spatial details of the video frames.

[0146] The judgment unit 711 is configured to determine whether a video frame needs filtering based on the frame type and quantization parameters.

[0147] This disclosure provides a video frame filtering apparatus. Since video frames need to reference other adjacent video frames during filtering, multiple reference video frames can be determined from a video frame queue before filtering. When the amount of free memory in the free memory pool is not less than the total number of video frames and reference video frames to be used to construct the pyramid structure, the data in the first memory space of the free memory pool can be overwritten with the video frames and at least one reference video frame; that is, the free memory in the free memory pool is reused. Then, the video frames are filtered based on the video frames in the first memory and the reference video frames stored in the second memory. By reusing the free memory in the free memory pool before filtering the video frames, additional memory overhead can be reduced, frequent memory allocation can be avoided, and video encoding efficiency can be improved.

[0148] It should be noted that the video frame filtering device provided in the above embodiments is only illustrated by the division of the above functional units when running the application. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the video frame filtering device and the video frame filtering method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0149] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0150] When an electronic device is provided as a terminal, Figure 9 This is a block diagram illustrating a terminal 900 according to an exemplary embodiment. The terminal... Figure 9 A structural block diagram of a terminal 900 provided in an exemplary embodiment of this disclosure is shown. The terminal 900 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal 900 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0151] Typically, terminal 900 includes a processor 901 and a memory 902.

[0152] Processor 901 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 901 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0153] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 are used to store at least one program code, which is executed by the processor 901 to implement the video frame filtering method provided in the method embodiments of this disclosure.

[0154] In some embodiments, the terminal 900 may also optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 903 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, and a power supply 908.

[0155] Peripheral device interface 903 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 901 and memory 902. In some embodiments, processor 901, memory 902 and peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 901, memory 902 and peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0156] The radio frequency (RF) circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 904 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuitry related to NFC (Near Field Communication), which is not limited in this disclosure.

[0157] Display screen 905 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 905 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 901 for processing. In this case, display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 905, which serves as the front panel of terminal 900; in other embodiments, there may be at least two display screens 905, respectively disposed on different surfaces of terminal 900 or in a folded design; in still other embodiments, display screen 905 may be a flexible display screen, disposed on a curved or folded surface of terminal 900. Furthermore, display screen 905 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 905 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0158] The camera assembly 906 is used to acquire images or videos. Optionally, the camera assembly 906 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0159] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting them into electrical signals that are input to the processor 901 for processing, or to the radio frequency circuit 904 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 900. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0160] Power supply 908 is used to power the various components in terminal 900. Power supply 908 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 908 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0161] In some embodiments, the terminal 900 further includes one or more sensors 909. The one or more sensors 909 include, but are not limited to, an accelerometer 910, a gyroscope 911, a pressure sensor 912, an optical sensor 99, and a proximity sensor 914.

[0162] Accelerometer 910 can detect the magnitude of acceleration on the three coordinate axes of a coordinate system established by terminal 900. For example, accelerometer 910 can be used to detect the components of gravitational acceleration on the three coordinate axes. Processor 901 can control display screen 905 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 910. Accelerometer 910 can also be used for games or for acquiring user motion data.

[0163] The gyroscope sensor 911 can detect the orientation and rotation angle of the terminal 900. The gyroscope sensor 911, in conjunction with the accelerometer sensor 910, can collect the user's 3D movements on the terminal 900. Based on the data collected by the gyroscope sensor 911, the processor 901 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0164] The pressure sensor 912 can be disposed on the side bezel of the terminal 900 and / or the lower layer of the display screen 905. When the pressure sensor 912 is disposed on the side bezel of the terminal 900, it can detect the user's grip signal on the terminal 900, and the processor 901 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 912. When the pressure sensor 912 is disposed on the lower layer of the display screen 905, the processor 901 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 905. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0165] An optical sensor 913 is used to collect ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the display screen 905 based on the ambient light intensity collected by the optical sensor 913. Optionally, when the ambient light intensity is high, the display brightness of the display screen 905 is increased; when the ambient light intensity is low, the display brightness of the display screen 905 is decreased. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera assembly 906 based on the ambient light intensity collected by the optical sensor 913.

[0166] The proximity sensor 914, also known as the distance sensor, is installed on the front panel of the terminal 900. The proximity sensor 914 is used to detect the distance between the user and the front of the terminal 900. In one embodiment, when the proximity sensor 914 detects that the distance between the user and the front of the terminal 900 is gradually decreasing, the processor 901 controls the display screen 905 to switch from a screen-on state to a screen-off state; when the proximity sensor 914 detects that the distance between the user and the front of the terminal 900 is gradually increasing, the processor 901 controls the display screen 905 to switch from a screen-off state to a screen-on state.

[0167] Those skilled in the art will understand that Figure 9 The structure shown does not constitute a limitation on terminal 900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0168] When electronic devices are provided as servers, Figure 10This is a block diagram illustrating a server 1000 according to an exemplary embodiment. The server 1000 can vary significantly due to differences in configuration or performance. It may include one or more Central Processing Units (CPUs) 1001 and one or more memories 1002. The memories 1002 store at least one line of program code, which is loaded and executed by the processor 1001 to implement the video frame filtering methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 1000 may also include other components for implementing device functions, which will not be elaborated upon here.

[0169] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 902 or a memory 1002 including instructions, which can be executed by a processor 901 of a terminal 900 or a processor 1001 of a server 1000 to perform the above method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0170] A computer program product includes a computer program / instructions that, when executed by a processor, implement the aforementioned video frame filtering method.

[0171] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0172] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method of video frame filtering, the method comprising: The method comprises: For any video frame in the video frame queue, determining a plurality of reference video frames of the video frame from the video frame queue, the plurality of reference video frames being used to provide reference information in filtering the video frame; In a case where the number of free memories in the free memory pool is not less than a first number, covering data in a first memory in the free memory pool with the video frame and at least one reference video frame, the free memories in the free memory pool being used to store encoded video frames in the video frame queue in a pyramid structure, the first number being the number of the video frame plus the number of reference video frames to be constructed in the pyramid structure, the number of the first memory being the first number; Filtering the video frame based on the video frame in the first memory and the video frame in a second memory, the second memory being used to store at least one reference video frame in a pyramid structure.

2. The method of Claim 1, wherein, The method further comprises: For any video frame in the video frame queue, determining a plurality of reference video frames of the video frame from the video frame queue, the plurality of reference video frames being used to provide reference information in filtering the video frame; For any video frame in the video frame queue, determining a plurality of reference video frames of the video frame from the video frame queue, the plurality of reference video frames being used to provide reference information in filtering the video frame; For any video frame in the video frame queue, determining a plurality of reference video frames of the video frame from the video frame queue, the plurality of reference video frames being used to provide reference information in filtering the video frame; For any video frame in the video frame queue, determining a plurality of reference video frames of the video frame from the video frame queue, the plurality of reference video frames being used to provide reference information in filtering the video frame.

3. The method of Claim 1, wherein, The method further comprises: Based on the frame identifiers of the plurality of reference video frames, obtaining at least one second memory from the free memory pool; Based on the number of the at least one second memory and the number of the plurality of reference video frames, determining the first number.

4. The method of Claim 1, wherein, The method further comprises: In a case where the number of free memories in the free memory pool is less than the first number, covering data in a free memory in the free memory pool with the video frame and at least one reference video frame, the number of the free memory being a second number; Creating a third number of third memories, the third number being the difference between the first number and the second number, the third memories being used to store at least one reference video frame in a pyramid structure; Filtering the video frame based on the video frame in the first memory, the video frame in the second memory and the video frame in the third memory.

5. The method of Claim 1, wherein, The method further comprises: In response to completion of filtering the video frame, encoding the video frame; In response to completion of encoding the video frame, putting the video frame into an encoding completion queue, the encoding completion queue being used to store video frames in the video frame queue that have completed encoding; In response to the encoding completion queue not including all video frames in the video frame queue and the video frame not needing to be referenced by a video frame in the video frame queue that has not been encoded, putting a memory storing the video frame into the free memory pool as a free memory.

6. The method of Claim 5, wherein, The method further comprises: In response to the encoding completion queue including all video frames in the video frame queue, memory is released from the free memory pool.

7. The method of Claim 1, wherein, The method further includes: pre-analyzing the video frames to obtain frame types and quantization parameters of the video frames, the quantization parameters being used to reflect spatial detail conditions of the video frames; based on the frame types and the quantization parameters, determining whether the video frames need to be filtered.

8. A video frame encoding apparatus characterized by comprising: The apparatus includes: a determination unit configured to determine, for any video frame in a video frame queue, a plurality of reference video frames of the video frame from the video frame queue, the plurality of reference video frames being used to provide reference information in filtering the video frame; a covering unit configured to, in a case where a number of free memories in a free memory pool is not less than a first number, cover data in a first memory in the free memory pool with the video frame and at least one reference video frame, the free memories in the free memory pool being used to store encoded video frames in the video frame queue in a pyramid structure, the first number being a number of the video frame plus a number of reference video frames to be constructed in the pyramid structure, and the first memory being of the first number; a filtering unit configured to filter the video frame based on video frames in the first memory and video frames in a second memory, the second memory being used to store at least one reference video frame in the pyramid structure.

9. An electronic device, comprising: The electronic device includes: one or more processors; a memory for storing program codes executable by the processors; wherein the processors are configured to execute the program codes to implement the video frame filtering method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When instructions in the computer readable storage medium are executed by the processors of the electronic device, the electronic device is enabled to perform the video frame filtering method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processors, implements the video frame filtering method according to any one of claims 1 to 7.

Images