Training method and device of reference frame screening model and computer program product

Abstract

The present disclosure provides a training method and device of a reference frame screening model, a method and device for screening a reference frame, an electronic device, a storage medium and a program product, relates to the field of artificial intelligence, specifically to the field of cloud computing, video coding, media cloud technology, and can be applied to an intelligent cloud scene. The specific implementation scheme is as follows: obtaining a training sample set, wherein the training sample in the training sample set comprises a video frame sequence and a label corresponding to a video frame in the video frame sequence, and the label is used to represent whether the video frame corresponding to the label is a reference frame of other video frames in the video frame sequence; using a machine learning method, taking the video frame sequence as input, and taking the label corresponding to the input video frame sequence as expected output, a reference frame screening model is trained. The present disclosure improves the accuracy of the reference frame screening model, increases the selection range of the reference frame, and improves the adaptability between the screened reference frame and the video frame and the accuracy of the motion estimation result.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, specifically to the fields of cloud computing, video encoding and decoding, and media cloud technology. In particular, it relates to a training method and apparatus for a reference frame filtering model, a method and apparatus for filtering reference frames, electronic devices, storage media, and computer program products that can be applied to intelligent cloud scenarios. Background Technology

[0002] HEVC (High Efficiency Video Coding) employs a Reference Picture Set (RPS) technique to manage decoded frames, which are then used as references for subsequent video frames. In the HEVC standard, all potentially used reference frames are stored in a linked list (reference frame set). The current reference frame selection strategy involves selecting a sufficient number of reference frames sequentially from the head of the linked list. Because each newly added reference frame is added to the head of the list, the final selected reference frames are always the closest to the current reference frame, preventing the selection of frames that are slightly farther away. This results in suboptimal motion estimation. Summary of the Invention

[0003] This disclosure provides a method and apparatus for training a reference frame filtering model, a method and apparatus for filtering reference frames, an electronic device, a storage medium, and a computer program product.

[0004] According to the first aspect, a training method for a reference frame selection model is provided, comprising: obtaining a training sample set, wherein the training samples in the training sample set include video frame sequences and labels corresponding to video frames in the video frame sequences, the labels being used to characterize whether the video frame corresponding to the label is a reference frame for other video frames in the video frame sequence; and using a machine learning method, taking the video frame sequences as input and the labels corresponding to the input video frame sequences as the expected output, to train the reference frame selection model.

[0005] According to the second aspect, a method for filtering reference frames is provided, comprising: acquiring a target video sequence; for each video frame in the target video sequence, filtering the target reference frame corresponding to the video frame from the candidate reference frame set corresponding to the video frame through a pre-trained reference frame filtering model, thereby obtaining a target reference frame set, wherein the reference frame filtering model is trained by any implementation of the first aspect.

[0006] According to a third aspect, a training apparatus for a reference frame selection model is provided, comprising: a first acquisition unit configured to acquire a training sample set, wherein the training samples in the training sample set include a video frame sequence and labels corresponding to the video frames in the video frame sequence, the labels being used to characterize whether the video frame corresponding to the label is a reference frame for other video frames in the video frame sequence; and a training unit configured to use a machine learning method, taking the video frame sequence as input and the labels corresponding to the input video frame sequence as the desired output, to train the reference frame selection model.

[0007] According to the fourth aspect, an apparatus for filtering reference frames is provided, comprising: a second acquisition unit configured to acquire a target video sequence; and a filtering unit configured to, for each video frame in the target video sequence, filter out a target reference frame corresponding to the video frame from a set of candidate reference frames corresponding to the video frame using a pre-trained reference frame filtering model, thereby obtaining a set of target reference frames, wherein the reference frame filtering model is trained by any implementation of the third aspect.

[0008] According to a fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first or second aspect.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the method described in any implementation of the first or second aspect.

[0010] According to the seventh aspect, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the method as described in either the first or second aspect.

[0011] According to the technology disclosed herein, a training method for a reference frame selection model and a method for selecting reference frames are provided. The method employs machine learning, using a sequence of video frames in the training samples as input and labels representing whether a video frame is a reference frame for other video frames in the sequence as the expected output, to train a reference frame selection model, thereby improving the accuracy of the model. By selecting the reference frame set corresponding to each video frame in the video sequence through the reference frame selection model, the range of reference frame selection is increased, and the adaptability of the selected reference frames to the video frames and the accuracy of motion estimation results are improved.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0014] Figure 1 This is an exemplary system architecture diagram that can be applied to an embodiment of this disclosure;

[0015] Figure 2 This is a flowchart of an embodiment of a training method for a reference frame selection model according to the present disclosure;

[0016] Figure 3 This is a schematic diagram illustrating an application scenario of the training method for the reference frame filtering model according to this embodiment;

[0017] Figure 4 This is a flowchart of yet another embodiment of the training method for the reference frame selection model according to the present disclosure;

[0018] Figure 5 This is a flowchart of one embodiment of the method for filtering reference frames according to the present disclosure;

[0019] Figure 6 This is a flowchart of yet another embodiment of the method for filtering reference frames according to the present disclosure;

[0020] Figure 7 This is a structural diagram of an embodiment of a training apparatus for a reference frame filtering model according to the present disclosure;

[0021] Figure 8 This is a structural diagram of one embodiment of the apparatus for filtering reference frames according to the present disclosure;

[0022] Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0024] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0025] Figure 1 An exemplary architecture 100 is shown for a training method and apparatus for applying the reference frame filtering model of this disclosure, and for a method and apparatus for filtering reference frames.

[0026] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The communication connections between terminal devices 101, 102, and 103 form a network topology. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0027] Terminal devices 101, 102, and 103 can be hardware or software that supports network connectivity for data interaction and processing. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices that support network connectivity, information acquisition, interaction, display, and processing functions, including but not limited to smartphones, tablets, e-book readers, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as, for example, multiple software programs or software modules to provide distributed services, or as a single software program or software module. No specific limitations are imposed here.

[0028] Server 105 can be a server providing various services, such as a backend processing server that receives training sample sets provided by terminal devices 101, 102, and 103, and trains a reference frame selection model using machine learning methods; or, for example, a backend processing server that selects candidate reference frames from the candidate reference frame set corresponding to each video frame in the target video sequence provided by terminal devices 101, 102, and 103 using a pre-trained reference frame selection model. As an example, server 105 can be a cloud server.

[0029] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0030] It should also be noted that the training method for the reference frame filtering model and the method for filtering reference frames provided in the embodiments of this disclosure can be executed by a server, by a terminal device, or by a combination of both. Accordingly, the various parts (e.g., units) included in the training device for the reference frame filtering model and the device for filtering reference frames can all be located in the server, all in the terminal device, or separately in the server and the terminal device.

[0031] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. When the training method for the reference frame filtering model and the electronic devices on which the method for filtering reference frames runs do not require data transmission with other electronic devices, the system architecture may only include the training method for the reference frame filtering model and the electronic devices (e.g., servers or terminal devices) on which the method for filtering reference frames runs.

[0032] Please refer to Figure 2 , Figure 2 A flowchart illustrating a training method for a reference frame selection model provided in this embodiment of the disclosure. Flowchart 200 includes the following steps:

[0033] Step 201: Obtain the training sample set.

[0034] In this embodiment, the execution entity of the training method for the reference frame filtering model (e.g., Figure 1 The terminal device or server in the process can obtain the training sample set from a remote location or from a local location via a wired or wireless network connection.

[0035] The training samples in the training sample set include video frame sequences and labels corresponding to the video frames in the video frame sequences. The labels are used to characterize whether the video frame corresponding to the label is a reference frame for other video frames in the video frame sequence.

[0036] As an example, a video frame sequence includes 50 consecutive video frames f1-f50. For video frame f20, its reference frames are f21, f24, and f25. Then, for video frame f20, its corresponding label indicates that video frames f21, f24, and f25 are the reference frames of video frame f20.

[0037] In this implementation, for each video frame in the video frame sequence that needs to be referenced from other frames for encoding and decoding operations, a corresponding label can be set to represent the reference frame corresponding to that video frame.

[0038] To improve the richness of training samples in the training sample set and the generalization ability of the reference frame selection model after training, the aforementioned execution entity can select multiple video sequences with significant differences based on the temporal and spatial complexity of the video frame sequences to generate a training sample set.

[0039] As an example, for each complexity in the time domain and spatial domain, the aforementioned execution entity can set a corresponding complexity level, with different complexity levels corresponding to different complexity value ranges; then, for each complexity level, a video sequence belonging to that complexity level is selected to generate a training sample; finally, multiple training samples with different time domain complexities and different spatial domain complexities are combined to obtain a training sample set.

[0040] Temporal complexity can be calculated using the frame rate or duration of the video sequence, while spatial complexity can be calculated using the number of pixels in each video frame of the video sequence.

[0041] In some optional implementations of this embodiment, for each video frame in the video frame sequence, the tag corresponding to that video frame is determined in the following way:

[0042] First, in response to the completion of the encoding operation of the video frame, determine the number of times each reference frame in the reference frame set corresponding to the video frame is referenced in the encoding operation of the video frame.

[0043] A video sequence typically includes I-frames, P-frames, and B-frames. I-frames are intra-coded frames, meaning their encoding and decoding do not require reference to other video frames. P-frames are forward predictive coded frames, requiring reference to preceding video frames during encoding and decoding. B-frames are bidirectional predictive coded frames, generally requiring reference to both preceding and / or following video frames during encoding and decoding. In inter-frame predictive coding, because objects in neighboring video frames have a certain correlation, the video frame can be divided into several coded blocks. The position of each coded block in neighboring video frames is then searched, and the relative spatial offset between the two is calculated. This relative offset is commonly referred to as the motion vector, and the process of obtaining the motion vector is called motion estimation.

[0044] The motion vector and the prediction error obtained after motion matching are sent together to the decoder. At the decoder, the corresponding coded block is found from the neighboring decoded reference frame image according to the position indicated by the motion vector. The position of the coded block in the current coded frame is obtained by adding the motion vector and the prediction error. Motion estimation can remove inter-frame redundancy, which greatly reduces the number of bits transmitted in the video.

[0045] As can be seen, for coded blocks in B-frames and P-frames, motion vectors are generally determined by referencing other video frames. Typically, a video frame contains multiple coded blocks, each of which references other video frames to determine its motion vectors; different coded blocks may reference the same video frames. The process of a coded block referencing other video frames can be viewed as the other video frames being referenced once by that video frame.

[0046] As an example, video frame f1 includes 16 coding blocks, of which coding blocks 1, 4 and 5 all reference video frame f2. Therefore, the number of times reference frame f2 is referenced in the coding operation of video frame f1 is 3.

[0047] The reference frame set corresponding to each video frame can be a reference frame set of video frames determined based on HEVC technology.

[0048] Second, the tag corresponding to the video frame is determined based on the number of times each reference frame in the reference frame set is referenced.

[0049] As an example, the aforementioned execution entity can set a number threshold, and in response to determining that the reference frame has been referenced more than the number of times during the encoding process of the video frame, set a label to indicate that the reference frame is the reference frame obtained from the video frame.

[0050] The number of times threshold can be set according to the actual situation, and is not limited here.

[0051] This implementation provides a method for determining the tags corresponding to video frames. By determining the number of times each reference frame in the reference frame set is referenced in the encoding operation of the video frame, it is determined whether a reference frame in the reference frame set is a reference frame of the video frame, thus improving the accuracy of the tags.

[0052] In some optional implementations of this embodiment, the execution entity may perform the first step as follows: in response to completing the encoding operation of the video frame based on a coding block of a preset size, determine the number of times each reference frame in the reference frame set corresponding to the video frame is referenced in the encoding operation of the video frame.

[0053] The preset size can be set according to the actual situation. For example, the preset size can be 4×4, 8×8, and 16×16.

[0054] In this implementation, the aforementioned execution entity can perform the second step as follows:

[0055] First, for each reference frame in the reference frame set, the reference ratio of the reference frame in the encoding operation of the video frame is determined based on the number of times the reference frame is referenced and the number of coded blocks in the video frame.

[0056] Specifically, for each reference frame in the reference frame set, the number of times the reference frame is referenced is divided by the number of coded blocks in the video frame to obtain the reference ratio of the reference frame in the encoding operation of the video frame.

[0057] Taking video frame f1 with a size of 1920×1080 and a preset size of 8×8 as an example, the number of coded blocks in this video frame is 32400. If reference frame f2 is referenced 10000 times, then the reference ratio of reference frame f2 in the encoding operation of video frame f1 is approximately 31%.

[0058] Second, determine the tag corresponding to the video frame based on the reference ratio corresponding to each reference frame in the reference frame set.

[0059] In this implementation, the execution entity can pre-set a ratio threshold. In response to determining that the reference ratio of the reference frame in the encoding process of the video frame is greater than the ratio threshold, a label representing that the reference frame is the reference frame of the video frame is set.

[0060] The ratio threshold can be set according to the actual situation, and is not limited here.

[0061] In this implementation, the label corresponding to the video frame is determined by the reference ratio, which further improves the accuracy of the label.

[0062] Step 202: Using machine learning methods, a reference frame selection model is trained by taking the video frame sequence as input and the corresponding label of the input video frame sequence as the expected output.

[0063] In this embodiment, the aforementioned execution entity can use machine learning methods to train a reference frame selection model by taking a video frame sequence as input and the label corresponding to the input video frame sequence as the expected output.

[0064] As an example, in response to the failure to meet the preset termination condition, the following training operation is executed repeatedly:

[0065] Select untrained target training samples from the training sample set, input the video frame sequence from the target training samples into the initial reference frame selection model to obtain the actual output; calculate the loss between the actual output and the labels in the target training samples; determine the update gradient of the model based on the loss, and then use the stochastic gradient descent method to update the parameters of the initial reference frame selection model to obtain the initial reference frame selection model corresponding to the next training operation.

[0066] Among them, the preset termination conditions can be, for example, training time exceeding a preset time threshold, training times exceeding a preset number of times threshold, or training loss tending to converge.

[0067] The reference frame selection model can use a neural network model with classification function, such as XGBoost (Extreme Gradient Boosting) model, fully connected neural network model, recurrent neural network model, residual network model, etc.

[0068] In some optional implementations of this embodiment, the execution entity can perform step 202 as follows:

[0069] Using machine learning methods, for each video frame in the video frame sequence, the reference frame selection model is trained by taking the video frame as input and the label corresponding to the video frame as the expected output.

[0070] As an example, in response to the failure to meet the preset termination condition, the following training operation is executed repeatedly:

[0071] Select untrained target training samples from the training sample set, input each video frame in the video frame sequence of the target training samples into the initial reference frame selection model, and obtain the actual output corresponding to each video frame; calculate the loss between the actual output and the label corresponding to each video frame in the target training samples; determine the update gradient of the model based on the loss, and then use the stochastic gradient descent method to update the parameters of the initial reference frame selection model to obtain the initial reference frame selection model corresponding to the next training operation.

[0072] In this implementation, machine learning of the reference frame selection model is performed on a per-video-frame basis, which helps to further improve the accuracy of the obtained reference frame selection model.

[0073] In some optional implementations of this embodiment, the execution entity can perform the above training steps in the following ways:

[0074] First, based on a preset feature set, extract the feature dataset for each video frame in the video frame sequence.

[0075] The preset feature set includes multiple feature types that indicate the video frames in the video frame sequence for feature acquisition. As an example, the preset feature set includes the number of inter blocks and intra blocks of the video frame and its corresponding reference frame in the lookahead process, the intra-frame cost, the quantization parameter (QP), the reference frame type, the current video frame type, the mean brightness of the image, and the brightness variance.

[0076] Second, using machine learning methods, for each video frame in the video frame sequence, the feature dataset corresponding to that video frame is used as input and the label corresponding to that video frame is used as the expected output to train a reference frame selection model.

[0077] In this implementation, the preset feature set can instruct the execution entity to extract the feature dataset of video frames, which improves the richness of data features and helps to improve the accuracy of the reference frame selection model after training. Furthermore, by specifying the feature type, the training process of the model can be guided, which helps to improve the training speed of the model.

[0078] In some optional implementations of this embodiment, the above-mentioned execution entity may also perform the following operation: during the training process, determine the importance of each feature data in the feature dataset.

[0079] In this implementation, the aforementioned execution entity can perform the second step as follows:

[0080] First, target feature data is selected from the feature dataset based on the importance of the feature data determined during training. Then, a machine learning method is used to train a reference frame selection model for each video frame in the video frame sequence, with the target feature dataset corresponding to that video frame as input and the label corresponding to that video frame as the expected output.

[0081] As an example, during training, corresponding learnable importance parameters can be set for features under various feature types, and the importance parameters can be updated as the model trains.

[0082] During the training process, the importance of feature data can be determined based on the value of the importance parameter, thereby allowing the selection of target feature data for model training.

[0083] For example, the filtered target feature data are feature data corresponding to feature types such as the number of inter blocks, the number of intra blocks, and QP.

[0084] In this implementation, more important target feature data are selected for training during the training process. While ensuring training accuracy, the amount of data in the training process is reduced, which helps to further improve the training speed of the model.

[0085] See also Figure 3 , Figure 3 This is a schematic diagram 300 illustrating an application scenario of the training method for the reference frame filtering model according to this embodiment. Figure 3In this application scenario, an initial reference frame selection model is deployed on server 301. To train this model, a training sample set is obtained from terminal device 302. The training samples in the training sample set include video frame sequences and labels corresponding to the video frames within those sequences. The labels characterize whether the video frame corresponding to the label is a reference frame for other video frames in the sequence. During training, a machine learning method is used, with the video frame sequences as input and the labels corresponding to the input video frame sequences as the desired output, to train the reference frame selection model.

[0086] In this embodiment, a training method for a reference frame selection model is provided. The method uses a machine learning approach, taking the video frame sequence in the training sample as input and the labels in the training sample that represent whether a video frame is a reference frame of other video frames in the video frame sequence as the expected output, to train the reference frame selection model and improve its accuracy.

[0087] Continue to refer to Figure 4 The illustration shows a schematic flow 400 of another embodiment of the training method for the reference frame selection model according to the present disclosure. Flow 400 includes the following steps:

[0088] Step 401: Obtain multiple video frame sequences with different temporal and spatial complexities.

[0089] Step 402: For each video frame in each video frame sequence, in response to completing the encoding operation of the video frame based on a coding block of a preset size, determine the number of times each reference frame in the reference frame set corresponding to the video frame is referenced in the encoding operation of the video frame.

[0090] Step 403: For each reference frame in the reference frame set, determine the reference ratio of the reference frame in the encoding operation of the video frame based on the number of times the reference frame is referenced and the number of coded blocks in the video frame.

[0091] Step 404: Determine the label corresponding to the video frame based on the reference ratio corresponding to each reference frame in the reference frame set, and finally obtain the training sample set.

[0092] Step 405: Extract the feature dataset of each video frame in the video frame sequence according to the preset feature set.

[0093] The preset feature set includes multiple feature types that indicate the video frames in the video frame sequence to be feature-collected.

[0094] Step 406: Using machine learning methods, for each video frame in the video frame sequence, the feature dataset corresponding to that video frame is used as input and the label corresponding to that video frame is used as the expected output to train a reference frame selection model.

[0095] As can be seen from this embodiment, with Figure 2 Compared with the corresponding embodiments, the process 400 of the training method for the reference frame selection model in this embodiment specifically describes the process of determining the labels in the training samples and the training process of the reference frame selection model, which further improves the accuracy of the obtained reference frame selection model.

[0096] Continue to refer to Figure 5 , Figure 5 A flowchart of a method for filtering reference frames provided in this disclosure embodiment. In process 500, the following steps are included:

[0097] Step 501: Obtain the target video sequence.

[0098] In this embodiment, the execution entity of the method for filtering reference frames (e.g., Figure 1 The terminal device or server in the process can obtain the target video sequence from a remote location or from a local location via a wired or wireless network connection.

[0099] A video frame sequence consists of multiple consecutive video frames used to represent video data.

[0100] Step 502: For each video frame in the target video sequence, the target reference frame corresponding to the video frame is selected from the candidate reference frame set corresponding to the video frame through the pre-trained reference frame selection model, so as to obtain the target reference frame set.

[0101] In this embodiment, for each video frame in the target video sequence, the aforementioned execution entity can use a pre-trained reference frame filtering model to filter out the target reference frame corresponding to the video frame from the candidate reference frame set corresponding to that video frame, thereby obtaining the target reference frame set.

[0102] The reference frame filtering model is obtained through the above embodiments 200 and 400.

[0103] As an example, for each video frame in the target video sequence, the candidate reference frames in the candidate reference frame set corresponding to the video frame are classified by a pre-trained reference frame screening model, and the target reference frame corresponding to the video frame is determined based on the classification results, thus obtaining the target reference frame set.

[0104] Among them, the candidate reference frame set corresponding to the video frame can be the reference frame set corresponding to the video frame determined based on HEVC technology.

[0105] In some optional implementations of this embodiment, the execution entity can perform step 502 as follows: in response to determining that the number of target reference frames in the target reference frame set does not exceed a preset number threshold, the following determination operation is performed cyclically:

[0106] First, candidate reference frames are extracted from the candidate reference frame set corresponding to the video frame, and target feature data corresponding to the candidate reference frames are extracted. Then, the target feature data is input into the reference frame filtering model to determine whether the candidate reference frame is the target reference frame corresponding to the video frame. Finally, in response to the determination that it is, the candidate reference frame is added to the target reference frame set.

[0107] The preset quantity threshold can be set according to the actual situation, and is not limited here.

[0108] Please continue to refer to this. Figure 6 The diagram illustrates a method for filtering reference frames. Specifically, in process 600, for each video frame in the target video frame sequence, the target reference frames for that video frame can be filtered through the following steps:

[0109] 1. Extract a candidate reference frame from the reference frame list (reference frame set). The reference frame list corresponding to the video frame can be determined based on HEVC technology.

[0110] 2. Determine whether the selected candidate reference frame is a forward reference frame.

[0111] 3. In response to determining that it is a forward reference frame, further determine whether the number of forward reference frames corresponding to the current video frame exceeds the preset forward reference frame threshold.

[0112] 4. In response to determining that it is a backward reference frame, further determine whether the number of backward reference frames corresponding to the current video frame exceeds the preset backward reference frame threshold.

[0113] 5. In response to the condition that the corresponding preset threshold is not exceeded (specifically, the preset forward reference frame threshold or the preset backward reference frame threshold), extract the feature data of the forward reference frame or the backward reference frame.

[0114] 6. Input the feature data into the reference frame filtering model to determine whether to add the selected reference frame to the target reference frame set (specifically, the target forward reference frame set or the target backward reference frame set).

[0115] 7. Determine if the end of the reference frame list has been reached.

[0116] 8. In response to the failure to reach the end of the reference frame list, or the number of forward reference frames exceeding a preset forward reference frame threshold, or the number of backward reference frames exceeding a preset backward reference frame threshold, continue executing step 1.

[0117] 9. In response to reaching the end of the reference frame list, determine whether the number of target reference frames in the target reference frame set is 0; if it is 0, select the reference frame that is closer to the current video frame from the reference frame list and add it to the target reference frame set; if it is not 0, directly output the target reference frame set.

[0118] After determining the target reference frame set corresponding to the video frame, encoding and decoding operations can be performed on the video frame based on the determined target reference frame set.

[0119] In this embodiment, a method for filtering reference frames is provided. The reference frame filtering model filters the reference frame set corresponding to each video frame in the video sequence, which increases the selection range of reference frames and improves the adaptability of the filtered reference frames to the video frames and the accuracy of motion estimation results.

[0120] Continue to refer to Figure 7 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a training apparatus for a reference frame filtering model, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0121] like Figure 7 As shown, the training device 700 for the reference frame selection model includes: a first acquisition unit 701, configured to acquire a training sample set, wherein the training samples in the training sample set include a video frame sequence and labels corresponding to the video frames in the video frame sequence, the labels being used to characterize whether the video frame corresponding to the label is a reference frame for other video frames in the video frame sequence; and a training unit 702, configured to use a machine learning method, taking the video frame sequence as input and the labels corresponding to the input video frame sequence as the desired output, to train the reference frame selection model.

[0122] In some optional implementations of this embodiment, the training unit 702 is further configured to: use a machine learning method to train a reference frame selection model for each video frame in the video frame sequence, with the video frame as input and the label corresponding to the video frame as the expected output.

[0123] In some optional implementations of this embodiment, the training unit 702 is further configured to: extract feature datasets for each video frame in the video frame sequence according to a preset feature set, wherein the preset feature set includes multiple feature types that indicate the video frames in the video frame sequence to be feature collected; and use machine learning methods to train a reference frame selection model for each video frame in the video frame sequence, with the feature dataset corresponding to the video frame as input and the label corresponding to the video frame as the expected output.

[0124] In some optional implementations of this embodiment, the above apparatus further includes: a first determining unit (not shown in the figure), configured to determine the importance of each feature data in the feature dataset during the training process; and a training unit 702, further configured to: select target feature data from the feature dataset according to the importance of the feature data determined during the training process; and use a machine learning method to train a reference frame selection model for each video frame in the video frame sequence, with the target feature dataset corresponding to the video frame as input and the label corresponding to the video frame as the expected output.

[0125] In some optional implementations of this embodiment, the above apparatus further includes: a second determining unit (not shown in the figure), configured to: for each video frame in the video frame sequence, determine the tag corresponding to the video frame in the following manner: in response to completing the encoding operation of the video frame, determine the number of times each reference frame in the reference frame set corresponding to the video frame is referenced in the encoding operation of the video frame; determine the tag corresponding to the video frame based on the number of times each reference frame in the reference frame set is referenced.

[0126] In some optional implementations of this embodiment, the second determining unit (not shown in the figure) is further configured to: in response to completing the encoding operation of the video frame based on a coding block of a preset size, determine the number of times each reference frame in the reference frame set corresponding to the video frame is referenced in the encoding operation of the video frame; for each reference frame in the reference frame set, determine the reference ratio of the reference frame in the encoding operation of the video frame based on the number of times the reference frame is referenced and the number of coding blocks in the video frame; and determine the tag corresponding to the video frame based on the reference ratio of each reference frame in the reference frame set.

[0127] In this embodiment, a training device for a reference frame selection model is provided. The device employs a machine learning method, taking the video frame sequence in the training sample as input and the labels in the training sample that represent whether a video frame is a reference frame of other video frames in the video frame sequence as the expected output, thereby training the reference frame selection model and improving its accuracy.

[0128] Continue to refer to Figure 8 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a training device for a reference frame filtering model, which is similar to... Figure 5 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0129] like Figure 8As shown, the training device 800 for the reference frame filtering model includes: a second acquisition unit 801, configured to acquire a target video sequence; and a filtering unit 802, configured to, for each video frame in the target video sequence, filter out the target reference frame corresponding to the video frame from the candidate reference frame set corresponding to the video frame through a pre-trained reference frame filtering model, thereby obtaining a target reference frame set. The reference frame filtering model is trained through any of the implementation methods in the above embodiment 700.

[0130] In some optional implementations of this embodiment, the filtering unit 802 is further configured to: in response to determining that the number of target reference frames in the target reference frame set does not exceed a preset number threshold, repeatedly perform the following determination operations: extract candidate reference frames from the candidate reference frame set corresponding to the video frame, and extract the target feature data corresponding to the candidate reference frames; input the target feature data into the reference frame filtering model to determine whether the candidate reference frame is the target reference frame corresponding to the video frame; in response to determining that it is, add the candidate reference frame to the target reference frame set.

[0131] In this embodiment, an apparatus for filtering reference frames is provided. The reference frame filtering model filters the reference frame set corresponding to each video frame in the video sequence, which increases the selection range of reference frames and improves the adaptability of the filtered reference frames to the video frames and the accuracy of motion estimation results.

[0132] According to embodiments of this disclosure, this disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to implement the training method of the reference frame screening model and the method for screening reference frames described in any of the above embodiments.

[0133] According to embodiments of this disclosure, this disclosure also provides a readable storage medium storing computer instructions that enable a computer to implement the training method for the reference frame filtering model and the method for filtering reference frames as described in any of the above embodiments when executed.

[0134] This disclosure provides a computer program product that, when executed by a processor, can implement the training method for the reference frame filtering model and the method for filtering reference frames as described in any of the above embodiments.

[0135] Figure 9A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0136] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0137] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0138] The computing unit 901 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the training method for a reference frame screening model. For example, in some embodiments, the training method for a reference frame screening model can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the training method for the reference frame screening model described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured in any other suitable manner (e.g., by means of firmware) to perform a training method for the reference frame screening model.

[0139] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0140] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0141] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0143] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0144] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are hosting products within the cloud computing service system to address the management difficulties and weak business scalability inherent in traditional physical hosts and Virtual Private Servers (VPS) services; they can also be servers for distributed systems or servers integrated with blockchain technology.

[0145] According to the technical solution of the embodiments of this disclosure, a training method for a reference frame selection model and a method for selecting reference frames are provided. The method employs a machine learning approach, using a video frame sequence in the training samples as input and labels in the training samples that characterize whether a video frame is a reference frame for other video frames in the video frame sequence as the expected output, thereby training a reference frame selection model and improving the accuracy of the reference frame selection model. By selecting the reference frame set corresponding to each video frame in the video sequence through the reference frame selection model, the selection range of reference frames is increased, and the adaptability of the selected reference frames to the video frames and the accuracy of motion estimation results are improved.

[0146] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0147] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a reference frame selection model, comprising: Obtain a training sample set, wherein the training samples in the training sample set include video frame sequences and labels corresponding to the video frames in the video frame sequences, and the labels are used to characterize whether the video frame corresponding to the label is a reference frame for other video frames in the video frame sequence; A reference frame selection model is trained using machine learning methods, with the video frame sequence as input and the labels corresponding to the input video frame sequence as the expected output. For each video frame in the video frame sequence, the corresponding tag is determined as follows: In response to the completion of the encoding operation of the video frame based on the encoding block of the preset size, determine the number of times each reference frame in the reference frame set corresponding to the video frame is referenced in the encoding operation of the video frame; For each reference frame in the reference frame set, the reference ratio of the reference frame in the encoding operation of the video frame is determined according to the number of times the reference frame is referenced and the number of coded blocks in the video frame. The tag corresponding to the video frame is determined based on the reference ratio corresponding to each reference frame in the reference frame set.

2. The method according to claim 1, wherein, The method employs machine learning, using the video frame sequence as input and the labels corresponding to the input video frame sequence as the expected output, to train a reference frame selection model, including: Using machine learning methods, for each video frame in the video frame sequence, the reference frame selection model is trained by taking the video frame as input and the label corresponding to the video frame as the expected output.

3. The method according to claim 2, wherein, The method employs machine learning, whereby for each video frame in the video frame sequence, the reference frame selection model is trained using that video frame as input and its corresponding label as the desired output. This training includes: Based on a preset feature set, feature datasets are extracted for each video frame in the video frame sequence, wherein the preset feature set includes multiple feature types that indicate the video frames in the video frame sequence to be feature collected; Using machine learning methods, for each video frame in the video frame sequence, the reference frame selection model is trained by taking the feature dataset corresponding to the video frame as input and the label corresponding to the video frame as the expected output.

4. The method according to claim 3, wherein, Also includes: During training, the importance of each feature in the feature dataset is determined; as well as The method employs machine learning, whereby for each video frame in the video frame sequence, the reference frame selection model is trained using the feature dataset corresponding to that video frame as input and the label corresponding to that video frame as the expected output. This training includes: Based on the importance of the feature data determined during the training process, target feature data is selected from the feature dataset; Using machine learning methods, for each video frame in the video frame sequence, the target feature dataset corresponding to the video frame is used as input and the label corresponding to the video frame is used as the expected output to train the reference frame selection model.

5. A method for filtering reference frames, comprising: Obtain the target video sequence; For each video frame in the target video sequence, the target reference frame corresponding to the video frame is selected from the candidate reference frame set corresponding to the video frame by a pre-trained reference frame selection model, thereby obtaining the target reference frame set. The reference frame selection model is trained by any one of claims 1-4.

6. The method according to claim 5, wherein, The pre-trained reference frame filtering model filters out the target reference frame corresponding to the video frame from the candidate reference frame set corresponding to the video frame, resulting in a target reference frame set, including: In response to determining that the number of target reference frames in the target reference frame set does not exceed a preset number threshold, the following determination operation is performed cyclically: Candidate reference frames are extracted from the candidate reference frame set corresponding to the video frame, and target feature data corresponding to the candidate reference frames are extracted. The target feature data is input into the reference frame filtering model to determine whether the candidate reference frame is the target reference frame corresponding to the video frame. In response to the determination that yes, the candidate reference frame is added to the target reference frame set.

7. A training device for a reference frame selection model, comprising: The first acquisition unit is configured to acquire a training sample set, wherein the training samples in the training sample set include a video frame sequence and labels corresponding to the video frames in the video frame sequence, and the labels are used to characterize whether the video frame corresponding to the label is a reference frame for other video frames in the video frame sequence. The training unit is configured to use machine learning methods, taking the video frame sequence as input and the labels corresponding to the input video frame sequence as the expected output, to train a reference frame selection model. The second determining unit is configured to determine the tag corresponding to each video frame in the video frame sequence in the following manner: In response to the completion of the encoding operation of the video frame based on the encoding blocks of a preset size, the number of times each reference frame in the reference frame set corresponding to the video frame is referenced in the encoding operation of the video frame is determined; for each reference frame in the reference frame set, the reference ratio of the reference frame in the encoding operation of the video frame is determined according to the number of times the reference frame is referenced and the number of the encoding blocks in the video frame; and the tag corresponding to the video frame is determined according to the reference ratio of each reference frame in the reference frame set.

8. The apparatus according to claim 7, wherein, The training unit is further configured as follows: Using machine learning methods, for each video frame in the video frame sequence, the reference frame selection model is trained by taking the video frame as input and the label corresponding to the video frame as the expected output.

9. The apparatus according to claim 8, wherein, The training unit is further configured as follows: Based on a preset feature set, feature datasets are extracted for each video frame in the video frame sequence. The preset feature set includes multiple feature types that indicate the video frames in the video frame sequence to be feature collected. Using a machine learning method, for each video frame in the video frame sequence, the reference frame selection model is trained by taking the feature dataset corresponding to the video frame as input and the label corresponding to the video frame as the expected output.

10. The apparatus according to claim 9, wherein, Also includes: The first determining unit is configured to determine the importance of each feature data in the feature dataset during the training process; as well as The training unit is further configured as follows: Based on the importance of the feature data determined during training, target feature data is selected from the feature dataset; using machine learning methods, for each video frame in the video frame sequence, the target feature dataset corresponding to that video frame is used as input, and the label corresponding to that video frame is used as the expected output to train the reference frame selection model.

11. An apparatus for filtering reference frames, comprising: The second acquisition unit is configured to acquire the target video sequence; The filtering unit is configured to, for each video frame in the target video sequence, filter out the target reference frame corresponding to the video frame from the candidate reference frame set corresponding to the video frame through a pre-trained reference frame filtering model, thereby obtaining a target reference frame set, wherein the reference frame filtering model is trained by any one of claims 7-10.

12. The apparatus according to claim 11, wherein, The filtering unit is further configured to: In response to determining that the number of target reference frames in the target reference frame set does not exceed a preset number threshold, the following determination operation is performed cyclically: Candidate reference frames are extracted from the candidate reference frame set corresponding to the video frame, and target feature data corresponding to the candidate reference frames is extracted; the target feature data is input into the reference frame filtering model to determine whether the candidate reference frame is the target reference frame corresponding to the video frame; In response to the determination that yes, the candidate reference frame is added to the target reference frame set.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

15. A computer program product comprising: A computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images