A data processing method, device, equipment and readable storage medium

Abstract

The present application discloses a data processing method, device, equipment and readable storage medium, the method comprising: inputting sample video data into a video coding application deployed and updated for the kth time, generating first training data through the video coding application deployed and updated for the kth time and the sample video data; training a filter model to be trained in the video coding application deployed and updated for the kth time based on the sample original video frame and the first sample to be filtered and reconstructed frame in the first training data to obtain a second filter model, updating and deploying the second filter model in the video coding application deployed and updated for the kth time to obtain a video coding application deployed and updated for the k+1th time; when the video coding application deployed and updated for the k+1th time meets the filtering quality requirement condition, determining the video coding application deployed and updated for the k+1th time as the target video coding application. By adopting the present application, filtering performance can be improved, image distortion of the encoded video can be reduced, and image quality of the encoded video can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, device and readable storage medium. Background Art

[0002] At present, in the process of encoding and compressing videos, video encoding and compression algorithms all adopt lossy compression methods, that is, there is a certain distortion between the image of the encoded compressed video and the image of the original video, and when the compression rate is high, the distortion and distortion of the image are more serious.

[0003] The existing video coding standards have introduced loop filters to filter the compressed video images, thereby reducing the degree of distortion and making the image quality of the compressed video infinitely close to the image quality of the original video. However, since the traditional loop filter mainly designs the filter coefficients manually, it is too dependent on manual experience, the accuracy is not high, and it cannot effectively reduce the degree of distortion. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device and readable storage medium, which can improve filtering performance, reduce image distortion of encoded video, and improve image quality of encoded video.

[0005] On the one hand, an embodiment of the present application provides a data processing method, including:

[0006] The sample video data is input into the video encoding application including the first filtering model that is deployed and updated for the kth time, and the first training data is generated through the video encoding application that is deployed and updated for the kth time and the sample video data; the first training data includes the sample original video frame as a training label, and the first sample reconstructed frame to be filtered corresponding to the sample original video frame; the first sample reconstructed frame to be filtered refers to a reconstructed frame that has not been filtered by the first filtering model during the process of reconstructing the sample original video frame through the video encoding application that is deployed and updated for the kth time; the sample original video frame is a video frame in the sample video data; k is a positive integer;

[0007] Based on the sample original video frame and the first sample to-be-filtered reconstructed frame, the filter model to be trained in the video coding application deployed and updated for the kth time is trained to obtain a second filter model in a training convergence state, and the second filter model is updated and deployed in the video coding application deployed and updated for the kth time to obtain a video coding application deployed and updated for the k+1th time;

[0008] When the video encoding application updated and deployed for the k+1th time meets the filtering quality requirement condition, the video encoding application updated and deployed for the k+1th time is determined as a target video encoding application for performing video encoding processing on the video data.

[0009] On the one hand, an embodiment of the present application provides another data processing method, including:

[0010] The video data is input into the target video coding application, and the video data is subjected to video coding processing by the target video coding application, and the video data corresponds to a video compression code stream; the target video coding application refers to the video coding application updated for the k+1th deployment that meets the filtering quality requirement; the video coding application updated for the k+1th deployment includes a second filtering model in a training convergence state; the second filtering model is obtained by training the filtering model to be trained in the video coding application updated for the kth deployment that includes the first filtering model based on the sample original video frame as a training label in the first training data and the first sample to-be-filtered reconstructed frame corresponding to the sample original video frame; the first training data is generated by the video coding application updated for the kth deployment and the sample video data; the first sample to-be-filtered reconstructed frame refers to a reconstructed frame that has not been filtered by the first filtering model during the process of reconstructing the sample original video frame by the video coding application updated for the kth deployment; the sample original video frame is a video frame in the sample video data; k is a positive integer;

[0011] The video compression code stream is sent to a receiving device so that the receiving device decodes the video compression code stream.

[0012] An embodiment of the present application provides a data processing device, including:

[0013] A training data generation module, configured to input sample video data into a video encoding application including a first filtering model that is deployed and updated for the kth time;

[0014] The training data generation module is further used to generate first training data through the video encoding application updated by the k-th deployment and the sample video data; the first training data includes a sample original video frame as a training label, and a first sample to-be-filtered reconstructed frame corresponding to the sample original video frame; the first sample to-be-filtered reconstructed frame refers to a reconstructed frame that has not been filtered by the first filtering model during the process of reconstructing the sample original video frame through the video encoding application updated by the k-th deployment; the sample original video frame is a video frame in the sample video data; k is a positive integer;

[0015] A model training module, used for training the filter model to be trained in the video coding application deployed and updated for the kth time based on the sample original video frame and the first sample to-be-filtered reconstructed frame, to obtain a second filter model in a training convergence state;

[0016] An application updating module, configured to update and deploy the second filter model in the video coding application updated and deployed for the kth time, to obtain the video coding application updated and deployed for the k+1th time;

[0017] The target application determination module is used to determine the video coding application deployed and updated for the k+1th time as the target video coding application for performing video coding processing on the video data when the video coding application deployed and updated for the k+1th time meets the filtering quality requirement condition.

[0018] In one embodiment, the first filtering model includes a first intra-frame filtering model for filtering a reconstructed frame to be filtered that belongs to an intra-frame prediction type; the first intra-frame filtering model is trained based on an initial intra-frame filtering model in a video coding application updated for a k-1th deployment, and the video coding application updated for a kth deployment also includes an untrained initial inter-frame filtering model; the initial inter-frame filtering model is used to filter the reconstructed frame to be filtered that belongs to an inter-frame prediction type, and the filtering model to be trained in the video coding application updated for a kth deployment includes the initial inter-frame filtering model and the first intra-frame filtering model;

[0019] The model training module includes:

[0020] A video label acquisition unit is used to acquire an inter-frame reconstructed frame to be filtered that belongs to the inter-frame prediction type in the first sample inter-frame reconstructed frame to be filtered, and use the sample original video frame corresponding to the inter-frame reconstructed frame to be filtered in the sample original video frame as the first video frame label;

[0021] The video label acquisition unit is further used to acquire a first intra-frame reconstructed frame to be filtered that belongs to the intra-frame prediction type in the first sample intra-frame reconstructed frame to be filtered, and use the sample original video frame corresponding to the first intra-frame reconstructed frame to be filtered in the sample original video frame as the second video frame label;

[0022] A model training unit, used to train an initial inter-frame filtering model based on the inter-frame reconstructed frame to be filtered and the first video frame label, to obtain a first inter-frame filtering model in a training convergence state;

[0023] The model training unit is further used to train the first intra-frame filtering model based on the reconstructed frame in the frame to be filtered and the second video frame label to obtain a second intra-frame filtering model in a training convergence state;

[0024] The model determination unit is used to determine the first inter-frame filtering model and the second intra-frame filtering model as the second filtering model.

[0025] In one embodiment, the application update module includes:

[0026] An intra-frame model replacement unit, configured to replace and update the first intra-frame filter model in the video coding application deployed and updated for the kth time with a second intra-frame filter model;

[0027] The inter-frame model replacement unit is used to replace and update the initial inter-frame filtering model in the video encoding application deployed and updated for the kth time with the first inter-frame filtering model.

[0028] In one embodiment, the first filtering model includes a first intra-frame filtering model for filtering the reconstructed frame to be filtered that belongs to the intra-frame prediction type; the first intra-frame filtering model is trained based on the initial intra-frame filtering model in the video coding application updated for the k-1th deployment, and the video coding application updated for the kth deployment also includes an untrained first-type initial inter-frame filtering model and a second-type initial inter-frame filtering model; the first-type initial inter-frame filtering model is used to filter the reconstructed frame to be filtered that belongs to the first inter-frame prediction type, and the second-type initial inter-frame filtering model is used to filter the reconstructed frame to be filtered that belongs to the second inter-frame prediction type;

[0029] The model training module includes:

[0030] A to-be-trained model determination unit, used to determine the first intra-frame filter model and the first type of initial inter-frame filter model in the video coding application deployed and updated for the kth time as the to-be-trained filter model of the video coding application deployed and updated for the kth time;

[0031] a video frame label determination unit, configured to obtain a first type of inter-frame reconstructed frame to be filtered belonging to a first inter-frame prediction type in a first sample inter-frame reconstructed frame to be filtered, and use the sample original video frame corresponding to the first type of inter-frame reconstructed frame to be filtered in the sample original video frame as a third video frame label;

[0032] The video frame label determination unit is further configured to obtain a second intra-frame reconstructed frame to be filtered that belongs to the intra-frame prediction type in the first sample intra-frame reconstructed frame to be filtered, and use the sample original video frame corresponding to the second intra-frame reconstructed frame to be filtered in the sample original video frame as a fourth video frame label;

[0033] An inter-frame model training unit, configured to train a first-type initial inter-frame filtering model based on a first-type inter-frame reconstructed frame to be filtered and a third video frame label, to obtain a first-type inter-frame filtering model in a training convergence state;

[0034] An intra-frame model training unit, configured to train the first intra-frame filtering model based on the reconstructed frame within the second frame to be filtered and the fourth video frame label, to obtain a second intra-frame filtering model in a training convergence state;

[0035] The filtering model determining unit is used to determine the first type of inter-frame filtering model and the second intra-frame filtering model as the second filtering model.

[0036] In one embodiment, the data processing device further includes:

[0037] A data generation module is used to generate second training data through the video coding application deployed and updated for the k+1th time and the sample video data when the video coding application deployed and updated for the k+1th time does not meet the filtering quality requirement condition; the second training data includes the sample original video frame as a training label, and the second sample to-be-filtered reconstructed frame corresponding to the sample original video frame, and the second sample to-be-filtered reconstructed frame refers to a video frame that has not been filtered by the second filtering model during the process of reconstructing the sample original video frame through the video coding application deployed and updated for the k+1th time;

[0038] A filter model training module, used to train the filter model to be trained in the video coding application deployed and updated for the k+1th time based on the sample original video frame and the second sample to-be-filtered reconstructed frame, to obtain a third filter model in a training convergence state;

[0039] A deployment model module is used to update and deploy a third filter model in the video coding application that is updated and deployed for the k+1th time, so as to obtain a video coding application that is updated and deployed for the k+2th time;

[0040] The target application determination module is used to determine the video coding application deployed and updated for the k+2th time as the target video coding application when the video coding application deployed and updated for the k+2th time meets the filtering quality requirement condition.

[0041] In one embodiment, the filter model to be trained in the video coding application for the k+1th deployment update includes a second type of initial inter-frame filter model;

[0042] The filter model training module includes:

[0043] a label determination unit, configured to obtain a second type of inter-frame reconstructed frame to be filtered belonging to a second inter-frame prediction type in the second sample inter-frame reconstructed frame to be filtered, and use the sample original video frame corresponding to the second type of inter-frame reconstructed frame to be filtered in the sample original video frame as a fifth video frame label;

[0044] The second type inter-frame model training unit is used to train the second type initial inter-frame filtering model based on the second type inter-frame reconstructed frame to be filtered and the fifth video frame label to obtain the second type inter-frame filtering model in a training convergence state;

[0045] The filtering model determining unit is used to determine the second type of inter-frame filtering model as the third filtering model.

[0046] In one embodiment, the model training module includes:

[0047] A filtering frame output unit, used for inputting the first sample to-be-filtered reconstructed frame into the first filtering model, and outputting a sample filtering reconstructed frame corresponding to the first sample to-be-filtered reconstructed frame through the first filtering model;

[0048] A parameter adjustment unit, used to determine an error value between a sample filtered reconstructed frame and a sample original video frame;

[0049] The parameter adjustment unit is further used to adjust the model parameters of the first filtering model according to the error value to obtain the first filtering model with the adjusted model parameters;

[0050] The parameter adjustment unit is further used to determine the first filtering model with the adjusted model parameters as the second filtering model in a training convergence state when the first filtering model with the adjusted model parameters meets the model convergence condition.

[0051] In one embodiment, the parameter adjustment unit includes:

[0052] A function acquisition subunit, used to acquire the loss function corresponding to the video encoding application updated by the k-th deployment;

[0053] an error value determination subunit, used for obtaining the original image quality corresponding to the original video frame of the sample based on the loss function, and using the original image quality as an image quality label;

[0054] The error value determination subunit is also used to obtain the filtered image quality corresponding to the sample filtered reconstructed frame based on the loss function, and determine the error value between the sample filtered video frame and the sample original video frame through the loss function, the image quality label and the filtered image quality.

[0055] In one embodiment, the loss function includes an absolute value loss function and a square error loss function;

[0056] The error value determination subunit is further specifically used to determine a first error value between a sample filtered reconstructed frame and a sample original video frame by using a square error loss function, an image quality label, and a filtered image quality;

[0057] The error value determination subunit is further specifically used to determine a second error value between the sample filtered reconstructed frame and the sample original video frame through an absolute value loss function, an image quality label, and a filtered image quality;

[0058] The error value determination subunit is further specifically used to perform operation processing on the first error value and the second error value, and determine the result of the operation as the error value between the sample filtered reconstructed frame and the sample original video frame.

[0059] In one embodiment, the data processing device further comprises:

[0060] A filtered video output module, used for inputting sample video data into the video encoding application updated for the k+1th deployment, and outputting the filtered video data to be detected through the video encoding application updated for the k+1th deployment;

[0061] A video quality acquisition module, used to acquire the filtered video quality corresponding to the filtered video data to be detected, and the original video quality corresponding to the sample video data;

[0062] The application detection module is used to detect the video encoding application updated by the k+1th deployment according to the filtered video quality and the original video quality.

[0063] In one embodiment, the application detection module includes:

[0064] a difference quality determination unit, configured to determine a difference video quality between a filtered video quality and an original video quality;

[0065] a detection result determination unit, configured to determine that the second video encoding application meets a filter quality requirement condition if the difference video quality is less than a difference quality threshold;

[0066] The detection result determination unit is further configured to determine that the second video encoding application does not meet the filtering quality requirement condition if the difference video quality is greater than the difference quality threshold.

[0067] On the one hand, an embodiment of the present application provides another data processing device, including:

[0068] The bitstream generation module is used to input the video data into the target video coding application, perform video coding processing on the video data through the target video coding application, and obtain a video compression bitstream corresponding to the video data; the target video coding application refers to the video coding application updated for the k+1th deployment that meets the filtering quality requirement condition; the video coding application updated for the k+1th deployment includes a second filtering model in a training convergence state; the second filtering model is obtained by training the filtering model to be trained in the video coding application updated for the kth deployment that includes the first filtering model based on the sample original video frame as a training label in the first training data and the first sample to-be-filtered reconstructed frame corresponding to the sample original video frame; the first training data is generated by the video coding application updated for the kth deployment and the sample video data; the first sample to-be-filtered reconstructed frame refers to a reconstructed frame that has not been filtered by the first filtering model during the process of reconstructing the sample original video frame through the video coding application updated for the kth deployment; the sample original video frame is a video frame in the sample video data; k is a positive integer;

[0069] The code stream sending module is used to send the video compression code stream to the receiving device so that the receiving device can decode the video compression code stream.

[0070] An embodiment of the present application provides a computer device, including: a processor and a memory;

[0071] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method in the embodiment of the present application.

[0072] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the method in the embodiment of the present application is executed.

[0073] In one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in one aspect of the embodiments of the present application.

[0074] In an embodiment of the present application, the sample video data can be input into a video coding application (a video coding application deployed with a first filtering model) that is deployed and updated for the kth time (e.g., the 1st, 2nd, 3rd, etc.), and the first training data corresponding to the sample video data can be output through the video coding application that is deployed and updated for the kth time; the first filtering model can be retrained based on the first training data to obtain a second filtering model, and the second filtering model can be used to deploy and update the video coding application that is deployed and updated for the kth time again, so as to obtain a video coding application that is deployed and updated for the k+1th time; at this time, the video coding application that is deployed and updated for the k+1th time can be detected, and when the video coding application that is deployed and updated for the k+1th time meets the filtering quality requirement conditions, the video coding application that is deployed and updated for the k+1th time can be determined as the target video coding application for performing video encoding processing on the video data. It should be understood that deploying the filter model (for example, the first filter model) to the video coding application is a deployment update of the video coding application. After generating training data based on the video coding application, the filter model can be trained based on machine learning, and then the trained filter model can be deployed and updated to the video coding application to obtain an updated video coding application; then, the present application can continue to generate new training data based on the updated video coding application, and then train the filter model again based on the new training data, and then the trained filter model can be deployed and updated to the video coding application again, and the video coding application can be updated again until the video coding meets the filtering quality requirement. That is to say, the present application can iteratively update the video coding application after deploying the filter model to the video coding application, repeat the training process of the filter model by updating the training data, and continuously deploy and update the video coding application, which can improve the consistency between the training effect and the test effect of the filter model, improve the coding efficiency, and can also improve the filtering quality of the filter model deployed in the video coding application without relying on manual experience, and reduce the distortion of the encoded video. In summary, the present application can improve filtering performance, reduce image distortion of encoded video, and improve image quality of encoded video. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0076] Figure 1 This is a network architecture diagram provided by an embodiment of the present application;

[0077] Figure 2 It is a method flow diagram of a data processing method provided by an embodiment of the present application;

[0078] Figure 3 is a schematic diagram of a coding reference relationship provided in an embodiment of the present application;

[0079] Figure 4 is a schematic diagram of a coding reference relationship provided in an embodiment of the present application;

[0080] Figure 5 It is a system architecture diagram of iterative training provided in an embodiment of the present application;

[0081] Figure 6 It is a flowchart of a training filter model provided in an embodiment of the present application;

[0082] Figure 7 It is a schematic diagram of a scenario of model training provided in an embodiment of the present application;

[0083] Figure 8 It is a schematic diagram of a scenario of model training provided in an embodiment of the present application;

[0084] Fig. 9 It is a flowchart of a training filter model provided in an embodiment of the present application;

[0085] Fig.10 A schematic diagram of another model training scenario provided in an embodiment of the present application;

[0086] Fig.11 It is a method flow diagram of a data processing method provided by an embodiment of the present application;

[0087] Fig.12 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0088] Fig.13 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0090] See also Figure 1 , Figure 1 Schematic diagram of a network architecture provided by an embodiment of the present application. Figure 1 As shown, the network architecture may include a service server 1000 and a terminal device cluster. The terminal device cluster may include one or more terminal devices, and the number of terminal devices is not limited here. Figure 1 As shown, the multiple terminal devices may specifically include terminal device 100a, terminal device 100b, terminal device 100c, ..., terminal device 100n. Figure 1 As shown, the terminal device 100a, the terminal device 100b, the terminal device 100c, ..., the terminal device 100n can be respectively connected to the above-mentioned business server 1000 through a network, so that each terminal device can exchange data with the business server 1000 through the network connection. The network connection here does not limit the connection method, and can be directly or indirectly connected through a wired communication method, directly or indirectly connected through a wireless communication method, or through other methods, which are not limited in this application.

[0091] Each terminal device can be integrated with a target application. When the target application runs in each terminal device, it can be used together with the above Figure 1 Data interaction is performed between the business servers 1000 shown. Among them, the target application may include an application with data information functions such as displaying text, images, audio and video. Among them, the application may include social applications, multimedia applications (for example, video applications), entertainment applications (for example, game applications), educational applications, live broadcast applications and other applications with video encoding functions. Of course, the application can also be other applications with data information display functions and video encoding functions, which are not given examples one by one here. Among them, the application can be an independent application or an embedded sub-application integrated in an application (for example, a social application, an educational application, and a multimedia application, etc.), which is not limited here.

[0092] For ease of understanding, the embodiments of the present application can be Figure 1 Select one terminal device from the multiple terminal devices shown as the target terminal device. Figure 1 The terminal device 100a shown is used as a target terminal device, and a target application with a video encoding function may be integrated in the target terminal device. At this time, the target terminal device can realize data interaction with the service server 1000 through the service data platform corresponding to the application client.

[0093] It should be understood that the computer device with video encoding function in the embodiment of the present application (for example, terminal device 100a, service server 1000) can realize data encoding and data transmission of multimedia data (for example, video data) through cloud technology. Among them, cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local area network to realize data calculation, storage, processing, and sharing.

[0094] Cloud technology can be a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.

[0095] For example, the data processing method provided in the embodiment of the present application can be applied to high-resolution and high-frame-rate scenes such as video viewing scenes, video call scenes, video transmission scenes, cloud conference scenes, and live broadcast scenes. Among them, cloud conference is an efficient, convenient, and low-cost conference form based on cloud computing technology. Users only need to use the Internet interface to perform simple and easy-to-use operations to quickly and efficiently share voice, data files, and videos with teams and users in various places, while the complex technologies such as data transmission and processing in the conference are operated by cloud conference service providers to help users. At present, cloud conferences are mainly concentrated on service content based on the SaaS (Software as a Service) model, including telephone, network, video and other service forms. Video conferences based on cloud computing are called cloud conferences. In the era of cloud conferences, data transmission, processing, and storage are all handled by the computer resources of video conference manufacturers. Users no longer need to purchase expensive hardware and install cumbersome software. They only need to open a browser and log in to the corresponding interface to conduct efficient remote conferences. The cloud conference system supports dynamic cluster deployment of multiple servers and provides multiple high-performance servers, which greatly improves the stability, security, and availability of conferences. In recent years, video conferencing has been welcomed by many users because it can greatly improve communication efficiency, continuously reduce communication costs, and upgrade internal management levels. It has been widely used in various fields such as transportation, finance, operators, education, and enterprises. There is no doubt that after the use of cloud computing, video conferencing has a stronger appeal in terms of convenience, speed, and ease of use, which will surely stimulate the arrival of a new wave of video conferencing applications.

[0096] It should be understood that a computer device with a video encoding function can encode video data through a video encoder to obtain a video code stream corresponding to the video data, thereby improving the transmission efficiency of the video data. The video encoder can be an AV1 video encoder, an H.266 video encoder, an AVS3 video encoder, etc., and examples are not given one by one here. The video encoder needs to comply with the corresponding video encoding compression standard. For example, the video compression standard of the AV1 video encoder is the first generation video encoding standard developed by the Alliance for Open Media (AOM).

[0097] To facilitate understanding of the process of video encoder encoding video data, the specific process of video encoding is described below. The process of video encoding may include at least the following steps 1 to 5:

[0098] Step 1: Block partition structure of the video frame. It is understandable that after the video data is input into the video encoder, the video encoder can encode each video frame of the video data, and for a certain video frame, it can be divided into several non-overlapping processing units according to the size of the video frame, and each processing unit will perform similar compression operations. Among them, this processing unit can be called a coding tree unit (Coding TreeUint, CTU) or a largest coding unit (Largest Code Unit, LCU). For each CTU, a more refined division can be continued to obtain one or more basic coding units, which can be called a coding unit (CodingUint, CU). Among them, each CU is the most basic element in a coding link. Steps 2 to 4 below may describe various encoding methods that may be used for each CU.

[0099] Step 2: Predictive Coding of CU. It is understandable that after the video frame is divided into one or more coding units (i.e., CU), each coding unit can be predictively coded, wherein the prediction method for predicting the coding unit can include intra-frame prediction (the predicted signals all come from the areas that have been encoded and reconstructed in the same image) and inter-frame prediction (the predicted signals come from other images that have been encoded and are different from the current image (i.e., other video frames, which can be called reference frames after reference images)). It should be understood that the original video signal (which can be understood as the original video frame) can be predicted by the selected reconstructed video signal (which can be a video frame that has been decoded and reconstructed by the decoding end) to obtain a residual video signal (i.e., a residual value), and the encoder in the video encoding application can determine the most suitable one among many possible predictive coding methods for the current CU, and inform the decoder in the video encoding application of the predictive coding method for the current CU, so that the decoder decodes and reconstructs the current video frame to generate a reconstructed frame (the reconstructed frame can be used as a reference frame when the encoder performs predictive coding on a certain original video frame).

[0100] Step 3: Transform coding and quantization of the residual video signal (Transform & Quantization): The residual video signal obtained above can be transformed (such as discrete cosine transform (Discrete Cosine Transform, DCT)) and the residual video signal can be converted into the transform domain (which can be called transform coefficient) through the transform process. Subsequently, the signal in the transform domain can be further subjected to lossy quantization operation, losing certain information, so that the quantized signal is conducive to compression expression. Among them, in some video coding standards, one or more transform methods may be selected for transform processing. Therefore, the encoder also needs to select one of the transforms for the current encoding CU and inform the decoder. The degree of quantization is usually determined by the quantization parameter (Quantization Parameter, QP). The larger the QP value, the larger the coefficients of the larger value range will be quantized to the same output, which usually brings greater distortion and lower bit rate; on the contrary, the smaller the QP value, the smaller the coefficients of the smaller value range will be quantized to the same output, which usually brings less distortion and corresponds to a higher bit rate.

[0101] Step 4: Entropy coding or statistical coding of the quantized signal: It can be understood that the quantized transform domain signal obtained above can be statistically compressed and coded according to the frequency of occurrence of each value, and finally output a binary (including the value 0 or the value 1) compressed code stream. At the same time, other information generated by the encoding, such as the selected mode, motion vector, etc., also needs to be entropy coded to reduce the bit rate. Among them, statistical coding is a lossless coding method that can effectively reduce the bit rate required to express the same signal. Common statistical coding methods include variable length coding (VLC) or context-based binary arithmetic coding (CABAC).

[0102] Step 5: Loop Filtering operation: It can be understood that the encoded image can be subjected to inverse quantization, inverse transformation and prediction compensation operations (which can be understood as the reverse operations of 2 to 4 above), thereby obtaining a reconstructed decoded image. Compared with the original image, the reconstructed image has some information that is different from the original image due to the influence of quantization, resulting in distortion. The reconstructed image can be filtered, such as deblocking filter (DBF), SAO (Sample Adaptive Offset) or ALF (Adaptive Loop Filter) and other filters, which can effectively reduce the degree of distortion caused by quantization. The reconstructed image after filtering will be used as a reference for the subsequent encoding of a certain video frame and for predicting future signals. Therefore, the above filtering operation is also called loop filtering, and the filtering operation in the encoding loop.

[0103] It can be understood that in order to improve the video coding efficiency and the filtering performance of the filter deployed in the video encoder, the present application proposes a method for iteratively training the filter (also referred to as the filtering model) and iteratively updating the video encoder (also referred to as the video coding application). The specific method can be found in the subsequent Figure 2 The description in the corresponding embodiment.

[0104] It is understandable that the method provided in the embodiments of the present application can be executed by a computer device, including but not limited to a terminal device or a business server. Among them, the business server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0105] Optionally, it can be understood that the above-mentioned computer device (such as the above-mentioned business server 1000, terminal device 100a, terminal device 100b, etc.) can be a node in a distributed system, wherein the distributed system can be a blockchain system, and the blockchain system can be a distributed system formed by connecting the multiple nodes through network communication. Among them, the nodes can form a peer-to-peer (P2P, Peer To Peer) network, and the P2P protocol is an application layer protocol running on the transmission control protocol (TCP, Transmission Control Protocol) protocol. In a distributed system, any form of computer equipment, such as business servers, terminal devices and other electronic devices, can become a node in the blockchain system by joining the peer-to-peer network. For ease of understanding, the concept of blockchain will be explained below: Blockchain is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism and encryption algorithm, which is mainly used to organize data in chronological order and encrypt it into an account book so that it cannot be tampered with or forged, and can verify, store and update data at the same time. When the computer device is a blockchain node, due to the tamper-proof and anti-counterfeiting characteristics of the blockchain, the data in this application (such as video data, encoded video data, related parameters in the encoding process, etc.) can be authentic and secure, so that the results obtained after related data processing based on these data can be more reliable.

[0106] For further information, see Figure 2 , Figure 2 is a method flow diagram of a data processing method provided by an embodiment of the present application. The method may be performed by a terminal device (such as the above Figure 1 The method may also be executed by a service server (such as the above-mentioned Figure 1 The method may also be performed by the terminal device and the service server together. For example, the method is performed by the service server. Figure 2As shown, the method flow may at least include the following steps S101-S103:

[0107] Step S101, input the sample video data into the video encoding application that is deployed and updated for the kth time and includes the first filtering model, and generate the first training data through the video encoding application that is deployed and updated for the kth time and the sample video data; the first training data includes the sample original video frame as a training label, and the first sample reconstructed frame to be filtered corresponding to the sample original video frame; the first sample reconstructed frame to be filtered refers to the reconstructed frame that has not been filtered by the first filtering model during the process of reconstructing the sample original video frame through the video encoding application that is deployed and updated for the kth time; the sample original video frame is the video frame in the sample video data; k is a positive integer.

[0108] In the present application, sample video data may refer to video data used to train the filter model. A computer device (e.g., a terminal device) may obtain video data collected by an image collector (e.g., a camera of a terminal device) in a data transmission scenario. The video data may be used as sample video data, and the terminal device may input the sample video data into a video coding application. In other words, when a video coding application is deployed to a computer device, the computer device has a video coding function, the computer device may collect video data through an image collector, and the video coding application may also obtain the video data. It is understood that the video data here may also be any video data that needs to be coded in other scenarios. For example, the video data may be video data collected by a camera of a terminal device in an audio and video call scenario, the video data may also be video data stored in an album of a terminal device, or the video data may also be video data downloaded by a terminal device on the Internet, which is not limited here.

[0109] It is understandable that the video encoding application can be used to encode video data (for example, sample video data) to obtain a video code stream corresponding to the video data. When the video encoding application encodes the video data, it usually encodes each video frame in the video data. For example, the video encoding application can obtain a video frame to be encoded from the sample video data, and the video frame can be referred to as a target video frame. The video encoding application can perform image block division processing on the target video frame to obtain one or more image blocks of the target video frame, and can obtain a unit to be encoded from one or more image blocks, and perform prediction processing on the unit to be encoded. Among them, when the target video frame is encoded, the frame type to which the target video frame belongs can be obtained, and when the unit to be encoded is predicted, the unit to be encoded can be predicted based on the frame type to which the target video frame belongs. The frame type may include an intra-frame prediction type and an inter-frame prediction type. For example, the frame type of an intra-frame coded frame (intra picture, referred to as an I frame) is an intra-frame prediction type, and the frame type of a bi-directional interpolated prediction frame (B frame) and a predictive-frame (P frame) is an inter-frame prediction type. When the frame type of the target video frame is an intra-frame prediction type, when predicting the unit to be encoded, only the encoded and reconstructed area in the target video frame can be referenced (it should be noted that the intra-frame prediction type here may refer to the full intra-frame prediction type, that is, when predicting all the units to be encoded in the target video frame, only the area in the target video frame is referenced); and when the frame type of the target video frame is an inter-frame prediction type, when predicting the unit to be encoded, it is necessary to refer to other video frames that have been encoded and are different from the target video frame (which can be called reference video frames). When selecting the reference video frame of the target video frame, the selection can be made according to the frame type of the target video frame. For example, when the target video frame is a bidirectional prediction coding frame, the reference video frame for the target video frame can be the previous frame and the next frame of the target video frame; when the target video frame is a forward prediction coding frame, the reference video frame for the target video frame can be the previous frame of the target video frame. Figure 1 The description in the corresponding embodiment.

[0110] To understand the encoding dependencies between video frames, please refer to Figure 3 , Figure 3 Schematic diagram of a coding reference relationship provided by an embodiment of the present application. Figure 3In the coding configuration shown, the coding order of the video frames is consistent with the display order, the arrows point to the reference frames, and the numbers are used to represent the coding order. That is, the 0th frame (i.e., the I frame) can be encoded first, and then the B frame of the 1st frame can be encoded, wherein when encoding the B frame of the 1st frame, it is necessary to refer to the I frame of the 0th frame that has been reconstructed by encoding; then, the B frame of the 2nd frame can be encoded, and when encoding the B frame of the 2nd frame, it is necessary to refer to the B frame of the 1st frame that has been reconstructed by encoding. And so on, until the encoding process of all video frames is completed.

[0111] To understand the encoding dependencies between video frames, please refer to Figure 4 , Figure 4 Schematic diagram of a coding reference relationship provided by an embodiment of the present application. Figure 4 In the coding configuration shown, the coding order of the video frames is inconsistent with the display order, the arrows point to the reference frames, and the numbers are used to represent the coding order. That is, the 0th frame (i.e., the I frame) can be encoded first, and then the B frame of the 1st frame can be encoded, wherein when encoding the B frame of the 1st frame, it is necessary to refer to the I frame of the 0th frame that has been reconstructed by encoding; then, the B frame of the 2nd frame can be encoded, and when encoding the B frame of the 2nd frame, the arrows point to the 0th frame and the 1st frame, and it is necessary to refer to the 1st frame and the 0th frame that have been reconstructed by encoding. And so on, until the encoding process of all video frames is completed.

[0112] It can be understood that, from the foregoing, when encoding the video frames according to the video frame encoding sequence configured for encoding, for a certain video frame, the following can be used: Figure 1 The encoding process shown is to determine the reference frame according to the frame type of the frame, perform prediction processing on the reference frame to obtain the prediction unit, and then determine the residual value between the prediction unit and the unit to be encoded. Based on the prediction unit and the residual value, the reconstructed frame corresponding to the video frame can be determined (for example, the prediction unit and the residual value are added, and then filtered to obtain the reconstructed frame). The reconstructed frame can enter the reference frame queue to provide reference data for subsequent video frame encoding. It should be understood that the prediction unit can be obtained by performing prediction processing on the encoding unit of the video frame, and the residual value can be determined based on the prediction unit. Subsequently, the reconstructed image (or the pre-filtered reconstructed frame) can be determined based on the prediction unit and the residual value. The reconstructed image is filtered to obtain the filtered reconstructed frame corresponding to the video frame.

[0113] It can be understood that after the sample video data is input into the video encoding application, the video encoding application can encode and reconstruct each video frame to obtain a reconstructed frame corresponding to each video frame, and the present application can obtain the reconstructed frame that has not been filtered (which can be called the sample reconstructed frame to be filtered) during the encoding and reconstruction process, and each sample original video frame and its corresponding sample reconstructed frame to be filtered can be combined into a group of training data pairs, thereby obtaining multiple groups of training data pairs, and the multiple groups of training data pairs can constitute a training data set, which can be used to train the filtering model in the video encoding application (that is, the model used to filter the reconstructed frame after inverse transformation) so that the filtering model of the video encoding application can have higher filtering performance.

[0114] In the present application, when the filter model in the video coding application is trained and is in a model convergence state, the trained filter model can be deployed and updated (for example, replaced) for the filter model deployed in the video coding application, and the process of deploying and updating the filter model deployed in the video coding application can be understood as a deployment update of the video coding application. The present application can continuously generate new training data through the updated video coding application, and then use the new training data to repeatedly train the filter model, and then deploy and update the video coding application again based on the retrained filter model until the filtering performance of the video coding application meets the filtering quality requirements. For example, taking a video coding application that has not been deployed and updated as an example, the video coding application that has not been deployed and updated includes an initial filtering model that has not been trained. Sample video data can be input into the video coding application, and training data corresponding to the sample video data can be generated through the video coding application. The initial filtering model can be trained and adjusted through the training data to obtain a trained and adjusted filtering model; then, the trained and adjusted filtering model can be updated and deployed in the video coding application to obtain a video application that has been deployed and updated for the first time; then, new training data can be generated again through the video application that has been deployed and updated for the first time, and the filtering model can be trained and adjusted again based on the new training data to obtain a new trained filtering model, and then the trained and adjusted filtering model can be updated and deployed in the video coding application that has been deployed and updated for the first time to obtain a video coding application that has been deployed and updated for the second time, until the filtering performance of the video coding application meets the required conditions.

[0115] That is to say, taking the current video coding application as the video coding application that has been deployed and updated for the kth time (with the first filtering processing model deployed), and the video coding application that has been deployed and updated for the kth time not meeting the filtering quality requirement as an example, at this time, the sample video data can be input into the video coding application that has been deployed and updated for the kth time, and the training data (which can be referred to as the first training data) can be generated by the video coding application that has been deployed and updated for the kth time, and the first training data includes the sample original video frame as the training label, and the sample reconstructed frame to be filtered corresponding to the sample original video frame (which can be referred to as the first sample reconstructed frame to be filtered). The first training data can be used to train and adjust the first filtering model. Among them, it can be known from the above that the process of reconstructing the sample original video frame can include predicting and processing to obtain a prediction unit, determining a residual value based on the prediction unit, determining a reconstructed image (or called a reconstructed frame) based on the prediction unit and the residual value, and filtering the reconstructed image, then the sample reconstructed frame to be filtered can be understood as a reconstructed frame that has not been filtered in the process of reconstructing the sample original video frame. A set of training data pairs can be formed according to a sample original video frame and its corresponding sample reconstructed frame to be filtered, and the first training data can be formed according to multiple sets of training data pairs.

[0116] Step S102, based on the sample original video frame and the first sample reconstructed frame to be filtered, the filter model to be trained in the video coding application deployed and updated for the kth time is trained to obtain a second filter model in a training convergence state, and the second filter model is updated and deployed in the video coding application deployed and updated for the kth time to obtain a video coding application deployed and updated for the k+1th time.

[0117] In the present application, it can be known from the above that when encoding the video frames, the frame type to which each video frame belongs can be determined first, and then the encoding unit to be encoded can be subjected to prediction processing and other encoding processing based on the frame type. The filter model in the video encoding application in the present application can be used to filter the reconstructed frames belonging to the intra-frame prediction type (for example, the reconstructed frames corresponding to the I frame), and can also be used to filter the reconstructed frames belonging to the inter-frame prediction type (the reconstructed frames corresponding to the B frame). That is to say, the above-mentioned first filter model may refer to the filter model corresponding to the I frame and the B frame. In this case, the filter model to be trained may refer to the first filter model, and the first filter model may be trained based on the first training data to obtain the second filter model in the training convergence state. Subsequently, the first filter model in the video encoding application updated for the kth deployment can be replaced by the second filter model, so that the video encoding application updated for the k+1th deployment can be obtained.

[0118] Among them, for the filter model to be trained in the video coding application that is deployed and updated for the kth time based on the sample original video frame and the first sample to be filtered and reconstructed frame, the specific implementation method of obtaining the second filter model in the training convergence state can be: the first sample to be filtered and reconstructed frame can be input into the first filter model, and the sample filtered and reconstructed frame corresponding to the first sample to be filtered and reconstructed frame can be output through the first filter model; then, the error value between the sample filtered and reconstructed frame and the sample original video frame can be determined, and the model parameters of the first filter model can be adjusted according to the error value to obtain the first filter model with adjusted model parameters; when the first filter model with adjusted model parameters meets the model convergence condition, the first filter model with adjusted model parameters can be determined as the second filter model in the training convergence state. It should be noted that the model convergence condition here can refer to the number of training adjustments reaching the preset number of adjustments, or it can refer to the image quality of the filtered image output by the adjusted first filter model meeting the quality requirement condition. For example, taking the model convergence condition that the number of training adjustments reaches the preset number of adjustments as an example, assuming that the preset number of adjustments is 10 times, after adjusting the model parameters once based on the error value generated for the first time, the first sample to be filtered and reconstructed frame can be input into the first filtering model that has been adjusted once again, and the sample filtered and reconstructed frame can be output again, and a new error value is generated again, and the model parameters are adjusted again until the model parameters are adjusted 10 times. At this time, the first filtering model after 10 adjustments can be determined to meet the model convergence condition (that is, it is in a training convergence state).

[0119] Among them, the specific implementation method for determining the error value between the sample filtered reconstructed frame and the sample original video frame can be: the loss function corresponding to the video encoding application deployed and updated for the kth time can be obtained; then, the original image quality corresponding to the sample original video frame can be obtained based on the loss function, and the original image quality is used as the image quality label; then, the filtered image quality corresponding to the sample filtered reconstructed frame can be obtained based on the loss function, and the error value between the sample filtered video frame and the sample original video frame can be determined through the loss function, the image quality label and the filtered image quality.

[0120] Among them, it can be understood that in the present application, each iterative training of the filter model is performed independently, that is, after generating the training data, the loss function of the filter model trained based on the training data can be a different independent loss function. For example, when the filter model is trained based on the training data generated for the first time, the loss function L1 (absolute value loss function) can be used to determine the error value, and the model parameters are adjusted based on the error value; when the filter model is trained based on the training data generated for the second time, the loss function L2 (square error loss function) can be used to determine the error value, and the model parameters are adjusted based on the error value; when the filter model is trained based on the training data generated for the third time, an error value can be determined by first using the loss function L1, and then an error value can be determined by using the loss function L2, and the model parameters can be adjusted based on the error value of L1, and then the error value of L2 can be adjusted on the basis of the adjustment of L1; of course, when the filter model is trained with the training data generated subsequently, the error value of L1 and the error value of L2 can also be added, and the model parameters can be adjusted based on the total error value obtained by adding. There is no limitation on the loss function of each iteration.

[0121] Among them, for each loss function of the present application, in addition to the absolute value loss function L1 and the square error loss function L2, it can also be any other loss function that can determine the error between the label and the output data, for example, the loss function can include a cross entropy loss function, a mean square error loss function, a logarithmic loss function, an exponential loss function, etc. Taking the loss function corresponding to the video encoding application updated for the kth deployment as an example, including the absolute value loss function and the square error loss function, the specific implementation method for determining the error value between the sample filtered reconstructed frame and the sample original video frame through the loss function, the image quality label and the filtered image quality can be: the first error value between the sample filtered reconstructed frame and the sample original video frame can be determined through the square error loss function, the image quality label and the filtered image quality; then, the second error value between the sample filtered reconstructed frame and the sample original video frame can be determined through the absolute value loss function, the image quality label and the filtered image quality; then, the first error value and the second error value can be processed by operation (for example, by addition operation), and the result obtained by the operation (for example, the result of addition) can be determined as the error value between the sample filtered reconstructed frame and the sample original video frame.

[0122] Step S103: when the video coding application updated and deployed for the k+1th time meets the filtering quality requirement condition, the video coding application updated and deployed for the k+1th time is determined as a target video coding application for performing video coding processing on the video data.

[0123] In the present application, each time after the video coding application is deployed and updated, the video coding application can be detected to determine whether it meets the filtering quality requirement conditions. When it is determined that it meets the filtering quality requirement conditions, the video coding application does not need to be iteratively updated, and the video coding application can be put into use (for example, as a target video coding application to perform video coding processing on subsequent video data). Taking the detection of the video coding application deployed and updated for the k+1th time as an example, the specific implementation method for detecting the video coding application can be: the sample video data can be input into the video coding application deployed and updated for the k+1th time, and the video coding application deployed and updated for the k+1th time outputs the filtered video data to be detected; then, the filtered video quality corresponding to the filtered video data to be detected and the original video quality corresponding to the sample video data can be obtained; based on the filtered video quality and the original video quality, the video coding application deployed and updated for the k+1th time can be detected.

[0124] Among them, the specific implementation method of detecting the video coding application deployed and updated for the k+1th time based on the filtered video quality and the original video quality can be as follows: the difference video quality between the filtered video quality and the original video quality can be determined; if the difference video quality is less than the difference quality threshold, it can be determined that the second video coding application meets the filtering quality requirement conditions; and if the difference video quality is greater than the difference quality threshold, it can be determined that the second video coding application does not meet the filtering quality requirement conditions.

[0125] It can be understood that each filtered reconstructed frame output by the video encoding application updated and deployed for the k+1th time (the reconstructed frame after filtering by the second filtering model) can be obtained, and the filtered image quality of the filtered reconstructed frame can be compared with the original image quality of the corresponding original video frame. If the difference in quality between the two is less than a preset threshold, it can be determined that after the second filtering model is deployed in the video encoding application, the video frames processed by the model meet the filtering requirements.

[0126] It can be understood that when the video coding application updated by the k+1th deployment does not meet the filtering quality requirement, new training data can be generated again based on the video coding application updated by the k+1th deployment, and the second filtering model can be trained again based on the new training data to obtain a new filtering model (such as a third filtering model). Subsequently, the second filtering model deployed in the video coding application updated by the k+1th deployment can be updated to the third filtering model to obtain the video coding application updated by the k+2th deployment, and the updated video coding application can be deployed again for the k+3th time, and the application can be tested again until the video coding application meets the filtering quality requirement, thereby obtaining the target video coding application.

[0127] To understand the specific process of model training and application iterative update in this application, please refer to Figure 5 , Figure 5 : is a system architecture diagram of iterative training provided by an embodiment of the present application. Figure 5 As shown, the system architecture may include a data set generation module, a model training module, and an application integration module. For ease of understanding, each module will be described below:

[0128] Data set generation module: mainly used to generate training data sets (also called training data). It should be understood that the video data can be encoded and reconstructed through the video encoding application. During the encoding and reconstruction process, the data set generation module can use the reconstructed frame to be filtered (that is, the reconstructed frame that has not been filtered) and its corresponding original video frame as a set of training data pairs, thereby obtaining training data including multiple sets of training data pairs.

[0129] Model training module: mainly used to train the filter model. It should be understood that the training data generated by the above-mentioned data set generation module can be transmitted to the model training module, in which the filter model can be trained based on the training data so that the filter model can meet the training objectives.

[0130] Application integration module: mainly used to integrate and deploy the trained filter model into the video coding application. It should be understood that the application integration module can integrate the above-mentioned trained filter model deployment into the video coding application. At this time, the video data can be input into the video coding application integrated with the trained filter model to detect whether the video coding application can meet the filtering requirements in the actual application (i.e., whether the filtering performance of the application meets the preset requirements). If it is satisfied, the video coding application can be put into use. If it is not satisfied, the data set generation module can be used again to regenerate the training data set based on the new video coding application, and train the filter model again based on the new training data set, and then perform application integration, and then perform actual filtering processing until the video coding application meets the filtering requirements. That is to say, when the trained filter model meets the training target (i.e., meets the filtering requirements), after it is integrated and deployed into the video coding application, due to the different reference relationships between the coding frames of the inter-frame prediction type (inter-frame prediction coding frames), the reconstructed frames before filtering corresponding to the inter-frame prediction coding frames will be inconsistent with the reconstructed frames before filtering in the training process, and the results of training and testing are inconsistent. The filtering performance of the filter model in the video coding application will not be the expected effect. The present application regenerates the training data set each time an updated video coding application is deployed, trains the filter model again, and integrates it into the video coding application again to perform an iterative training process for actual filtering processing. This can continuously improve the consistency of training and testing until the filter model in the video coding application meets the filtering requirements, thereby improving coding efficiency.

[0131] It should be noted that when the data set generation module generates a training data set, the training data set can also be generated by software integrated with a filter of comparable performance. That is, for the current filter model, software integrated with a filter having comparable performance to the current model can be used to generate training data, and then the current filter model is trained based on the training data. For example, before the first iterative update of the application, a filter having comparable performance to the initial filter model that has not been trained (a traditional filter, such as a filter whose filter parameters are determined based on artificial experience) can be used to generate a training data set, and the training data set can be used to complete the model training process in the first iterative update (i.e., the training data set is used to train the initial filter model that has not been trained).

[0132] In an embodiment of the present application, the sample video data can be input into a video coding application (a video coding application deployed with a first filtering model) that is deployed and updated for the kth time (e.g., the 1st, 2nd, 3rd, etc.), and the first training data corresponding to the sample video data can be output through the video coding application that is deployed and updated for the kth time; the first filtering model can be retrained based on the first training data to obtain a second filtering model, and the second filtering model can be used to deploy and update the video coding application that is deployed and updated for the kth time again, so as to obtain a video coding application that is deployed and updated for the k+1th time; at this time, the video coding application that is deployed and updated for the k+1th time can be detected, and when the video coding application that is deployed and updated for the k+1th time meets the filtering quality requirement conditions, the video coding application that is deployed and updated for the k+1th time can be determined as the target video coding application for performing video encoding processing on the video data. It should be understood that deploying the filter model (for example, the first filter model) to the video coding application is a deployment update of the video coding application. After generating training data based on the video coding application, the filter model can be trained based on machine learning, and then the trained filter model can be deployed and updated to the video coding application to obtain an updated video coding application; then, the present application can continue to generate new training data based on the updated video coding application, and then train the filter model again based on the new training data, and then the trained filter model can be deployed and updated to the video coding application again, and the video coding application can be updated again until the video coding meets the filtering quality requirement. That is to say, the present application can iteratively update the video coding application after deploying the filter model to the video coding application, repeat the training process of the filter model by updating the training data, and continuously deploy and update the video coding application, which can improve the consistency between the training effect and the test effect of the filter model, improve the coding efficiency, and can also improve the filtering performance of the filter model deployed in the video coding application without relying on manual experience, and reduce the distortion of the encoded video. In summary, the present application can improve filtering performance, reduce image distortion of encoded video, and improve encoding efficiency.

[0133] From the above, it can be known that when encoding the video frames, the frame type to which each video frame belongs can be determined first, and then the encoding unit to be encoded can be subjected to prediction processing and other encoding processing based on the frame type. Since the prediction mode of the inter-frame prediction encoding frame (in the encoding process, each video frame can be called an encoding frame, and the inter-frame prediction encoding frame can be understood as an encoding frame whose frame type is the inter-frame prediction type, such as a B frame) is the inter-frame prediction mode, it can refer to other reconstructed frames for prediction, and the prediction accuracy is usually higher; while the prediction mode of the intra-frame prediction encoding frame (which can be understood as an encoding frame whose frame type is the intra-frame prediction type, such as an I frame) is the intra-frame prediction mode, which refers to other areas of the frame, and the prediction accuracy is usually lower than that of the inter-frame prediction encoding frame. In other words, for the filtering model, the image features before filtering of the intra-frame prediction encoding frame and the inter-frame prediction encoding frame (which can be understood as the Figure 1 In the corresponding process, the features of the reconstructed frame before filtering (which can be called the reconstructed frame to be filtered) will be different and there will be obvious differences. Therefore, if the same filtering model is used for unified filtering, there will be a certain impact on the image quality after filtering. So in order to make the filtering effect of the intra-frame prediction coding frame and the inter-frame prediction coding frame better, that is, to improve the image quality after filtering, the present application can train different filtering models for the intra-frame prediction coding frame and the inter-frame prediction coding frame, and use different filtering models to filter the reconstructed frame to be filtered corresponding to the intra-frame prediction coding frame and the reconstructed frame to be filtered corresponding to the inter-frame prediction coding frame. For example, the intra-frame filtering model is used to filter the reconstructed frame to be filtered corresponding to the intra-frame prediction coding frame, and the inter-frame filtering model is used to filter the reconstructed frame to be filtered corresponding to the inter-frame prediction coding frame.

[0134] It can be understood that when different filtering models are used to filter intra-frame prediction coding frames and inter-frame prediction coding frames, the video coding application in the present application may include an intra-frame filtering model for filtering reconstructed frames belonging to the intra-frame prediction type (reconstructed frames to be filtered corresponding to intra-frame prediction coding frames, for example, reconstructed frames corresponding to I frames), and an inter-frame filtering model for filtering reconstructed frames belonging to the inter-frame prediction type (reconstructed frames to be filtered corresponding to inter-frame prediction coding frames, for example, reconstructed frames corresponding to B frames). The present application can determine a model to be trained between the intra-frame filtering model and the inter-frame filtering model, and then train the model to be trained based on the training data. For example, taking the first filtering model deployed in the above-mentioned k-th deployed and updated video coding application as an example, including the first intra-frame filtering model for filtering the reconstructed frames to be filtered that belong to the intra-frame prediction type (the first intra-frame filtering model is trained based on the initial intra-frame filtering model in the video coding application deployed and updated for the k-1th time), and the video coding application deployed and updated for the kth time also includes an untrained initial inter-frame filtering model (the initial inter-frame filtering model is used to filter the reconstructed frames to be filtered that belong to the inter-frame prediction type), the present application can determine the filtering model to be trained in the video coding application deployed and updated for the kth time as the initial inter-frame filtering model and the first intra-frame filtering model, or can determine the filtering model to be trained as only the initial inter-frame filtering model.

[0135] For easier understanding, please also refer to Figure 6 , Figure 6 1 is a flow chart of a training filter model provided by an embodiment of the present application. The flow chart takes the filter model to be trained in the video coding application deployed and updated for the kth time, including the initial inter-frame filter model and the first intra-frame filter model, as an example, and trains the filter model to be trained in the video coding application deployed and updated for the kth time based on the sample original video frame and the first sample to be filtered reconstructed frame, and obtains the specific flow of the second filter model in the training convergence state. Figure 6 As shown, the process may include at least the following steps S501 to S505:

[0136] Step S501, obtaining an inter-frame reconstructed frame to be filtered that belongs to the inter-frame prediction type in a first sample inter-frame reconstructed frame to be filtered, and taking the sample original video frame corresponding to the inter-frame reconstructed frame to be filtered in the sample original video frame as the first video frame label.

[0137] Specifically, it can be known from the above that the training data generated by the video encoding application may include each sample original video frame and its corresponding sample reconstructed frame to be filtered, and in the process of encoding and reconstructing each sample original video frame, it is necessary to determine the frame type to which each sample original video frame belongs, and then perform prediction processing based on the frame type, so the obtained sample reconstructed frame to be filtered also corresponds to different frame types. For example, in the process of encoding and reconstructing the sample original video frame a, it is determined that the frame type of the encoded frame of the sample original video frame a is an intra-frame prediction type, then the frame type of the reconstructed frame corresponding to the sample original video frame a can also be understood as an intra-frame prediction type. In the present application, for the intra-frame filtering model, video frames with a frame type of intra-frame prediction type can be used for training, and for the inter-frame filtering model, video frames with a frame type of inter-frame prediction type can be used for training. Then, when the filtering model to be trained in the updated video encoding application is deployed for the kth time, including the initial inter-frame filtering model and the first intra-frame filtering model, for the initial inter-frame filtering model, a reconstructed frame to be filtered (which can be called an inter-frame reconstructed frame to be filtered) belonging to the inter-frame prediction type can be obtained in the first training data, and the sample original video frame corresponding to the inter-frame reconstructed frame to be filtered in the sample original video frame can be used as the first video frame label.

[0138] Step S502, obtaining a first intra-frame reconstructed frame to be filtered that belongs to the intra-frame prediction type in the first sample intra-frame reconstructed frame to be filtered, and taking the sample original video frame corresponding to the first intra-frame reconstructed frame to be filtered in the sample original video frame as the second video frame label.

[0139] Specifically, similarly, for the first intra-frame filtering model, a reconstructed frame belonging to the intra-frame prediction type (which can be called the first intra-frame reconstructed frame to be filtered) can also be obtained in the first training data, and the sample original video frame corresponding to the first intra-frame reconstructed frame to be filtered in the sample original video frame can be used as the second video frame label.

[0140] Step S503 , based on the inter-frame reconstructed frame to be filtered and the first video frame label, the initial inter-frame filtering model is trained to obtain a first inter-frame filtering model in a training convergence state.

[0141] Specifically, the initial inter-frame filtering model can be trained and adjusted through the inter-frame reconstructed frame to be filtered and its corresponding first video frame label to obtain an inter-frame filtering model in a training convergence state (which can be called a first inter-frame filtering model).

[0142] Step S504: Based on the reconstructed frame within the to-be-filtered frame and the second video frame label, the first intra-frame filtering model is trained to obtain a second intra-frame filtering model in a training convergence state.

[0143] Specifically, the first intra-frame filtering model can be trained and adjusted through the reconstructed frame within the first frame to be filtered and its corresponding second video frame label to obtain an intra-frame filtering model in a training convergence state (which can be called a second intra-frame filtering model).

[0144] Step S505: determine the first inter-frame filtering model and the second intra-frame filtering model as the second filtering model.

[0145] Specifically, the second intra-frame filtering model and the first inter-frame filtering model can be used as the second filtering model. Figure 3 The description of the training process for training the filter model in the corresponding embodiment will not be repeated here.

[0146] It should be noted that in the above Figure 2 In the corresponding embodiment, when the first filter model is a filter model corresponding to both the inter-frame prediction coding frame and the intra-frame prediction coding frame, the specific process of updating and deploying the second filter model in the video coding application updated for the kth deployment to obtain the video coding application updated for the k+1th deployment can be to replace only the first filter model with the second filter model. Then in this embodiment, when the filter model to be trained in the video coding application updated for the kth deployment includes the initial inter-frame filter model and the first intra-frame filter model, the specific method of updating and deploying the second filter model in the video coding application updated for the kth deployment to obtain the video coding application updated for the k+1th deployment can be: the first intra-frame filter model in the video coding application updated for the kth deployment can be replaced and updated with the second intra-frame filter model; the initial inter-frame filter model in the video coding application updated for the kth deployment can be replaced and updated with the first inter-frame filter model. That is to say, the two trained models are updated and replaced separately.

[0147] Optionally, it can be understood that when the video coding application updated for the kth deployment includes an initial inter-frame filtering model and a first intra-frame filtering model, and the initial inter-frame filtering model is an untrained filtering model, the model to be trained in the video coding application updated for the kth deployment can be determined as only the initial inter-frame filtering model, that is, the first intra-frame filtering model does not need to be trained again. After the initial inter-frame filtering model is trained and adjusted to obtain the first inter-frame filtering model through the inter-frame reconstruction frame to be filtered and the first video frame label in the above-mentioned first training data, the first inter-frame filtering model can be determined as the second filtering model. Subsequently, the initial inter-frame filtering model in the video coding application updated for the kth deployment can be replaced and updated with the first inter-frame filtering model to obtain the video coding application updated for the k+1th deployment. In the subsequent iterative update process of the video coding application, new training data can be generated based on the video coding application updated for the k+1th deployment, and then the above-mentioned first intra-frame filtering model and the first inter-frame filtering model are trained together based on the new training data, and then the video coding application updated for the k+1th deployment is deployed and updated based on the trained intra-frame or inter-frame filtering model.

[0148] It can be understood that, for the case where intra-frame prediction coding frames (i.e., coding frames or video frames belonging to the intra-frame prediction type, such as I frames) have a type of model (i.e., intra-frame filtering model, such as I frame model) and inter-frame prediction coding frames (i.e., coding frames or video frames belonging to the inter-frame prediction type, such as B frames) have a type of model, for the training of the filtering model, the following can be used: Figure 6 For easier understanding, see Figure 7 , Figure 7 Schematic diagram of a model training scenario provided in an embodiment of the present application. Figure 7 As shown, taking the intra-frame filtering model including the I-frame model and the inter-frame filtering model including the B-frame model as an example, in this case, the iterative training process of the model may include the following steps (1) to (4):

[0149] Step (1): When the video encoding application includes an initial I frame model and a B frame model that have not been trained, sample video data can be input into the video encoding application, and the training data can be output through the video encoding application.

[0150] Step (2): Take the initial I frame model in the video coding application as the model to be trained, train the I frame model based on the training data (train the I frame model based on the I frame original video frame and the I frame to be filtered and reconstructed frame in the training data), and obtain an I frame model in a training convergence state (i.e., the trained I frame model). Subsequently, the initial I frame model in the video coding application can be replaced with the trained I frame model, and the initial B frame model is kept unchanged, thereby obtaining the first deployment and update of the video coding application.

[0151] Step (3): input the sample video data into the video encoding application deployed and updated for the first time, and output new training data through the video encoding application deployed and updated for the first time.

[0152] Step (4): Take the initial B frame model in the video coding application that is deployed and updated for the first time as the model to be trained, and train the B frame model based on the new training data (train the B frame model based on the B frame original video frame and the B frame to be filtered and reconstructed frame in the training data), and obtain a B frame model in a training convergence state (i.e., a trained B frame model). The initial B frame model in the video coding application that is deployed and updated for the first time can be replaced with the trained B frame model to obtain a video coding application that is deployed and updated for the second time. It should be noted that when the B frame model is trained based on the new training data, the trained I frame model can also be trained at the same time to obtain a trained B frame model and a trained I frame model, and then the I frame model and the B frame model in the video coding application are updated and replaced respectively.

[0153] Step (5): Input the sample video data into the second deployment and update of the video coding application, and the second deployment and update of the video coding application can output new training data again. At this time, the I frame model and the B frame model in the first deployment and update of the video coding application can be trained again based on the new training data to obtain the trained I frame model and B frame model, and then the trained I frame model and B frame model are used to replace and update the models in the second deployment and update of the video coding application, respectively, to obtain the third deployment and update of the video coding application.

[0154] It should be noted that each time the video encoding application is deployed and updated, it can be tested on the application to determine whether it meets the filtering requirements. If the filtering requirements are not met, a new training data set is generated, and the model is trained again based on the new training data set. The video encoding application is deployed and updated again based on the trained model until the video encoding application meets the filtering requirements.

[0155] Optionally, in a feasible embodiment, the filter model to be trained in the video coding application updated for the kth deployment may include a first inter-frame filter model and a first intra-frame filter model, and the first inter-frame filter model is trained based on the initial inter-frame filter model in the video coding application updated for the k-1th deployment. That is to say, the present application may train the initial inter-frame filter model and the initial intra-frame filter model together to obtain the first inter-frame filter model and the first intra-frame filter model, and then respectively update and replace the initial inter-frame filter model and the initial intra-frame filter model in the video coding application updated for the k-1th deployment to obtain the video coding application updated for the kth deployment including the first inter-frame filter model and the first intra-frame filter model. For ease of understanding, please refer to Figure 8 , Figure 8 Schematic diagram of a model training scenario provided in an embodiment of the present application. Figure 8 As shown, taking the intra-frame filtering model including the I-frame model and the inter-frame filtering model including the B-frame model as an example, in this case, the iterative training process of the model may include the following steps (81) to (84):

[0156] Step (81): When the video encoding application includes an initial I frame model and a B frame model that have not been trained, sample video data can be input into the video encoding application, and training data can be output through the video encoding application.

[0157] Step (82): Take the initial I frame model in the video coding application as the model to be trained, train the I frame model based on the training data (train the I frame model based on the I frame original video frame in the training data and the I frame to be filtered and reconstructed frame), and train the B frame model based on the training data (train the B frame model based on the B frame original video frame in the training data and the B frame to be filtered and reconstructed frame), and obtain a B frame model in a training convergence state (i.e., a trained B frame model). Subsequently, the initial I frame model in the video coding application can be replaced with the trained I frame model, and the initial B frame model can be replaced with the trained B frame model, thereby obtaining the first deployment and update of the video coding application.

[0158] Step (83): input the sample video data into the video encoding application deployed and updated for the first time, and output new training data through the video encoding application deployed and updated for the first time.

[0159] Step (84): The I frame model and the B frame model in the video coding application that was deployed and updated for the first time are used as models to be trained (or only the B frame model can be used as the model to be trained), and the I frame model and the B frame model are trained based on the new training data to obtain an I frame model and a B frame model in a training convergence state. The model in the video coding application that was deployed and updated for the first time can be replaced and updated accordingly to obtain a video coding application that was deployed and updated for the second time.

[0160] After each deployment and update of the video encoding application, it can be checked for application to determine whether it meets the filtering requirements. If the filtering requirements are not met, a new training data set is generated, and the model is trained again based on the new training data set. The video encoding application is deployed and updated again based on the trained model until the video encoding application meets the filtering requirements.

[0161] It should be understood that, from the above, when encoding the video frames, the frame type to which each video frame belongs can be determined first, and then the encoding unit to be encoded can be subjected to prediction processing and other encoding processing based on the frame type. For inter-frame prediction coding frames (such as B frames), since they require other reconstructed frames as references for prediction coding, the coupling relationship of the reference relationship of the inter-frame prediction coding frames affects the consistency of training and actual testing. In other words, even if the trained filtering model is in a model convergence state and the training goal is achieved, after it is deployed to the video coding application, since the inter-frame prediction coding frames require forward reference or bidirectional reference, in the actual test process, the filtered reconstructed frame output by the video coding application is not the expected filtering effect. In other words, the results of training and testing are not equivalent. For example, for a certain B frame, its corresponding reference frame is the filtered reconstructed frame of the previous I frame. After the trained filtering model of the I frame and the filtering model of the B frame are deployed and updated to the video encoding application, the quality of the filtered reconstructed frame of the I frame will be improved. Then, due to the improvement in the quality of the filtered reconstructed frame of the I frame, the prediction result of the B frame will also be improved, that is, the characteristics of the B frame before filtering will be improved. Then, after being processed by the filtering model of the B frame, the filtering effect of the reconstructed frame of the B frame is not the filtering effect in the training process. That is, the training process and the testing process are not equivalent. By the iterative training method provided in the embodiment of the present application, the B frame model or the I frame model is repeatedly trained by updating the training data set, so that the consistency of the training and testing of the B frame can be improved, so that the filtering performance of the video encoding application can be improved while improving the encoding efficiency, and the distortion of the encoded video can be reduced.

[0162] In an embodiment of the present application, by repeatedly training the inter-frame filtering model by updating the training data set, the consistency in training and testing of the inter-frame prediction coding frames can be improved, thereby improving the coding efficiency, improving the filtering performance of the video coding application, and reducing the distortion of the encoded video.

[0163] Furthermore, it is understandable that the above Figure 6 The corresponding embodiment can be understood as a model training process in which the inter-frame prediction coding frame corresponds to only one type of filtering model (i.e., the inter-frame filtering model). In fact, some frames in the inter-frame prediction coding frame can be filtered by one type of filtering model (for example, taking B-1, B-3, and B-5 frames as an example, they can be filtered by the B-1 model), and some frames can be filtered by another type of filtering model (for example, taking B-2, B-4, and B-6 frames as an example, they can be filtered by the B-2 model). In other words, the inter-frame filtering model can include two There are two or more types of inter-frame filtering models (various inter-frame filtering models may refer to models with different network structures and model parameters; of course, various inter-frame filtering models may also refer to models with the same network structure but different model parameters). For the sake of distinction, the inter-frame prediction type (e.g., B frame type) may be subdivided into a first inter-frame prediction type (e.g., B-1, B-3, B-5 ​​frame types) and a second inter-frame prediction type (e.g., B-2, B-4, B-6 frame types), and a certain type of inter-frame filtering model may be referred to as a type inter-frame filtering model. For example, the inter-frame filtering model corresponding to the first inter-frame prediction type may be referred to as a first type inter-frame filtering model, and the inter-frame filtering model corresponding to the second inter-frame prediction type may be referred to as a second type inter-frame filtering model.

[0164] It can be understood that when different types of inter-frame filtering models are used to filter different inter-frame prediction coding frames, in the video coding application of the present application, it can include an intra-frame filtering model for filtering reconstructed frames belonging to the intra-frame prediction type (reconstructed frames to be filtered corresponding to intra-frame prediction coding frames, for example, reconstructed frames corresponding to I frames), a first type of inter-frame filtering model for filtering reconstructed frames belonging to the first inter-frame prediction type, and a second type of inter-frame filtering model for filtering reconstructed frames belonging to the second inter-frame prediction type. The present application can determine a model to be trained between the intra-frame filtering model and different types of inter-frame filtering models, and then train the model to be trained based on the training data. For example, taking the case where the first filtering model includes a first intra-frame filtering model for filtering the reconstructed frames to be filtered that belong to the intra-frame prediction type (the first intra-frame filtering model is trained based on the initial intra-frame filtering model in the video coding application updated for the k-1th deployment), and the video coding application updated for the kth deployment also includes an untrained first-type initial inter-frame filtering model and a second-type initial inter-frame filtering model (the first-type initial inter-frame filtering model is used to filter the reconstructed frames to be filtered that belong to the first inter-frame prediction type, and the second-type initial inter-frame filtering model is used to filter the reconstructed frames to be filtered that belong to the second inter-frame prediction type), the present application can first determine the filtering model to be trained in the video coding application updated for the kth deployment as the first intra-frame filtering model and the first-type initial inter-frame filtering model that has not been trained, or can determine the filtering model to be trained only as the first-type initial inter-frame filtering model that has not been trained.

[0165] For easier understanding, please also refer to Fig. 9 , Fig. 9 Schematic diagram of a flow chart of a training filter model provided in an embodiment of the present application. Fig. 9 As shown, the process may include at least the following steps S801 to S806:

[0166] Step S801, determining the first intra-frame filtering model and the first type of initial inter-frame filtering model in the video coding application deployed and updated for the kth time as the filtering models to be trained for the video coding application deployed and updated for the kth time.

[0167] Specifically, when determining the model to be trained each time, if there is a certain filter model (including an intra-frame filter model and an inter-frame filter model) that has not been trained, a certain filter model can be preferentially selected from the models that have not been trained as the model to be trained until all the filter models have been trained. At this time, when all the filter models have been trained, when determining the model to be trained, all the filter models that have been trained can be used as the models to be trained, and all the models are trained and adjusted simultaneously based on the new training data. Of course, when determining the model to be trained each time, if there is a certain filter model (including an intra-frame filter model and an inter-frame filter model) that has not been trained, a certain filter model can be preferentially selected from the models that have not been trained, and it can be used together with the trained models as the models to be trained. Optionally, when determining the model to be trained each time, if there is a certain filter model (including an intra-frame filter model and an inter-frame filter model) that has not been trained, all the filter models that have not been trained can also be used together as the models to be trained. Then in an embodiment of the present application, the first intra-frame filtering model in the video coding application that is deployed and updated for the kth time is a trained model, and the first type of initial inter-frame filtering model and the second type of initial inter-frame filtering model are both models that have not been trained. In an embodiment of the present application, the first intra-frame filtering model and the first type of initial inter-frame filtering model (or the second type of initial inter-frame filtering model) can be used together as models to be trained. Of course, only the first type of initial inter-frame filtering model (or the second type of initial inter-frame filtering model) can be used as the model to be trained; the first type of initial inter-frame filtering model and the second type of initial inter-frame filtering model can also be used together as models to be trained.

[0168] Step S802, obtaining a first type of inter-frame reconstructed frame to be filtered belonging to a first inter-frame prediction type in the first sample inter-frame reconstructed frame to be filtered, and using the sample original video frame corresponding to the first type of inter-frame reconstructed frame to be filtered in the sample original video frame as a third video frame label.

[0169] Specifically, for the intra-frame filtering model, video frames with a frame type of intra-frame prediction type can be used for training, and for the inter-frame filtering model, video frames with a frame type of inter-frame prediction type can be used for training. Then, when the filtering model to be trained in the video encoding application updated for the kth deployment includes a first type of initial inter-frame filtering model, a reconstructed frame to be filtered belonging to the first inter-frame prediction type (which can be referred to as a first type of inter-frame reconstructed frame to be filtered) can be obtained in the first training data, and the sample original video frame corresponding to the first type of inter-frame reconstructed frame to be filtered in the sample original video frame is used as its corresponding video frame label (which can be referred to as a third video frame label).

[0170] Step S803, obtaining a second intra-frame reconstructed frame to be filtered that belongs to the intra-frame prediction type in the first sample intra-frame reconstructed frame to be filtered, and taking the sample original video frame corresponding to the second intra-frame reconstructed frame to be filtered in the sample original video frame as a fourth video frame label.

[0171] Specifically, in the same way, when the filtering model to be trained in the updated video encoding application deployed for the kth time includes the first intra-frame filtering model, a reconstructed frame to be filtered (which can be referred to as the second intra-frame reconstructed frame to be filtered) belonging to the intra-frame prediction type can be obtained in the first training data, and the sample original video frame corresponding to the second intra-frame reconstructed frame to be filtered in the sample original video frame is used as its corresponding video frame label (which can be referred to as the fourth video frame label).

[0172] Step S804: Based on the first type of inter-frame reconstructed frame to be filtered and the third video frame label, the first type of initial inter-frame filtering model is trained to obtain a first type of inter-frame filtering model in a training convergence state.

[0173] Specifically, the first type of initial inter-frame filtering model can be trained and adjusted through the first type of inter-frame reconstructed frame to be filtered and the third video frame label to obtain the first type of inter-frame filtering model in a training convergence state.

[0174] Step S805 , training the first intra-frame filtering model based on the reconstructed frame within the second frame to be filtered and the fourth video frame label to obtain a second intra-frame filtering model in a training convergence state.

[0175] Specifically, the first intra-frame filtering model can be trained by using the reconstructed frame in the second frame to be filtered and the fourth video frame label to obtain a second intra-frame filtering model in a training convergence state.

[0176] Step S806: determine the first type of inter-frame filtering model and the second intra-frame filtering model as the second filtering model.

[0177] Specifically, the second intra-frame filtering model and the first type inter-frame filtering model can be used as the second filtering model. Figure 3 The description of the training process for training the filter model in the corresponding embodiment will not be repeated here.

[0178] It should be noted that when the model to be trained of the video coding application updated for the kth deployment includes the first type of initial inter-frame filtering model and the first intra-frame filtering model, the specific method for updating and deploying the second filtering model in the video coding application updated for the kth deployment to obtain the video coding application updated for the k+1th deployment can be: the first intra-frame filtering model in the video coding application updated for the kth deployment can be replaced and updated with the second intra-frame filtering model; the first type of initial inter-frame filtering model in the video coding application updated for the kth deployment can be replaced and updated with the first type of inter-frame filtering model. In other words, the two trained models are updated and replaced respectively.

[0179] Optionally, it can be understood that, in the case where the model to be trained of the video coding application updated in the k-th deployment includes a first type of initial inter-frame filtering model and a first intra-frame filtering model, when it is detected that the video coding application updated in the k+1-th deployment does not meet the filtering quality requirement conditions, the second training data can be generated through the video coding application updated in the k+1-th deployment and the sample video data; wherein the second training data includes a sample original video frame as a training label, and a second sample to-be-filtered reconstructed frame corresponding to the sample original video frame, and the second sample to-be-filtered reconstructed frame refers to the process of encoding and reconstructing the sample original video frame through the video coding application updated in the k+1-th deployment. In the process, a video frame that has not been filtered by the second filtering model is obtained; subsequently, the filtering model to be trained (for example, the second type of initial inter-frame filtering model) in the video coding application updated for the k+1th deployment can be trained based on the sample original video frame and the second sample reconstructed frame to be filtered, so as to obtain a third filtering model in a training convergence state; subsequently, the third filtering model can be updated and deployed in the video coding application updated for the k+1th deployment to obtain a video coding application updated for the k+2th deployment; when the video coding application updated for the k+2th deployment meets the filtering quality requirement condition, the video coding application updated for the k+2th deployment can be determined as the target video coding application.

[0180] Taking the second type of initial inter-frame filtering model as an example, the filtering model to be trained in the video coding application that is deployed and updated for the k+1th time is trained based on the sample original video frame and the second sample reconstructed frame to be filtered, and the specific implementation method of obtaining the third filtering model in the training convergence state can be: the second type of inter-frame reconstructed frame to be filtered belonging to the second inter-frame prediction type can be obtained in the second sample reconstructed frame to be filtered, and the sample original video frame corresponding to the second type of inter-frame reconstructed frame to be filtered in the sample original video frame is used as the fifth video frame label; the second type of initial inter-frame filtering model can be trained based on the second type of inter-frame reconstructed frame to be filtered and the fifth video frame label to obtain the second type of inter-frame filtering model in the training convergence state; the second type of inter-frame filtering model is determined as the third filtering model.

[0181] It can be understood that, for the case where intra-frame prediction coding frames (e.g., I frames) have one type of model (i.e., intra-frame filtering model, such as I frame model) and inter-frame prediction coding frames (e.g., B frames) have two or more types of models, for the training of the filtering model, the following can be used: Fig. 9 For easier understanding, please also refer to Fig.10 , Fig.10 A schematic diagram of another model training scenario provided in an embodiment of the present application. Fig.10 As shown, the intra-frame filtering model includes the I frame model, and the inter-frame filtering model includes the B-1 and B-2 models. In this case, combined with Fig.10 In the scenario shown, the process of iterative model training can include the following steps 1-6:

[0182] Step 1: When the video encoding application includes an initial I frame model, a B-1 model, and a B-2 model that have not been trained, sample video data can be input into the video encoding application, and the training data can be output through the video encoding application.

[0183] Step 2: Use the initial I frame model in the video coding application as the model to be trained, train the I frame model based on the training data (train the I frame model based on the I frame original video frame and the I frame to be filtered and reconstructed frame in the training data), and obtain an I frame model in a training convergence state (i.e., the trained I frame model). The initial I frame model in the video coding application can be replaced with the trained I frame model, keeping the initial B-1 and B-2 models unchanged, to obtain the first deployed updated video coding application.

[0184] Step 3: Input the sample video data into the video encoding application deployed and updated for the first time, and output new training data through the video encoding application deployed and updated for the first time.

[0185] Step 4: Use the initial B-1 model in the video coding application that is deployed and updated for the first time as the model to be trained, and train the initial B-1 model based on the new training data (train the B-1 model based on the original video frames corresponding to the B-1 model in the training data and the reconstructed frames to be filtered), and obtain a B-1 model in a training convergence state (the trained B-1 model). The initial B-1 model in the video coding application that is deployed and updated for the first time can be replaced with the trained B-1 model to obtain a video coding application that is deployed and updated for the second time.

[0186] Step 5: Input the sample video data into the video encoding application that is deployed and updated for the second time, and output new training data through the video encoding application that is deployed and updated for the second time.

[0187] Step 6: Take the initial B-2 model in the video coding application updated for the second deployment as the model to be trained, and train the B-2 model based on the new training data (train the B-2 model based on the original video frames corresponding to the B-2 model in the training data and the reconstructed frames to be filtered), and obtain a B-2 model in a training convergence state. The initial B-2 model in the video coding application updated for the second deployment can be replaced with the trained B-2 model to obtain a video coding application updated for the third deployment. Of course, the sample video data can then be input again into the video coding application updated for the third deployment, and new training data can be output again based on the video coding application updated for the third deployment. At this time, each filter model in the video coding application updated for the third deployment can be trained based on the new training data to obtain a trained model, and then the video coding application updated for the third deployment can be updated and deployed based on the trained model to obtain a video coding application updated for the fourth deployment.

[0188] Of course, it is understandable that in the above step 4, the initial B-1 model and the initial B-2 model can also be trained based on the training data. After obtaining the trained B-1 model and B-2 model, the initial B-1 model and B-2 model in the first deployment and update video coding application are respectively updated and replaced to obtain the second deployment and update video coding application. After the I frame model, B-1 model and B-2 model are trained, new training data can be generated again, and the I frame model, B-1 model and B-2 model can be trained together based on the new training data.

[0189] It should be noted that each time the video encoding application is deployed and updated, it can be tested on the application to determine whether it meets the filtering requirements. If the filtering requirements are not met, a new training data set is generated, and the model is trained again based on the new training data set. The video encoding application is deployed and updated again based on the trained model until the video encoding application meets the filtering requirements.

[0190] In an embodiment of the present application, by repeatedly training the inter-frame filtering model by updating the training data set, the consistency in training and testing of the inter-frame prediction coding frames can be improved, thereby improving the coding efficiency, improving the filtering quality of the video coding application, and reducing the distortion of the encoded video.

[0191] It should be noted that the model iterative training process proposed in this application can also be applied to the iterative training of other models, for example, iterative training of inter-frame prediction models and iterative training of intra-frame prediction models. In other words, the model training method proposed in this application, which regenerates training data based on the application after model deployment and repeatedly trains the model, and then updates and deploys the application until the application meets the performance requirements, is not limited to iterative training of the filter model.

[0192] It can be understood that after the target video encoding application is determined through the model iteration training method proposed in the embodiment of the present application, the target video encoding application can be put into use, that is, the target video encoding application can be used to perform video encoding processing on the video data. For example, in a video call scenario, when two users are making a video call, the terminal device corresponding to user a can perform video encoding processing on the video data associated with user a based on the target video encoding application, and after obtaining the video compression code stream, the video compression code stream is transmitted to the terminal device corresponding to user b (the user making a video call with user a), so that the terminal device corresponding to user b can decode and output the video data associated with user a in the display interface. For ease of understanding, please refer to Fig.11 , Fig.11 1 is a schematic diagram of a method flow of a data processing method provided by an embodiment of the present application, and the flow can be an application flow of a target video encoding application. Fig.11 As shown, the process may include at least the following steps S201-S202:

[0193] Step S201, inputting video data into a target video coding application, performing video coding processing on the video data through the target video coding application, and obtaining a video compression code stream corresponding to the video data; the target video coding application refers to a video coding application updated for the k+1th deployment that meets the filtering quality requirement; the video coding application updated for the k+1th deployment includes a second filtering model in a training convergence state; the second filtering model is obtained by training the filtering model to be trained in the video coding application updated for the kth deployment that includes the first filtering model based on the sample original video frame as a training label in the first training data and the first sample to-be-filtered reconstructed frame corresponding to the sample original video frame; the first training data is generated by the video coding application updated for the kth deployment and the sample video data; the first sample to-be-filtered reconstructed frame refers to a reconstructed frame that has not been filtered by the first filtering model during the process of reconstructing the sample original video frame through the video coding application updated for the kth deployment; the sample original video frame is a video frame in the sample video data; k is a positive integer.

[0194] For details on the specific process of determining the target video encoding application, please refer to the previous article Figure 2 The description in the corresponding embodiment will not be repeated here.

[0195] Step S202: sending the video compression code stream to a receiving device, so that the receiving device decodes the video compression code stream.

[0196] Specifically, the computer device may send the video compression code stream to a receiving device (eg, a terminal device that receives the video compression code stream), and the receiving device may decode the video compression code stream.

[0197] In an embodiment of the present application, by repeatedly training the inter-frame filtering model by updating the training data set, the consistency in training and testing of the inter-frame prediction coding frames can be improved, thereby improving the coding efficiency, improving the filtering quality of the video coding application, and reducing the distortion of the encoded video.

[0198] For further information, see Fig.12 , Fig.12 1 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The data processing device may be a computer program (including program code) running in a computer device, for example, the data processing device is an application software; the data processing device may be used to execute Figure 3 As shown in the method. Fig.12 As shown, the data processing device 1 may include: a training data generating module 11, a model training module 12, an application updating module 13 and a target application determining module 14.

[0199] A training data generating module 11, configured to input sample video data into a video encoding application including a first filtering model that is deployed and updated for the kth time;

[0200] The training data generating module 11 is further used to generate first training data through the video encoding application and the sample video data deployed and updated for the kth time; the first training data includes a sample original video frame as a training label, and a first sample to-be-filtered reconstructed frame corresponding to the sample original video frame; the first sample to-be-filtered reconstructed frame refers to a reconstructed frame that has not been filtered by the first filtering model during the process of reconstructing the sample original video frame through the video encoding application deployed and updated for the kth time; the sample original video frame is a video frame in the sample video data; k is a positive integer;

[0201] A model training module 12 is used to train the filter model to be trained in the video coding application deployed and updated for the kth time based on the sample original video frame and the first sample to-be-filtered reconstructed frame, to obtain a second filter model in a training convergence state;

[0202] An application updating module 13 is used to update and deploy the second filter model in the video coding application that is updated and deployed for the kth time, so as to obtain the video coding application that is updated and deployed for the k+1th time;

[0203] The target application determination module 14 is used to determine the video coding application deployed and updated for the k+1th time as the target video coding application for performing video coding processing on the video data when the video coding application deployed and updated for the k+1th time meets the filtering quality requirement condition.

[0204] The specific implementation of the training data generation module 11, the model training module 12, the application update module 13 and the target application determination module 14 can be found in the above Figure 2 The description of step S101 to step S103 in the corresponding embodiment will not be repeated here.

[0205] In one embodiment, the first filtering model includes a first intra-frame filtering model for filtering a reconstructed frame to be filtered that belongs to an intra-frame prediction type; the first intra-frame filtering model is trained based on an initial intra-frame filtering model in a video coding application updated for a k-1th deployment, and the video coding application updated for a kth deployment also includes an untrained initial inter-frame filtering model; the initial inter-frame filtering model is used to filter the reconstructed frame to be filtered that belongs to an inter-frame prediction type, and the filtering model to be trained in the video coding application updated for a kth deployment includes the initial inter-frame filtering model and the first intra-frame filtering model;

[0206] The model training module 12 may include: a video tag acquiring unit 121 , a model training unit 122 and a model determining unit 123 .

[0207] The video label acquisition unit 121 is used to acquire an inter-frame reconstructed frame to be filtered that belongs to the inter-frame prediction type in the first sample inter-frame reconstructed frame to be filtered, and use the sample original video frame corresponding to the inter-frame reconstructed frame to be filtered in the sample original video frame as the first video frame label;

[0208] The video label acquisition unit 121 is further configured to acquire a first intra-frame reconstructed frame to be filtered that belongs to the intra-frame prediction type in the first sample intra-frame reconstructed frame to be filtered, and use the sample original video frame corresponding to the first intra-frame reconstructed frame to be filtered in the sample original video frame as the second video frame label;

[0209] A model training unit 122 is used to train the initial inter-frame filtering model based on the inter-frame reconstructed frame to be filtered and the first video frame label to obtain a first inter-frame filtering model in a training convergence state;

[0210] The model training unit 122 is further used to train the first intra-frame filtering model based on the reconstructed frame in the frame to be filtered and the second video frame label to obtain a second intra-frame filtering model in a training convergence state;

[0211] The model determination unit 123 is used to determine the first inter-frame filtering model and the second intra-frame filtering model as the second filtering model.

[0212] The specific implementation of the video tag acquisition unit 121, the model training unit 122 and the model determination unit 123 can be found in the above Figure 6 The description of steps S501 to S505 in the corresponding embodiment will not be repeated here.

[0213] In one embodiment, the application updating module 13 may include: an intra-frame model replacing unit 131 and an inter-frame model replacing unit 132 .

[0214] An intra-frame model replacement unit 131 is used to replace and update the first intra-frame filter model in the video coding application deployed and updated for the kth time with a second intra-frame filter model;

[0215] The inter-frame model replacement unit 132 is used to replace and update the initial inter-frame filtering model in the video encoding application deployed and updated for the kth time with the first inter-frame filtering model.

[0216] The specific implementation of the intra-frame model replacement unit 131 and the inter-frame model replacement unit 132 can be found in the above Figure 6 The description of step S505 in the corresponding embodiment will not be repeated here.

[0217] In one embodiment, the first filtering model includes a first intra-frame filtering model for filtering the reconstructed frame to be filtered that belongs to the intra-frame prediction type; the first intra-frame filtering model is trained based on the initial intra-frame filtering model in the video coding application updated for the k-1th deployment, and the video coding application updated for the kth deployment also includes an untrained first-type initial inter-frame filtering model and a second-type initial inter-frame filtering model; the first-type initial inter-frame filtering model is used to filter the reconstructed frame to be filtered that belongs to the first inter-frame prediction type, and the second-type initial inter-frame filtering model is used to filter the reconstructed frame to be filtered that belongs to the second inter-frame prediction type;

[0218] The model training module 12 may include: a model to be trained determining unit 124 , a video frame label determining unit 125 , an inter-frame model training unit 126 , an intra-frame model training unit 127 and a filtering model determining unit 128 .

[0219] The to-be-trained model determining unit 124 is used to determine the first intra-frame filtering model and the first type of initial inter-frame filtering model in the video coding application deployed and updated for the kth time as the to-be-trained filtering model of the video coding application deployed and updated for the kth time;

[0220] The video frame label determination unit 125 is used to obtain a first type of inter-frame reconstructed frame to be filtered that belongs to the first inter-frame prediction type in the first sample inter-frame reconstructed frame to be filtered, and use the sample original video frame corresponding to the first type of inter-frame reconstructed frame to be filtered in the sample original video frame as a third video frame label;

[0221] The video frame label determination unit 125 is further configured to obtain a second intra-frame reconstructed frame to be filtered that belongs to the intra-frame prediction type in the first sample intra-frame reconstructed frame to be filtered, and use the sample original video frame corresponding to the second intra-frame reconstructed frame to be filtered in the sample original video frame as a fourth video frame label;

[0222] The inter-frame model training unit 126 is used to train the first type of initial inter-frame filtering model based on the first type of inter-frame reconstructed frame to be filtered and the third video frame label to obtain the first type of inter-frame filtering model in a training convergence state;

[0223] The intra-frame model training unit 127 is used to train the first intra-frame filtering model based on the reconstructed frame within the second frame to be filtered and the fourth video frame label to obtain a second intra-frame filtering model in a training convergence state;

[0224] The filtering model determining unit 128 is used to determine the first type of inter-frame filtering model and the second intra-frame filtering model as a second filtering model.

[0225] For the specific implementation method, please refer to the above Figure 7The description of steps S501 to S505 in the corresponding embodiment will not be repeated here.

[0226] The specific implementation of the to-be-trained model determination unit 124, the video frame label determination unit 125, the inter-frame model training unit 126, the intra-frame model training unit 127, and the filter model determination unit 128 can be found in the above Fig. 9 The description of steps S801 to S806 in the corresponding embodiment will not be repeated here.

[0227] In one embodiment, the data processing device 1 may further include: a data generating module 15 , a filtering model training module 16 , a deployment model module 17 and a target application determining module 18 .

[0228] The data generation module 15 is used to generate second training data through the video coding application deployed and updated for the k+1th time and the sample video data when the video coding application deployed and updated for the k+1th time does not meet the filtering quality requirement condition; the second training data includes the sample original video frame as a training label, and the second sample to-be-filtered reconstructed frame corresponding to the sample original video frame, and the second sample to-be-filtered reconstructed frame refers to a video frame that has not been filtered by the second filtering model during the process of encoding and reconstructing the sample original video frame through the video coding application deployed and updated for the k+1th time;

[0229] A filter model training module 16 is used to train the filter model to be trained in the video coding application updated for the k+1th time based on the sample original video frame and the second sample to-be-filtered reconstructed frame, to obtain a third filter model in a training convergence state;

[0230] A deployment model module 17, configured to update and deploy a third filter model in the video coding application that is updated and deployed for the k+1th time, to obtain a video coding application that is updated and deployed for the k+2th time;

[0231] The target application determination module 18 is configured to determine the video coding application deployed and updated for the k+2th time as the target video coding application when the video coding application deployed and updated for the k+2th time meets the filtering quality requirement condition.

[0232] The specific implementation of the data generation module 15, the filter model training module 16, the deployment model module 17 and the target application determination module 18 can be found in the above Fig. 9 The description of step S806 in the corresponding embodiment will not be repeated here.

[0233] In one embodiment, the filter model to be trained in the video coding application for the k+1th deployment update includes a second type of initial inter-frame filter model;

[0234] The filter model training module 16 may include: a label determination unit 161 , a type inter-frame model training unit 162 and a filter model determination unit 163 .

[0235] The label determination unit 161 is used to obtain a second type of inter-frame reconstructed frame to be filtered that belongs to the second inter-frame prediction type in the second sample inter-frame reconstructed frame to be filtered, and use the sample original video frame corresponding to the second type of inter-frame reconstructed frame to be filtered in the sample original video frame as the fifth video frame label;

[0236] The second type inter-frame model training unit 162 is used to train the second type initial inter-frame filtering model based on the second type inter-frame reconstructed frame to be filtered and the fifth video frame label to obtain the second type inter-frame filtering model in a training convergence state;

[0237] The filtering model determining unit 163 is configured to determine the second type inter-frame filtering model as the third filtering model.

[0238] The specific implementation of the label determination unit 161, the type frame model training unit 162 and the filter model determination unit 163 can be found in the above Fig. 9 The description of step S806 in the corresponding embodiment will not be repeated here.

[0239] In one embodiment, the model training module 12 may include: a filter frame output unit 129 and a parameter adjustment unit 120 .

[0240] The filtering frame output unit 129 is used to input the first sample to-be-filtered reconstructed frame into the first filtering model, and output the sample filtering reconstructed frame corresponding to the first sample to-be-filtered reconstructed frame through the first filtering model;

[0241] A parameter adjustment unit 120, configured to determine an error value between a sample filtered reconstructed frame and a sample original video frame;

[0242] The parameter adjustment unit 120 is further configured to adjust the model parameters of the first filtering model according to the error value to obtain the first filtering model having the adjusted model parameters;

[0243] The parameter adjustment unit 120 is further configured to determine the first filtering model with the adjusted model parameters as the second filtering model in a training convergence state when the first filtering model with the adjusted model parameters meets a model convergence condition.

[0244] The specific implementation of the filter frame output unit 129, the parameter adjustment unit 120 and the model determination unit 121 can be found in the above Figure 3 The description of step S102 in the corresponding embodiment will not be repeated here.

[0245] In one embodiment, the parameter adjustment unit 120 includes: a function acquisition subunit 1201 and an error value determination subunit 1202 .

[0246] The function acquisition subunit 1201 is used to acquire the loss function corresponding to the video encoding application updated by the k-th deployment;

[0247] The error value determination subunit 1202 is used to obtain the original image quality corresponding to the sample original video frame based on the loss function, and use the original image quality as an image quality label;

[0248] The error value determination subunit 1202 is also used to obtain the filtered image quality corresponding to the sample filtered reconstructed frame based on the loss function, and determine the error value between the sample filtered video frame and the sample original video frame through the loss function, the image quality label and the filtered image quality.

[0249] The specific implementation of the function acquisition subunit 1201 and the error value determination subunit 1202 can be found in the above Figure 2 The description of step S102 in the corresponding embodiment will not be repeated here.

[0250] In one embodiment, the loss function includes an absolute value loss function and a square error loss function;

[0251] The error value determination subunit 1202 is further specifically used to determine a first error value between a sample filtered reconstructed frame and a sample original video frame by using a square error loss function, an image quality label, and a filtered image quality;

[0252] The error value determination subunit 1202 is further specifically used to determine a second error value between the sample filtered reconstructed frame and the sample original video frame by using an absolute value loss function, an image quality label, and a filtered image quality;

[0253] The error value determination subunit 1202 is further specifically configured to perform operation processing on the first error value and the second error value, and determine the result of the operation as the error value between the sample filtered reconstructed frame and the sample original video frame.

[0254] In one embodiment, the data processing device 1 may further include: a filtered video output module 19 , a video quality acquisition module 20 and an application detection module 21 .

[0255] A filtered video output module 19, configured to input the sample video data into the video encoding application updated for the k+1th deployment, and output the filtered video data to be detected through the video encoding application updated for the k+1th deployment;

[0256] The video quality acquisition module 20 is used to acquire the filtered video quality corresponding to the filtered video data to be detected, and the original video quality corresponding to the sample video data;

[0257] The application detection module 21 is used to detect the video encoding application updated for the k+1th deployment according to the filtered video quality and the original video quality.

[0258] The specific implementation of the filtered video output module 19, the video quality acquisition module 20 and the application detection module 21 can be found in the above Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0259] In one embodiment, the application detection module 21 may include: a difference quality determination unit 211 and a detection result determination unit 212 .

[0260] A difference quality determination unit 211, configured to determine a difference video quality between the filtered video quality and the original video quality;

[0261] A detection result determination unit 212, configured to determine that the second video encoding application meets the filter quality requirement condition if the difference video quality is less than the difference quality threshold;

[0262] The detection result determination unit 212 is further configured to determine that the second video encoding application does not meet the filtering quality requirement condition if the difference video quality is greater than the difference quality threshold.

[0263] The specific implementation of the difference quality determination unit 211 and the detection result determination unit 212 can be found in the above Figure 2 The description of step S103 in the corresponding embodiment will not be repeated here.

[0264] In an embodiment of the present application, by repeatedly training the inter-frame filtering model by updating the training data set, the consistency of the inter-frame prediction coding frames in training and testing can be improved, thereby improving the coding efficiency, improving the filtering performance of the video coding application, and reducing the distortion of the encoded video.

[0265] For further information, see Fig.13 , Fig.13 1 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The data processing device may be a computer program (including program code) running in a computer device, for example, the data processing device is an application software; the data processing device may be used to execute Fig.11 As shown in the method. Fig.13 As shown, the data processing device 2 may include: a code stream generating module 31 and a code stream sending module 32 .

[0266] The bitstream generation module 31 is used to input the video data into the target video coding application, perform video coding processing on the video data through the target video coding application, and obtain a video compression bitstream corresponding to the video data; the target video coding application refers to the video coding application updated for the k+1th deployment that meets the filtering quality requirement; the video coding application updated for the k+1th deployment includes a second filtering model in a training convergence state; the second filtering model is obtained by training the filtering model to be trained in the video coding application updated for the kth deployment that includes the first filtering model based on the sample original video frame as a training label in the first training data and the first sample to-be-filtered reconstructed frame corresponding to the sample original video frame; the first training data is generated by the video coding application updated for the kth deployment and the sample video data; the first sample to-be-filtered reconstructed frame refers to a reconstructed frame that has not been filtered by the first filtering model during the process of reconstructing the sample original video frame through the video coding application updated for the kth deployment; the sample original video frame is a video frame in the sample video data; k is a positive integer;

[0267] The code stream sending module 32 is used to send the video compression code stream to the receiving device so that the receiving device can decode the video compression code stream.

[0268] For the specific implementation of the code stream generating module 31 and the code stream sending module 32, please refer to the above Fig.11 The description in the corresponding embodiment will not be repeated here.

[0269] For further information, see Fig.13 , Fig.13 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.13 As shown above Fig.11 The device 1 or Fig.12The device 2 in the corresponding embodiment can be applied to the above-mentioned computer device 8000, and the above-mentioned computer device 8000 may include: a processor 8001, a network interface 8004 and a memory 8005. In addition, the above-mentioned computer device 8000 also includes: a user interface 8003, and at least one communication bus 8002. Among them, the communication bus 8002 is used to realize the connection and communication between these components. Among them, the user interface 8003 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 8003 may also include a standard wired interface and a wireless interface. The network interface 8004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 8005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 8005 may optionally also be at least one storage device located away from the aforementioned processor 8001. As Fig.13 As shown, the memory 8005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application.

[0270] exist Fig.13 In the computer device 8000 shown, the network interface 8004 can provide a network communication function; the user interface 8003 is mainly used to provide an input interface for the user; and the processor 8001 can be used to call the device control application stored in the memory 8005 to achieve:

[0271] The sample video data is input into the video encoding application including the first filtering model that is deployed and updated for the kth time, and the first training data is generated through the video encoding application that is deployed and updated for the kth time and the sample video data; the first training data includes the sample original video frame as a training label, and the first sample reconstructed frame to be filtered corresponding to the sample original video frame; the first sample reconstructed frame to be filtered refers to a reconstructed frame that has not been filtered by the first filtering model during the process of reconstructing the sample original video frame through the video encoding application that is deployed and updated for the kth time; the sample original video frame is a video frame in the sample video data; k is a positive integer;

[0272] Based on the sample original video frame and the first sample to-be-filtered reconstructed frame, the filter model to be trained in the video coding application deployed and updated for the kth time is trained to obtain a second filter model in a training convergence state, and the second filter model is updated and deployed in the video coding application deployed and updated for the kth time to obtain a video coding application deployed and updated for the k+1th time;

[0273] When the video encoding application updated and deployed for the k+1th time meets the filtering quality requirement condition, the video encoding application updated and deployed for the k+1th time is determined as a target video encoding application for performing video encoding processing on the video data.

[0274] Or implement:

[0275] The video data is input into the target video coding application, and the video data is subjected to video coding processing by the target video coding application, and the video data corresponds to a video compression code stream; the target video coding application refers to the video coding application updated for the k+1th deployment that meets the filtering quality requirement; the video coding application updated for the k+1th deployment includes a second filtering model in a training convergence state; the second filtering model is obtained by training the filtering model to be trained in the video coding application updated for the kth deployment that includes the first filtering model based on the sample original video frame as a training label in the first training data and the first sample to-be-filtered reconstructed frame corresponding to the sample original video frame; the first training data is generated by the video coding application updated for the kth deployment and the sample video data; the first sample to-be-filtered reconstructed frame refers to a reconstructed frame that has not been filtered by the first filtering model during the process of reconstructing the sample original video frame by the video coding application updated for the kth deployment; the sample original video frame is a video frame in the sample video data; k is a positive integer;

[0276] The video compression code stream is sent to a receiving device so that the receiving device decodes the video compression code stream.

[0277] It should be understood that the computer device 8000 described in the embodiment of the present application can execute the above Figure 2 or Fig.11 The description of the data processing method in the corresponding embodiment can also be performed as described above. Fig.11 In the corresponding embodiment, the data processing device 1 or Fig.12 The description of the data processing device 2 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0278] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the computer device 1000 for data processing mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, the computer program can execute the above-mentioned data processing. Figure 3 or Fig.11The description of the above data processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0279] The computer-readable storage medium may be the data processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0280] In one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in one aspect of the embodiments of the present application.

[0281] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0282] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0283] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0284] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: include: Inputting the sample video data into a video encoding application that is deployed and updated for the kth time and includes a first filtering model, and generating first training data for the sample video data through the video encoding application that is deployed and updated for the kth time; the first training data includes a sample original video frame as a training label, and a first sample to-be-filtered reconstructed frame corresponding to the sample original video frame; The first sample to-be-filtered reconstructed frame refers to a reconstructed frame that has not been filtered by the first filtering model during the process of reconstructing the sample original video frame by the video encoding application updated by the kth deployment; the sample original video frame is a video frame in the sample video data; the sample video data refers to video data used for training the filtering model; k is a positive integer; Based on the sample original video frame and the first sample to-be-filtered reconstructed frame, training the filter model to be trained in the video coding application deployed and updated for the kth time to obtain a second filter model in a training convergence state, and updating and deploying the second filter model in the video coding application deployed and updated for the kth time to obtain a video coding application deployed and updated for the k+1th time; When the video encoding application updated for the k+1th deployment meets the filtering quality requirement condition, the video encoding application updated for the k+1th deployment is determined as a target video encoding application for performing video encoding processing on the video data.

2. The method according to claim 1, characterized in that The first filtering model includes a first intra-frame filtering model for filtering the reconstructed frames to be filtered that belong to the intra-frame prediction type; the first intra-frame filtering model is trained based on the initial intra-frame filtering model in the video coding application updated for the k-1th deployment, and the video coding application updated for the kth deployment also includes an untrained initial inter-frame filtering model; the initial inter-frame filtering model is used to filter the reconstructed frames to be filtered that belong to the inter-frame prediction type, and the filtering models to be trained in the video coding application updated for the kth deployment include the initial inter-frame filtering model and the first intra-frame filtering model.

3. The method according to claim 2, characterized in that The step of training the filter model to be trained in the video coding application updated for the kth deployment based on the sample original video frame and the first sample to-be-filtered reconstructed frame to obtain a second filter model in a training convergence state includes: Obtaining an inter-frame reconstructed frame to be filtered that belongs to an inter-frame prediction type from the first sample inter-frame reconstructed frame to be filtered, and using a sample original video frame corresponding to the inter-frame reconstructed frame to be filtered in the sample original video frame as a first video frame label; Acquire a first intra-frame reconstructed frame to be filtered that belongs to the intra-frame prediction type from the first sample intra-frame reconstructed frame to be filtered, and use the sample original video frame corresponding to the first intra-frame reconstructed frame to be filtered in the sample original video frame as a second video frame label; Based on the inter-frame reconstructed frame to be filtered and the first video frame label, the initial inter-frame filtering model is trained to obtain a first inter-frame filtering model in a training convergence state; Based on the reconstructed frame within the to-be-filtered frame and the second video frame label, the first intra-frame filtering model is trained to obtain a second intra-frame filtering model in a training convergence state; The first inter-frame filtering model and the second intra-frame filtering model are determined as the second filtering model.

4. The method according to claim 2, characterized in that: The step of updating and deploying the second filtering model in the video coding application updated and deployed for the kth time to obtain the video coding application updated and deployed for the k+1th time includes: Replacing and updating the first intra-frame filtering model in the video encoding application deployed and updated for the kth time with a second intra-frame filtering model; The initial inter-frame filtering model in the video encoding application deployed and updated for the kth time is replaced and updated with the first inter-frame filtering model.

5. The method according to claim 1, characterized in that: The first filtering model includes a first intra-frame filtering model for filtering the reconstructed frames to be filtered that belong to the intra-frame prediction type; the first intra-frame filtering model is trained based on the initial intra-frame filtering model in the video coding application updated for the k-1th deployment, and the video coding application updated for the kth deployment also includes an untrained first type initial inter-frame filtering model and a second type initial inter-frame filtering model; the first type initial inter-frame filtering model is used to filter the reconstructed frames to be filtered that belong to the first inter-frame prediction type, and the second type initial inter-frame filtering model is used to filter the reconstructed frames to be filtered that belong to the second inter-frame prediction type.

6. The method according to claim 5, characterized in that The step of training the filter model to be trained in the video coding application updated for the kth deployment based on the sample original video frame and the first sample to-be-filtered reconstructed frame to obtain a second filter model in a training convergence state includes: Determine the first intra-frame filtering model and the first type of initial inter-frame filtering model in the video coding application deployed and updated for the kth time as the filtering models to be trained for the video coding application deployed and updated for the kth time; Acquire a first type of inter-frame reconstructed frame to be filtered belonging to a first inter-frame prediction type from the first sample inter-frame reconstructed frame to be filtered, and use the sample original video frame corresponding to the first type of inter-frame reconstructed frame to be filtered in the sample original video frame as a third video frame label; Acquire a second intra-frame reconstructed frame to be filtered that belongs to the intra-frame prediction type from the first sample intra-frame reconstructed frame to be filtered, and use the sample original video frame corresponding to the second intra-frame reconstructed frame to be filtered in the sample original video frame as a fourth video frame label; Based on the first type of inter-frame reconstructed frame to be filtered and the third video frame label, the first type of initial inter-frame filtering model is trained to obtain a first type of inter-frame filtering model in a training convergence state; Based on the reconstructed frame within the second frame to be filtered and the fourth video frame label, the first intra-frame filtering model is trained to obtain a second intra-frame filtering model in a training convergence state; The first type of inter-frame filtering model and the second intra-frame filtering model are determined as the second filtering model.

7. The method according to claim 4, characterized in that The method further comprises: When the video encoding application updated by the k+1th deployment does not meet the filtering quality requirement condition, second training data is generated by using the video encoding application updated by the k+1th deployment and the sample video data; the second training data includes the sample original video frame as a training label, and a second sample to-be-filtered reconstructed frame corresponding to the sample original video frame, the second sample to-be-filtered reconstructed frame refers to a video frame that has not been filtered by the second filtering model during the process of reconstructing the sample original video frame by the video encoding application updated by the k+1th deployment; Based on the sample original video frame and the second sample to-be-filtered reconstructed frame, training the to-be-trained filter model in the video coding application updated by the k+1th deployment to obtain a third filter model in a training convergence state; The third filter model is updated and deployed in the video coding application updated and deployed for the k+1th time, so as to obtain the video coding application updated and deployed for the k+2th time; When the video coding application updated for the k+2th deployment meets the filtering quality requirement condition, the video coding application updated for the k+2th deployment is determined as the target video coding application.

8. The method according to claim 7, characterized in that The filter model to be trained in the video coding application deployed and updated for the k+1th time includes a second type of initial inter-frame filter model; The method of training the filter model to be trained in the video coding application updated for the k+1th time based on the sample original video frame and the second sample to-be-filtered reconstructed frame to obtain a third filter model in a training convergence state includes: Obtaining a second type of inter-frame reconstructed frame to be filtered belonging to a second inter-frame prediction type from the second sample inter-frame reconstructed frame to be filtered, and using the sample original video frame corresponding to the second type of inter-frame reconstructed frame to be filtered in the sample original video frame as a fifth video frame label; Based on the second type of inter-frame reconstructed frame to be filtered and the fifth video frame label, training the second type of initial inter-frame filtering model to obtain a second type of inter-frame filtering model in a training convergence state; The second type inter-frame filtering model is determined as the third filtering model.

9. The method according to any one of claims 1 to 8, characterized in that: The step of training the filter model to be trained in the video coding application updated for the kth deployment based on the sample original video frame and the first sample to-be-filtered reconstructed frame to obtain a second filter model in a training convergence state includes: Inputting the first sample to-be-filtered reconstructed frame into the first filtering model, and outputting a sample filtered reconstructed frame corresponding to the first sample to-be-filtered reconstructed frame through the first filtering model; Determine an error value between the sample filtered reconstructed frame and the sample original video frame, and adjust a model parameter of the first filtering model according to the error value to obtain a first filtering model with adjusted model parameters; When the first filtering model with the adjusted model parameters meets the model convergence condition, the first filtering model with the adjusted model parameters is determined as the second filtering model in the training convergence state.

10. The method according to claim 9, characterized in that The determining of the error value between the sample filtered reconstructed frame and the sample original video frame comprises: Obtaining a loss function corresponding to the video encoding application deployed and updated for the kth time; Acquire the original image quality corresponding to the sample original video frame based on the loss function, and use the original image quality as an image quality label; The filtered image quality corresponding to the sample filtered reconstructed frame is obtained based on the loss function, and the error value between the sample filtered reconstructed frame and the sample original video frame is determined by the loss function, the image quality label and the filtered image quality.

11. The method according to claim 10, characterized in that The loss function includes an absolute value loss function and a square error loss function; The determining the error value between the sample filtered reconstructed frame and the sample original video frame by using the loss function, the image quality label and the filtered image quality comprises: Determining a first error value between the sample filtered reconstructed frame and the sample original video frame by using the square error loss function, the image quality label and the filtered image quality; Determining a second error value between the sample filtered reconstructed frame and the sample original video frame by using the absolute value loss function, the image quality label and the filtered image quality; The first error value and the second error value are processed by operation, and a result obtained by the operation is determined as an error value between the sample filtered reconstructed frame and the sample original video frame.

12. The method according to claim 1, characterized in that The method further comprises: Inputting the sample video data into the video encoding application updated for the k+1th deployment, and outputting the to-be-detected filtered video data through the video encoding application updated for the k+1th deployment; Acquire the filtered video quality corresponding to the filtered video data to be detected, and the original video quality corresponding to the sample video data; The video encoding application updated for the k+1th deployment is detected according to the filtered video quality and the original video quality.

13. The method according to claim 12, characterized in that The detecting the filtering quality in the video encoding application updated for the k+1th deployment according to the filtered video quality and the original video quality comprises: determining a difference video quality between the filtered video quality and the original video quality; If the difference video quality is less than the difference quality threshold, determining that the second video encoding application meets the filtering quality requirement condition; If the difference video quality is greater than the difference quality threshold, it is determined that the second video encoding application does not meet the filtering quality requirement condition.

14. A data processing method, characterized in that: include: Inputting the video data into a target video encoding application, performing video encoding processing on the video data through the target video encoding application, and obtaining a video compression code stream corresponding to the video data; The target video coding application refers to the video coding application updated for the k+1th deployment that meets the filtering quality requirement condition; the video coding application updated for the k+1th deployment includes a second filtering model in a training convergence state, the second filtering model is based on the sample original video frame as a training label in the first training data, and the first sample to-be-filtered reconstructed frame corresponding to the sample original video frame, and is obtained by training the to-be-trained filtering model in the video coding application updated for the kth deployment that includes the first filtering model; the first training data is generated for the sample video data by the video coding application updated for the kth deployment; the first sample to-be-filtered reconstructed frame refers to a reconstructed frame that has not been filtered by the first filtering model during the process of reconstructing the sample original video frame by the video coding application updated for the kth deployment; the sample original video frame is a video frame in the sample video data; the sample video data refers to the video data used to train the filtering model; k is a positive integer; The video compression code stream is sent to a receiving device so that the receiving device decodes the video compression code stream.

15. A data processing device, characterized in that: include: A training data generation module, configured to input sample video data into a video encoding application updated for the kth deployment; The training data generating module is further configured to generate first training data for the sample video data by applying the video encoding application including the first filtering model updated by the k-th deployment; the first training data includes a sample original video frame as a training label, and a first sample to-be-filtered reconstructed frame corresponding to the sample original video frame; The first sample to-be-filtered reconstructed frame refers to a reconstructed frame that has not been filtered by the first filtering model during the process of reconstructing the sample original video frame by the video encoding application updated by the kth deployment; the sample original video frame is a video frame in the sample video data; the sample video data refers to video data used for training the filtering model; k is a positive integer; A model training module, used for training the filter model to be trained in the video coding application deployed and updated for the kth time based on the sample original video frame and the first sample reconstructed frame to be filtered, to obtain a second filter model in a training convergence state; An application updating module, configured to update and deploy the second filtering model in the video coding application updated with the k-th deployment, to obtain the video coding application updated with the k+1-th deployment; The target application determination module determines the video coding application deployed and updated for the k+1th time as a target video coding application for performing video coding processing on the video data when the video coding application deployed and updated for the k+1th time meets the filtering quality requirement condition.

16. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a network communication function, the memory is used to store program code, and the processor is used to call the program code so that the computer device executes the method described in any one of claims 1-14.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the method according to any one of claims 1 to 14.

Images