Video processing method and device, equipment and storage medium

Abstract

The present disclosure relates to a video processing method, device and equipment and storage medium, and belongs to the technical field of computers, which is used for deinterlacing processing of interlaced videos to improve video quality and increase video definition. The video processing method comprises the following steps: obtaining a video to be processed; determining a video type of the video to be processed based on a pre-trained target type detection model; the video type is a non-interlaced type or a target interlaced type in multiple interlaced types; when the video type is the target interlaced type in the multiple interlaced types, inputting multiple image frames to be processed into a target deinterlacing model corresponding to the target interlaced type for deinterlacing processing to obtain a non-interlaced video corresponding to the video to be processed.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a video processing method, apparatus, device, and storage medium. Background Technology

[0002] Interlaced video frames can have various interlacing types, easily leading to complex interlacing phenomena, i.e., horizontal jagged edges. Common techniques typically perform simple deinterlacing processing on interlaced video, which cannot effectively improve the quality of interlaced video, resulting in low video clarity. Summary of the Invention

[0003] This disclosure provides a video processing method, apparatus, device, and storage medium for deinterlacing interlaced video to improve video quality and enhance video clarity.

[0004] The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a video processing method is provided, comprising: acquiring a video to be processed. The video to be processed includes multiple image frames to be processed. Based on a pre-trained target type detection model, the video type of the video to be processed is determined. The video type is either a non-interlaced type or a target interlaced type among multiple interlaced types. When the video type is a target interlaced type among multiple interlaced types, the multiple image frames to be processed are input to a target deinterlacing model corresponding to the target interlacing type for deinterlacing processing to obtain a non-interlaced video corresponding to the video to be processed.

[0006] Optionally, the video processing method further includes: acquiring multiple sets of interleaved samples and multiple non-interleaved sample image frames. The multiple sets of interleaved samples correspond one-to-one with various interleaving types. Each set of interleaved samples includes multiple sample image frames of interleaved types corresponding to the interleaved sample sets. An initial type detection model is trained based on the multiple non-interleaved sample image frames and the multiple sets of interleaved samples to obtain a target type detection model.

[0007] Optionally, the interlaced sample image frames possess at least one distortion feature among compression distortion and blurring distortion. Multiple interlacing types include single-interlacing and multi-interlacing types. The method for obtaining multiple interlaced sample sets specifically includes: updating parameters of a preset type within the sample non-interlaced video to obtain a single-interlaced interlaced video; transcoding and compressing the single-interlaced interlaced video to obtain a compressed single-interlaced interlaced video; determining a candidate set corresponding to the single-interlacing type based on motion information between image frames in the sample non-interlaced video and the compressed interlaced video; the candidate set corresponding to the single-interlacing type includes multiple candidate samples of the single-interlacing type; performing super-resolution processing at different magnifications on the candidate samples in the candidate set corresponding to the single-interlacing type to obtain candidate sets corresponding to each multi-interlacing type; and blurring multiple candidate sets that correspond one-to-one with multiple interlacing types to obtain multiple interlaced sample sets.

[0008] Optionally, the method for determining the candidate set corresponding to a single interlacing type based on the motion information between image frames in the sample non-interlaced video and the compressed interlaced video specifically includes: decoding the compressed interlaced video and the sample non-interlaced video to obtain multiple image frames of single interlacing types and multiple non-interlaced video frames. Based on the multiple non-interlaced video frames, determining the motion information corresponding to each image frame of a single interlacing type. The motion information is used to characterize the relative motion information between a non-interlaced image frame and its immediately following non-interlaced image frame. Target regions within the image frames of single interlacing types are determined as candidate samples of the single interlacing type, obtaining the candidate set corresponding to the single interlacing type. The target regions are used to characterize regions where the average value of the motion information within the image frames of single interlacing types is greater than a first preset threshold.

[0009] Optionally, the video processing method further includes: obtaining an interlaced sample set and a set of real images corresponding to the interlaced type to be trained. The set of real images includes real image regions in the non-interlaced video that correspond to each sample image frame in the interlaced sample set. Based on the interlaced sample set and the set of real images, an initial deinterlacing model corresponding to the interlaced type to be trained is trained to obtain a target deinterlacing model corresponding to the interlaced type to be trained.

[0010] Optionally, when the interleaving type to be trained is a single interleaving type, the initial deinterleaving model is used to divide the input sample image frame into scan fields, and then perform deinterleaving processing on the upper scan field and the lower scan field respectively. When the interleaving type to be trained is a multiple interleaving type, the initial deinterleaving model is used to sequentially perform multiple downsampling processing, deinterleaving processing, multiple upsampling processing, and residual summation processing on the input sample image frame.

[0011] Optionally, a method for determining the video type of the video to be processed based on a pre-trained target type detection model specifically includes: dividing each image frame to be processed into multiple image blocks; inputting each image block into the target type detection model to obtain detection information for each image block; the detection information for each image block includes a type label and a confidence level of the type label; determining the average confidence level of the type labels for each type; and identifying type labels whose average confidence level is greater than or equal to a second preset threshold as the video type of the video to be processed.

[0012] Optionally, the video processing method further includes: performing quality assessments on the non-interlaced video corresponding to the video to be processed and the video to be processed, respectively, to obtain an assessment result after deinterlacing and an assessment result before deinterlacing. A comparative assessment result is then determined based on the assessment results after deinterlacing and the assessment results before deinterlacing.

[0013] According to a second aspect of the present disclosure, a video processing apparatus is provided, comprising: an acquisition unit, a determination unit, and a processing unit.

[0014] The acquisition unit is configured to acquire the video to be processed. The video to be processed includes multiple image frames to be processed.

[0015] The determination unit is configured to execute a pre-trained target type detection model to determine the video type of the video to be processed. The video type is either a non-interlaced type or a target interlaced type among multiple interlaced types.

[0016] The processing unit is configured to, when the video type is the target interlacing type among multiple interlacing types, input multiple image frames to be processed into the target deinterlacing model corresponding to the target interlacing type for deinterlacing processing, and obtain the non-interlaced video corresponding to the video to be processed.

[0017] Optionally, the acquisition unit is further configured to acquire multiple interleaved sample sets and multiple non-interleaved sample image frames. The multiple interleaved sample sets correspond one-to-one with multiple interleaved types. Each interleaved sample set includes multiple sample image frames of the interleaved type corresponding to the interleaved sample set. The processing unit is configured to train an initial type detection model based on the multiple non-interleaved sample image frames and the multiple interleaved sample sets to obtain a target type detection model.

[0018] Optionally, the interlaced sample image frames possess at least one distortion feature among compression distortion and blurring distortion. Multiple interlacing types include single-interlacing and multi-interlacing types. The acquisition unit is specifically configured to perform the following: update parameters of a preset type within the sample non-interlaced video to obtain a single-interlaced interlaced video; transcode and compress the single-interlaced interlaced video to obtain a compressed single-interlaced interlaced video; determine a candidate set corresponding to the single-interlacing type based on motion information between image frames in the sample non-interlaced video and the compressed interlaced video; the candidate set corresponding to the single-interlacing type includes multiple candidate samples of the single-interlacing type; perform super-resolution processing at different magnifications on the candidate samples in the candidate set corresponding to the single-interlacing type to obtain candidate sets corresponding to each multi-interlacing type; and blur the multiple candidate sets that correspond one-to-one with the multiple interlacing types to obtain multiple interlaced sample sets.

[0019] Optionally, the acquisition unit is specifically configured to perform the following: decoding processing on the compressed interlaced video and sample non-interlaced video to obtain multiple image frames of single-interlaced type and multiple non-interlaced video frames. Based on the multiple non-interlaced video frames, determine the motion information corresponding to each image frame of single-interlaced type. The motion information is used to characterize the relative motion information between the non-interlaced image frame and the immediately following non-interlaced image frame. Target regions within the image frames of single-interlaced type are determined as candidate samples of single-interlaced type, obtaining a candidate set corresponding to the single-interlaced type. The target regions are used to characterize regions where the average value of the motion information within the image frames of single-interlaced type is greater than a first preset threshold.

[0020] Optionally, the acquisition unit is further configured to acquire the set of interlaced samples and the set of real images corresponding to the interlaced type to be trained. The set of real images includes the real image regions in the non-interlaced video samples that correspond to each sample image frame in the interlaced sample set. The processing unit is further configured to train the initial deinterlacing model corresponding to the interlaced type to be trained based on the set of interlaced samples and the set of real images, to obtain the target deinterlacing model corresponding to the interlaced type to be trained.

[0021] Optionally, when the interleaving type to be trained is a single interleaving type, the initial deinterleaving model is used to divide the input sample image frame into scan fields, and then perform deinterleaving processing on the upper scan field and the lower scan field respectively. When the interleaving type to be trained is a multiple interleaving type, the initial deinterleaving model is used to sequentially perform multiple downsampling processing, deinterleaving processing, multiple upsampling processing, and residual summation processing on the input sample image frame.

[0022] Optionally, the determining unit is specifically configured to perform the following: dividing each image frame to be processed into multiple image blocks to be processed; inputting each image block to be processed into a target type detection model to obtain detection information for each image block to be processed; the detection information for each image block to be processed includes the type label of the image block to be processed and the confidence level of the type label; determining the average confidence level of the type labels for various types; and determining the type labels whose average confidence level is greater than or equal to a second preset threshold as the video type of the video to be processed.

[0023] Optionally, the processing unit is further configured to perform quality assessments on the non-interlaced video corresponding to the video to be processed and the video to be processed, respectively, to obtain an evaluation result after deinterlacing and an evaluation result before deinterlacing. The determining unit is further configured to determine a comparative evaluation result based on the evaluation result after deinterlacing and the evaluation result before deinterlacing.

[0024] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement any of the optional video processing methods described in the first aspect.

[0025] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, which stores instructions that, when executed by a processor of an electronic device, enable the electronic device to perform any of the optional video processing methods described in the first aspect.

[0026] According to a fifth aspect of the present disclosure, a computer program product is provided, including instructions that, when executed on a processor in an electronic device, cause the electronic device to perform any of the optional video processing methods of the first aspect.

[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0028] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0029] Based on any of the above aspects, in the video processing method provided by this disclosure, after the server acquires a video to be processed including multiple image frames to be processed, it can determine the video type of the video to be processed based on a pre-trained target type detection model. When the video type is a target interlacing type among multiple interlacing types, it can further input the multiple image frames to be processed into a target deinterlacing model corresponding to the target interlacing type for deinterlacing processing to obtain a non-interlaced video corresponding to the video to be processed.

[0030] Based on this, this disclosure can effectively improve the video quality of the video to be processed by accurately detecting the video type and adopting deinterlacing processing corresponding to the video type, thus avoiding the problem of video quality degradation caused by mismatch between the deinterlacing processing method and the video type. Therefore, this disclosure can be used to perform deinterlacing processing on interlaced videos to improve video quality and enhance video clarity. Attached Figure Description

[0031] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0032] Figure 1 A schematic diagram of the structure of a video processing system provided in an embodiment of this disclosure is shown;

[0033] Figure 2 This illustration shows a flowchart of a video processing method provided in an embodiment of the present disclosure. Figure 1 ;

[0034] Figure 3 This illustration shows a flowchart of a video processing method provided in an embodiment of the present disclosure. Figure 2 ;

[0035] Figure 4 This illustration shows a flowchart of a video processing method provided in an embodiment of the present disclosure. Figure 3 ;

[0036] Figure 5 This illustration shows a flowchart of a video processing method provided in an embodiment of the present disclosure. Figure 4 ;

[0037] Figure 6 This illustration shows a flowchart of a video processing method provided in an embodiment of the present disclosure. Figure 5 ;

[0038] Figure 7 This illustration shows a flowchart of a video processing method provided in an embodiment of the present disclosure. Figure 6 ;

[0039] Figure 8 This illustration shows a flowchart of a video processing method provided in an embodiment of the present disclosure. Figure 7 ;

[0040] Figure 9 This illustration shows a flowchart of a video processing method provided in an embodiment of the present disclosure. Figure 8 ;

[0041] Figure 10 A schematic diagram of an image frame provided in an embodiment of this disclosure is shown;

[0042] Figure 11 A schematic diagram of the structure of a video processing apparatus provided in an embodiment of this disclosure is shown;

[0043] Figure 12 A schematic diagram of the structure of a terminal provided in an embodiment of this disclosure is shown;

[0044] Figure 13 A schematic diagram of the structure of another server provided in an embodiment of this disclosure is shown. Detailed Implementation

[0045] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0046] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0047] It should also be understood that the term "comprising" indicates the presence of the described feature, whole, step, operation, element and / or component, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements and / or components.

[0048] The data disclosed herein may be data authorized by the user or fully authorized by all parties.

[0049] Deinterlacing is a video processing technique that converts interlaced video into non-interlaced video, used to restore and enhance the image quality of interlaced video.

[0050] Each image frame in interlaced video consists of two scan fields, which can be divided into an upper field and a lower field. Each scan field contains half the number of pixels in the horizontal direction of the image frame. Interlaced video can be specifically divided into single-interlaced video and multi-interlaced video. In a single-interlaced video image frame, the upper field contains all odd-numbered rows of pixels, and the lower field contains all even-numbered rows of pixels. For multi-interlaced video, taking 2x interlaced video as an example, the upper field of the image frame contains multiple sets of two consecutive rows of pixels, and the lower field contains multiple sets of two consecutive rows of pixels.

[0051] Image frames within interlaced videos typically exhibit various types of distortion, including compression and blurring, which can lead to complex interlacing phenomena, known as lateral jagged edges. Common techniques usually deinterlacing single-fold interlaced videos, making it difficult to handle multi-fold interlaced videos. Furthermore, it's challenging to correct compression and blurring distortions present in interlaced videos, thus failing to effectively improve their quality.

[0052] Based on this, the present disclosure provides a video processing method. After the server acquires a video to be processed, which includes multiple image frames to be processed, it can determine the video type of the video to be processed based on a pre-trained target type detection model. When the video type is a target interlacing type among multiple interlacing types, the server can further input the multiple image frames to be processed into a target deinterlacing model corresponding to the target interlacing type for deinterlacing processing to obtain a non-interlaced video corresponding to the video to be processed.

[0053] Based on this, this disclosure can effectively improve the video quality of the video to be processed by accurately detecting the video type and adopting deinterlacing processing corresponding to the video type, thus avoiding the problem of video quality degradation caused by mismatch between the deinterlacing processing method and the video type. Therefore, this disclosure can be used to perform deinterlacing processing on interlaced videos to improve video quality and enhance video clarity.

[0054] Figure 1 This is a schematic diagram of a video processing system provided in an embodiment of the present disclosure, such as... Figure 1 As shown, the video processing system 100 may include a terminal 101 and a server 102. The terminal 101 can establish a connection with the server 102 through a wired network or a wireless network.

[0055] Optional, Figure 1 The terminal 101 and server 102 can be functional modules integrated into the same device, or they can be two independently configured devices. This disclosure does not impose any limitations on this.

[0056] It is easy to understand that when terminal 101 and server 102 are functional modules integrated within the same device, the communication method between terminal 101 and server 102 is the same as the communication method between modules within the device. In this case, the communication process between the two is the same as that between terminal 101 and server 102 when they are set up independently.

[0057] For ease of understanding, this disclosure is mainly illustrated by the example of terminal 101 and server 102 being configured to operate independently.

[0058] Figure 1Terminal 101 can be used to collect various data, such as interlaced and non-interlaced video, and provide them to the server.

[0059] In one possible configuration, terminal 101 may be configured with an input module and a display module. The input module may be a mouse, keyboard, or touchscreen, etc., used to support users in performing various operations (such as data sending operations and click operations). The display module may be an LCD screen, etc., used to display various interfaces (such as data editing interfaces).

[0060] Optionally, terminal 101 can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, etc., which can install and use applications such as video processing and / or content community. This disclosure does not impose any special restrictions on the specific form of the terminal. It can interact with the user through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device.

[0061] Figure 1 Server 102 can be configured with resource files for various models, such as object type detection models and object deinterlacing models, to provide video processing services. For example, it can provide type detection and deinterlacing services for the video to be processed.

[0062] Optional, Figure 1 The server 102 can be a single server or a server cluster consisting of multiple servers. In some embodiments, the server cluster can also be a distributed cluster. This disclosure does not limit the specific implementation of the server.

[0063] Optionally, the above Figure 1 In the video processing system shown, server 102 can be connected to at least one terminal 101. This disclosure does not limit the number or type of terminal 101.

[0064] It should be noted that both the terminal 101 and the server 102 mentioned above can be referred to as electronic devices.

[0065] The video processing method provided in this disclosure can be applied to... Figure 1 The server 102 in the video processing system 100 shown.

[0066] The video processing method provided in the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0067] like Figure 2 As shown, the video processing method may include: S201-S203.

[0068] S201. The server retrieves the video to be processed.

[0069] The video to be processed may include multiple image frames to be processed.

[0070] In one possible approach, the video to be processed can be a video built into the server. Specifically, the server can be configured with a storage module. This storage module can be used to store multiple videos, including the video to be processed, so that the server can read and process them. Based on this, the server can read the video to be processed from the storage module.

[0071] Alternatively, the video to be processed can also be a video from the terminal. Specifically, the user can perform deinterlacing processing on the video to be processed through the input module configured on the terminal (e.g., by clicking the deinterlacing button). In response to this deinterlacing processing operation, the terminal can send a deinterlacing request message carrying the video to be processed to the server. Accordingly, the server can receive the deinterlacing request message from the terminal and parse it to obtain the video to be processed.

[0072] Optionally, the video to be processed can be user-generated content (UGC) video. This UGC video can be used to showcase specific content. For example, the specific content could be dance content, game content, or news content.

[0073] S202. The server determines the video type of the video to be processed based on a pre-trained target type detection model.

[0074] The video type to be processed can be a non-interlaced type, or a target interlaced type from among multiple interlaced types. Multiple interlaced types can include single interlaced and multi-interlaced types. The target interlaced type can be any of the multiple interlaced types. For example, the target interlaced type can be a single interlaced type, a double interlaced type, or a triple interlaced type, etc.

[0075] In one possible approach, the target type detection model can be used to determine the detection information of the input image frame. The detection information of the input image frame can include a type label and a confidence level for the type label. The type label can be used to characterize whether the input image frame is a non-interlaced type or any of several interlaced types.

[0076] In one possible approach, the target type detection model can be pre-trained based on non-interleaved sample image frames and sample image frames of various interleaved types. Specifically, the process of training the target type detection model can be understood by referring to S301-S302 below, and will not be elaborated upon here.

[0077] In one possible approach, after acquiring the video to be processed, the server can decode it to obtain multiple image frames to be processed. Next, the server can input each image frame into a target type detection model for type detection, obtaining detection information for each frame. Based on this, the server can determine the type label and confidence level of each image frame. Furthermore, the server can determine the average confidence level of each type label and identify the type label with the highest average confidence level as the video type of the video to be processed.

[0078] Alternatively, to improve the efficiency of type detection, the server can extract N image frames from the multiple image frames to be processed in the video to be processed using a preset extraction method, and further determine the video type of the video to be processed based on the detection information of the N image frames. For example, the preset extraction method can be a random extraction method, or it can be a method of extracting one image frame every M image frames, etc., where N and M are both positive integers.

[0079] Alternatively, to improve the accuracy of type detection, the server can divide each image frame to be processed into image blocks of a preset size, resulting in multiple image blocks. Based on the detection information of these multiple image blocks, the video type of the video to be processed can then be determined. The preset size can be pre-set by the operator to be smaller than the size of the image frame to be processed, yet large enough to cover the entire image frame.

[0080] In one possible approach, if the video type of the video to be processed is used to indicate that the video is non-interlaced, it suggests that the video has high resolution. In this case, the server does not need to perform subsequent S203 processing on the video to be processed.

[0081] S203. When the video type is the target interlacing type among multiple interlacing types, the server inputs multiple image frames to be processed into the target deinterlacing model corresponding to the target interlacing type for deinterlacing processing to obtain the non-interlaced video corresponding to the video to be processed.

[0082] In one possible approach, the target deinterlacing model corresponding to the target interlacing type can be pre-trained based on sample image frames of the target interlacing type. Specifically, the training process of the target deinterlacing model can be found in the descriptions in S601-S602 below, and will not be repeated here.

[0083] In one possible approach, the server's storage module can store a pre-defined set of models. This pre-defined set of models can include multiple target deinterlacing models that correspond one-to-one with various interlacing types. For example, target deinterlacing models corresponding to single-interlacing types, double-interlacing types, triple-interlacing types, etc.

[0084] It should be noted that during the training of the deinterlacing models for each objective, sample image frames of various interlacing types can possess at least one distortion feature, namely compression distortion and blur distortion. Based on this, the deinterlacing models for each objective can better repair image distortion problems such as compression distortion and blur distortion in each image frame to be processed when performing deinterlacing processing on each image frame to be processed in the video, thereby improving the clarity of the video after deinterlacing processing.

[0085] In one possible approach, if the video type of the video to be processed is used to characterize the video to be processed as a target interlaced type, it indicates that the video to be processed is based on interlaced scanning acquisition or transmission, which is prone to image distortion problems such as compression distortion and blur distortion, as well as horizontal jagged edges (also known as jagged edges), resulting in poor clarity.

[0086] In this case, deinterlacing the video to be processed can be considered to convert it into a non-interlaced video. Furthermore, to improve the deinterlacing effect, the server can match a target deinterlacing model from a pre-defined model set that corresponds to the target interlacing type.

[0087] Next, the server can input each image frame to be processed into the target deinterlacing model corresponding to the target interlacing type for deinterlacing processing, obtaining deinterlaced image frames. Then, the server can encode each deinterlaced image frame according to its temporal order in the video to be processed, obtaining the non-interlaced video corresponding to the video to be processed.

[0088] The technical solution provided by the above embodiments brings at least the following beneficial effects: As can be seen from S201-S203, after the server obtains a video to be processed including multiple image frames to be processed, it can determine the video type of the video to be processed based on a pre-trained target type detection model. When the video type is the target interlacing type among multiple interlacing types, it can further input the multiple image frames to be processed into the target deinterlacing model corresponding to the target interlacing type for deinterlacing processing to obtain the non-interlaced video corresponding to the video to be processed.

[0089] Based on this, this disclosure can effectively improve the video quality of the video to be processed by accurately detecting the video type and adopting deinterlacing processing corresponding to the video type, thus avoiding the problem of video quality degradation caused by mismatch between the deinterlacing processing method and the video type. Therefore, this disclosure can be used to perform deinterlacing processing on interlaced videos to improve video quality and enhance video clarity.

[0090] In one embodiment, such as Figure 3 As shown, the video processing method provided in this embodiment of the present disclosure further includes: S301-S302.

[0091] S301. The server acquires multiple sets of interleaved samples and multiple non-interleaved sample image frames.

[0092] In this context, multiple interleaved sample sets correspond one-to-one with various interleaving types. An interleaved sample set may include multiple sample image frames of the interleaving type corresponding to that single interleaved sample set.

[0093] In one possible approach, the server's storage module can pre-store high-resolution, distortion-free non-interlaced video. Based on this, the server can read the non-interlaced video from the storage module and further decode it to obtain multiple non-interlaced sample image frames.

[0094] Optionally, the sample non-interlaced video can be one or multiple.

[0095] In one possible approach, the server can convert the non-interlaced video into a single-interlaced interlaced video by modifying relevant parameters. Then, the server can obtain multiple single-interlaced sample image frames based on the single-interlaced video. Further, the server can perform multi-fold upsampling (e.g., quadruple and triple upsampling) on ​​these multiple single-interlaced sample image frames to obtain multi-fold interlaced sample image frames.

[0096] In one possible approach, to improve the accuracy of type detection, the sample image frames within each interleaved sample set can also exhibit image distortion issues such as compression distortion and blurring distortion. For example, the server can perform transcoding, compression, and blurring processing on each sample image frame, causing the sample image frames to exhibit image distortion issues such as compression distortion and blurring distortion.

[0097] S302. The server trains the initial type detection model based on multiple non-interleaved sample image frames and multiple interleaved sample sets to obtain the target type detection model.

[0098] In one possible approach, the backbone network of the initial type detection model can be a pre-trained classification model. The pre-trained classification model can be a convolutional neural network such as an 18-layer residual network (ResNet18) or a 34-layer residual network (ResNet34). Two parallel fully connected layers can be connected after the backbone network. One fully connected layer has a 2-dimensional output dimension, used to output non-interleaved and interleaved types. The other fully connected layer has a multi-dimensional output dimension, used to output the multiples of interleaved types.

[0099] One possible approach is to use pre-trained parameters from a classification dataset in the database when initializing the hyperparameters of the initial type detection model.

[0100] In one possible approach, the server can mix multiple non-interleaved sample image frames with sample image frames from multiple interleaved sample sets to obtain a mixed sample set. Then, the server can extract a predetermined number of sample image frames from the mixed sample set each time and input them into an initial type detection model for type detection processing, obtaining a predetermined number of detection results corresponding one-to-one with the predetermined number of sample image frames. Based on this, the server can compare each detection result with the true type of the corresponding sample image frame to determine the correctness of each detection result and further determine the loss value.

[0101] If the loss value is greater than the loss threshold (i.e., the model has not converged), it indicates that the initial type detection model does not yet meet the accuracy requirements. In this case, staff can adjust the hyperparameters of the initial type detection model on the server via a terminal. If the loss value is less than or equal to the loss threshold (i.e., the model has converged), it indicates that the initial type detection model has met the accuracy requirements. In this case, the server can obtain the target type detection model.

[0102] Optionally, the server may determine the loss value based on the cross-entropy loss function, or it may determine the loss value based on other suitable loss functions. This disclosure does not limit this approach.

[0103] The technical solution provided by the above embodiments brings at least the following beneficial effects: As shown in S301-S302, the server can train a target type detection model based on multiple non-interlaced sample image frames and multiple interlaced sample image frames. Based on this, the target type detection model can identify multiple interlaced image frames with high accuracy. Therefore, this disclosure can improve the recognition accuracy of multiple interlaced image frames and avoid false detections and other errors.

[0104] In one embodiment, when the server obtains multiple interleaved sample sets, such as Figure 4 As shown, this disclosure provides an optional implementation, including: S401-S405.

[0105] S401. The server updates the parameters of the preset type in the sample non-interlaced video to obtain an interlaced video of the single interlaced type.

[0106] The preset type parameters can be time-series parameters such as tinterlace, or field order parameters (also known as fieldorder parameters). Furthermore, updating preset type parameters can involve replacing them with preset parameter values. The preset type and preset parameter values ​​can be pre-set in the server by staff.

[0107] In one possible approach, the server can read the non-interlaced video from the storage module and convert the sample non-interlaced video into a single-interlaced interlaced video by updating the parameters of a preset type within the sample non-interlaced video.

[0108] In one possible example, the preset duration of the sample non-interlaced video is 10 seconds, and the frame rate per second (FPS) is 30. That is, the sample non-interlaced video can include 300 image frames. The server can update the temporal class parameters and field order parameters of the sample non-interlaced video, combining every two consecutive image frames, and concatenating the odd-numbered rows of pixels from one image frame with the even-numbered rows of pixels from the other image frame to obtain the image frames in a single-interlaced type interlaced video. In other words, each image frame in the interlaced video can be composed of two consecutive image frames from the sample non-interlaced video. Based on this, the server updates the sample non-interlaced video, and the resulting single-interlaced type interlaced video can include 150 image frames.

[0109] S402. The server performs transcoding and compression processing on the single-interlaced type interlaced video to obtain a compressed single-interlaced type interlaced video.

[0110] In one possible approach, in order to obtain sample image frames with compression distortion, the server can perform one or more transcoding compression processes on the single-interlaced type interlaced video to obtain a compressed single-interlaced type interlaced video.

[0111] Optionally, in the case of multiple transcoding and compression processes, each transcoding and compression process can be based on different encoding parameters and compression bitrates.

[0112] S403. The server determines the candidate set corresponding to the single interlacing type based on the motion information between image frames in the sample non-interlaced video and the compressed interlaced video.

[0113] Among them, the candidate set corresponding to the haploid staggered type can include multiple candidate samples of haploid staggered type;

[0114] It should be noted that since the image frames in the single-interlaced type of interlaced video are obtained by combining two consecutive image frames from the sample non-interlaced video, if there is no relative motion between two consecutive image frames in the sample non-interlaced video (i.e., the two consecutive image frames are identical), then the corresponding image frames in the single-interlaced type of interlaced video will not produce interlaced horizontal stripes, i.e., there will be no horizontal jaggedness problem. Therefore, this disclosure considers determining the motion information of each image frame in the sample non-interlaced video and then filtering each image frame in the obtained interlaced video to obtain sample image frames with interlaced horizontal stripes.

[0115] In one possible approach, the server can decode the sample non-interlaced video and the compressed interlaced video of the single-interlaced type to obtain multiple image frames included in the sample non-interlaced video and multiple image frames included in the compressed interlaced video of the single-interlaced type.

[0116] Next, the server can determine the motion information between every two consecutive image frames in the sample non-interlaced video using the inter-frame difference method. Further, based on the motion information between every two consecutive image frames in the sample non-interlaced video, the server can crop each image frame in the compressed interlaced video of the single-interlaced type accordingly, identifying target regions with a higher average value of motion information in the image frames of the compressed interlaced video of the single-interlaced type, thus obtaining multiple target regions. Based on this, the server can construct a candidate set corresponding to the single-interlaced type from the multiple cropped target regions.

[0117] Alternatively, the server can use optical flow to determine the motion information between every two consecutive image frames in the sample non-interlaced video. Further, based on the motion information between every two consecutive image frames in the sample non-interlaced video, the server can crop each image frame in the compressed interlaced video of the single-interlaced type accordingly, identifying target regions with higher average motion information values ​​among the image frames of the compressed interlaced video of the single-interlaced type, thus obtaining multiple target regions. Based on this, the server can construct a candidate set corresponding to the single-interlaced type from the multiple cropped target regions.

[0118] S404. The server performs super-resolution processing of different multiples on the candidate samples in the candidate set corresponding to the single-fold interlacing type to obtain the candidate set corresponding to each multi-fold interlacing type.

[0119] Alternatively, super-resolution processing can be upsampling or interpolation.

[0120] In one possible approach, the server can perform super-resolution processing on candidate samples in the candidate set corresponding to a single-fold interlacing type at different ratios to obtain candidate sets corresponding to other interlacing types. Specifically, when the multi-fold interlacing type is a two-fold interlacing type, the server can perform two-fold super-resolution processing on candidate samples in the candidate set corresponding to a single-fold interlacing type to obtain a candidate set corresponding to a two-fold interlacing type. When the multi-fold interlacing type is a three-fold interlacing type, the server can also perform three-fold super-resolution processing on candidate samples in the candidate set corresponding to a single-fold interlacing type to obtain a candidate set corresponding to a three-fold interlacing type, and so on.

[0121] S405. The server performs fuzzing on multiple candidate sets that correspond one-to-one with various interleaving types to obtain multiple interleaved sample sets.

[0122] In one possible approach, in order to obtain sample image frames with blur distortion, the server can perform blur processing on multiple candidate sets that correspond one-to-one with multiple interlacing types, namely the candidate set corresponding to the single interlacing type and the candidate set corresponding to each multiple interlacing type, so as to simulate the blur distortion effect present in real interlaced videos and obtain multiple interlaced sample sets that correspond one-to-one with multiple interlacing types.

[0123] Optionally, the server can perform multiple fuzzing processes on the candidate sets corresponding to each interlacing type based on different fuzzing kernels.

[0124] The technical solution provided by the above embodiments brings at least the following beneficial effects: As shown in S401-S405, the server can obtain interlaced video by converting sample non-interlaced video, and further obtain sample image frames with image distortion features such as compression distortion and blurring distortion by transcoding, compression, and blurring of the interlaced video. Based on this, the server can improve the training effect of the initial type detection model and the initial deinterlacing model, improve the detection accuracy of the target type detection model, and improve the distortion repair performance of the target deinterlacing model.

[0125] In one embodiment, combined with Figure 4 In S403 above, when the server determines the candidate set corresponding to the single interlacing type based on the motion information between image frames in the sample non-interlaced video and the compressed interlaced video, such as Figure 5 As shown, this disclosure provides an optional implementation method, including: S501-S503.

[0126] S501. The server decodes the compressed interlaced video and the sample non-interlaced video to obtain multiple single-interlaced image frames and multiple non-interlaced image frames.

[0127] In one possible approach, the server can decode the single-interlaced compressed interlaced video and the sample non-interlaced video to obtain multiple image frames comprising the single-interlaced compressed interlaced video and multiple image frames comprising the sample non-interlaced video. The multiple image frames comprising the single-interlaced compressed interlaced video are equivalent to multiple single-interlaced image frames. The multiple image frames comprising the sample non-interlaced video are equivalent to multiple non-interlaced image frames.

[0128] S502. The server determines the motion information corresponding to each single-interlaced image frame based on multiple non-interlaced image frames.

[0129] Motion information can be used to characterize the relative motion between a non-interlaced image frame and its immediately following non-interlaced image frame. That is, it represents the relative motion information between pixels within two consecutive image frames in a sample non-interlaced video.

[0130] In one possible approach, the server can determine the motion information between any two temporally consecutive non-interlaced image frames within a plurality of non-interlaced image frames. When a single-interlaced image frame is obtained by stitching together every two non-interlaced image frames, the server can determine the motion information between any two temporally consecutive non-interlaced image frames as the motion information corresponding to each single-interlaced image frame.

[0131] In one possible example, a set of ten non-interlaced image frames are pre-defined, arranged in chronological order as A1, A2, B1, B2, C1, C2, D1, D2, E1, and E2. In this case, the corresponding single-interlaced image frames can be five: A, B, C, D, and E. Specifically, single-interlaced image frame A is obtained by stitching together non-interlaced image frames A1 and A2. Single-interlaced image frame B is obtained by stitching together non-interlaced image frames B1 and B2. Single-interlaced image frame C is obtained by stitching together non-interlaced image frames C1 and C2. Single-interlaced image frame D is obtained by stitching together non-interlaced image frames D1 and D2. Single-interlaced image frame E is obtained by stitching together non-interlaced image frames E1 and E2. The server can then determine the relative motion information between non-interlaced image frames A1 and A2 as the motion information corresponding to single-interlaced image frame A. Similarly, the server can obtain motion information for five single-interlaced image frames: A, B, C, D, and E.

[0132] Optionally, when the server determines the motion information between two non-interlaced image frames, it can obtain the pixel difference between two non-interlaced image frames according to the inter-frame difference method, or it can obtain the optical flow map or motion vector map between two non-interlaced image frames according to the optical flow method.

[0133] S503. The server determines the target region within each image frame of the single interlacing type as a candidate sample of the single interlacing type, and obtains the candidate set corresponding to the single interlacing type.

[0134] The target region can be used to characterize the region where the average value of intra-frame motion information of a single-interlaced image is greater than a first preset threshold.

[0135] In one possible approach, to ensure that the candidate samples primarily contain regions with horizontally textured patterns, the server can divide a single-interlaced image frame into multiple regions of preset sizes and determine the average value of motion information for each pixel within each region. Next, the server can identify the region with the highest average value of pixel motion information as the target region. Then, the server can crop the single-interlaced image frame to the target region, obtaining candidate samples for that single-interlaced type. Based on this, the server can obtain a candidate set corresponding to the single-interlaced type. In this case, the first preset threshold can be dynamically adjusted by the server to the maximum value among the average values ​​corresponding to each region.

[0136] Alternatively, the server can identify target regions as those where the average value of the motion information corresponding to a pixel is greater than a first preset threshold. In this case, a single-interlaced image frame can correspond to multiple target regions. The server can then crop out these multiple target regions corresponding to the single-interlaced image frame. Based on this, the server can obtain a candidate set corresponding to the single-interlaced type.

[0137] The technical solution provided by the above embodiments brings at least the following beneficial effects: As can be seen from S501-S503, since the candidate samples are obtained by cropping the regions with larger running information (i.e., target regions) in each image frame, they mainly include regions with horizontal string textures. Therefore, the sample image frames obtained based on the candidate samples can have more horizontal string texture information, thereby improving the training effect of the deinterlacing model.

[0138] In one embodiment, such as Figure 6 As shown, the video processing method provided in this embodiment of the present disclosure further includes: S601-S602.

[0139] S601. The server obtains the set of interlaced samples and the set of real images corresponding to the interlaced type to be trained.

[0140] The real image set can include the real image regions in the non-interlaced video samples that correspond to the image frames of each sample in the interlaced sample set.

[0141] Furthermore, the reference ground truth (i.e. the real image region) corresponding to the sample image frame obtained by cropping the target region in the interlaced video can be obtained by cropping the same region in the sample non-interlaced video.

[0142] It should be understood that the interleaving type to be trained can be any interleaving type, such as single interleaving type or double interleaving type. Furthermore, the specific description of how the server obtains the interleaving sample set corresponding to the interleaving type to be trained can be found in the descriptions in S401-S405 above, and will not be repeated here.

[0143] S602. The server trains the initial deinterlacing model corresponding to the interlacing type to be trained based on the interlaced sample set and the real image set, and obtains the target deinterlacing model corresponding to the interlacing type to be trained.

[0144] In one possible approach, after the server obtains the set of interlaced samples and the set of real images corresponding to the interlaced type to be trained, it can combine the non-interlaced image frames (i.e., real image regions) and the corresponding interlaced image frames (i.e., sample image frames) into a set of data pairs to obtain multiple sets of data pairs.

[0145] Next, the server can input the obtained multiple sets of data pairs in batches into the initial deinterlacing model corresponding to the interlacing type to be trained for training. Specifically, the server can use the initial deinterlacing model to deinterlac the sample image frames in the data pair to obtain the processing result, and determine the loss value based on the real image region in the data pair and the obtained processing result. If the loss value is large, the operator can adjust the hyperparameters in the initial deinterlacing model through the terminal. If the loss value converges, the server can obtain the target deinterlacing model corresponding to the interlacing type to be trained.

[0146] In one possible approach, when the server determines the loss value based on the real image region in the data pair and the obtained processing result, it can use a point-by-point pixel loss function (also known as Charbonnier loss), or other suitable loss functions.

[0147] In one possible approach, the backbone network of the initial deinterlacing model could be a processing-repair type of network such as an enhanced deep super-resolution network (EDSR) or a deep residual channel attention network (RCAN).

[0148] It should be noted that, since the scan field of a single-interlaced image frame can be easily divided—that is, odd-numbered rows of pixels belong to the upper scan field, and even-numbered rows belong to the lower scan field (or vice versa)—in practical applications, when processing single-interlaced interlaced video, it is also possible to choose to extract only the information from the upper scan field or the lower scan field to improve the flexibility of deinterlacing processing.

[0149] Therefore, when the interleaving type to be trained is a single interleaving type, the staff can pre-set the scan field partitioning method in the initial deinterleaving model based on prior knowledge. In this case, the initial deinterleaving model can also be used to partition the input sample image frame into scan fields, and then perform deinterleaving processing on the upper scan field and the lower scan field respectively, thereby training a target deinterleaving model that only processes the upper scan field or the lower scan field.

[0150] Optionally, since there may be mixed interlacing types among two-sided and above interlacing types, such as single-fold interlacing and double-fold interlacing, or single-fold interlacing, double-fold interlacing, and triple-fold interlacing, etc., in order to improve the processing effect for two-sided and above interlacing types, the scanning field division process can be cancelled in the deinterlacing model of two-sided and above interlacing types.

[0151] In one possible approach, when the interleaving type to be trained is a double-interleaving type, the hyperparameters in the initial deinterleaving model corresponding to the double-interleaving type can be initialized by referring to the hyperparameters in the target deinterleaving model corresponding to the trained single-interleaving type. Furthermore, the initial deinterleaving model corresponding to the double-interleaving type can be used to directly deinterleave sample image frames corresponding to the double-interleaving type. Moreover, by modifying the output layer logic, the output of the initial deinterleaving model corresponding to the double-interleaving type can also be divided into an upper scan field and a lower scan field.

[0152] Alternatively, the initial deinterlacing model corresponding to the double-interlacing type can also be used to sequentially perform double downsampling, deinterlacing, double upsampling, and residual summation on the input sample image frames to obtain the output result.

[0153] In one possible approach, when the interlacing type to be trained is a high-magnification interlacing type (more than twice the size of the target interlacing type), the initial deinterlacing model corresponding to this high-magnification interlacing type can be used to first perform high-magnification downsampling on the input sample image frames, restoring the high-magnification interlacing type sample image frames to approximately single-magnification interlacing type sample image frames. These frames are then input into a single-magnification interlacing type processing network for deinterlacing. The hyperparameters of the single-magnification interlacing type processing network can be initialized by referring to the hyperparameters in the trained target deinterlacing model corresponding to the single-magnification interlacing type.

[0154] Furthermore, after processing the single-fold interleaving network, two convolutional layers can be stacked to achieve super-resolution processing (e.g., upsampling), and a residual layer is connected after the two convolutional layers to achieve residual addition. Based on this, the initial deinterleaving model corresponding to this high-fold interleaving type can perform high-fold upsampling on the deinterleaving result to restore the output to the high-fold interleaving type. Then, the initial deinterleaving model corresponding to this high-fold interleaving type can perform residual addition on the output result and then enter the final convolutional layer for output. Residual addition adds the original input sample image frame and the result after high-fold upsampling to better preserve the spatial texture information of the original input data in the output result.

[0155] The technical solution provided by the above embodiments brings at least the following beneficial effects: As shown in S601-S602, the server can train the initial deinterlacing model corresponding to various interlacing types using the interlaced sample set and the real image set corresponding to various interlacing types, thereby obtaining the target deinterlacing model corresponding to various interlacing types. Based on this, each target deinterlacing model can be trained based on the interlaced sample set of the corresponding interlacing type, and can have a better deinterlacing processing effect when processing image frames of the corresponding interlacing type.

[0156] In one embodiment, combined with Figure 2 In S202 above, when the server determines the video type of the video to be processed based on a pre-trained target type detection model, such as... Figure 7 As shown, this disclosure provides an optional implementation, including: S701-S704.

[0157] S701, The server divides each image frame to be processed into multiple image blocks to be processed.

[0158] It should be noted that, considering the potentially complex mixed interlacing types in real interlaced videos, to improve the accuracy of type detection in the video to be processed, the image frames to be processed can be divided into smaller image blocks for detection. Based on this, the server can divide each image frame to be processed according to a preset size, resulting in multiple image blocks to be processed.

[0159] S702. The server inputs each image block to be processed into the target type detection model to obtain the detection information of each image block to be processed.

[0160] The detection information for the image patch to be processed may include the type label of the image patch and the confidence level of the type label. The type label of the image patch to be processed may be a non-interlaced type or any of the interlaced types.

[0161] In one possible approach, the server can input each image block to be processed into the target type detection model to obtain the detection information of each image block, so as to determine the type label and type label confidence of each image block.

[0162] S703, The server determines the average confidence level of various types of type labels.

[0163] In one possible approach, the server can classify the detection information of multiple image patches to be processed, i.e., count the number of type labels for each type. Then, the server can determine the average confidence level of each type of type label.

[0164] S704. The server determines the type label of the video to be processed as the type of the video whose average confidence level is greater than or equal to the second preset threshold.

[0165] In one possible approach, after determining the average confidence level of various type labels, the server can set the average value with the highest confidence level as a second preset threshold. Based on this, the server can determine the type labels with an average confidence level greater than or equal to the second preset threshold, i.e., the type labels with the highest average confidence level, as the video type of the video to be processed.

[0166] The technical solution provided by the above embodiments brings at least the following beneficial effects: As shown in S701-S704, the server can divide the image frame to be processed into smaller image blocks for detection, and further determine the video type of the video to be processed based on the average confidence of the type labels of multiple image blocks. Therefore, in practical application scenarios where complex mixed interlacing types may exist in real interlaced videos, this disclosure can more accurately determine the video type of the video to be processed, avoiding video degradation problems easily caused by false detections and other errors.

[0167] In one embodiment, such as Figure 8 As shown, the video processing method provided in this embodiment of the present disclosure further includes: S801-S802.

[0168] S801 and the server perform quality assessments on the non-interlaced video corresponding to the video to be processed and the video to be processed, respectively, to obtain the assessment results after deinterlacing and the assessment results before deinterlacing.

[0169] The evaluation results after deinterlacing can be used to characterize the video quality information of the non-interlaced video corresponding to the video to be processed in multiple dimensions such as brightness, contrast, and distortion. The evaluation results before deinterlacing can be used to characterize the video quality information of the video to be processed in multiple dimensions such as brightness, contrast, and distortion.

[0170] In one possible approach, the server can input each image frame in the non-interlaced video corresponding to the video to be processed, as well as each image frame to be processed in the video to be processed, into a preset evaluation model for quality evaluation, and obtain the evaluation results after deinterlacing and the evaluation results before deinterlacing.

[0171] Alternatively, the server can input the image frames with partial timestamps from the non-interlaced video corresponding to the video to be processed, as well as the image frames with timestamps that are the same as those in the video to be processed, into a preset evaluation model for quality evaluation, and obtain the evaluation results after deinterlacing and the evaluation results before deinterlacing.

[0172] Optionally, the preset evaluation model can be a general model such as the natural image quality evaluator (NIQE) model and the blind video quality assessment (BVQA) model.

[0173] S802. The server determines the comparative evaluation result based on the evaluation result after deinterleaving and the evaluation result before deinterleaving.

[0174] Among them, the comparative evaluation results can be used to characterize whether the quality of the video to be processed is higher than that of the corresponding non-interlaced video, or whether the quality of the video to be processed is lower than that of the corresponding non-interlaced video.

[0175] It should be noted that to avoid image quality degradation caused by false detections in the target type detection model—for example, misdetecting a 3x interlacing type as a 4x interlacing type—the image quality of the video to be processed will degrade after deinterlacing according to the 4x interlacing type. In such cases, it is advisable to evaluate the original video to be processed and the deinterlaced video. If the image quality decreases, the result after deinterlacing should not be used. If the image quality improves, the result after deinterlacing should be used.

[0176] Based on this, the server can compare the evaluation results after deinterlacing with the evaluation results before deinterlacing to determine the quality of the video to be processed and the quality of the corresponding non-interlaced video.

[0177] If the quality of the video to be processed is higher than that of its corresponding non-interlaced video, it indicates that the quality of the video to be processed has decreased after deinterlacing. In this case, the server can delete the corresponding non-interlaced video and use the video to be processed as the output.

[0178] If the quality of the video to be processed is lower than that of its corresponding non-interlaced video, it indicates that the quality of the video to be processed has been improved after deinterlacing. In this case, the server can use the corresponding non-interlaced video as the output.

[0179] The technical solution provided by the above embodiments brings at least the following beneficial effects: As shown in S801-S802, after the server performs deinterlacing processing on the video to be processed, it can also perform quality assessments on the non-interlaced video corresponding to the video to be processed and the video to be processed, respectively, to further determine the comparative assessment results. Based on this, this disclosure can provide a backup for the effect of deinterlacing processing, avoid video degradation problems that are easily caused by errors such as false detections during the deinterlacing processing, and improve the stability of deinterlacing processing.

[0180] In one embodiment, such as Figure 9 The diagram shown illustrates a video processing method provided in this disclosure. After obtaining the video to be processed, the server can execute a video type detection process. That is, the server can perform video type detection on the video to be processed based on a target type detection model. If the video to be processed is not an interlaced type, i.e., a non-interlaced video, no processing is performed. If the video to be processed is an interlaced type, i.e., an interlaced video, the interlaced type is output, i.e., the target interlaced type corresponding to the video to be processed.

[0181] Next, the server can execute the video deinterlacing process. That is, the server can select a target deinterlacing model corresponding to the target interlacing type, perform deinterlacing processing on the video to be processed, and output the processing result. For example, when the target interlacing type is 1x interlacing, the server can select a target deinterlacing model corresponding to the single interlacing type and perform single-level deinterlacing processing on the video to be processed. When the target interlacing type is 2x interlacing, the server can select a target deinterlacing model corresponding to the double interlacing type and perform double-level deinterlacing processing on the video to be processed. When the target interlacing type is 3x interlacing, the server can select a target deinterlacing model corresponding to the triple interlacing type and perform triple-level deinterlacing processing on the video to be processed. When the target interlacing type is 4x interlacing, the server can select a target deinterlacing model corresponding to the quadruple interlacing type and perform quadruple-level deinterlacing processing on the video to be processed.

[0182] Next, the server can execute a video deinterlacing quality evaluation process. This means the server can evaluate the quality of the video deinterlacing process output compared to the video to be processed. If the deinterlacing process output shows an improvement in quality compared to the video to be processed, the server can determine that the deinterlacing process output is the output. If the deinterlacing process output does not show an improvement in quality compared to the video to be processed, the server can decide not to process it and determine the video to be processed as the output.

[0183] Table 1 below shows comparative experimental data for the method of this disclosure, the Yet Another Deinterlacing Filter (YADIF) method in Fast Forward MPEG (FFMPEG), and the Bob Weaver Deinterlacing Filter (BWDIF) method in FFMPEG. In the comparative experiments, the method of this disclosure, the YADIF method, and the BWDIF method were used to deinterlace interlaced videos of the same interlacing type. The comparative experimental data includes the results of each method in terms of peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

[0184] Table 1

[0185]

[0186] As can be seen, the processing effect of the method disclosed herein is superior to that of the YADIF and BWDIF methods for both single-interlaced and double-interlaced types. Especially for double-interlaced types, neither the YADIF nor BWDIF methods are suitable for deinterlacing. Furthermore, as shown in Table 1 below, the method disclosed herein can achieve better deinterlacing results for double-interlaced types. However, since the YADIF and BWDIF methods can only handle single-interlaced types, they exacerbate the interlacing streaking phenomenon in double-interlaced videos, leading to further degradation of video quality.

[0187] Furthermore, such as Figure 10 The image shown is a schematic diagram of an image frame provided in an embodiment of this application. Figure 10 (a) in the image shows a raw image frame. Figure 10 (b) shows the image frame after deinterlacing the original image frame based on the YADIF method. Figure 10 Figure (c) shows an image frame after deinterlacing the original image frame using the method of this disclosure. As indicated by the arrows in the figure, the image frame processed using the method of this disclosure no longer has image distortion and has better clarity.

[0188] It is understood that, in actual implementation, the server described in the embodiments of this disclosure may include one or more hardware structures and / or software modules for implementing the aforementioned corresponding video processing methods, and these hardware structures and / or software modules may constitute an electronic device. Those skilled in the art should readily recognize that, based on the algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0189] Based on this understanding, this disclosure also provides a video processing apparatus. Figure 11 A schematic diagram of the structure of a video processing apparatus provided in an embodiment of this disclosure is shown. Figure 11 As shown, the video processing device may include: an acquisition unit 901, a determination unit 902, and a processing unit 903.

[0190] The acquisition unit 901 is configured to acquire a video to be processed. The video to be processed includes multiple image frames to be processed.

[0191] The determination unit 902 is configured to execute a pre-trained target type detection model to determine the video type of the video to be processed. The video type is either a non-interlaced type or a target interlaced type among multiple interlaced types.

[0192] The processing unit 903 is configured to perform deinterlacing processing on multiple image frames to be processed when the video type is the target interlacing type among multiple interlacing types, by inputting the deinterlacing processing on the target deinterlacing model corresponding to the target interlacing type, and obtaining the non-interlaced video corresponding to the video to be processed.

[0193] Optionally, the acquisition unit 901 is further configured to acquire multiple interleaved sample sets and multiple non-interleaved sample image frames. The multiple interleaved sample sets correspond one-to-one with multiple interleaved types. Each interleaved sample set includes multiple sample image frames of the interleaved type corresponding to the interleaved sample set. The processing unit 903 is configured to train the initial type detection model based on the multiple non-interleaved sample image frames and the multiple interleaved sample sets to obtain a target type detection model.

[0194] Optionally, the interlaced sample image frames possess at least one distortion feature among compression distortion and blurring distortion. Multiple interlacing types include single-interlacing and multi-interlacing types. The acquisition unit 901 is specifically configured to perform the following: update parameters of a preset type within the sample non-interlaced video to obtain a single-interlaced interlaced video; transcode and compress the single-interlaced interlaced video to obtain a compressed single-interlaced interlaced video; determine a candidate set corresponding to the single-interlacing type based on the motion information between image frames in the sample non-interlaced video and the compressed interlaced video; the candidate set corresponding to the single-interlacing type includes multiple candidate samples of the single-interlacing type; perform super-resolution processing at different magnifications on the candidate samples in the candidate set corresponding to the single-interlacing type to obtain candidate sets corresponding to each multi-interlacing type; and blur the multiple candidate sets that correspond one-to-one with the multiple interlacing types to obtain multiple interlaced sample sets.

[0195] Optionally, the acquisition unit 901 is specifically configured to perform the following: decoding processing on the compressed interlaced video and sample non-interlaced video to obtain multiple single-interlaced image frames and multiple non-interlaced video frames. Based on the multiple non-interlaced video frames, motion information corresponding to each single-interlaced image frame is determined. The motion information is used to characterize the relative motion information between the non-interlaced image frame and its immediately following non-interlaced image frame. Target regions within the single-interlaced image frames are determined as candidate samples for the single-interlaced type, resulting in a candidate set corresponding to the single-interlaced type. The target regions are used to characterize regions where the average value of the motion information within the single-interlaced image frames is greater than a first preset threshold.

[0196] Optionally, the acquisition unit 901 is further configured to acquire the set of interlaced samples and the set of real images corresponding to the interlaced type to be trained. The set of real images includes the real image regions in the non-interlaced video samples that correspond to each sample image frame in the interlaced sample set. The processing unit 903 is further configured to train the initial deinterlacing model corresponding to the interlaced type to be trained based on the set of interlaced samples and the set of real images, to obtain the target deinterlacing model corresponding to the interlaced type to be trained.

[0197] Optionally, when the interleaving type to be trained is a single interleaving type, the initial deinterleaving model is used to divide the input sample image frame into scan fields, and then perform deinterleaving processing on the upper scan field and the lower scan field respectively. When the interleaving type to be trained is a multiple interleaving type, the initial deinterleaving model is used to sequentially perform multiple downsampling processing, deinterleaving processing, multiple upsampling processing, and residual summation processing on the input sample image frame.

[0198] Optionally, the determining unit 902 is specifically configured to perform the following: dividing each image frame to be processed into multiple image blocks to be processed; inputting each image block to be processed into a target type detection model to obtain detection information for each image block to be processed; the detection information for each image block to be processed includes the type label of the image block to be processed and the confidence level of the type label; determining the average confidence level of the type labels for various types; and determining the type labels whose average confidence level is greater than or equal to a second preset threshold as the video type of the video to be processed.

[0199] Optionally, the processing unit 903 is further configured to perform quality assessments on the non-interlaced video corresponding to the video to be processed and the video to be processed, respectively, to obtain an assessment result after deinterlacing and an assessment result before deinterlacing. The determining unit 902 is further configured to determine a comparative assessment result based on the assessment result after deinterlacing and the assessment result before deinterlacing.

[0200] As described above, the embodiments of this disclosure can divide the terminal into functional modules according to the above method examples. The integrated modules can be implemented in hardware or as software functional modules. Furthermore, it should be noted that the module division in these embodiments is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into a single processing module.

[0201] The specific methods by which each module performs its operation and the beneficial effects of the video processing apparatus in the above embodiments have been described in detail in the foregoing method embodiments, and will not be repeated here.

[0202] This disclosure also provides a terminal, which may be a user terminal such as a mobile phone or a computer. Figure 12 A schematic diagram of the structure of a terminal provided in an embodiment of this disclosure is shown. The terminal may be a video processing device and may include at least one processor 61, a communication bus 62, a memory 63, and at least one communication interface 64.

[0203] Processor 61 may be a processor (central processing unit, CPU), microprocessor unit, ASIC, or one or more integrated circuits for controlling the execution of programs according to the present disclosure.

[0204] The communication bus 62 may include a path for transmitting information between the aforementioned components.

[0205] Communication interface 64 uses any transceiver-like device for communicating with other devices or communication networks, such as servers, Ethernet, radio access networks (RAN), wireless local area networks (WLAN), etc.

[0206] Memory 63 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory may exist independently and be connected to the processing unit via a bus. Memory may also be integrated with the processing unit.

[0207] The memory 63 stores the application code that executes the present invention, and its execution is controlled by the processor 61. The processor 61 executes the application code stored in the memory 63 to implement the functions of the method of the present invention.

[0208] In a specific implementation, as one example, processor 61 may include one or more CPUs, for example... Figure 12CPU0 and CPU1 in the CPU.

[0209] In a specific implementation, as one example, the terminal may include multiple processors, for example... Figure 12 Processors 61 and 65 are included. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0210] In a specific implementation, as one embodiment, the terminal may further include an input device 66 and an output device 67. The input device 66 and output device 67 communicate and can accept user input in various ways. For example, the input device 66 may be a mouse, keyboard, touchscreen device, or sensing device. The output device 67 communicates with the processor 61 and can display information in various ways. For example, the output device 61 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, etc.

[0211] Those skilled in the art will understand that Figure 12 The structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0212] This disclosure also provides a server. Figure 13 A schematic diagram of the server structure provided in an embodiment of this disclosure is shown. This server can be a video processing device. The server can vary significantly due to differences in configuration or performance, and may include one or more processors 71 and one or more memories 72. The memory 72 stores at least one instruction, which is loaded and executed by the processor 71 to implement the video processing methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0213] This disclosure also provides a computer-readable storage medium including instructions stored thereon, which, when executed by a processor of a computer device, enable the computer to perform the video processing method provided in the embodiments shown above. For example, the computer-readable storage medium may be a memory 63 including instructions, which may be executed by a processor 61 of a terminal to complete the method. As another example, the computer-readable storage medium may be a memory 72 including instructions, which may be executed by a processor 71 of a server to complete the method. Optionally, the computer-readable storage medium may be a non-transitory computer-readable storage medium, such as ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0214] This disclosure also provides a computer program product including computer instructions that, when executed on an electronic device, cause the electronic device to perform the above-described actions. Figures 2-9 The video processing method is shown in any of the attached figures.

[0215] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0216] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video processing method, characterized in that, include: Get the video to be processed; The video to be processed includes multiple image frames to be processed; Acquire multiple sets of interleaved samples and multiple non-interleaved sample image frames; The multiple interleaved sample sets correspond one-to-one with various interleaving types; each interleaved sample set includes multiple sample image frames of interleaving types corresponding to the interleaved sample set. The initial type detection model is trained based on the multiple non-interleaved sample image frames and the multiple interleaved sample sets to obtain the target type detection model; Based on a pre-trained target type detection model, the video type of the video to be processed is determined; the video type is a non-interlaced type, or a target interlaced type among multiple interlaced types. When the video type is the target interlacing type among the multiple interlacing types, the multiple image frames to be processed are input into the target deinterlacing model corresponding to the target interlacing type for deinterlacing processing to obtain the non-interlaced video corresponding to the video to be processed; The interlaced sample image frames possess at least one distortion feature between compression distortion and blurring distortion; the multiple interlacing types include single interlacing and multiple interlacing; obtaining multiple interlaced sample sets includes: The parameters of the preset type in the non-interlaced sample video are updated to obtain the interlaced video of the single interlaced type. The interlaced video of the single interlacing type is transcoded and compressed to obtain the compressed interlaced video of the single interlacing type. Based on the motion information between image frames in the non-interlaced video sample and the compressed interlaced video, a candidate set corresponding to the single interlacing type is determined; the candidate set corresponding to the single interlacing type includes multiple candidate samples of the single interlacing type. For the candidate samples in the candidate set corresponding to the single-fold interleaving type, super-resolution processing of different multiples is performed to obtain the candidate set corresponding to each of the multiple-fold interleaving types; The multiple candidate sets corresponding one-to-one with the various interleaving types are fuzzed to obtain the multiple interleaved sample sets.

2. The video processing method according to claim 1, characterized in that, The step of determining the candidate set corresponding to the single interlacing type based on the motion information between image frames in the sample non-interlaced video and the compressed interlaced video includes: The compressed interlaced video and the sample non-interlaced video are decoded to obtain multiple single-interlaced image frames and multiple non-interlaced image frames; Based on the plurality of non-interlaced image frames, motion information corresponding to each of the single-interlaced image frames is determined; the motion information is used to characterize the relative motion information between the non-interlaced image frame and the next adjacent non-interlaced image frame. The target region within the image frame of the single-interlaced type is determined as a candidate sample of the single-interlaced type, and a candidate set corresponding to the single-interlaced type is obtained; the target region is used to characterize the region where the average value of the motion information within the image frame of the single-interlaced type is greater than a first preset threshold.

3. The video processing method according to claim 1, characterized in that, Also includes: Obtain the set of interleaved samples and the set of real images corresponding to the interleaved type to be trained; the set of real images includes the real image regions in the non-interleaved video samples that correspond to each sample image frame in the set of interleaved samples. Based on the set of interleaved samples and the set of real images, the initial deinterleaving model corresponding to the interleaving type to be trained is trained to obtain the target deinterleaving model corresponding to the interleaving type to be trained.

4. The video processing method according to claim 3, characterized in that, When the interleaving type to be trained is a single interleaving type, the initial deinterleaving model is used to divide the input sample image frame into scan fields, and then perform deinterleaving processing on the upper scan field and the lower scan field respectively. When the interleaving type to be trained is a multiple interleaving type, the initial deinterleaving model is used to sequentially perform multiple downsampling, deinterleaving, multiple upsampling, and residual addition on the input sample image frames.

5. The video processing method according to claim 1, characterized in that, The process of determining the video type of the video to be processed based on a pre-trained target type detection model includes: Each of the image frames to be processed is divided into multiple image blocks to be processed; Each of the image blocks to be processed is input into the target type detection model to obtain the detection information of each image block to be processed; the detection information of the image block to be processed includes the type label of the image block to be processed and the confidence level of the type label; Determine the average confidence level for each type of type label; The type label whose average confidence level is greater than or equal to the second preset threshold is determined as the video type of the video to be processed.

6. The video processing method according to claim 1, characterized in that, Also includes: The quality of the non-interlaced video corresponding to the video to be processed and the video to be processed are evaluated respectively to obtain the evaluation results after deinterlacing and the evaluation results before deinterlacing. The comparison evaluation result is determined based on the evaluation results after deinterlacing and the evaluation results before deinterlacing.

7. A video processing apparatus, characterized in that, include: Acquisition unit, determination unit, and processing unit; The acquisition unit is configured to acquire the video to be processed; The video to be processed includes multiple image frames to be processed; The acquisition unit is further configured to acquire multiple interleaved sample sets and multiple non-interleaved sample image frames; the multiple interleaved sample sets correspond one-to-one with multiple interleaved types; the interleaved sample sets include multiple sample image frames of interleaved types corresponding to the interleaved sample sets. The processing unit is configured to train an initial type detection model based on the plurality of non-interleaved sample image frames and the plurality of interleaved sample sets to obtain a target type detection model; The determining unit is configured to execute a pre-trained target type detection model to determine the video type of the video to be processed. The video type is either a non-interlaced type or a target interlaced type among multiple interlaced types; The processing unit is configured to, when the video type is a target interlacing type among the multiple interlacing types, input the multiple image frames to be processed into a target deinterlacing model corresponding to the target interlacing type for deinterlacing processing, so as to obtain a non-interlaced video corresponding to the video to be processed; The interlaced sample image frames possess at least one distortion feature, either compression distortion or blurring distortion; the various interlacing types include single interlacing and multiple interlacing. The acquisition unit is specifically configured to perform parameter update processing on a preset type of sample non-interlaced video to obtain an interlaced video of single interlaced type. The interlaced video of the single interlacing type is transcoded and compressed to obtain the compressed interlaced video of the single interlacing type. Based on the motion information between image frames in the non-interlaced video sample and the compressed interlaced video, a candidate set corresponding to the single interlacing type is determined; the candidate set corresponding to the single interlacing type includes multiple candidate samples of the single interlacing type. For the candidate samples in the candidate set corresponding to the single-fold interleaving type, super-resolution processing of different multiples is performed to obtain the candidate set corresponding to each of the multiple-fold interleaving types; The multiple candidate sets corresponding one-to-one with the various interleaving types are fuzzed to obtain the multiple interleaved sample sets.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the video processing method as described in any one of claims 1-6.

9. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the video processing method as described in any one of claims 1-6.

Images