Video deinterlacing method and deinterlacing model generation method

Abstract

The present application relates to a video deinterlacing method and a method for generating a deinterlacing model. The video deinterlacing method comprises: obtaining an interlaced video to be processed, segmenting each interlaced frame in the interlaced video to be processed, and obtaining a plurality of fields to be processed; based on the plurality of fields to be processed, determining a set to be processed for each field to be processed; for any field to be processed, inputting the set to be processed for any field to be processed into a spatial domain deinterlacing module in a trained deinterlacing model, and obtaining a plurality of upsampled feature maps corresponding to any field to be processed; inputting the set to be processed for any field to be processed, and the plurality of upsampled feature maps corresponding to any field to be processed, into a temporal domain deinterlacing module in the trained deinterlacing model, and obtaining a target frame corresponding to any field to be processed, and generating a target deinterlaced video according to each target frame. The method has a small amount of calculation and low hardware requirements.

Description

Technical Field

[0001] The present application relates to the technical field of video processing, and in particular to a video deinterlacing method and a method for generating a deinterlacing model. Background Art

[0002] In the past analog TV broadcasting systems, interlaced scanning technology was used to reduce the amount of data and ensure the frame rate. Each frame of the image was completed through two field scans. In the first field of the two field scans, only odd-numbered lines were scanned, and lines 1, 3, 5, etc. were scanned in sequence. In the second field, only even-numbered lines were scanned, and lines 2, 4, 6, etc. were scanned in sequence. Interlaced scanning technology played a great role in the case of insufficient transmission signal bandwidth.

[0003] In order to make the interlaced video display better in the progressive scanning device, the interlaced video needs to be deinterlaced. The basic principle of deinterlacing is to insert the pixel values ​​of the missing lines in the interlaced image according to the correlation of image information in time and space.

[0004] Existing deinterlacing processing methods include two categories: non-motion compensation and motion compensation. The accuracy of non-motion compensation algorithms is relatively low. The motion compensation method makes full use of the image information within and between fields. The image information interpolated by it is smooth and natural in motion, without obvious steps and jagged edges. The motion compensation accuracy is relatively high, but the motion compensation method is complex to calculate, labor-intensive, and has high hardware requirements, resulting in high implementation costs. Summary of the invention

[0005] Based on this, it is necessary to provide a video deinterlacing method and a deinterlacing model generation method with small computational complexity and low hardware requirements to address the above technical problems.

[0006] In a first aspect, the present application provides a video deinterlacing method. The video deinterlacing method comprises:

[0007] Acquire an interlaced video to be processed, and perform segmentation processing on each interlaced frame in the interlaced video to be processed to obtain a plurality of fields to be processed, wherein the plurality of fields to be processed include a plurality of odd fields and a plurality of even fields;

[0008] Based on the multiple fields to be processed, determining a set to be processed for each field to be processed, wherein each set to be processed includes at least one odd field and one even field;

[0009] For any field to be processed, inputting the processing set of any field to be processed into a spatial domain deinterlacing module in a trained deinterlacing model to obtain a plurality of upsampled feature maps corresponding to any field to be processed;

[0010] The processing set of any field to be processed and the multiple up-sampled feature maps corresponding to any field to be processed are input into the time domain deinterlacing module in the trained deinterlacing model to obtain the target frame corresponding to any field to be processed, and generate a target deinterlaced video according to each target frame.

[0011] In a second aspect, the present application further provides a method for generating a deinterlacing model. The method for generating a deinterlacing model comprises:

[0012] Acquire an interlaced training video and a reference progressive video corresponding to the interlaced training video, and segment each training interlaced frame in the interlaced training video to obtain a plurality of training fields, wherein the plurality of training fields include a plurality of odd-numbered training fields and a plurality of even-numbered training fields;

[0013] Based on the multiple training fields, determining a training set for each training field, wherein each training set includes at least one odd-numbered training field and one even-numbered training field;

[0014] Based on the reference progressive video and the training set of each training field, iteratively train the initial spatial de-interlacing module to obtain a spatial de-interlacing module;

[0015] Based on the reference progressive video, the training set of each training field and the spatial de-interlacing module, iteratively train the initial temporal de-interlacing module to obtain a temporal de-interlacing module;

[0016] A trained deinterlacing model is determined according to the spatial domain deinterlacing module and the temporal domain deinterlacing module, wherein the trained deinterlacing model is a trained deinterlacing model in the above-mentioned video deinterlacing method.

[0017] In a third aspect, the present application further provides a video deinterlacing device, the video deinterlacing device comprising:

[0018] A to-be-processed field determination module is used to obtain an interlaced video to be processed, and to segment each interlaced frame in the interlaced video to be processed to obtain a plurality of to-be-processed fields, wherein the plurality of to-be-processed fields include a plurality of odd fields and a plurality of even fields;

[0019] A to-be-processed set determination module, used to determine a to-be-processed set of each to-be-processed field based on the plurality of to-be-processed fields, wherein each to-be-processed set includes at least one odd-numbered field and one even-numbered field;

[0020] An up-sampled feature map determination module is used for inputting a processing set of any field to be processed into a spatial domain de-interlacing module in a trained de-interlacing model to obtain a plurality of up-sampled feature maps corresponding to any field to be processed;

[0021] The target deinterlaced video generation module is used to input the processing set of any field to be processed and the multiple up-sampled feature maps corresponding to any field to be processed into the time domain deinterlacing module in the trained deinterlacing model, obtain the target frame corresponding to any field to be processed, and generate a target deinterlaced video according to each target frame.

[0022] In a fourth aspect, the present application further provides a device for generating a de-interlacing model, the device for generating a de-interlacing model comprising:

[0023] A training field determination module is used to obtain an interlaced training video and a reference progressive video corresponding to the interlaced training video, and to segment each training interlaced frame in the interlaced training video to obtain a plurality of training fields, wherein the plurality of training fields include a plurality of odd-numbered training fields and a plurality of even-numbered training fields;

[0024] A training set determination module, configured to determine a training set for each training field based on the plurality of training fields, wherein each training set includes at least one odd-numbered training field and one even-numbered training field;

[0025] A first training module is used to iteratively train an initial spatial de-interlacing module based on the reference progressive video and a training set of each training field to obtain a spatial de-interlacing module;

[0026] A second training module is used to iteratively train the initial temporal deinterlacing module based on the reference progressive video, the training set of each training field and the spatial deinterlacing module to obtain a temporal deinterlacing module;

[0027] The deinterlacing model determination module is used to determine a trained deinterlacing model according to the spatial domain deinterlacing module and the temporal domain deinterlacing module, wherein the trained deinterlacing model is the trained deinterlacing model in the above-mentioned video deinterlacing method.

[0028] In a fifth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the video deinterlacing method described in the first aspect or the deinterlacing model generation method described in the second aspect is implemented.

[0029] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the video deinterlacing method described in the first aspect or the method for generating a deinterlacing model described in the second aspect is implemented.

[0030] In a seventh aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the video deinterlacing method described in the first aspect, or the deinterlacing model generation method described in the second aspect.

[0031] The above-mentioned video deinterlacing method, deinterlacing model generation method, video deinterlacing device, deinterlacing model generation device, computer equipment, storage medium and computer program product, the video deinterlacing method divides each interlaced frame of the interlaced video to be processed into multiple fields to be processed, determines a processing set for each field to be processed among the multiple fields to be processed, determines the target frame of each field to be processed based on the processing set of each field to be processed and the trained deinterlacing model, and then obtains the target deinterlaced video. Compared with the motion compensation method in the prior art, the above-mentioned video deinterlacing method effectively extracts spatial information through a spatial deinterlacing module to obtain multiple up-sampled feature maps of the field to be processed, effectively extracts temporal information through a temporal deinterlacing module to obtain a spatiotemporal residual map, interpolates the field to be processed through the temporal deinterlacing module to obtain an estimated map, and uses the spatiotemporal residual map to correct the estimated map to obtain a target frame. The amount of calculation is much smaller than that of the existing motion compensation method. Deinterlacing processing is performed by the video deinterlacing method, the workload is small, the hardware requirements are low, and the cost is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic flow chart of a video deinterlacing method in one embodiment;

[0033] Figure 2 A schematic diagram of obtaining an interlaced training video according to a reference progressive video scan in one embodiment;

[0034] Figure 3 A schematic diagram of dividing a training interlaced frame into an even-numbered training field and an odd-numbered training field in one embodiment;

[0035] Figure 4 A schematic diagram of determining a first training progressive frame according to a first training field, a training set of the first training field and an initial spatial de-interlacing module in an embodiment;

[0036] Figure 5 It is a schematic diagram of obtaining a first up-sampled training feature map according to a spatial training feature map and an initial pixel reorganization layer in one embodiment;

[0037] Figure 6 A schematic diagram of determining a second training progressive frame according to a second training field, a training set of the second training field, a spatial de-interlacing module and an initial temporal de-interlacing module in an embodiment;

[0038] Figure 7is a schematic flow chart of a method for generating a de-interlacing model in an embodiment;

[0039] Figure 8 A schematic diagram of obtaining a target frame of any field to be processed according to any field to be processed, any set to be processed of the field to be processed and a trained de-interlacing model in an embodiment;

[0040] Fig. 9 is a structural block diagram of a video deinterlacing device in one embodiment;

[0041] Fig.10 is a structural block diagram of a device for generating a de-interlacing model in an embodiment;

[0042] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] In one embodiment, Figure 1 As shown, a method for generating a deinterlacing model is provided. This embodiment takes the method applied to a terminal as an example for explanation. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method for generating a deinterlacing model includes the following steps:

[0045] S101, obtaining an interlaced training video and a reference progressive video corresponding to the interlaced training video, and segmenting each training interlaced frame in the interlaced training video to obtain a plurality of training frames.

[0046] The interlaced training video includes a plurality of training interlaced frames, and the plurality of training fields include a plurality of odd-numbered training fields and a plurality of even-numbered training fields.

[0047] Specifically, an interlaced training video is obtained from a training video set, the training video set including multiple interlaced training videos and a reference progressive video corresponding to each interlaced training video. The interlaced training video may be obtained by interlacing a reference progressive video. The reference progressive video includes multiple reference progressive frames, the number of which is twice the number of training interlaced frames; two adjacent reference progressive frames include a first reference progressive frame and a second reference progressive frame, and a training interlaced frame is obtained by obtaining all even-numbered pixel points in the first reference progressive frame, obtaining all odd-numbered pixel points in the second reference progressive frame, all even-numbered pixel points and all odd-numbered pixel points.

[0048] For example, a reference progressive video I includes 2t reference progressive frames: I 1 ,I 2 ,……,I 2t , the interlaced training video S consists of t reference progressive frames: S 1 ,S 2 ,……,S t . See Figure 2 , get I 1 All even-numbered rows (I 1 The pixel point marked as 1+ in the row is obtained by 2 All odd-numbered rows (I 2 The pixel points marked as 2- in the row are obtained by S 1 ; Get I 3 All pixels in the even-numbered rows in I 4 All the pixels in the odd rows in the 3 All even-numbered rows (I 3 The pixels in the row marked as 3+ in the figure, and I 4 All odd-numbered rows (I 4 The pixel points marked as 4- in the row are obtained by S 2 ; ...; Get I 2t-1 All pixels in the even-numbered rows in I 2t All the pixels in the odd rows in the 2t-1 All pixels in the even-numbered rows, and I 2t All the pixels in the odd rows in S t .

[0049] The reference progressive video is interlaced to obtain an interlaced training video, as shown in formula (1).

[0050]

[0051] Among them, S n Indicates the nth training interlaced frame in the interlaced training video, I 2n-1is the 2n-1th reference progressive frame in the reference progressive video, I 2n is the 2nth reference progressive frame in the reference progressive video, (i, j) is the coordinate of the pixel point, 2h is the maximum number of rows of pixel point coordinates in the training interlaced frame (reference progressive frame), and w is the maximum number of columns of pixel point coordinates in the training interlaced frame (reference progressive frame).

[0052] For any training interlaced frame, obtain the pixel points of all even rows in any training interlaced frame to obtain the even training field, and determine the field number of the even training field; obtain the pixel points of all odd rows in any training interlaced frame to obtain the odd training field, and determine the field number of the odd training field.

[0053] S102: Determine a training set for each training field based on the multiple training fields.

[0054] Each training set includes at least one odd-numbered training field and one even-numbered training field, and the training set of any training field includes the training field itself.

[0055] Specifically, multiple training fields are sorted according to the field sequence number of each training field to obtain a training field sequence, and a training set of each training field is determined in the training field sequence. The training set of any training field at least includes: any training field, and training fields adjacent to the any training field in the training field sequence.

[0056] S103, based on the reference progressive video and the training set of each training field, iteratively train the initial spatial de-interlacing module to obtain a spatial de-interlacing module.

[0057] Specifically, for any training field, a target reference frame corresponding to any training field is determined in a reference progressive video; a training set corresponding to any training field is input into an initial spatial deinterlacing module to obtain a plurality of first upsampled training feature maps, and the plurality of first upsampled training feature maps are input into a second convolutional layer to obtain a spatial training residual map corresponding to any training field; interpolation processing is performed on any training field to obtain a first training estimation map; a first training progressive frame of any training field is obtained based on the first training estimation map and the spatial training residual map of any training field; parameters of the initial spatial deinterlacing module are modified based on the first training progressive frame and the target reference frame of any training field to complete one training; multiple iterations of training are performed according to the above process to obtain a spatial deinterlacing module.

[0058] S104, iteratively training an initial temporal deinterlacing module based on the reference progressive video, the training set of each training field and the spatial deinterlacing module to obtain a temporal deinterlacing module.

[0059] Specifically, after obtaining the spatial domain de-interlacing module, the parameters of the spatial domain de-interlacing module are fixed, and then the initial temporal domain de-interlacing module is trained.

[0060] For any training field, a target reference frame corresponding to any training field is determined in a reference progressive video; a training set corresponding to any training field is input into an initial spatial deinterlacing module to obtain a plurality of second upsampled training feature maps; the plurality of second upsampled training feature maps and any training field are input into an initial temporal deinterlacing module to obtain a second training progressive frame; parameters of the initial temporal deinterlacing module are modified according to the second training progressive frame of any training field and the target reference frame to complete a training; multiple iterations of training are performed according to the above process to obtain a temporal deinterlacing module.

[0061] S105: Determine a trained de-interlacing model according to the spatial domain de-interlacing module and the temporal domain de-interlacing module.

[0062] Specifically, the trained deinterlacing model includes a spatial domain deinterlacing module and a temporal domain deinterlacing module. The trained deinterlacing model is used to perform deinterlacing processing on interlaced video.

[0063] In the above-mentioned method for generating a deinterlacing model, each training interlaced frame in an interlaced training video is divided into an odd training field and an even training field to obtain multiple training fields of the interlaced training video, a training set of each training field is determined in the multiple training fields, and an initial spatial deinterlacing module is iteratively trained based on the training set of each training field and a reference progressive video to obtain a spatial deinterlacing module, and the parameters of the spatial deinterlacing module are fixed, and an initial temporal deinterlacing module is iteratively trained based on the training set of each training field and a reference progressive video, a reference progressive video and a spatial deinterlacing module to obtain a temporal deinterlacing module, and the trained deinterlacing model includes a spatial deinterlacing module and a temporal deinterlacing module. Through step-by-step training, the difficulty of training is reduced, which is more conducive to the convergence of the loss function, and the accuracy of the trained deinterlacing model is improved. Through the trained deinterlacing model, a better deinterlacing effect can be obtained.

[0064] In one embodiment, in S101, segmenting each training interlaced frame in the interlaced training video to obtain a plurality of training fields includes:

[0065] S1021, for any training interlaced frame, obtain all even-numbered row pixels of the any training interlaced frame to obtain the even-numbered training fields of the any training interlaced frame, and determine the field number of the even-numbered training fields of the any training interlaced frame according to the frame number of the any training interlaced frame.

[0066] Specifically, the frame number of any training interlaced frame is determined according to the playing time of the interlaced training video. When playing the interlaced training video, the frame number of the first played training interlaced frame is smaller than the frame number of the later played training interlaced frame.

[0067] The field number of an even-numbered training field in any training interlaced frame is an odd number. The field number of an even-numbered training field in any training interlaced frame multiplied by 2 plus 1 equals the frame number of any training interlaced frame. "+" represents an even number. For an even-numbered training field: Determine the field number of each even-numbered training field according to formula (2):

[0068] m is an odd number, 0≤i <h,0≤j<w,1≤m≤2t-1 (2)

[0069] When m is an odd number, is the frame number of the training interlaced frame, m is the field number of the even training field, (i, j) is the pixel coordinates in the even training field, h is the maximum number of rows of pixel coordinates in the even training field, and w is the maximum number of columns of pixel coordinates in the even training field.

[0070] S1022, acquiring all odd-numbered row pixels of the any training interlaced frame to obtain the odd-numbered training field of the any training interlaced frame, and determining the field number of the any odd-numbered training field according to the frame number of the any training interlaced frame.

[0071] Specifically, the field number of the odd-numbered training field of any training interlaced frame is an even number, and the field number of the odd-numbered training field of any training interlaced frame multiplied by 2 is equal to the frame number of any training interlaced frame. "-" represents an odd number, for an odd-numbered training field: Determine the field number of each odd-numbered training field according to formula (3):

[0072] m is an even number, 0≤i <h,0≤j<w,1≤m≤2t (3)

[0073] When m is an even number, is the frame number of the training interlaced frame, m is the field number of the odd training field, (i,j) is the pixel coordinates in the odd training field, h is the maximum number of rows of pixel coordinates in the odd training field, and w is the maximum number of columns of pixel coordinates in the odd training field.

[0074] For example, see Figure 3 , for training interlaced frames: S 1 , get S 1 All even-numbered rows (S 1 The pixels in the row marked as 1+ in the figure are used to obtain the even-numbered training field. (m=1, m is an odd number, determined according to formula (2) ), get S 1 All odd-numbered rows (S 1 The pixels in the rows marked as 2- in the figure are used to obtain the odd-numbered training ground. (m=2, m is an even number, determined according to formula (3) ), for training interlaced frames: S 2 , get S 2 All even-numbered rows (S 2 The pixels in the row marked as 3+ in the figure are used to obtain the even-numbered training field. (m=3, m is an odd number, determined according to formula (2) ), get S 2 All odd-numbered rows (S 2 The pixels in the rows marked as 4- in the figure are used to obtain the odd-numbered training ground. (m=4, m is an even number, determined according to formula (3) ).

[0075] In one embodiment, S102 includes:

[0076] S1021A, sort the multiple training fields according to the field serial numbers of the multiple training fields to obtain a training field sequence.

[0077] Specifically, multiple training fields are sorted in ascending order of field numbers to obtain a training field sequence. The first and last training fields in the training field sequence are used as boundary training fields, and the other training fields in the training field sequence except the boundary training fields are used as non-boundary training fields.

[0078] For example, the training ground sequence includes: in, and For border training ground, It is a non-border training ground.

[0079] S1022A: Determine a training set of any non-boundary training field and a training set of any boundary training field based on the training field sequence.

[0080] Among them, the training set of any non-boundary training field includes: any non-boundary training field, and two training fields adjacent to any non-boundary training field in the training field sequence; the training set of any boundary training field includes: two of any boundary training fields, and one training field adjacent to any boundary training field in the training field sequence.

[0081] Specifically, the training set of any training field includes three training fields, and the three training fields include the any training field itself. For any non-border training field, the other two training fields adjacent to it can be determined in the training field sequence, and the training set of any non-border training field includes: any non-border training field, a training field with a field number before the any non-border training field, and a training field with a field number after the any non-border training field. For any boundary training field, only one training field adjacent to it can be determined in the training field sequence; the training set of any boundary training field includes: two of any boundary training fields, and a training field with a field number adjacent to it.

[0082] For example, when m is an odd number, The training set is: When m is an even number, The training set is: When m=1, The training set is: When m = 2t, The training set is:

[0083] In one embodiment, S102 includes:

[0084] S1021B, sort the multiple training fields according to their field numbers to obtain a training field sequence, and determine a training set for any training field based on the training field sequence.

[0085] Specifically, multiple training fields are sorted in order of field numbers from small to large to obtain a training field sequence, and a training set for each training field is determined based on the training field sequence. For any training field, the training set of any training field includes: any training field, and another training field adjacent to any training field in the training field sequence. In the training field sequence, another training field can be arranged before any training field, and another training field can also be arranged after any training field.

[0086] For example, the training ground sequence includes: in, The training set is: The training set is: or, The training set is:

[0087] In one embodiment, S103 includes:

[0088] S1031, determining a target reference frame corresponding to each training field in the reference progressive video.

[0089] Specifically, the field sequence number of each training field is obtained, and the target reference frame corresponding to the training field is determined in the reference progressive video according to the field sequence number, and the frame sequence number of the target reference frame is the same as the field sequence number.

[0090] For example, the training ground is The field number is 1, and the reference progressive frame with frame number 1 in the progressive video is referenced: I 1 As The corresponding target reference frame.

[0091] S1032: Input the training set of the first training field among the multiple training fields into an initial spatial de-interlacing module to obtain multiple first up-sampled training feature maps.

[0092] Specifically, a training field can be randomly selected from the multiple training fields as the first training field. The initial spatial deinterlacing module includes: an initial feature extraction layer and an initial pixel reorganization layer; the training set of the first training field is input into the initial feature extraction layer to obtain multiple spatial training feature maps; the multiple spatial training feature maps are input into the initial pixel reorganization layer to obtain multiple first up-sampled training feature maps.

[0093] The training set of the first training field includes multiple training fields; the initial feature extraction layer is a two-dimensional convolution layer (2Dconvolution), the convolution kernel size is 3×3, the padding is 1, the step size is 1, the size of each training field is (h, w, c1), c1 is the number of channels of the training field, c1=1, the training set of the first training field is input into the initial feature extraction layer, and the spatial training feature map corresponding to each training field in the training set of the first training field is obtained. The size of the spatial training feature map is (h, w, c2), c2 is the number of channels of the spatial training feature map, c2=6, and each spatial training feature map is input into the initial pixel reorganization layer. The initial pixel reorganization layer merges the feature maps of multiple channels in each spatial training feature map in pairs to obtain the first up-sampled training feature map corresponding to each spatial training map. The size of the first up-sampled training feature map is (2h, w, c3), c3 is the number of channels of the first up-sampled training feature, c3=3. Upsampling is performed through the initial pixel reorganization layer, and the amount of calculation is low.

[0094] For example, see Figure 4 , the first training ground is: The training set for the first training ground includes: in, The number of channels is 1. and Input the initial feature extraction layer (3×3, 2Dconvolution), and get The corresponding spatial training feature map F11, The corresponding spatial training feature map F12, and The corresponding spatial training feature map F13, where the number of channels of F11, F12 and F13 are all 6, is input into the initial pixel reorganization layer (pixel shuffle). The pixel shuffle merges the feature maps of the 6 channels in each spatial training feature map in pairs to obtain The corresponding first up-sampled training feature map Fup11, The corresponding first up-sampled training feature map Fup12 and The corresponding first up-sampled training feature map is Fup13.

[0095] See also Figure 5 , the feature maps of the 6 channels of the spatial training feature map are The spatial training feature map is input into the initial pixel reorganization layer to obtain the first up-sampled training feature map. The feature maps of the three channels of the first up-sampled training feature map are: According to formula (4), The feature maps of the two channels in are merged pairwise to determine

[0096]

[0097] For example, m = 0, is based on and Sure, Each row in ( The pixels in the rows marked as 0 in Even-numbered rows ( The pixel points of the row marked as 0 in Each row in ( The pixels in the rows marked as 1 in Odd-numbered fields ( The pixels in the row marked as 1 in .

[0098] S1033: Input the plurality of first up-sampled training feature maps into a second convolutional layer to obtain a spatial training residual map.

[0099] Specifically, the second convolution layer is a two-dimensional convolution layer (2Dconvolution), the convolution kernel size of the second convolution layer is 1×1, the padding is 0, the step size is 1, the size of the first upsampling training feature map is: (2h, w, c3), c3 is the number of channels of the first upsampling training feature map, c3=3, the size of the spatial training residual map is: (2h, w, c4), c4 is the number of channels of the spatial training residual map, c4=1.

[0100] For example, see Figure 4 , the first training ground The training set includes: Will The corresponding first up-sampled training feature map Fup11, The corresponding first up-sampled training feature map Fup12 and The corresponding first up-sampled training feature map Fup13 is input into the second convolutional layer to obtain the spatial training residual map Fst11.

[0101] S1034: Perform linear interpolation processing on the training set of the first training field to obtain a first training estimation image, and correct the first training estimation image based on the spatial training residual image to obtain a first training progressive frame corresponding to the first training field.

[0102] Specifically, a training set of the first training field is subjected to linear interpolation processing to obtain a first training estimation map, and the first training interpolation map and the spatial training residual map are added to correct the first training estimation map based on the spatial training residual map to obtain a first training progressive frame. By correcting the first training estimation map through the spatial training residual map, the learning difficulty of the initial spatial deinterlacing module is reduced, and the training effect is better.

[0103] For example, see Figure 4 , the first training ground The training set includes: right Perform linear interpolation to obtain the first training estimation graph Fil11, and add Fil11 and The spatial training residual map Fst11 is input into ADD to add Fil11 and Fst11 to get the first training field The first training progressive frame

[0104] S1035: Modify parameters of the initial spatial de-interlacing module according to the target reference frame and the first training progressive frame corresponding to the first training scene.

[0105] Specifically, a first loss value is determined according to the first loss function, the target reference frame corresponding to the first training scene, and the first training progressive frame, and the parameters of the initial spatial deinterlacing module are modified according to the first loss value. The first loss value is calculated according to formula (5).

[0106]

[0107] Among them, L1 is the first loss value, The first training ground (or ) corresponds to the first training progressive frame, It is the first training field (or ) corresponds to the target reference frame, and ∈ is the regularization term.

[0108] S1036: Based on the training fields other than the first training field among the multiple training fields, repeat the above process of modifying the parameters of the initial spatial de-interlacing module to obtain a spatial de-interlacing module.

[0109] Specifically, after completing one training according to S1032 to S1035, based on the training fields other than the first training field in the multiple training fields, S1031 to S1035 are repeatedly executed to complete the training again, so as to perform iterative training until the initial spatial deinterlacing module converges to obtain a spatial deinterlacing module. The spatial deinterlacing module includes: a feature extraction layer (3×3, 2D convolution) and a pixel reorganization layer (pixel shuffle).

[0110] In one embodiment, S104 includes:

[0111] S1041, determining a target reference frame corresponding to each training field in the reference progressive video.

[0112] Specifically, the field sequence number of each training field is obtained, and the target reference frame corresponding to the training field is determined in the reference progressive video according to the field sequence number, and the frame sequence number of the target reference frame is the same as the field sequence number.

[0113] S1042: Input the training set of the second training field among the multiple training fields into the spatial deinterlacing module to obtain multiple second up-sampled training feature maps.

[0114] Specifically, a training field can be randomly selected from the multiple training fields as the second training field. The process of inputting the training set of the second training field into the spatial deinterlacing module to obtain multiple second up-sampled training feature maps is the same as the process of inputting the training set of the first training field into the initial spatial deinterlacing module to obtain multiple first up-sampled training feature maps in S1032. Therefore, the specific description of S1042 can refer to the specific description of S1032.

[0115] S1043: Input the plurality of second up-sampled training feature maps and the training set of the second training field into an initial time domain de-interlacing module to obtain a second training progressive frame corresponding to the second training field.

[0116] Specifically, the initial time domain deinterlacing module includes: an initial three-dimensional convolution layer, an initial first convolution layer, an initial interpolation layer and an initial addition layer. Input multiple second upsampled training feature maps into the initial three-dimensional convolution layer to obtain spatiotemporal training feature maps, input the multiple spatiotemporal training feature maps into the initial first convolution layer to obtain spatiotemporal training residual maps; input the training set of the second training field into the initial difference layer to obtain a second training estimation map, input the spatiotemporal training residual map and the second training estimation map into the initial addition layer to obtain a second training progressive frame corresponding to the second training field.

[0117] The initial three-dimensional convolution layer is a three-dimensional convolution layer (3Dconvolution), the convolution kernel size is 1×1×3, the padding is 0, and the stride is 1; the initial first convolution layer is a two-dimensional convolution layer (2Dconvolution), the convolution kernel size is 1×1, the padding is 0, and the stride is 1.

[0118] See also Figure 6 , the second training ground is: The training set for the second training ground includes: Will Input the spatial deinterlacing module and get The corresponding second up-sampled training feature map Fup21, The corresponding second up-sampled training feature map Fup22, The corresponding second up-sampled training feature map Fup23 is input into the initial three-dimensional convolutional layer to obtain the spatiotemporal training feature map F211 corresponding to Fup21, the spatiotemporal training feature map F221 corresponding to Fup22 and the spatiotemporal training feature map F231 corresponding to Fup23. The spatiotemporal training feature maps: F211, F221 and F231 are input into the initial first convolutional layer to obtain The corresponding spatiotemporal training residual graph Fst21; Input the initial interpolation layer to obtain the second training estimation graph Fil21. The corresponding second training estimation graph Fil21 and spatiotemporal training residual graph Fst21 are input into the initial addition layer to obtain The corresponding second training progressive frame

[0119] S1044: Modify the parameters of the initial time-domain de-interlacing module according to the target reference frame and the second training progressive frame corresponding to the second training set.

[0120] Specifically, a second loss value is determined according to the second loss function, the target reference frame corresponding to the second training scene, and the second training progressive frame, and the parameters of the initial temporal deinterlacing module are modified according to the second loss value. The second loss value is calculated according to formula (6).

[0121]

[0122] Among them, L2 is the second loss value, The second training ground (or ) corresponding to the second training progressive frame, I t The second training ground (or ) corresponds to the target reference frame, and ∈ is the regularization term.

[0123] S1045: Based on the training sets other than the second training set in the multiple training sets, repeat the above process of modifying the parameters of the initial time domain de-interlacing module to obtain a time domain de-interlacing module.

[0124] Specifically, after completing one training according to S1042 to S1044, based on the training fields other than the second training field in the multiple training fields, S1042 to S1044 are repeatedly executed to complete the training again, so as to perform iterative training until the initial time domain deinterlacing module converges to obtain the time domain deinterlacing module. The time domain deinterlacing module includes: a three-dimensional convolution layer (1×1×3, 3Dconvolution), a first convolution layer (1×1, 2Dconvolution), an interpolation layer and an addition layer.

[0125] In the above-mentioned method for generating a deinterlacing model, the initial spatial deinterlacing module is firstly iteratively trained to obtain a spatial deinterlacing module, the parameters of the spatial deinterlacing module are fixed, and the initial temporal deinterlacing module is iteratively trained based on the training set of each training field and the reference progressive video, the reference progressive video and the spatial deinterlacing module to obtain a temporal deinterlacing module. The trained deinterlacing model includes a spatial deinterlacing module and a temporal deinterlacing module. Through step-by-step training, the difficulty of training is reduced, which is more conducive to the convergence of the loss function, and the accuracy of the trained deinterlacing model is improved. Through the trained deinterlacing model, a better deinterlacing effect can be obtained.

[0126] In one embodiment, Figure 7 As shown, a video deinterlacing method is provided. This embodiment takes the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the video deinterlacing method includes the following steps:

[0127] S201, obtaining an interlaced video to be processed, and segmenting each interlaced frame in the interlaced video to be processed to obtain a plurality of fields to be processed.

[0128] The interlaced video to be processed includes a plurality of interlaced frames, and the plurality of fields to be processed include a plurality of odd fields and a plurality of even fields.

[0129] Specifically, for any interlaced frame, all pixel points of the even-numbered rows in any interlaced frame are obtained to obtain the even field, and the field number of the even field is determined; all pixel points of the odd-numbered rows in any interlaced frame are obtained to obtain the odd field, and the field number of the odd field is determined.

[0130] S202: Determine a to-be-processed set for each to-be-processed field based on the multiple to-be-processed fields.

[0131] Each to-be-processed set includes at least one odd-numbered field and one even-numbered field, and the to-be-processed set of any to-be-processed field includes the any to-be-processed field itself.

[0132] Specifically, multiple fields to be processed are sorted according to the field sequence number of each field to be processed to obtain a sequence of fields to be processed, and a set to be processed of each field to be processed is determined in the sequence of fields to be processed. The set to be processed of any field to be processed at least includes: any field to be processed, and fields to be processed adjacent to the field to be processed in the sequence of fields to be processed.

[0133] S203: For any field to be processed, input the processing set of any field to be processed into a spatial deinterlacing module in a trained deinterlacing model to obtain a plurality of upsampled feature maps corresponding to any field to be processed.

[0134] Specifically, the trained deinterlacing model is obtained based on the above-mentioned generation method of a deinterlacing model. The spatial domain deinterlacing module includes: a feature extraction layer and a pixel reorganization layer, inputting a processing set of any field to be processed into the feature extraction layer to obtain a plurality of spatial feature maps corresponding to any field to be processed, and inputting the plurality of spatial feature maps into the pixel reorganization layer to obtain a plurality of up-sampled feature maps.

[0135] S204, inputting the processing set of any field to be processed and the multiple up-sampled feature maps corresponding to any field to be processed into the time domain deinterlacing module in the trained deinterlacing model, obtaining the target frame corresponding to any field to be processed, and generating a target deinterlaced video according to each target frame.

[0136] Specifically, the temporal deinterlacing module includes: a three-dimensional convolution layer, a first convolution layer, an interpolation layer and an addition layer, a plurality of upsampled feature maps corresponding to any field to be processed are input into the three-dimensional convolution layer, and spatiotemporal fusion is performed through the three-dimensional convolution layer to obtain a plurality of spatiotemporal feature maps, and the plurality of spatiotemporal feature maps are input into the first convolution layer to obtain a spatiotemporal residual map; a processing set of any field to be processed is input into the interpolation layer to obtain an estimated map, and the estimated map and the spatiotemporal residual map are input into the addition layer to realize correction of the estimated map using the spatiotemporal residual map to obtain a target frame corresponding to any field to be processed. A target deinterlaced video is generated according to the target frame corresponding to each field to be processed.

[0137] See also Figure 8 The trained deinterlacing model includes: feature extraction layer C1 (3×3, 2Dconvolution) and pixel reorganization layer C2 (pixel shuffle), three-dimensional convolution layer C3 (1×1×3, 3Dconvolution), first convolution layer C4 (1×1, 2Dconvolution), interpolation layer C5 and addition layer C6. The trained deinterlacing model is used to deinterlace interlaced video. The processing set of any field to be processed is input into the trained deinterlacing model to obtain the target frame corresponding to any field to be processed.

[0138] In the above-mentioned video deinterlacing method, each interlaced frame of the interlaced video to be processed is divided into a plurality of fields to be processed, a set to be processed of each field to be processed is determined among the plurality of fields to be processed, and a target frame of each field to be processed is determined based on the set to be processed of each field to be processed and the trained deinterlacing model, thereby obtaining a target deinterlaced video. Compared with the motion compensation method in the prior art, the above-mentioned video deinterlacing method effectively extracts spatial information through a spatial deinterlacing module to obtain a plurality of upsampled feature maps of the field to be processed, effectively extracts temporal information through a temporal deinterlacing module to obtain a spatiotemporal residual map, interpolates the field to be processed through the temporal deinterlacing module to obtain an estimated map, and uses the spatiotemporal residual map to correct the estimated map to obtain a target frame. The amount of calculation is much less than that of the existing motion compensation method. The deinterlacing method is used for deinterlacing, which has a small workload, low hardware requirements, and low cost.

[0139] In one embodiment, in S201, segmenting each interlaced frame in the interlaced video to be processed to obtain a plurality of fields to be processed includes:

[0140] S2011, for any interlaced frame, obtain all even-numbered row pixels of the interlaced frame to obtain the even-numbered field of the interlaced frame, and determine the field number of the even-numbered field of the interlaced frame according to the frame number of the interlaced frame.

[0141] Specifically, the frame number of any interlaced frame is determined according to the playback time of the interlaced video to be processed. When playing the interlaced video to be processed, the frame number of the interlaced frame played first is smaller than the frame number of the interlaced frame played later.

[0142] The field number of an even field in any interlaced frame is an odd number. The field number of an even field in any interlaced frame multiplied by 2 plus 1 equals the frame number of any interlaced frame. "+" represents an even number. For an even field: The field number of each even field is determined according to formula (8):

[0143]

[0144] When m is an odd number, is the frame number of the interlaced frame, and m is the field number of the even field.

[0145] S2012, acquiring all odd-numbered row pixels of the any interlaced frame to obtain the odd-numbered field of the any interlaced frame, and determining the field sequence number of the odd-numbered field of the any interlaced frame according to the frame sequence number of the any interlaced frame.

[0146] The field number of the odd field of any interlaced frame is an even number. The field number of the odd field of any interlaced frame multiplied by 2 is equal to the frame number of any interlaced frame. "-" represents an even number. For an even field: The field number of each odd field is determined according to formula (9):

[0147]

[0148] When m is an even number, is the frame number of the interlaced frame, and m is the field number of the odd field.

[0149] In one embodiment, S202 includes:

[0150] S2021A, sort the multiple training fields according to their field serial numbers to obtain a training field sequence.

[0151] Specifically, multiple to-be-processed fields are sorted in ascending order of field numbers to obtain a to-be-processed field sequence. The first and last to-be-processed fields in the to-be-processed field sequence are taken as boundary fields, and the other to-be-processed fields in the to-be-processed field sequence except the boundary fields are taken as non-boundary fields.

[0152] For example, the sequence of fields to be processed includes: in, and is the boundary field, It is a non-boundary field.

[0153] S2022A: Determine a to-be-processed set of any non-boundary field and a to-be-processed set of any boundary field based on the to-be-processed field sequence.

[0154] Among them, the to-be-processed set of any non-boundary field includes: any non-boundary field, and two to-be-processed fields adjacent to any non-boundary field in the to-be-processed field sequence; the to-be-processed set of any boundary field includes: two of any boundary fields, and one to-be-processed field adjacent to any boundary field in the to-be-processed field sequence.

[0155] Specifically, the to-be-processed set of any to-be-processed field includes three to-be-processed fields, and the three to-be-processed fields include the any to-be-processed field itself. For any non-border field, the to-be-processed set of any non-border field includes: the any non-border field, the to-be-processed field with a field sequence number that is one digit before the any non-border field, and the to-be-processed field with a field sequence number that is one digit after the any non-border field. For any border field, only one to-be-processed field adjacent to it can be determined in the to-be-processed field sequence; the to-be-processed set of any border to-be-processed field includes: two of the any border fields, and one to-be-processed field with a field sequence number that is adjacent to it.

[0156] For example, when m is an odd number, The set to be processed is: When m is an even number, The set to be processed is: When m=1, The set to be processed is: When m = 2t, The set to be processed is:

[0157] In one embodiment, S202 includes:

[0158] S2022B: Sort the multiple to-be-processed fields according to their field serial numbers to obtain a to-be-processed field sequence, and determine a to-be-processed set of any to-be-processed field based on the to-be-processed field sequence.

[0159] Specifically, multiple fields to be processed are sorted in order of field numbers from small to large to obtain a sequence of fields to be processed, and a set to be processed of each field to be processed is determined based on the sequence of fields to be processed. For any field to be processed, the set to be processed of any field to be processed includes: any field to be processed, and another field to be processed adjacent to any field to be processed in the sequence of fields to be processed. In the sequence of fields to be processed, another field to be processed can be arranged before any field to be processed, and another field to be processed can also be arranged after any field to be processed.

[0160] For example, the sequence of fields to be processed includes: The set to be processed is: The set to be processed is: or, The set to be processed is:

[0161] In one embodiment, S203 includes:

[0162] S2031, inputting the processing set of any of the processing fields into the feature extraction layer to obtain a plurality of spatial feature maps.

[0163] Specifically, the processing set of any field to be processed includes multiple fields to be processed, the convolution kernel size of the feature extraction layer is 3×3, the padding is 1, and the step size is 1. The processing set is input into the feature extraction layer to obtain multiple spatial feature maps, which correspond one to one to the multiple fields to be processed included in the processing set. The size of the field to be processed is (h, w, c1), c1 is the number of channels of the field to be processed, c1=1, and the size of the spatial feature map is (h, w, c2), c2 is the number of channels of the spatial feature map, c2=6.

[0164] S2032, input the multiple spatial feature maps to the pixel recombination layer to obtain multiple up-sampled feature maps corresponding to any field to be processed.

[0165] Specifically, the pixel reorganization layer combines the feature maps of multiple channels in each spatial feature map in pairs to upsample the spatial feature map to obtain an upsampled feature map corresponding to each spatial feature map. The size of the upsampled feature map is (2h, w, c3), where c3 is the number of channels of the upsampled feature map, c3 = 3. Upsampling through the pixel reorganization layer has low computational complexity.

[0166] For example, the treatment site is The set to be processed is Will and Input feature extraction layer, get The corresponding spatial feature map T11, The corresponding spatial feature map T12, and The corresponding spatial feature map T13, T11, T12 and T13 are input into the pixel recombination layer to obtain the up-sampled feature map T21 corresponding to T11, the up-sampled feature map T22 corresponding to T12, and the up-sampled feature map T23 corresponding to T13.

[0167] The specific process of S2031 and S2032 is the same as the specific process of S1032 in the above-mentioned method for generating the deinterlacing model. Therefore, the specific process of S2031 and S2032 can refer to the specific description of S1032 in the above-mentioned method for generating the deinterlacing model.

[0168] In one embodiment, S204 includes:

[0169] S2041, inputting a plurality of upsampled feature maps corresponding to any of the fields to be processed into the three-dimensional convolutional layer to obtain a plurality of spatiotemporal feature maps.

[0170] Specifically, the three-dimensional convolution layer is a 3D convolution layer with a convolution kernel size of 1×1×3, padding of 0, and a step size of 1; the convolution kernel size is 1×1×3, padding of 0, and a step size of 1, and multiple upsampled feature maps corresponding to any field to be processed are input into the three-dimensional convolution layer to obtain a spatiotemporal feature map corresponding to each upsampled feature map.

[0171] For example, in the above example, The corresponding multiple upsampled feature maps: T21, T22 and T23 are input into the three-dimensional convolutional layer to obtain the spatiotemporal feature map T31 corresponding to T21, the spatiotemporal feature map T32 corresponding to T22, and the spatiotemporal feature map T33 corresponding to T23.

[0172] S2042: Input the multiple spatiotemporal feature maps into the first convolutional layer to obtain a spatiotemporal residual map of any field to be processed.

[0173] Specifically, the first convolution layer is a 2D convolution layer, the convolution kernel size is 1×1, the padding is 0, and the step size is 1.

[0174] For example, in the above example, The corresponding multiple spatiotemporal feature maps: T31, T32 and T33 are input to the first convolutional layer to obtain Corresponding spatiotemporal residual graph

[0175] S2043, inputting the processing set of any field to be processed into the interpolation layer to obtain an estimated image of any field to be processed, and inputting the spatiotemporal residual image and the estimated image of any field to be processed into the addition layer to obtain a target frame corresponding to any field to be processed.

[0176] Specifically, through the interpolation layer, linear interpolation processing is performed on any field to be processed to obtain an estimated map of any field to be processed. The spatiotemporal residual map and the estimated map of any field to be processed are added to correct the estimated map of any field to be processed to obtain a target frame corresponding to any field to be processed.

[0177] For example, in the above example, The pending set Input to the interpolation layer and get Estimation graph of Will and Add together and get The corresponding target frame T 2 .

[0178] In one embodiment, a target deinterlaced video is generated according to each target frame, including: using the field number of the to-be-processed field corresponding to each target frame as the frame number of each target frame, and generating the target deinterlaced video according to each target frame and the frame number of each target frame.

[0179] For example, in the above example, the field to be processed is obtained The corresponding target frame T 2 , similarly, we can get: the field to be processed The corresponding target frame T 1 , ..., waiting for treatment The corresponding target frame T 2t According to T 1 ,T 2 ,……,T 2t Get the target deinterlaced video.

[0180] In the above-mentioned video deinterlacing method, each interlaced frame of the interlaced video to be processed is divided into multiple fields to be processed, and a processing set for each field to be processed is determined among the multiple fields to be processed. Based on the processing set of each field to be processed and the trained deinterlacing model, a target frame for each field to be processed is determined, thereby obtaining a target deinterlaced video. Compared with the motion compensation method in the prior art, the above-mentioned video deinterlacing method effectively extracts spatial information through a spatial deinterlacing module to obtain multiple upsampled feature maps of the field to be processed, effectively extracts temporal information through a temporal deinterlacing module to obtain a spatiotemporal residual map, interpolates the field to be processed through the temporal deinterlacing module to obtain an estimated map, and uses the spatiotemporal residual map to correct the estimated map to obtain a target frame; the number of layers of the trained deinterlacing model is shallow, and the size of the convolution kernel is small, which reduces the amount of calculation; upsampling is performed through a pixel recombination layer to reduce the amount of calculation; the amount of calculation of the above-mentioned video deinterlacing method is much smaller than that of the existing motion compensation method, the above-mentioned video deinterlacing method has a small workload, and has low hardware requirements and low cost. Through the trained deinterlacing model, a better deinterlacing effect can be obtained.

[0181] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0182] Based on the same inventive concept, the embodiment of the present application also provides a video deinterlacing device for implementing the above-mentioned video deinterlacing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more video deinterlacing device embodiments provided below can refer to the above-mentioned limitations on the video deinterlacing method, and will not be repeated here.

[0183] In one embodiment, Fig. 9 As shown, a video de-interlacing device is provided, comprising:

[0184] A training field determination module is used to obtain an interlaced training video and a reference progressive video corresponding to the interlaced training video, and to segment each training interlaced frame in the interlaced training video to obtain a plurality of training fields, wherein the plurality of training fields include a plurality of odd-numbered training fields and a plurality of even-numbered training fields;

[0185] A training set determination module, configured to determine a training set for each training field based on the plurality of training fields, wherein each training set includes at least one odd-numbered training field and one even-numbered training field;

[0186] A first training module is used to iteratively train an initial spatial de-interlacing module based on the reference progressive video and a training set of each training field to obtain a spatial de-interlacing module;

[0187] A second training module is used to iteratively train the initial temporal deinterlacing module based on the reference progressive video, the training set of each training field and the spatial deinterlacing module to obtain a temporal deinterlacing module;

[0188] The deinterlacing model determination module is used to determine a trained deinterlacing model according to the spatial domain deinterlacing module and the temporal domain deinterlacing module, wherein the trained deinterlacing model is the trained deinterlacing model in the above-mentioned video deinterlacing method.

[0189] Based on the same inventive concept, the embodiment of the present application also provides a deinterlacing model generation device for implementing the deinterlacing model generation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more deinterlacing model generation devices provided below can refer to the limitations of the deinterlacing model generation method above, and will not be repeated here.

[0190] Each module in the above-mentioned video deinterlacing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0191] In one embodiment, Fig.10 As shown, a device for generating a de-interlacing model is provided, comprising:

[0192] A training field determination module is used to obtain an interlaced training video and a reference progressive video corresponding to the interlaced training video, and to segment each training interlaced frame in the interlaced training video to obtain a plurality of training fields, wherein the plurality of training fields include a plurality of odd-numbered training fields and a plurality of even-numbered training fields;

[0193] A training set determination module, configured to determine a training set for each training field based on the plurality of training fields, wherein each training set includes at least one odd-numbered training field and one even-numbered training field;

[0194] A first training module is used to iteratively train an initial spatial de-interlacing module based on the reference progressive video and a training set of each training field to obtain a spatial de-interlacing module;

[0195] A second training module is used to iteratively train the initial temporal deinterlacing module based on the reference progressive video, the training set of each training field and the spatial deinterlacing module to obtain a temporal deinterlacing module;

[0196] The deinterlacing model determination module is used to determine a trained deinterlacing model according to the spatial domain deinterlacing module and the temporal domain deinterlacing module, wherein the trained deinterlacing model is the trained deinterlacing model in the above-mentioned deinterlacing model generation method.

[0197] Each module in the above-mentioned deinterlacing model generation device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0198] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a video deinterlacing method or a method for generating a deinterlacing model is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0199] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0200] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the above-mentioned video deinterlacing method or a deinterlacing model generation method are implemented.

[0201] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned video deinterlacing method or a deinterlacing model generation method are implemented.

[0202] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the above-mentioned video deinterlacing method or a deinterlacing model generation method.

[0203] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0204] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0205] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0206] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video deinterlacing method, characterized in that: The video deinterlacing method comprises: Acquire an interlaced video to be processed, and perform segmentation processing on each interlaced frame in the interlaced video to be processed to obtain a plurality of fields to be processed, wherein the plurality of fields to be processed include a plurality of odd fields and a plurality of even fields; Based on the multiple fields to be processed, determining a set to be processed for each field to be processed, wherein each set to be processed includes at least one odd field and one even field; For any field to be processed, inputting the processing set of any field to be processed into a spatial domain deinterlacing module in a trained deinterlacing model to obtain a plurality of upsampled feature maps corresponding to any field to be processed; Inputting the processing set of any field to be processed and a plurality of up-sampled feature maps corresponding to any field to be processed into the time domain deinterlacing module in the trained deinterlacing model to obtain a target frame corresponding to any field to be processed, and generating a target deinterlaced video according to each target frame; The spatial domain deinterlacing module includes: a feature extraction layer and a pixel reorganization layer; the processing set of any field to be processed is input into the spatial domain deinterlacing module in the trained deinterlacing model to obtain multiple up-sampled feature maps corresponding to any field to be processed, including: Inputting the processing set of any one of the processing fields into the feature extraction layer to obtain a plurality of spatial feature maps; Inputting the plurality of spatial feature maps into the pixel reorganization layer to obtain a plurality of up-sampled feature maps corresponding to any field to be processed; The temporal deinterlacing module comprises: a three-dimensional convolution layer, a first convolution layer, an interpolation layer and an addition layer; the processing set of any field to be processed and a plurality of upsampled feature maps corresponding to any field to be processed are input into the temporal deinterlacing module in the trained deinterlacing model to obtain a target frame corresponding to any field to be processed, comprising: Inputting a plurality of upsampled feature maps corresponding to any of the fields to be processed into the three-dimensional convolutional layer to obtain a plurality of spatiotemporal feature maps; Inputting the plurality of spatiotemporal feature maps into the first convolutional layer to obtain a spatiotemporal residual map of any field to be processed; The processing set of any field to be processed is input into the interpolation layer to obtain an estimated image of any field to be processed, and the spatiotemporal residual image and the estimated image of any field to be processed are input into the addition layer to obtain a target frame corresponding to any field to be processed.

2. The video deinterlacing method according to claim 1, characterized in that: The step of segmenting each interlaced frame in the interlaced video to be processed to obtain a plurality of fields to be processed includes: For any interlaced frame, all even-numbered row pixel points of the interlaced frame are acquired to obtain the even-numbered field of the interlaced frame, and the field sequence number of the even-numbered field of the interlaced frame is determined according to the frame sequence number of the interlaced frame; All odd-numbered row pixels of any interlaced frame are acquired to obtain an odd-numbered field of any interlaced frame, and a field sequence number of the odd-numbered field of any interlaced frame is determined according to the frame sequence number of any interlaced frame.

3. The video deinterlacing method according to claim 2, characterized in that: The step of determining a to-be-processed set for each to-be-processed field based on the plurality of to-be-processed fields comprises: Sorting the plurality of fields to be processed according to their field serial numbers to obtain a sequence of fields to be processed, wherein the sequence of fields to be processed includes: a plurality of non-boundary fields and two boundary fields; Based on the sequence of fields to be processed, a to-be-processed set of any non-boundary field and a to-be-processed set of any boundary field are determined, wherein the to-be-processed set of any non-boundary field includes: any non-boundary field, and two to-be-processed fields adjacent to any non-boundary field in the sequence of fields to be processed; and the to-be-processed set of any boundary field includes: two of any boundary fields, and one to-be-processed field adjacent to any boundary field in the sequence of fields to be processed.

4. A method for generating a de-interlacing model, characterized in that: The method for generating the de-interlacing model comprises: Acquire an interlaced training video and a reference progressive video corresponding to the interlaced training video, and segment each training interlaced frame in the interlaced training video to obtain a plurality of training fields, wherein the plurality of training fields include a plurality of odd-numbered training fields and a plurality of even-numbered training fields; Based on the multiple training fields, determining a training set for each training field, wherein each training set includes at least one odd-numbered training field and one even-numbered training field; Based on the reference progressive video and the training set of each training field, the initial spatial deinterlacing module is iteratively trained to obtain a spatial deinterlacing module, wherein the spatial deinterlacing module includes: a feature extraction layer and a pixel reorganization layer; Based on the reference progressive video, the training set of each training field and the spatial deinterlacing module, iteratively training the initial temporal deinterlacing module to obtain a temporal deinterlacing module, wherein the temporal deinterlacing module includes: a three-dimensional convolutional layer, a first convolutional layer, an interpolation layer and an addition layer; A trained deinterlacing model is determined according to the spatial deinterlacing module and the temporal deinterlacing module, wherein the trained deinterlacing model is applied to the video deinterlacing method according to any one of claims 1 to 3.

5. The method for generating a de-interlacing model according to claim 4, characterized in that: The step of segmenting each training interlaced frame in the interlaced training video to obtain a plurality of training fields comprises: For any training interlaced frame, all even-numbered row pixel points of the any training interlaced frame are obtained to obtain the even-numbered training field of the any training interlaced frame, and the field number of the even-numbered training field of the any training interlaced frame is determined according to the frame number of the any training interlaced frame; All odd-numbered row pixels of any training interlaced frame are acquired to obtain the odd-numbered training fields of any training interlaced frame, and the field numbers of the odd-numbered training fields of any training interlaced frame are determined according to the frame numbers of any training interlaced frame.

6. The method for generating a de-interlacing model according to claim 5, characterized in that: The step of determining a training set for each training field based on the plurality of training fields comprises: Sorting the plurality of training fields according to the field serial numbers of the plurality of training fields to obtain a training field sequence, wherein the training field sequence includes: a plurality of non-boundary training fields and two boundary training fields; Based on the training field sequence, a training set of any non-boundary training field and a training set of any boundary training field are determined, wherein the training set of any non-boundary training field includes: any non-boundary training field and two training fields adjacent to any non-boundary training field in the training field sequence; the training set of any boundary training field includes: two of any boundary training fields and one training field adjacent to any boundary training field in the training field sequence.

7. The method for generating a de-interlacing model according to claim 4, characterized in that: The iterative training of the initial spatial de-interlacing module based on the reference progressive video and the training set of each training field to obtain the spatial de-interlacing module includes: Determining a target reference frame corresponding to each training field in the reference progressive video; Inputting a training set of a first training field among the multiple training fields into an initial spatial deinterlacing module to obtain a plurality of first up-sampled training feature maps; Inputting the plurality of first up-sampled training feature maps into a second convolutional layer to obtain a spatial training residual map; Performing linear interpolation processing on the training set of the first training field to obtain a first training estimation map, and correcting the first training estimation map based on the spatial training residual map to obtain a first training line-by-line frame corresponding to the first training field; Modifying parameters of the initial spatial de-interlacing module according to the target reference frame and the first training progressive frame corresponding to the first training scene; Based on the training fields other than the first training field among the multiple training fields, the above process of modifying the parameters of the initial spatial de-interlacing module is repeated to obtain a spatial de-interlacing module.

8. The method for generating a de-interlacing model according to claim 7, characterized in that: The initial spatial deinterlacing module includes: an initial feature extraction layer and an initial pixel reorganization layer; the first training set among the multiple training sets is input into the initial spatial deinterlacing module to obtain multiple first up-sampled training feature maps, including: Inputting the first training set into an initial feature extraction layer to obtain a plurality of spatial training feature maps; The multiple spatial training feature maps are input into an initial pixel recombination layer to obtain multiple first up-sampled training feature maps.

9. The method for generating a de-interlacing model according to claim 4, characterized in that: The iterative training of the initial temporal deinterlacing module based on the reference progressive video, the training set of each training field and the spatial deinterlacing module to obtain the temporal deinterlacing module includes: Determining a target reference frame corresponding to each training field in the reference progressive video; Inputting a training set of a second training field among the multiple training fields into the spatial domain deinterlacing module to obtain a plurality of second up-sampled training feature maps; Inputting the plurality of second up-sampled training feature maps and the training set of the second training field into an initial time domain de-interlacing module to obtain a second training progressive frame corresponding to the second training field; Modifying the parameters of the initial temporal deinterlacing module according to the target reference frame and the second training progressive frame corresponding to the second training scene; Based on the training fields other than the second training field among the multiple training fields, the above process of modifying the parameters of the initial time domain de-interlacing module is repeated to obtain a time domain de-interlacing module.

10. The method for generating a de-interlacing model according to claim 9, characterized in that: The initial time domain deinterlacing module includes: an initial three-dimensional convolution layer, an initial first convolution layer, an initial interpolation layer and an initial addition layer; the inputting the plurality of second up-sampled training feature maps and the training set of the second training field into the initial time domain deinterlacing module to obtain the second training progressive frame corresponding to the second training field includes: Inputting the plurality of second up-sampled training feature maps into the initial three-dimensional convolutional layer to obtain a plurality of spatiotemporal training feature maps; Inputting the plurality of spatiotemporal training feature maps into the initial first convolutional layer to obtain a spatiotemporal training residual map; The training set of the second training field is input into the initial interpolation layer to obtain a second training estimation map, and the second training estimation map and the spatiotemporal training residual map are input into the initial addition layer to obtain a second training progressive frame corresponding to the second training field.

11. A video deinterlacing device, characterized in that: The video de-interlacing device comprises: A to-be-processed field determination module is used to obtain an interlaced video to be processed, and to segment each interlaced frame in the interlaced video to be processed to obtain a plurality of to-be-processed fields, wherein the plurality of to-be-processed fields include a plurality of odd fields and a plurality of even fields; A to-be-processed set determination module, used to determine a to-be-processed set of each to-be-processed field based on the plurality of to-be-processed fields, wherein each to-be-processed set includes at least one odd-numbered field and one even-numbered field; An up-sampled feature map determination module is used for inputting a processing set of any field to be processed into a spatial domain de-interlacing module in a trained de-interlacing model to obtain a plurality of up-sampled feature maps corresponding to any field to be processed; A target deinterlaced video generation module is used to input the to-be-processed set of any to-be-processed field and a plurality of up-sampled feature maps corresponding to any to-be-processed field into the time-domain deinterlacing module in the trained deinterlacing model, obtain a target frame corresponding to any to-be-processed field, and generate a target deinterlaced video according to each target frame; The spatial deinterlacing module includes: a feature extraction layer and a pixel reorganization layer; the up-sampled feature map determination module is used to input the to-be-processed set of any to-be-processed field into the feature extraction layer to obtain a plurality of spatial feature maps; input the plurality of spatial feature maps into the pixel reorganization layer to obtain a plurality of up-sampled feature maps corresponding to any to-be-processed field; The temporal deinterlacing module includes: a three-dimensional convolutional layer, a first convolutional layer, an interpolation layer and an addition layer; the target deinterlaced video generation module is used to input multiple upsampled feature maps corresponding to any field to be processed into the three-dimensional convolutional layer to obtain multiple spatiotemporal feature maps; input the multiple spatiotemporal feature maps into the first convolutional layer to obtain the spatiotemporal residual map of any field to be processed; input the processed set of any field to be processed into the interpolation layer to obtain the estimated map of any field to be processed, and input the spatiotemporal residual map and the estimated map of any field to be processed into the addition layer to obtain the target frame corresponding to any field to be processed.

12. A device for generating a de-interlacing model, characterized in that: The de-interlacing model generating device comprises: A training field determination module is used to obtain an interlaced training video and a reference progressive video corresponding to the interlaced training video, and to segment each training interlaced frame in the interlaced training video to obtain a plurality of training fields, wherein the plurality of training fields include a plurality of odd-numbered training fields and a plurality of even-numbered training fields; A training set determination module, configured to determine a training set for each training field based on the plurality of training fields, wherein each training set includes at least one odd-numbered training field and one even-numbered training field; A first training module is used to iteratively train an initial spatial deinterlacing module based on the reference progressive video and a training set of each training field to obtain a spatial deinterlacing module, wherein the spatial deinterlacing module includes a feature extraction layer and a pixel reorganization layer; A second training module is used to iteratively train the initial temporal deinterlacing module based on the reference progressive video, the training set of each training field and the spatial deinterlacing module to obtain a temporal deinterlacing module, wherein the temporal deinterlacing module includes: a three-dimensional convolutional layer, a first convolutional layer, an interpolation layer and an addition layer; A deinterlacing model determination module is used to determine a trained deinterlacing model according to the spatial domain deinterlacing module and the temporal domain deinterlacing module, wherein the trained deinterlacing model is applied to the video deinterlacing method according to any one of claims 1 to 3.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the processor implements the steps of the video deinterlacing method according to any one of claims 1 to 3, or the method for generating a deinterlacing model according to any one of claims 4 to 10.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the video deinterlacing method according to any one of claims 1 to 3, or the method for generating a deinterlacing model according to any one of claims 4 to 10.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the video deinterlacing method according to any one of claims 1 to 3, or the steps of the deinterlacing model generation method according to any one of claims 4 to 10.

Images