Video processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a video processing method and device, electronic equipment and storage medium, belonging to the technical field of multimedia. The method comprises: determining a first propagation path and a second propagation path based on quality information of a plurality of compressed video frames in a first video segment; performing feature extraction on the plurality of target video frames and the plurality of compressed video frames based on the first propagation path and the second propagation path, respectively, to obtain first features and second features; and generating a second video segment based on the first features and the second features. The above scheme enables the first features to be extracted in the order indicated by the first propagation path to generate a video segment, thereby reducing the influence of noise. Furthermore, the second features are extracted from the plurality of compressed video frames in the order indicated by the second propagation path, thereby fully utilizing the continuity of the compressed video frames and eliminating motion errors, so that a high-quality second video segment can be obtained after decompression.

Description

Technical Field

[0001] This disclosure relates to the field of multimedia technology, and in particular to a video processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] As video quality increases, transmitting high-quality video consumes significant bandwidth. To reduce bandwidth usage, videos are typically compressed before transmission and decompressed afterward. How to decompress videos to obtain high-quality results has become a key research focus in the industry.

[0003] In related technologies, bidirectional recurrent network reconstruction is commonly used to decompress compressed videos. By performing forward and backward propagation processing on multiple video frames in the compressed video, historically accumulated video frame information can be used to supplement the texture details of the video frames, thereby achieving a balanced distribution of information among the video frames and ultimately reconstructing a higher-quality video.

[0004] However, different video frames in the compressed video have varying degrees of compression. In the above-mentioned technical solution, the bidirectional recurrent network processes video frames with different degrees of compression in the same way, resulting in poor reconstruction of video frames with higher compression levels and thus lower quality of the reconstructed video. Summary of the Invention

[0005] This disclosure provides a video processing method, apparatus, electronic device, and storage medium capable of generating high-quality video clips. The technical solution of this disclosure is as follows:

[0006] According to one aspect of the embodiments of this disclosure, a video processing method is provided, comprising:

[0007] Based on the quality information of multiple compressed video frames in the first video segment, a first propagation path and a second propagation path are determined. The first propagation path is used to indicate the order of multiple target video frames, where the target video frames are compressed video frames whose quality meets the quality conditions. The second propagation path is used to indicate the order of the multiple compressed video frames.

[0008] Based on the first propagation path and the second propagation path, feature extraction is performed on the plurality of target video frames and the plurality of compressed video frames respectively to obtain a first feature and a second feature. The first feature is used to represent the features of the plurality of target video frames, and the second feature is used to represent the features of the plurality of compressed video frames.

[0009] A second video segment is generated based on the first feature and the second feature.

[0010] According to another aspect of the embodiments of this disclosure, a video processing apparatus is provided, comprising:

[0011] The determining unit is configured to perform a first propagation path and a second propagation path based on the quality information of multiple compressed video frames in a first video segment. The first propagation path is used to indicate the order of multiple target video frames, wherein the target video frames are compressed video frames whose quality meets the quality conditions, and the second propagation path is used to indicate the order of the multiple compressed video frames.

[0012] The feature extraction unit is configured to perform feature extraction on the plurality of target video frames and the plurality of compressed video frames based on the first propagation path and the second propagation path, respectively, to obtain a first feature and a second feature, wherein the first feature is used to represent the features of the plurality of target video frames and the second feature is used to represent the features of the plurality of compressed video frames;

[0013] The generation unit is configured to generate a second video segment based on the first feature and the second feature.

[0014] In some embodiments, the feature extraction unit is configured to perform the following actions: determining a target video frame sequence including the plurality of target video frames based on the first propagation path; determining a compressed video frame sequence including the plurality of compressed video frames based on the second propagation path; and performing feature extraction on the target video frame sequence and the compressed video frame sequence respectively based on a recurrent convolutional neural network to obtain the first feature and the second feature.

[0015] In some embodiments, the generating unit includes:

[0016] The splicing subunit is configured to splice the first feature and the second feature along the feature dimension to obtain a spliced ​​feature, wherein the number of feature dimensions of the spliced ​​feature is equal to the sum of the number of feature dimensions of the first feature and the number of feature dimensions of the second feature;

[0017] The generation subunit is configured to generate the second video segment based on the splicing features.

[0018] In some embodiments, the stitching subunit is configured to perform residual processing on the stitching features to obtain a first target feature; and to perform super-resolution processing on the first target feature to generate the second video segment.

[0019] In some embodiments, the splicing subunit is configured to perform a deformable convolution-based feature extraction operator to convolve the spliced ​​features to obtain a second target feature; and to perform super-resolution processing on the second target feature to generate the second video segment.

[0020] In some embodiments, the generation unit is configured to perform the following actions: obtaining the weight parameters of the first feature and the weight parameters of the second feature; performing weighted fusion of the first feature and the second feature using an attention mechanism based on the weight parameters of the first feature and the weight parameters of the second feature to obtain a third target feature; and performing super-resolution processing on the third target feature to generate the second video segment.

[0021] In some embodiments, the generation unit is configured to perform the following operations: obtain the weight parameters of the first feature and the weight parameters of the second feature; perform weighted fusion of the first feature and the second feature using an attention mechanism based on the weight parameters of the first feature and the weight parameters of the second feature to obtain a fused feature; perform residual processing on the fused feature to obtain a fourth target feature; and perform super-resolution processing on the fourth target feature to generate the second video segment.

[0022] In some embodiments, the determining unit is configured to perform the following actions: acquiring quality information of the plurality of compressed video frames, wherein the quality information is a quality parameter of the compressed video frame, and the quality parameter is inversely correlated with the quality of the corresponding compressed video frame; acquiring a plurality of target video frames from the plurality of compressed video frames whose quality parameters are lower than a quality parameter threshold; determining a first propagation path based on the timing of the plurality of target video frames; and determining a second propagation path based on the timing of the plurality of compressed video frames.

[0023] According to another aspect of the embodiments of this disclosure, an electronic device is provided, the electronic device comprising:

[0024] One or more processors;

[0025] Memory used to store the executable program code of the processor;

[0026] The processor is configured to execute the program code to implement the aforementioned video processing method.

[0027] According to another aspect of the present disclosure, a computer-readable storage medium is provided that, when program code in the computer-readable storage medium is executed by a processor of an electronic device, enables the electronic device to perform the video processing method described above.

[0028] According to another aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the video processing method described above.

[0029] This disclosure provides a video processing scheme. Since the quality of compressed video frames is inversely correlated with the noise they contain, a first propagation path is determined based on the quality information of multiple compressed video frames during the decompression of a first video segment. This path indicates the order of multiple target video frames, reducing the impact of noise when generating video segments by extracting first features according to the order indicated by the first propagation path. Because the multiple target video frames are discontinuous in time, the generated video segments contain motion errors. By adding second features extracted from the multiple compressed video frames according to the order indicated by the second propagation path to the first features—that is, generating second video segments using both the first and second features—not only can the impact of compression noise be reduced, but the continuity of the compressed video frames is also fully utilized to eliminate motion errors, thereby enabling the decompression to obtain a high-quality second video segment.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0031] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0032] Figure 1 This is a schematic diagram illustrating the implementation environment of a video processing method according to an exemplary embodiment.

[0033] Figure 2 This is a flowchart illustrating a video processing method according to an exemplary embodiment.

[0034] Figure 3 This is a flowchart illustrating another video processing method according to an exemplary embodiment.

[0035] Figure 4 This is a schematic diagram illustrating a propagation path according to an exemplary embodiment.

[0036] Figure 5 This is a schematic diagram illustrating quality parameters of a compressed video frame according to an exemplary embodiment.

[0037] Figure 6 This is a schematic diagram illustrating a method for processing splicing features using a residual network, according to an exemplary embodiment.

[0038] Figure 7 This is a schematic diagram illustrating a method for decompressing video using a residual network, according to an exemplary embodiment.

[0039] Figure 8This is an illustration of the effect of decompressing a video frame according to an exemplary embodiment.

[0040] Figure 9 This is a schematic diagram of the structure of a video processing apparatus according to an exemplary embodiment.

[0041] Figure 10 This is a schematic diagram of the structure of another video processing apparatus according to an exemplary embodiment.

[0042] Figure 11 This is a block diagram illustrating a terminal according to an exemplary embodiment.

[0043] Figure 12 This is a block diagram illustrating a server according to an exemplary embodiment. Detailed Implementation

[0044] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0045] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0046] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the video clips involved in this disclosure were all obtained with full authorization.

[0047] To better understand this plan, the following explanations are provided for the terms used in this plan.

[0048] I-frame: Represents a complete picture; decoding only requires the data of this frame.

[0049] P-frame: This indicates the difference between this frame and the previous I-frame or P-frame. Decoding a P-frame only requires superimposing the difference of this frame onto the previously cached image to generate the final image.

[0050] B-frame: Represents the difference between this frame and the frames before and after it. Decoding a B-frame requires not only obtaining the previous buffered frames, but also decoding the following frames. The data from the previous and following frames are superimposed with the data of this frame to obtain the final image.

[0051] Encoding is the process of converting information from one form or format to another. It involves encoding text, numbers, or other objects into digital codes using pre-defined methods, or converting information or data into specified electrical pulse signals. Encoding is widely used in electronic computers, televisions, remote controls, and communications. Decoding is the reverse process of encoding. In the embodiments of this disclosure, encoding a video segment is equivalent to compressing the video segment, and decoding a video segment is equivalent to decompressing the video segment.

[0052] Recurrent Convolutional Neural Networks (RCNs) are a type of recurrent neural network that takes sequential data as input, recursively processes data in the direction of sequence evolution, and connects all nodes (recurrent units) in a chain-like manner. In this embodiment of the present disclosure, the RCN takes a sequence of video frames as input and processes multiple video frames in the order of the video frames.

[0053] QP (Quantization Parameter): This indicates the degree of compression of spatial details in an image and reflects the image quality to a certain extent. In this embodiment, the QP value is used to represent the quality of the compressed video frame.

[0054] ReLU function: A commonly used activation function in artificial neural networks, usually referring to nonlinear functions represented by the ramp function and its variants.

[0055] Figure 1 This is a schematic diagram illustrating an implementation environment for a video processing method according to an exemplary embodiment. Taking execution by a terminal as an example, see [link to example]. Figure 1 The implementation environment includes: terminal 101 and server 102.

[0056] Terminal 101 is at least one of the following devices: smartphone, smartwatch, desktop computer, laptop, MP3 player, MP4 player, and laptop computer. Terminal 101 has an application for video processing installed and running. This application can be a multimedia application, a social application, or a monitoring application; this embodiment does not limit the specific type. Terminal 101 can connect to server 102 via a wireless or wired network, thereby obtaining encoded video segments from server 102. The encoded video segments include multiple compressed video frames. Terminal 101 generally refers to one of multiple terminals; this embodiment uses terminal 101 as an example. Those skilled in the art will understand that the number of terminals can be more or less. For example, there can be several terminals, or dozens or hundreds of terminals, or even more. This embodiment does not limit the number or type of terminals.

[0057] Server 102 can be at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 102 can connect to terminal 101 and other terminals via a wireless or wired network. Server 102 can send compressed video segments to terminal 101, which then decompresses the video segments using the video processing method provided in this embodiment. In some embodiments, the number of servers can be more or less, and this embodiment does not limit this. Of course, server 102 may also include other functional servers to provide more comprehensive and diversified services.

[0058] Figure 2 This is a flowchart illustrating a video processing method according to an exemplary embodiment, see [link to flowchart]. Figure 2 Taking execution by the terminal as an example, the steps include:

[0059] In step 201, the terminal determines a first propagation path and a second propagation path based on the quality information of multiple compressed video frames in the first video segment. The first propagation path is used to indicate the order of multiple target video frames, which are compressed video frames whose quality meets the quality conditions. The second propagation path is used to indicate the order of multiple compressed video frames.

[0060] In this embodiment of the disclosure, the terminal is Figure 1The terminal shown is capable of receiving an encoded first video segment sent by a server. This first video segment includes multiple compressed video frames. The first video segment can be a complete video or a portion extracted from a complete video; this embodiment does not limit the scope. The multiple compressed video frames contain noise generated during the encoding process. The amount of noise is inversely correlated with the quality of the compressed video frames. The more noise generated during encoding, the lower the quality of the compressed video frame; conversely, the less noise generated, the higher the quality. The terminal can select multiple target video frames whose quality meets certain conditions based on the quality of the multiple compressed video frames. These quality conditions can include the target video frame's quality reaching a quality threshold, the target video frame's quality being within a preset range, or the target video frame's quality parameter being lower than a quality parameter threshold; this embodiment does not limit the scope. The quality parameter represents the quality of the video frame.

[0061] In step 202, the terminal extracts features from multiple target video frames and multiple compressed video frames based on the first propagation path and the second propagation path, respectively, to obtain a first feature and a second feature. The first feature is used to represent the features of the multiple target video frames, and the second feature is used to represent the features of the multiple compressed video frames.

[0062] In this embodiment, the terminal can extract features from multiple target video frames based on a first propagation path to obtain first features of the multiple target video frames. The terminal can also extract features from these multiple compressed video frames based on a second propagation path to obtain second features of the multiple compressed video frames. Since the multiple target video frames contain less compression noise, the first features have less compression noise than the second features, and can more accurately reflect the characteristics of the video frames.

[0063] In step 203, the terminal generates a second video segment based on the first feature and the second feature.

[0064] In this embodiment of the disclosure, the terminal can perform super-resolution processing on the first feature and the second feature to generate a second video segment. The second video segment includes multiple decompressed video frames, and the second video segment is the video segment obtained by decompressing the first video segment.

[0065] The solution provided in this disclosure addresses the issue that the quality of compressed video frames is inversely correlated with the noise they contain. By determining a first propagation path indicating the order of multiple target video frames based on their quality information during the decompression of the first video segment, the impact of noise can be reduced when generating video segments by extracting first features according to the order indicated by the first propagation path. Since the multiple target video frames are not sequentially continuous, the generated video segments contain motion errors. By adding second features extracted from the multiple compressed video frames according to the order indicated by the second propagation path to the first features—that is, generating second video segments using both the first and second features—not only can the impact of compression noise be reduced, but the continuity of the compressed video frames can also be fully utilized to eliminate motion errors, thereby enabling the decompression to obtain high-quality second video segments.

[0066] In some embodiments, feature extraction is performed on multiple target video frames and multiple compressed video frames based on a first propagation path and a second propagation path, respectively, including:

[0067] Based on the first propagation path, a target video frame sequence including multiple target video frames is determined;

[0068] Based on the second propagation path, a compressed video frame sequence including multiple compressed video frames is determined;

[0069] Based on a recurrent convolutional neural network, feature extraction is performed on the target video frame sequence and the compressed video frame sequence to obtain the first feature and the second feature.

[0070] The solution provided in this disclosure constructs a target video frame sequence using multiple target video frames and a first propagation path indicating the order of the multiple target video frames. This allows the recurrent convolutional neural network to process the multiple target video frames sequentially according to their order during feature extraction, thereby ensuring that the first feature includes the features of the multiple target video frames. Similarly, a compressed video frame sequence is constructed using multiple compressed video frames and a second propagation path indicating the order of the multiple compressed video frames. This allows the recurrent convolutional neural network to process the multiple compressed video frames sequentially according to their order during feature extraction, thereby ensuring that the second feature includes the features of the multiple compressed video frames. This fully utilizes the information in the video frames. Furthermore, since the multiple target video frames in the target video frame sequence are high-quality compressed video frames, this disclosure allows for the processing of high-quality compressed video frames individually by extracting features from the target video frame sequence. Compared to processing multiple compressed video frames, this reduces the impact of noise in low-quality compressed video frames.

[0071] In some embodiments, generating a second video segment based on a first feature and a second feature includes:

[0072] The first feature and the second feature are concatenated along the feature dimension to obtain the concatenated feature. The number of feature dimensions of the concatenated feature is equal to the sum of the number of feature dimensions of the first feature and the number of feature dimensions of the second feature.

[0073] Based on the splicing features, a second video segment is generated.

[0074] The solution provided in this embodiment of the present disclosure is that the first feature is obtained from a high-quality target video frame, and the compression noise in the first feature is small. The second feature is obtained from multiple compressed video frames, and the second feature can reflect the continuity of the video frames. By splicing the first feature and the second feature, a second video segment is generated through the spliced ​​feature. This not only reduces the impact of compression noise, but also makes full use of the continuity of the compressed video frames to eliminate motion errors, thereby decompressing to obtain a high-quality second video segment.

[0075] In some embodiments, a second video segment is generated based on splicing features, including:

[0076] The spliced ​​features are processed by residual processing to obtain the first target features;

[0077] Super-resolution processing is performed on the features of the first target to generate the second video segment.

[0078] The solution provided in this embodiment of the present disclosure, by performing residual processing on the splicing features, enables the first target feature to fully integrate the characteristics of low noise and continuous video frames, thereby obtaining a high-quality second video segment through super-resolution processing of the first target feature.

[0079] In some embodiments, a second video segment is generated based on splicing features, including:

[0080] Based on the feature extraction operator of deformable convolution, the concatenated features are convolved to obtain the second target feature;

[0081] Super-resolution processing is performed on the features of the second target to generate a second video segment.

[0082] The solution provided in this embodiment of the present disclosure processes the spliced ​​features using a deformable convolution feature extraction operator. By leveraging the characteristics of deformable convolution, it can fully utilize the information contained in the features of the video frame, making the obtained second target feature more prominent in the video frame. This allows for the generation of a high-quality second video segment using the second target feature.

[0083] In some embodiments, generating a second video segment based on a first feature and a second feature includes:

[0084] Obtain the weight parameters of the first feature and the weight parameters of the second feature;

[0085] Based on the weight parameters of the first feature and the weight parameters of the second feature, the first feature and the second feature are weighted and fused through an attention mechanism to obtain the third target feature;

[0086] Super-resolution processing is performed on the features of the third target to generate a second video segment.

[0087] The solution provided in this embodiment introduces an attention mechanism to learn the importance of the first feature and the second feature based on the amount of detailed information in the features. Since the higher the importance, the larger the weight parameter, and the lower the importance, the smaller the weight parameter, the third target feature obtained based on the importance of the first feature and the second feature can make full use of the detailed information in the features and more accurately reflect the feature information of the video segment. Thus, a high-quality second video segment can be generated through the third target feature.

[0088] In some embodiments, generating a second video segment based on a first feature and a second feature includes:

[0089] Obtain the weight parameters of the first feature and the weight parameters of the second feature;

[0090] Based on the weight parameters of the first feature and the weight parameters of the second feature, the first feature and the second feature are weighted and fused through an attention mechanism to obtain the fused feature;

[0091] The fused features are processed using residuals to obtain the fourth target feature;

[0092] Super-resolution processing is performed on the features of the fourth target to generate the second video segment.

[0093] The solution provided in this embodiment introduces an attention mechanism, which fuses the first feature and the second feature based on their respective importance. This results in a fused feature that more accurately reflects the feature information of the video segment. Residual processing is then performed to ensure that the fourth target feature fully reflects the characteristics of low compression noise and the continuity of video frames. By processing the fourth target feature through super-resolution, a high-quality second video segment can be obtained.

[0094] In some embodiments, determining a first propagation path and a second propagation path based on quality information of multiple compressed video frames in a first video segment includes:

[0095] Obtain quality information for multiple compressed video frames. This quality information consists of quality parameters of the compressed video frames, which are inversely correlated with the quality of the corresponding compressed video frames.

[0096] From multiple compressed video frames, obtain multiple target video frames whose quality parameters are lower than the quality parameter threshold;

[0097] The first propagation path is determined based on the timing of multiple target video frames;

[0098] The second propagation path is determined based on the timing of multiple compressed video frames.

[0099] The solution provided in this disclosure means that the larger the quality parameter, the lower the quality of the compressed video frame, and the smaller the quality parameter, the higher the quality of the compressed video frame. Therefore, by measuring the quality of the compressed video frame through the quality parameter corresponding to the compressed video frame, it is possible to obtain target video frames with quality parameters lower than the quality parameter threshold from multiple compressed video frames, thereby improving the accuracy of obtaining high-quality target video frames.

[0100] The above Figure 2 The diagram illustrates the basic process of this disclosure. The following section, based on one implementation method, further elaborates on the solution provided in this disclosure. Figure 3 This is a flowchart illustrating another video processing method according to an exemplary embodiment. Taking execution by a terminal as an example, see [link to flowchart]. Figure 3 The method includes:

[0101] In step 301, the terminal obtains the quality information of multiple compressed video frames in the first video segment.

[0102] In this embodiment, the first video segment is encoded into a video segment comprising multiple compressed video frames. The quality information includes at least one of the following: the resolution of the compressed video frame, the quality parameters of the compressed video frame, or the contrast of the compressed video frame; this embodiment does not limit the specific quality of these parameters. Different compression levels result in different qualities of the compressed video frames. Since noise is generated during video segment encoding, compressed video frames with higher compression levels contain more noise and thus have lower quality; compressed video frames with lower compression levels contain less noise and thus have higher quality. The multiple compressed video frames may include three types of frames: I-frames, P-frames, and B-frames, or two types: P-frames and B-frames; this embodiment does not limit the specific types of frames. Specifically, for I-frames, P-frames, and B-frames, the compression level of I-frames is less than that of P-frames, and the compression level of P-frames is less than that of B-frames; that is, the quality of I-frames is higher than that of P-frames, and the quality of P-frames is higher than that of B-frames.

[0103] For example, this quality information includes the resolution of multiple compressed video frames. Compressed video frames with higher compression levels have lower resolutions, meaning they are of lower quality; conversely, compressed video frames with lower compression levels have higher resolutions, meaning they are of higher quality. The terminal can filter through these multiple compressed video frames using the resolution information.

[0104] In step 302, the terminal determines a first propagation path and a second propagation path based on the quality information of multiple compressed video frames. The first propagation path is used to indicate the order of multiple target video frames, which are compressed video frames whose quality meets the quality conditions. The second propagation path is used to indicate the order of multiple compressed video frames.

[0105] In this embodiment of the disclosure, in order to reduce the impact of noise generated during encoding during the decoding of video segments, the terminal can select compressed video frames whose quality meets the quality conditions from multiple compressed video frames based on the quality information used for multiple compressed video frames, and use the compressed video frames whose quality meets the quality conditions as target video frames. The terminal can obtain I-frames and P-frames as target video frames from the multiple video frames including I-frames, P-frames, and B-frames; or, the terminal can obtain P-frames as target video frames from the multiple video frames including P-frames and B-frames, and this embodiment of the disclosure is not limited in this regard. The quality conditions can be that the quality of the target video frame reaches a quality threshold, the quality of the target video frame is within a preset range, or the quality parameters of the target video frame are lower than a quality parameter threshold, etc., and this embodiment of the disclosure is not limited in this regard. The quality can be the resolution of the compressed video frame; the higher the resolution of the compressed video frame, the higher the quality of the compressed video frame; the lower the resolution of the compressed video frame, the lower the quality of the compressed video frame. The quality can also be the contrast or brightness of the compressed video frame, etc., and this embodiment of the disclosure is not limited in this regard. This quality parameter represents the quality of a compressed video frame, and it is inversely correlated with the quality of the corresponding compressed video frame. This quality parameter can be a quantization parameter used when compressing a video frame. A larger quality parameter indicates lower quality, and a smaller quality parameter indicates higher quality.

[0106] The terminal can determine a first propagation path based on the order of multiple target video frames. The order of the target video frames can be the temporal sequence of the target video frames or the reverse order; this embodiment does not limit this. Since the target video frames contain less noise from encoding, the first propagation path is a high-quality path for the target video frames. The first feature obtained through the first propagation path contains less noise, thus reducing the impact of noise when generating a decoded video segment based on the first feature. However, the I-frames and P-frames obtained through MPEG (Moving Picture Experts Group) video compression technology are discontinuous; that is, the multiple target video frames in the first propagation path are discontinuous, resulting in significant motion errors in the decoded video segment obtained based on these multiple target video frames. The terminal can introduce lower-quality compressed video frames between the target video frames based on the first propagation path to reduce motion errors caused by the discontinuity of the target video frames. These multiple target video frames and the lower-quality compressed video frames between the target video frames together constitute multiple compressed video frames in the first video segment. The terminal determines a second propagation path based on the order of these multiple compressed video frames. The order of the compressed video frames can be the temporal sequence of the compressed video frames or the reverse order of the temporal sequence of the compressed video frames; this disclosure does not limit this. Furthermore, the multiple compressed video frames in this disclosure can also be obtained using JPEG (Joint Photographic Experts Group) compression technology; this disclosure does not limit this either.

[0107] For example, Figure 4 This is a schematic diagram illustrating a propagation path according to an exemplary embodiment. See also... Figure 4 The first video segment contains multiple compressed video frames, namely frame 1, frame 2, frame 3, frame 4, frame 5, frame 6, and frame 7. Among them, frames 1, 3, 5, and 7 are high-quality compressed video frames, i.e., the target video frames. Frames 2, 4, and 6 are low-quality compressed video frames. The terminal determines a first propagation path based on the order of frames 1, 3, 5, and 7, which is "1→3→5→7". The terminal determines a second propagation path based on the order of frames 1, 2, 3, 4, 5, 6, and 7, which is "1→2→3→4→5→6→7".

[0108] In some embodiments, the quality of compressed video frames is represented by quality parameters. The terminal can obtain target video frames whose quality parameters meet certain conditions from multiple compressed video frames using these quality parameters, thereby determining a first propagation path and a second propagation path based on the order of the target video frames and the order of the multiple compressed frames, respectively. Accordingly, the process of the terminal determining the first and second propagation paths is as follows: the terminal obtains quality information of multiple compressed video frames, which is the quality parameter of the compressed video frames, and this quality parameter is inversely correlated with the quality of the corresponding compressed video frame. Then, the terminal obtains multiple target video frames from the multiple compressed video frames whose quality parameters are lower than a quality parameter threshold. Then, the terminal determines the first propagation path based on the timing of the multiple target video frames. Then, the terminal determines the second propagation path based on the timing of the multiple compressed video frames. Alternatively, the terminal can also determine the second propagation path based on the timing of the multiple compressed video frames before determining the first propagation path; or, the terminal can also determine the second propagation path based on the timing of the multiple compressed video frames before obtaining the quality information. This disclosure does not limit this approach. The solution provided in this disclosure means that the larger the quality parameter, the lower the quality of the compressed video frame, and the smaller the quality parameter, the higher the quality of the compressed video frame. This allows the quality of the compressed video frame to be measured by the quality parameter corresponding to the compressed video frame in the quality information. As a result, it is possible to obtain target video frames with quality parameters lower than the quality parameter threshold from multiple compressed video frames, thereby improving the accuracy of obtaining high-quality target video frames.

[0109] For example, a terminal can use the QP value of a compressed video frame as a quality parameter for that compressed video frame. Figure 5 This is a schematic diagram illustrating quality parameters of a compressed video frame according to an exemplary embodiment. See also... Figure 5 The first compressed video frame is an I-frame, which has the lowest QP value. Other compressed video frames with QP values ​​below 40 are P-frames, which also have relatively low QP values. Compressed video frames with QP values ​​above 40 are B-frames, which have relatively high QP values. The terminal can extract I-frames with QP values ​​below 40 and multiple P-frames from multiple video frames as target video frames.

[0110] In some embodiments, the terminal can select multiple target video frames from multiple compressed video frames based on quality parameters, and then determine a first propagation path based on the timing of the target video frames, and determine a second propagation path based on the timing of the multiple compressed video frames.

[0111] For example, multiple compressed video frames are sequentially named as frame 1, frame 2, frame 3, frame 4, frame 5, frame 6, frame 7, frame 8, frame 9, and frame 10. The quality parameter of frame 1 is 36, the quality parameters of frames 4, 7, and 10 are all 38, and the quality parameters of frames 2, 3, 5, 6, 8, and 9 are all 45. Therefore, frames 1, 4, 7, and 10 are high-quality compressed video frames, i.e., the target video frames. Based on the timing of these target video frames, the terminal can determine the first propagation path as "1→4→7→10". Based on the timing of the aforementioned compressed video frames, the terminal can determine the second propagation path as "1→2→3→4→5→6→7→8→9→10".

[0112] In step 303, the terminal performs feature extraction on multiple target video frames and multiple compressed video frames based on the first propagation path and the second propagation path, respectively, to obtain a first feature and a second feature. The first feature is used to represent the features of the multiple target video frames, and the second feature is used to represent the features of the multiple compressed video frames.

[0113] In this embodiment of the disclosure, the terminal can extract features from multiple target video frames in the order indicated by the first propagation path to obtain a first feature. The terminal can also extract features from multiple compressed video frames in the order indicated by the second propagation path to obtain a second feature. Both the first and second features can be represented as C*H*W. Here, C is the feature dimension, H and W are the number of rows and columns of a single feature vector, respectively, H is equal to the height of the compressed video frame, and W is equal to the width of the compressed video frame.

[0114] In some embodiments, the terminal can process multiple target video frames and compressed video frames separately using a recurrent convolutional neural network to obtain a first feature and a second feature. Accordingly, the terminal determines a target video frame sequence based on a first propagation path and multiple target video frames. Then, the terminal determines a compressed video frame sequence based on a second propagation path and multiple compressed video frames. Then, the terminal performs feature extraction on the target video frame sequence and the compressed video frame sequence respectively using a recurrent convolutional neural network to obtain the first feature and the second feature. The first feature is a feature of the target video frame sequence, and the second feature is a feature of the compressed video frame sequence. The solution provided in this disclosure constructs a target video frame sequence using multiple target video frames and a first propagation path indicating the order of the multiple target video frames. This allows the recurrent convolutional neural network to process the multiple target video frames sequentially according to their order during feature extraction, thereby ensuring that the first feature includes the features of the multiple target video frames. Similarly, a compressed video frame sequence is constructed using multiple compressed video frames and a second propagation path indicating the order of the multiple compressed video frames. This allows the recurrent convolutional neural network to process the multiple compressed video frames sequentially according to their order during feature extraction, thereby ensuring that the second feature includes the features of the multiple compressed video frames. This fully utilizes the information in the video frames. Furthermore, since the multiple target video frames in the target video frame sequence are high-quality compressed video frames, this disclosure allows for the processing of high-quality compressed video frames individually by extracting features from the target video frame sequence. Compared to processing multiple compressed video frames, this reduces the impact of noise in low-quality compressed video frames.

[0115] It should be noted that, in order to obtain higher quality video clips after decompression, the terminal can train a recurrent convolutional neural network to achieve higher quality extracted features. During this training process, both the first and second features can be represented as B*C*H*W. Here, B is the batch size of video frames during training, C is the feature dimension, and H and W are the number of rows and columns of a single feature vector, respectively.

[0116] In step 304, the terminal generates a second video segment based on the first feature and the second feature.

[0117] In this embodiment, the terminal can use super-resolution technology to process the first feature and the second feature to generate a second video segment. The second video segment includes multiple decompressed video frames. During this process, the terminal can perform sub-pixel rearrangement of the first and second features and upsample the processed features to generate the second video segment. The second video segment is the video segment after decompressing the first video segment. The terminal can process the first and second features using feature concatenation and generate the second video segment based on the concatenated features; alternatively, the terminal can fuse the first and second features based on an attention mechanism and generate the second video segment based on the fused features; or, the terminal can combine an attention mechanism and a residual network to process the first and second features and generate the second video segment based on the processed features.

[0118] In some embodiments, the terminal can generate a second video segment using a feature obtained by splicing a first feature and a second feature. Accordingly, the terminal splices the first feature and the second feature along a feature dimension to obtain a spliced ​​feature. Then, the terminal generates the second video segment based on the spliced ​​feature. The feature dimension of the spliced ​​feature is equal to the sum of the feature dimensions of the first feature and the second feature. The solution provided in this disclosure embodiment uses a first feature obtained from a high-quality target video frame with low noise, and a second feature obtained from multiple compressed video frames that reflects the continuity of the video frames. By splicing the first feature and the second feature, and generating the second video segment using the spliced ​​feature, not only can the influence of noise be reduced, but the continuity of the compressed video frames can also be fully utilized to eliminate motion errors, thereby enabling the decompression to obtain a high-quality second video segment.

[0119] In this embodiment, the terminal processes the stitched features using a residual network and then generates a second video segment based on the processed features. Accordingly, the terminal performs residual processing on the stitched features to obtain a first target feature. Then, the terminal performs super-resolution processing on the first target feature to generate the second video segment. The solution provided in this disclosure, by performing residual processing on the stitched features, enables the first target feature to fully integrate the characteristics of low noise and continuous video frames, thereby obtaining a high-quality second video segment through super-resolution processing of the first target feature.

[0120] For example, Figure 6 This is a schematic diagram illustrating the use of residual networks to process splicing features according to an exemplary embodiment. See also Figure 6The terminal inputs the concatenated feature into a residual network, which includes a first convolutional layer, an activation function, and a second convolutional layer. Both the first and second convolutional layers are two-dimensional convolutional layers. The concatenated feature undergoes sequential processing through the first convolutional layer, the activation function, and the second convolutional layer to obtain an intermediate feature. Then, the terminal adds this intermediate feature to the concatenated feature using the residual network to obtain the first target feature. The activation function used is the ReLU activation function.

[0121] To better understand this plan, it will be explained in its entirety again below. Figure 7 This is a schematic diagram illustrating a video decompression process using a residual network according to an exemplary embodiment. The terminal concatenates a first feature and a second feature to obtain a concatenated feature. Then, the terminal inputs this concatenated feature into a residual network, which processes the feature and outputs a first target feature. Next, the terminal inputs this first target feature into a reconstruction network, which performs super-resolution processing on the feature to obtain a high-quality second video segment.

[0122] It should be noted that the terminal can also use deformable convolution feature extraction operators to process the spliced ​​features, and then generate a second video segment based on the processed features. Accordingly, the terminal convolves the spliced ​​features using deformable convolution feature extraction operators to obtain the second target feature. Then, the terminal performs super-resolution processing on the second target feature to generate the second video segment. The solution provided in this embodiment, by processing the spliced ​​features using deformable convolution feature extraction operators, can fully utilize the information contained in the features of the video frames through the characteristics of deformable convolution, making the obtained second target feature more prominent in the video frame's details, thereby enabling the generation of a high-quality second video segment using this second target feature.

[0123] In some embodiments, during the decompression of video segments, the terminal can supplement the compressed video frame with more detailed information to generate the corresponding decompressed video frame. The terminal can introduce an attention mechanism to determine the importance of the first feature and the second feature based on the amount of detailed information of the video frame contained in the first feature and the second feature. Then, the terminal fuses the first feature and the second feature based on their importance, and generates a second video segment using the fused feature. The more detailed information a feature contains, the higher its importance; the less detailed information a feature contains, the lower its importance. Accordingly, the terminal obtains the weight parameters of the first feature and the second feature, and then, based on these weight parameters, the terminal performs a weighted fusion of the first feature and the second feature using the attention mechanism to obtain a third target feature. The terminal then performs super-resolution processing on the third target feature to generate the second video segment. The solution provided in this embodiment introduces an attention mechanism to learn the importance of the first feature and the second feature based on the amount of detailed information in the features. Since the higher the importance, the larger the weight parameter, and the lower the importance, the smaller the weight parameter, the third target feature obtained based on the importance of the first feature and the second feature can make full use of the detailed information in the features and more accurately reflect the feature information of the video segment. Thus, a high-quality second video segment can be generated through the third target feature.

[0124] In some embodiments, the terminal can also jointly process the first feature and the second feature through an attention mechanism and a residual network, and then generate a second video segment based on the processed features. Accordingly, the terminal obtains the weight parameters of the first feature and the weight parameters of the second feature. Then, the terminal performs weighted fusion of the first feature and the second feature through an attention mechanism according to the weight parameters of the first feature and the second feature, obtaining a fused feature. Then, the terminal performs residual processing on the fused feature to obtain a fourth target feature. Then, the terminal performs super-resolution processing on the fourth target feature to generate the second video segment. The solution provided by this disclosure, by introducing an attention mechanism, fuses the first feature and the second feature based on their importance, making the obtained fused feature more accurately reflect the feature information of the video segment. Further residual processing allows the fourth target feature to fully reflect the characteristics of low noise and continuous video frames, thereby obtaining a high-quality second video segment through super-resolution processing of the fourth target feature.

[0125] To clearly understand this solution, we will apply it to the process of decompressing videos, using compressed video frames as an example. Figure 8This is a diagram illustrating the effect of decompressing a video frame according to an exemplary embodiment. See also... Figure 8 , Figure 8 (a) shows the compressed video frame before decompression. The water ripples in the compressed video frame are affected by compression noise, and the area marked with a box appears blurred, indicating that the quality of the compressed video frame is low. Figure 8 (b) in the middle shows Figure 8 (a) is the decompressed video frame. This video frame eliminates the effects of compression noise and supplements the detailed information in the video frame, reducing the blur at the frame and making the video frame clearer and of higher quality. Figure 8 (c) shows the compressed video frame before decompression. The hill texture in the compressed video frame is affected by compression noise, block effect appears at the bounding box, and the overall image looks blurry, indicating that the quality of the compressed video frame is low. Figure 8 (d) in the text shows Figure 8 (c) is the decompressed video frame. This video frame eliminates the effects of compression noise and supplements the detailed information in the video frame, making the block effect at the frame disappear. This video frame is clearer and of higher quality.

[0126] The solution provided in this disclosure addresses the issue that the quality of compressed video frames is inversely correlated with the noise they contain. By determining a first propagation path indicating the order of multiple target video frames based on their quality information during the decompression of the first video segment, the impact of noise can be reduced when generating video segments by extracting first features according to the order indicated by the first propagation path. Since the multiple target video frames are not sequentially continuous, the generated video segments contain motion errors. By adding second features extracted from the multiple compressed video frames according to the order indicated by the second propagation path to the first features—that is, generating second video segments using both the first and second features—not only can the impact of compression noise be reduced, but the continuity of the compressed video frames can also be fully utilized to eliminate motion errors, thereby enabling the decompression to obtain high-quality second video segments.

[0127] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0128] Figure 9 This is a schematic diagram illustrating the structure of a video processing apparatus according to an exemplary embodiment. See also: Figure 9 The device includes: a determination unit 901, a feature extraction unit 902, and a generation unit 903.

[0129] The determining unit 901 is configured to perform a first propagation path and a second propagation path based on the quality information of multiple compressed video frames in the first video segment. The first propagation path is used to indicate the order of multiple target video frames, which are compressed video frames whose quality meets the quality conditions. The second propagation path is used to indicate the order of the multiple compressed video frames.

[0130] The feature extraction unit 902 is configured to perform feature extraction on multiple target video frames and multiple compressed video frames based on a first propagation path and a second propagation path, respectively, to obtain a first feature and a second feature. The first feature is used to represent the features of the multiple target video frames, and the second feature is used to represent the features of the multiple compressed video frames.

[0131] The generation unit 903 is configured to generate a second video segment based on the first feature and the second feature.

[0132] In some embodiments, Figure 10 This is a schematic diagram illustrating the structure of another video processing apparatus according to an exemplary embodiment. See also: Figure 10 The feature extraction unit 902 is configured to perform the following operations: based on a first propagation path, determine a target video frame sequence including multiple target video frames; based on a second propagation path, determine a compressed video frame sequence including multiple compressed video frames; and based on a recurrent convolutional neural network, extract features from the target video frame sequence and the compressed video frame sequence to obtain a first feature and a second feature.

[0133] In some embodiments, see continue to see Figure 10 The generating unit 903 includes:

[0134] The splicing subunit 1001 is configured to splice the first feature and the second feature in the feature dimension to obtain a spliced ​​feature, the number of feature dimensions of the spliced ​​feature being equal to the sum of the number of feature dimensions of the first feature and the number of feature dimensions of the second feature;

[0135] Subunit 1002 is configured to generate a second video segment based on splicing features.

[0136] In some embodiments, see continue to see Figure 10 The splicing subunit 1001 is configured to perform residual processing on the splicing features to obtain the first target feature; and to perform super-resolution processing on the first target feature to generate the second video segment.

[0137] In some embodiments, see continue to see Figure 10 The splicing subunit 1001 is configured to perform a feature extraction operator based on deformable convolution, convolve the spliced ​​features to obtain the second target features, and perform super-resolution processing on the second target features to generate the second video segment.

[0138] In some embodiments, see continue to see Figure 10 The generation unit 903 is configured to perform the following operations: obtain the weight parameters of the first feature and the weight parameters of the second feature; perform weighted fusion of the first feature and the second feature through an attention mechanism based on the weight parameters of the first feature and the weight parameters of the second feature to obtain the third target feature; and perform super-resolution processing on the third target feature to generate the second video segment.

[0139] In some embodiments, see continue to see Figure 10 The generation unit 903 is configured to perform the following operations: obtain the weight parameters of the first feature and the weight parameters of the second feature; perform weighted fusion of the first feature and the second feature through an attention mechanism based on the weight parameters of the first feature and the weight parameters of the second feature to obtain a fused feature; perform residual processing on the fused feature to obtain a fourth target feature; and perform super-resolution processing on the fourth target feature to generate a second video segment.

[0140] In some embodiments, see continue to see Figure 10 The determining unit 901 is configured to perform the following actions: acquiring quality information of multiple compressed video frames, wherein the quality information is the quality parameter of the compressed video frame, and the quality parameter is inversely correlated with the quality of the corresponding compressed video frame; acquiring multiple target video frames from the multiple compressed video frames whose quality parameter is lower than the quality parameter threshold; determining a first propagation path based on the timing of the multiple target video frames; and determining a second propagation path based on the timing of the multiple compressed video frames.

[0141] The apparatus provided in this disclosure reduces the impact of noise when generating video segments by extracting first features according to the order indicated by the first propagation path based on the quality information of multiple compressed video frames during the decompression of a first video segment. Since the multiple target video frames are discontinuous in time, resulting in motion errors in the generated video segments, the apparatus adds second features extracted from the multiple compressed video frames according to the order indicated by the second propagation path to the first features. This not only reduces the impact of compression noise but also fully utilizes the continuity of the compressed video frames to eliminate motion errors, thereby enabling the decompression of high-quality second video segments.

[0142] It should be noted that the video processing apparatus provided in the above embodiments is only illustrated by the division of the above functional units when processing video. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the video processing apparatus and video processing method embodiments provided in the above embodiments belong to the same concept, and their implementation process can be found in the method embodiments, which will not be repeated here.

[0143] Regarding the apparatus in the above embodiments, the manner in which each unit performs its operation has been described in the embodiments relating to the method, and will not be elaborated upon here.

[0144] When an electronic device is provided as a terminal, Figure 11 This is a block diagram illustrating a terminal 1100 according to an exemplary embodiment. The terminal... Figure 11 A structural block diagram of a terminal 1100 provided in an exemplary embodiment of this disclosure is shown. The terminal 1100 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal 1100 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0145] Typically, terminal 1100 includes a processor 1101 and a memory 1102.

[0146] Processor 1101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1101 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1101 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1101 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1101 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0147] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1102 are used to store at least one program code, which is executed by the processor 1101 to implement the video processing method provided in the method embodiments of this disclosure.

[0148] In some embodiments, the terminal 1100 may also optionally include: a peripheral device interface 1103 and at least one peripheral device. The processor 1101, memory 1102, and peripheral device interface 1103 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1103 via a bus, signal line, or circuit board. The peripheral device includes at least one of: a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, an audio circuit 1107, a positioning assembly 1108, and a power supply 1109.

[0149] Peripheral device interface 1103 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1101 and memory 1102. In some embodiments, processor 1101, memory 1102 and peripheral device interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1101, memory 1102 and peripheral device interface 1103 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0150] The radio frequency (RF) circuit 1104 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1104 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1104 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1104 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1104 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1104 may also include circuitry related to NFC (Near Field Communication), which is not limited in this disclosure.

[0151] Display screen 1105 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1105 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1101 for processing. In this case, display screen 1105 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1105, which serves as the front panel of terminal 1100; in other embodiments, there may be at least two display screens, respectively disposed on different surfaces of terminal 1100 or in a folded design; in still other embodiments, display screen 1105 may be a flexible display screen, disposed on a curved or folded surface of terminal 1100. Furthermore, display screen 1105 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1105 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0152] The camera assembly 1106 is used to acquire images or videos. Optionally, the camera assembly 1106 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1106 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0153] The audio circuit 1107 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1101 for processing, or input to the radio frequency circuit 1104 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1100. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1101 or the radio frequency circuit 1104 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1107 may also include a headphone jack.

[0154] The positioning component 1108 is used to locate the current geographical location of the terminal 1100 in order to enable navigation or LBS (Location Based Service). The positioning component 1108 can be a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, Russia's Granas system, or the European Union's Galileo system.

[0155] Power supply 1109 is used to power the various components in terminal 1100. Power supply 1109 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1109 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0156] In some embodiments, the terminal 1100 further includes one or more sensors 1110. The one or more sensors 1110 include, but are not limited to: an accelerometer 1111, a gyroscope 1112, a pressure sensor 1113, a fingerprint sensor 1114, an optical sensor 1115, and a proximity sensor 1116.

[0157] Accelerometer 1111 can detect the magnitude of acceleration along the three axes of a coordinate system established with terminal 1100. For example, accelerometer 1111 can be used to detect the components of gravitational acceleration along the three axes. Processor 1101 can control display screen 1105 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1111. Accelerometer 1111 can also be used for collecting game or user motion data.

[0158] The gyroscope sensor 1112 can detect the orientation and rotation angle of the terminal 1100. The gyroscope sensor 1112 can work in conjunction with the accelerometer sensor 1111 to collect the user's 3D movements on the terminal 1100. Based on the data collected by the gyroscope sensor 1112, the processor 1101 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0159] The pressure sensor 1113 can be disposed on the side bezel of the terminal 1100 and / or on the lower layer of the display screen 1105. When the pressure sensor 1113 is disposed on the side bezel of the terminal 1100, it can detect the user's grip signal on the terminal 1100, and the processor 1101 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1113. When the pressure sensor 1113 is disposed on the lower layer of the display screen 1105, the processor 1101 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1105. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0160] The fingerprint sensor 1114 is used to collect the user's fingerprint. The processor 1101 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1114, or the fingerprint sensor 1114 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as trusted, the processor 1101 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 1114 can be located on the front, back, or side of the terminal 1100. When the terminal 1100 has physical buttons or a manufacturer's logo, the fingerprint sensor 1114 can be integrated with the physical buttons or manufacturer's logo.

[0161] An optical sensor 1115 is used to collect ambient light intensity. In one embodiment, the processor 1101 can control the display brightness of the display screen 1105 based on the ambient light intensity collected by the optical sensor 1115. When the ambient light intensity is high, the display brightness of the display screen 1105 is increased; when the ambient light intensity is low, the display brightness of the display screen 1105 is decreased. In another embodiment, the processor 1101 can also dynamically adjust the shooting parameters of the camera assembly 1106 based on the ambient light intensity collected by the optical sensor 1115.

[0162] The proximity sensor 1116, also known as a distance sensor, is typically mounted on the front panel of the terminal 1100. The proximity sensor 1116 is used to detect the distance between the user and the front of the terminal 1100. In one embodiment, when the proximity sensor 1116 detects that the distance between the user and the front of the terminal 1100 is gradually decreasing, the processor 1101 controls the display screen 1105 to switch from a screen-on state to a screen-off state; when the proximity sensor 1116 detects that the distance between the user and the front of the terminal 1100 is gradually increasing, the processor 1101 controls the display screen 1105 to switch from a screen-off state to a screen-on state.

[0163] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on terminal 1100 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0164] When electronic devices are provided as servers, Figure 12 This is a block diagram illustrating a server 1200 according to an exemplary embodiment. The server 1200 can vary significantly due to differences in configuration or performance. It may include one or more Central Processing Units (CPUs) 1201 and one or more memories 1202. The memories 1202 store at least one line of program code, which is loaded and executed by the processor 1201 to implement the video processing methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 1200 may also include other components for implementing device functions, which will not be elaborated upon here.

[0165] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1102 or a memory 1202 including instructions. These instructions can be executed by the processor 1101 of the terminal 1100 or the processor 1201 of the server 1200 to complete the aforementioned video processing method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0166] A computer program product includes a computer program / instructions that, when executed by a processor, implement the aforementioned video processing method.

[0167] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0168] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method of video processing, the method comprising: The method comprises: determining a first propagation path and a second propagation path based on quantization parameters of a plurality of compressed video frames in a first video segment, the first propagation path being used to indicate an order of a plurality of target video frames, the target video frames being compressed video frames with quantization parameters lower than a parameter threshold, the quantization parameters being inversely related to quality of corresponding compressed video frames, the second propagation path being used to indicate an order of the plurality of compressed video frames; determining a target video frame sequence comprising the plurality of target video frames based on the first propagation path; determining a compressed video frame sequence comprising the plurality of compressed video frames based on the second propagation path; performing feature extraction on the target video frame sequence and the compressed video frame sequence respectively based on a recurrent convolutional neural network to obtain first features and second features, the first features being used to represent features of the target video frame sequence, the second features being used to represent features of the compressed video frame sequence; performing weighted fusion on the first features and the second features through an attention mechanism according to weight parameters of the first features and weight parameters of the second features, and generating a second video segment based on the fused features, the weight parameter of each feature being used to represent importance of the feature.

2. The video processing method of claim 1, wherein, The method of performing weighted fusion on the first features and the second features through an attention mechanism according to weight parameters of the first features and weight parameters of the second features, and generating a second video segment based on the fused features, comprises: obtaining third target features by performing weighted fusion on the first features and the second features through an attention mechanism according to the weight parameters of the first features and the weight parameters of the second features; performing super-resolution processing on the third target features to generate the second video segment.

3. The video processing method of claim 1, wherein, The method of performing weighted fusion on the first features and the second features through an attention mechanism according to weight parameters of the first features and weight parameters of the second features, and generating a second video segment based on the fused features, comprises: obtaining fused features by performing weighted fusion on the first features and the second features through an attention mechanism according to the weight parameters of the first features and the weight parameters of the second features; performing residual processing on the fused features to obtain fourth target features; performing super-resolution processing on the fourth target features to generate the second video segment.

4. The video processing method of claim 1, wherein, The method of determining a first propagation path and a second propagation path based on quantization parameters of a plurality of compressed video frames in a first video segment, comprises: obtaining quantization parameters of the plurality of compressed video frames; obtaining a plurality of target video frames with quantization parameters lower than a parameter threshold from the plurality of compressed video frames; determining the first propagation path based on time sequences of the plurality of target video frames; determining the second propagation path based on time sequences of the plurality of compressed video frames.

5. A video processing apparatus, comprising: The device comprises: The determining unit is configured to determine a first propagation path and a second propagation path based on quantization parameters of the plurality of compressed video frames in the first video segment, the first propagation path being used to indicate an order of a plurality of target video frames, the target video frames being compressed video frames whose quantization parameters are lower than a parameter threshold, the quantization parameters being inversely related to quality of corresponding compressed video frames, and the second propagation path being used to indicate an order of the plurality of compressed video frames. The feature extraction unit is configured to determine a target video frame sequence including the plurality of target video frames based on the first propagation path, determine a compressed video frame sequence including the plurality of compressed video frames based on the second propagation path, and perform feature extraction on the target video frame sequence and the compressed video frame sequence respectively based on a recurrent convolutional neural network to obtain first features and second features, the first features being used to represent features of the target video frame sequence, and the second features being used to represent features of the compressed video frame sequence. The generating unit is configured to perform weighting fusion on the first features and the second features through an attention mechanism according to weight parameters of the first features and weight parameters of the second features, and generate a second video segment based on the fused features, each feature's weight parameter being used to represent importance of the feature.

6. An electronic device, comprising: The electronic device comprises: one or more processors; a memory for storing program codes executable by the processors; wherein the processors are configured to execute the program codes to implement the video processing method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processors of the electronic device, the electronic device is enabled to perform the video processing method according to any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processors, implements the video processing method according to any one of claims 1 to 4.

Images