Video processing method, device, storage medium and computer program product

Abstract

The application discloses a video processing method and device, a storage medium and a computer program product, relates to the technical field of video coding, and comprises the following steps: acquiring an input video, determining the bit rates of each video frame of the input video in each coding path; determining a target coding path of the video frame according to the bit rates of the video frame in each coding path; encoding the video frame based on the target coding path to generate a first feature set corresponding to the video frame; and compressing the first feature set corresponding to each video frame to generate a compressed bit stream corresponding to each video frame. According to the application, a suitable coding path is adaptively selected according to the code rate of the video frame, so that the compression efficiency of the video is guaranteed, and the compression quality is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video coding, and particularly relates to a video processing method and device, a storage medium and a computer program product. BACKGROUND

[0002] With the popularity of high-definition and ultra-high-definition video content and the rapid development of the Internet and mobile networks, international standardization organizations have successively introduced video coding standards such as H.264 / AVC and H.265 / HEVC to improve video compression efficiency and quality.

[0003] At present, conventional algorithms use inter-frame prediction technology to reduce inter-frame redundancy, convert video blocks from a spatial domain to a frequency domain through transform coding to further compress spatial redundancy, use motion estimation technology to predict the motion trajectory of a video object to reduce motion redundancy, and use lossless coding on compressed data to reduce data volume, so that a higher compression ratio and lower bandwidth requirement can be achieved while ensuring video quality.

[0004] However, these conventional algorithms usually rely on fixed parameter settings and are difficult to adapt to dynamically changing scenes and bit rate requirements, resulting in difficulty in achieving optimal performance in the compression of complex video content and reduced video quality.

[0005] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0006] The main purpose of the present application is to provide a video processing method, device, storage medium and computer program product, which realizes dynamic balance of compression quality and video quality and improves video quality while ensuring compression quality.

[0007] To achieve the above purpose, the present application provides a video processing method, which comprises the following steps:

[0008] An input video is obtained, and the bit rates of each video frame of the input video in each coding path are determined;

[0009] According to the bit rates of each video frame in each coding path, the target coding path of the video frame is determined;

[0010] The video frame is coded based on the target coding path to generate a first feature set corresponding to the video frame;

[0011] The first feature set corresponding to each video frame is compressed to generate a compressed bit stream corresponding to each video frame.

[0012] In an embodiment, the step of determining the target encoding path of the video frame according to the bitrates of the video frame in each encoding path comprises:

[0013] determining a target bitrate of the video frame according to the preset bitrate of the video frame and the sum of the consumed bitrates of each encoding path;

[0014] determining the target encoding path of the video frame according to the bitrates of the video frame in each encoding path, the consumed bitrates of each encoding path and the target bitrate.

[0015] In an embodiment, the step of determining the target encoding path of the video frame according to the bitrates of the video frame in each encoding path, the consumed bitrates of each encoding path and the target bitrate comprises:

[0016] determining an allocated bitrate of the video frame according to the number of encoded frames of the input video and the preset bitrate of the video frame;

[0017] when the allocated bitrate is higher than the sum of the consumed bitrates and there is at least one encoding path whose generated bitrate is higher than the target bitrate, determining the encoding path with the smallest bitrate from the encoding paths with the bitrate higher than the target bitrate as the target encoding path of the video frame; or,

[0018] when the allocated bitrate is higher than the sum of the consumed bitrates and the generated bitrate of each encoding path is not higher than the target bitrate, determining the encoding path with the largest bitrate as the target encoding path of the video frame; or,

[0019] when the allocated bitrate is not higher than the sum of the consumed bitrates and there is at least one encoding path whose generated bitrate is not higher than the target bitrate, determining the encoding path with the largest bitrate from the encoding paths with the bitrate not higher than the target bitrate as the target encoding path of the video frame; or,

[0020] when the allocated bitrate is not higher than the sum of the consumed bitrates and the generated bitrate of each encoding path is higher than the target bitrate, determining the encoding path with the smallest bitrate as the target encoding path of the video frame.

[0021] In an embodiment, the target encoding path comprises a plurality of sub-encoding paths, the first feature set comprises sub-features of the video frame generated based on each sub-encoding path, and before the step of compressing the first feature set corresponding to each video frame, the method further comprises:

[0022] splicing the sub-features;

[0023] According to the spliced sub-features, the first feature set is updated.

[0024] In an embodiment, the step of splicing each of the sub-features comprises:

[0025] A first path is determined from each of the sub-encoding paths.

[0026] A sub-feature corresponding to the first path is determined as a first feature, and the first feature is modulated to obtain a second feature.

[0027] A second path is determined from a sub-encoding path of an unmodulated feature.

[0028] A sub-feature corresponding to the second path is determined as a third feature, and the second feature and the third feature are spliced to obtain a fourth feature.

[0029] The fourth feature is re-determined as the first feature, and the step of modulating the first feature to obtain the second feature and subsequent steps are performed until the sub-features corresponding to each of the sub-encoding paths are spliced.

[0030] In an embodiment, before the step of determining the bit rates of each video frame of the input video in each encoding path, the method further comprises:

[0031] The trade-off factors of each encoding path are adjusted, and the parameters of each encoding path are iteratively trained according to a preset training number until the rate-distortion ratios of the verification video set in each encoding path reach a maximum value and the bit rates of the verification video set in each encoding path reach a minimum value.

[0032] In an embodiment, after the step of generating the compressed bit stream, the method further comprises:

[0033] The compressed bit stream is decompressed to generate a second feature set.

[0034] The bit rates of each decoding path of the second feature set are determined.

[0035] According to the bit rates of each decoding path of the second feature set, a target decoding path is determined.

[0036] Based on the target decoding path, the second feature set is decoded to generate a restored video frame.

[0037] In addition, to achieve the above-mentioned purpose, the application further provides a video processing device, which comprises:

[0038] An acquisition module is configured to acquire an input video and determine the bit rates of each video frame of the input video in each encoding path.

[0039] determining a target encoding path of the video frame according to the bit rate of the video frame in each encoding path;

[0040] encoding the video frame based on the target encoding path to generate a first feature set corresponding to the video frame;

[0041] compressing the first feature set corresponding to each video frame to generate a compressed bit stream corresponding to each video frame.

[0042] In addition, to achieve the above object, the present application further provides a video processing device, which comprises a memory, a processor and a computer program stored in the memory and executable in the processor, and the computer program is configured to implement the steps of the video processing method as described above.

[0043] In addition, to achieve the above object, the present application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the video processing method as described above.

[0044] In addition, to achieve the above object, the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the video processing method as described above.

[0045] The one or more technical solutions provided by the embodiments of the present application have at least the following technical effects: by obtaining an input video, determining the bit rate of each video frame of the input video in each encoding path to obtain the code rate requirement of each video frame, providing a basis for subsequent path selection; then determining the target encoding path of the video frame according to the bit rate of the video frame in each encoding path, and encoding the video frame in the target encoding path to generate a first feature set corresponding to the video frame; by allocating an encoding path to the video frame according to the code rate, the allocated encoding path is more in line with the encoding requirement of each video frame, since the code rate requirement of the video frame is positively correlated with the richness of the content of the video frame, the complexity of the content of the video frame can be determined by the code rate requirement of the video frame, and then a path with a lower code rate is allocated to the video frame with simple content, so as to reduce the size of the compressed bit stream, or a path with a higher code rate is allocated to the video frame with high complexity, so as to retain as much video details as possible and improve the video quality; then the first feature set corresponding to each video frame is compressed to generate a compressed bit stream corresponding to each video frame, by compressing the features, the data redundancy is further reduced, the video compression quality is improved, and the video features are converted into the format of the compressed bit stream, which is more conducive to the transmission and storage of the video data. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0048] Figure 1 A flowchart of the first embodiment of the video processing method of this application is provided;

[0049] Figure 2 A video encoding diagram provided for the first embodiment of the video processing method of this application;

[0050] Figure 3 A video decoding diagram provided for the third embodiment of the video processing method of this application;

[0051] Figure 4 This is a schematic diagram of the module structure of the video processing device according to an embodiment of the present application;

[0052] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the video processing method in the embodiment of the present application. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of the embodiment of the present application is: obtain the input video, determine the bit rate of each video frame of the input video in each encoding path; determine the target encoding path of the video frame based on the bit rate of the video frame in each encoding path; encode the video frame based on the target encoding path to generate a first feature set corresponding to the video frame; compress the first feature set corresponding to each video frame to generate a compressed bit stream corresponding to each video frame.

[0056] In this embodiment, for ease of description, the following description is made with the terminal as the execution subject.

[0057] Since in the conventional technology, the size of video data can be effectively reduced by combining various algorithms such as inter-frame prediction, transform coding, quantization and entropy coding, so as to facilitate storage and transmission, but the conventional algorithm usually relies on fixed encoding parameters (such as λ value) to realize rate-distortion tradeoff, and it is difficult to adaptively adjust according to the dynamic change of video content and the code rate requirement, resulting in that in some cases where the complexity of video content changes greatly, the fixed parameters are difficult to adapt, the compression performance is reduced, and the video quality is damaged.

[0058] The present application provides a solution, obtaining an input video frame, calculating the bit rate of each input video frame in each encoding path, combining the current consumed bit number and the preset bit number of the video frame, adjusting the bit rate of the video frame, so that the target code rate of the video frame is more consistent with the overall code rate of the current video compression, avoiding the scene that the video quality is high and low caused by unstable code rate; Then the bit rate of the video frame in each encoding path and the adjusted target code rate can be compared, and a better encoding path is selected to encode the video frame to generate a first feature set, to balance the bit rate and distortion of the video frame as much as possible, to compress efficiently while ensuring video quality; The sub-features in the first feature set are spliced and modulated, and the first feature set after splicing and modulation is compressed to generate a compressed bit stream, to further reduce redundant feature information and improve video compression quality.

[0059] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions. The following will take the terminal as an example to illustrate the present embodiment and the following embodiments.

[0060] Based on this, the present application provides a video processing method, referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the video processing method of the present application is shown in the figure.

[0061] In the present embodiment, the video processing method comprises steps S10-S40:

[0062] Step S10, obtaining an input video, determining the bit rate of each video frame of the input video in each encoding path;

[0063] It should be noted that the video frame of the input video refers to a single image picture arranged in sequence or in a certain order in the video stream, and the frame is the basic data unit of video analysis and processing. The encoding path refers to a set of a series of encoding algorithms, parameter configurations and processing steps, representing a subset of the overall encoding network; the bit rate refers to the rate of data transmission, indicating the number of bits transmitted per second, and in the video encoding process, the bit rate can affect the clarity and file size of the video. The complexity refers to a quantitative indicator for measuring the richness of the content of the video frame or the processing difficulty, which is calculated based on various factors, including the change amount of the intra-pixel, the color change and the change of the motion vector, etc.

[0064] The original video is obtained, frame segmentation is performed thereon to obtain continuous input video frames, and since the complexity of each frame is different, the bit rates of different input video frames at each encoding path are also different. Therefore, the bit rates of each frame at each encoding path can be calculated considering the encoding requirements of each frame for subsequent video encoding and the like.

[0065] In step S20, the target encoding path of the video frame is determined according to the bit rate of the video frame at each encoding path.

[0066] It should be noted that the target encoding path refers to a series of "optimal" encoding strategies and algorithms selected in the video encoding process to achieve a specific encoding effect, so as to ensure the encoding quality and efficiency.

[0067] According to the bit rate of each input video frame, the most suitable encoding path is selected. Exemplarily, for a static background, a low bit rate encoding path (target encoding path) is selected for encoding compression, only the basic features of the background are retained, which can reduce the amount of calculation while ensuring the basic video quality; for a scene where a person or an animal moves quickly, a high bit rate encoding path is selected for encoding compression, and the generated features retain the characteristics of the movement of the person or the animal as much as possible, including the moving direction and the moving distance.

[0068] It can be understood that by distinguishing different complexities to match the appropriate encoding path, a more refined encoding strategy is selected for a high complexity video frame, which helps to retain more details and reduce distortion, thereby improving the video quality.

[0069] In step S30, the video frame is encoded based on the target encoding path to generate a first feature set corresponding to the video frame.

[0070] It should be noted that encoding refers to the process of converting a video frame into a specific format through a specific compression technology; the first feature refers to the core representation of the video data generated through the encoding process, which is used to reflect the characteristics of the video frame. These features can exist in the form of a matrix, a vector or a specific data structure to form the first feature set, which is not limited in the embodiment.

[0071] Step S40, compressing the first feature set corresponding to each video frame to generate a compressed bit stream corresponding to each video frame;

[0072] It should be noted that compression refers to the process of encoding and quantizing the first feature set, reducing the amount of video data while trying to retain useful information contained therein; the compressed bit stream refers to the bit stream generated after reducing the redundancy information of the video data through a specific compression algorithm, facilitating the transmission and storage of the video data.

[0073] By way of example, please refer to Figure 2 , Figure 2 A video encoding schematic diagram is provided, which first acquires an input video frame x t , assigns a target encoding path to the video frame by the code rate control agent module, encodes the input video frame x t according to the configuration of the target encoding path through the multiple scalable convolutional layers and scalable residual units of the adjustable encoder to generate different features; passes through the feature modulation module, Figure 2 where FM represents feature modulation, the circle represents splicing features from different encoding paths and modulated features, and the feature modulation module can output different spliced features by channel; then passes through the autoregressive model to the entropy bottleneck, enters the entropy encoder to compress the feature representation of the video frame, and generates a bit stream convenient for transmission and storage. After decompression, decoding, and other operations on the compressed bit stream corresponding to the video frame, a restored video frame is generated, and the restored video frames are spliced in order to obtain a restored video.

[0074] By way of example, each frame of the video is encoded and compressed to generate a compressed bit stream, and then the bit streams are spliced in order according to the frames to form a continuous compressed video stream for transmission on the network. The ordered splicing ensures the coherence and time sequence of video playback, without affecting the viewing experience.

[0075] In this embodiment, the bit rate of the video frame in the encoding path is calculated, and the target code rate of the single video frame is adaptively adjusted according to the bit rate of the frame and the code rate of the current encoding path, and then a suitable encoding path is selected according to the adjusted target code rate, thereby avoiding uneven distribution of the overall video code rate, resulting in high and low video quality, especially in the face of high complexity video frames, while maintaining the overall code rate of the video stable, the encoding path with higher bit rate is selected for encoding and compression, thereby reducing the video compression file while improving the video quality.

[0076] In a possible implementation, the step of determining the target encoding path of the video frame according to the bit rate of the video frame in each encoding path in step S20 can include:

[0077] Step S21, determining a target code rate of the video frame according to the preset code rate of the video frame and the sum of the consumed bit numbers of the encoding paths;

[0078] It should be noted that the preset code rate refers to the expected bit rate set for a group of pictures (GoP) in the encoding process, i.e., the average bit rate that the user expects the group of pictures to achieve, which is used to guide the encoder to approach the set value as much as possible while maintaining the video quality, so as to achieve the expected file size or transmission efficiency. The target code rate refers to the expected bit rate dynamically adjusted according to the consumed bit number and the preset code rate, which is used to ensure the smooth progress of the overall encoding process while ensuring the encoding quality.

[0079] In addition, it should be noted that the target code rate of the video frame is negatively correlated with the sum of the consumed bit numbers of the encoding paths; and the target code rate of the video frame is positively correlated with the preset code rate of the video frame.

[0080] First, the currently consumed bit number information of each encoding path needs to be obtained, then the content complexity of the current video frame is analyzed to determine the bit rate requirement, i.e., to determine the preset code rate. Based on the currently consumed bit number information of each encoding path and the preset code rate, the target code rate most suitable for the current frame is calculated through a certain algorithm, which guides the encoder to encode at the code rate.

[0081] For example, the sliding window algorithm is used to dynamically allocate the target code rate of the current frame, and the calculation formula is as follows, wherein SW represents the length of the sliding window, which helps to realize a smoother rate control process. First, the total number of bits that can be used in the sliding window is calculated, which is calculated by “preset code rate x (number of encoded frames + SW)”, which is used to ensure that the total bit rate in the entire sliding window is equal to the preset code rate; then, the sum of the consumed bit numbers is subtracted from the total number of bits to obtain the remaining number of bits in the sliding window; finally, the remaining number of bits is divided by the length of the sliding window to obtain the bit rate (target code rate) that should be allocated to the current frame.

[0082]

[0083] Using the sliding window algorithm can ensure that the total bit rate of the video in the sliding window remains at a stable level, thereby avoiding the unstable video quality caused by the dramatic fluctuation of the bit rate in the encoding process.

[0084] Step S22, determining the target encoding path of the video frame according to the bit rates of the video frame in the encoding paths, the consumed bit numbers of the encoding paths, and the target code rate.

[0085] For example, there are three encoding paths A, B, and C. The bit rates required for the current video frame in the three encoding paths are 0.3Mbps, 0.7Mbps, and 1Mbps, respectively, and the target bit rate allocated to the current frame is 0.5Mbps. Therefore, the current frame can be encoded in encoding path B or C. By comparing the number of consumed bits of encoding paths B and C, it is found that encoding path B has more remaining bits. Therefore, encoding path B is selected as the target encoding path for the current frame, and the current frame is encoded.

[0086] This implementation intelligently allocates bitrates based on the complexity of video frames, reducing unnecessary data transmission while ensuring the quality of key or high-complexity frames. This ensures sufficient encoding resources for critical portions of the video, thereby improving video quality. Furthermore, flexible encoding paths are selected based on current conditions, ensuring higher video quality within a limited bit budget. Therefore, the most appropriate encoding path is dynamically selected based on the encoding requirements of the video frame and network conditions, achieving an optimal balance between video quality and encoding efficiency.

[0087] In a feasible implementation, step S22 may include steps S221 to S225:

[0088] Step S221, determining the number of allocated bits for the video frame according to the number of encoded frames of the input video and the preset bit rate of the video frame;

[0089] It should be noted that the number of allocated bits refers to the number of bits that the current encoded video frame should consume under the ideal situation of the preset bit rate. The number of allocated bits is positively correlated with the number of encoded frames of the input video and the preset bit rate. For example, the number of allocated bits can be calculated by multiplying the current number of encoded frames by the preset bit rate of the current frame. This embodiment does not make any specific restrictions on this.

[0090] Step S222: If the number of allocated bits is greater than the sum of the number of consumed bits, and there is at least one encoding path whose bit rate is greater than the target bit rate, the encoding path with the lowest bit rate among the encoding paths having a bit rate greater than the target bit rate is determined as the target encoding path for the video frame; or

[0091] The number of allocated bits for the current frame is higher than the sum of the bits consumed by the encoding path, indicating that the number of bits consumed in the actual video encoding process is less than the ideal situation. The bit rate of the current frame can be appropriately increased to improve the video quality.

[0092] Exemplarily, there are three encoding paths A, B and C, the bit rates of the current frame in the three encoding paths are 0.3 Mbps, 0.7 Mbps and 1 Mbps respectively, and the target code rate allocated to the current frame is 0.5 Mbps. In the case that the allocated bit number of the current frame is higher than the sum of the consumed bit numbers, the bit rates of the encoding paths B and C are higher than the target code rate of the current frame, and the bit rate of the encoding path B is lower than that of the encoding path C, so the encoding path B is selected as the target encoding path of the current frame to encode the current frame.

[0093] In step S223, in the case that the allocated bit number is higher than the sum of the consumed bit numbers and the bit rate generated based on each encoding path is not higher than the target code rate, the encoding path with the highest bit rate is determined as the target encoding path of the video frame; or,

[0094] Exemplarily, in the case that the allocated bit number of the current frame is higher than the sum of the consumed bit numbers, there are three encoding paths A, B and C, the bit rates of the current frame in the three encoding paths are 0.3 Mbps, 0.7 Mbps and 1 Mbps respectively, and the target code rate allocated to the current frame is 1.5 Mbps, and no encoding path can meet the encoding condition of the current frame, so the encoding path C with the highest bit rate among the three encoding paths is selected as the target encoding path of the current frame to maximize the guarantee of the video quality.

[0095] In step S224, in the case that the allocated bit number is not higher than the sum of the consumed bit numbers and there is at least one encoding path with a bit rate not higher than the target code rate, the encoding path with the highest bit rate is determined as the target encoding path of the video frame from the encoding paths with the bit rates not higher than the target code rate; or,

[0096] The allocated bit number of the current frame is lower than the sum of the consumed bit numbers of the encoding paths, which indicates that the consumed bit number in the actual video encoding process is more than the ideal case, and the bit rate of the current frame needs to be appropriately reduced to reduce the size of the video compression file and ensure that the encoding of the subsequent video frames can be normally performed.

[0097] Exemplarily, there are three encoding paths A, B and C, the bit rates of the current frame in the three encoding paths are 0.3 Mbps, 0.7 Mbps and 1 Mbps respectively, and the target code rate allocated to the current frame is 0.8 Mbps. In the case that the allocated bit number of the current frame is not higher than the sum of the consumed bit numbers, the bit rates of the encoding paths A and B are lower than the target code rate of the current frame, and the bit rate of the encoding path B is higher than that of the encoding path A, so the encoding path A (target encoding path) is selected to encode the current frame to minimize the size of the video stream and maximize the guarantee of the video quality.

[0098] Step S225, in the case that the allocated bit number is not higher than the sum of the consumed bit numbers, and the bit rate generated based on each encoding path is higher than the target code rate, the encoding path with the smallest bit rate is determined as the target encoding path of the video frame.

[0099] For example, in the case that the allocated bit number of the current frame is not higher than the sum of the consumed bit numbers, there are three encoding paths A, B and C, the bit rates required by the current frame in the three encoding paths are 0.5 Mbps, 0.7 Mbps and 1 Mbps respectively, and the target code rate of the current frame is 0.3 Mbps. All the encoding paths can meet the encoding condition of the current frame, and the encoding path A with the smallest bit rate in the three encoding paths is selected as the target encoding path of the current frame.

[0100] In this embodiment, according to the encoding requirement of the current frame and the actual situation of the encoding path, an optimal encoding path is found, which can ensure that the bit rate allocation in the video encoding process is neither too much nor too little, thereby optimizing the overall video quality and file size.

[0101] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above-mentioned first embodiment can be referred to the above introduction, and will not be described hereinafter. On this basis, in the case that the target path includes multiple sub-encoding paths, the first feature set includes sub-features of the video frame generated based on each sub-encoding path, before step S40, comprising:

[0102] Step E10, splicing the sub-features;

[0103] It should be noted that the sub-encoding path refers to a component of the target encoding path, which focuses on processing a specific part or characteristic in the video frame or image; and the splicing refers to a process of merging the sub-features generated by multiple sub-encoding paths into a whole according to certain rules.

[0104] In the video encoding process, a video frame can contain multiple complex scenes or objects, and each sub-encoding path can be responsible for encoding one of these scenes or objects. For example, one sub-encoding path can focus on encoding the details of a person's face, and another can focus on the background texture. Therefore, it is necessary to splice the sub-features generated by these sub-encoding paths to reflect the complete image information.

[0105] Step E20, updating the first feature set according to the spliced sub-features.

[0106] Exemplarily, in the case that the target encoding path comprises a plurality of sub-encoding paths, the video frame image is processed by each sub-encoding path to obtain a plurality of sub-features of the frame, and a splicing operation is performed to combine the sub-features into a complete feature, so as to completely display the image details, and the spliced feature is updated to the original feature set for subsequent compression process.

[0107] In the embodiment, by splicing the image features generated by each sub-encoding path, the key information can be more accurately captured and retained, the image information of each video frame is enriched, and the quality of the encoded video is improved.

[0108] In an available implementation, the step E10 further comprises:

[0109] The step E11 determines a first path from the sub-encoding paths.

[0110] The step E12 determines a sub-feature corresponding to the first path as a first feature, and modulates the first feature to obtain a second feature.

[0111] It should be noted that modulation refers to transformation or processing of the feature data extracted from the video data to adapt to different encoding requirements.

[0112] The step E13 determines a second path from the sub-encoding paths without modulated features.

[0113] It should be noted that the sub-encoding path without modulated features refers to that the sub-feature generated in the sub-encoding path is not modulated, nor is it spliced with other features and then modulated.

[0114] The step E14 determines a sub-feature corresponding to the second path as a third feature, and splices the second feature and the third feature to obtain a fourth feature.

[0115] The step E15 re-determines the fourth feature as the first feature, and performs the step of modulating the first feature to obtain the second feature, and subsequent steps until the sub-features corresponding to each sub-encoding path are spliced.

[0116] Exemplarily, please refer to Figure 2 , Figure 2 In the embodiment, FM represents feature modulation, which is used to extract and emphasize the key information in the feature, Figure 2The circle in the figure represents splicing different features to merge into a new feature set. The static background feature generated by the first decoding path (the first feature generated by the first path) is modulated, and the key features of the video frame, such as the background color and the contained objects, are extracted to obtain the second feature. The second feature is spliced with the object detail feature (the third feature) of the video frame generated by the second path. The object detail feature can include the expression and clothing of a person or animal object. The fourth feature obtained by splicing contains complete background features and video object details. The fourth feature is re-established as the first feature, and the above steps are repeated, so that all the first features generated by different decoding paths are modulated and spliced step by step until the first features of all sub-encoding paths are spliced.

[0117] In this embodiment, by modulating the features, the key information is extracted and the redundant features are reduced, which effectively helps to reduce the size of the video stream file. At the same time, the "optimization processing" of the video features can enhance their effectiveness in subsequent applications. By modulating the features of different encoding paths, the feature information of different video frames is spliced and fused to obtain the overall image of the video frame, thereby effectively improving the video quality.

[0118] In a feasible implementation, before the step of determining the bit rate of each video frame of the input video in each encoding path in step S10, the method further comprises the step of:

[0119] In step S01, the trade-off factor of each encoding path is adjusted, and the parameters of each encoding path are iteratively trained according to a preset training number until the rate-distortion ratio of the verification video set in each encoding path reaches a maximum value, and the bit rate of the verification video set in each encoding path reaches a minimum value.

[0120] It should be noted that the trade-off factor is a preset value or parameter used to balance the relationship between the encoding data rate (such as bit rate) and the encoding quality (such as distortion) during encoding. The preset training number refers to the number of training iterations, which indicates how many times the parameters of the encoding path will be iteratively trained using the training video set. The training video set and the verification video set can be the same, and this embodiment does not make specific limitations.

[0121] In addition, it should be noted that the rate-distortion ratio refers to the ratio of the bit rate to the distortion after encoding. At the same time, the higher the rate-distortion ratio, the smaller the loss of video quality, and the higher the encoding efficiency.

[0122] Exemplarily, the overall neural network is composed of multiple convolutional layers, pooling layers, and fully connected layers, forming multiple encoding paths from input video frames to output features. Before training each encoding path, the training video set is used to train the overall neural network. Through the process of continuous iteration of forward propagation, loss calculation, and backpropagation, the model parameters are initialized and tested with the validation video set to determine the rate-distortion ratio and bit rate of the validation video set on the overall network. A large-scale pre-training is first performed on the entire video processing to ensure that each module or each encoding path can learn the common features of the video data, which will help to find the most suitable parameters more quickly when adjusting each encoding path later.

[0123] Exemplarily, the parameters of each coding path are appropriately adjusted in descending order of the Lagrange multiplier λ value (weighting factor) of each coding path; during the entire adjustment process, the coding path is trained N times (preset number of times) using the training video set, and the coding path parameters are updated after each training is completed; after N trainings are completed, the bit rate and rate-distortion ratio of the verification video set at this time are calculated and compared with the previous bit rate and rate-distortion ratio; if the bit rate at this time is higher than the previous bit rate and the rate-distortion ratio at this time is lower than the previous rate-distortion ratio, the bit rate and rate-distortion ratio of the verification video set are updated to the bit rate and rate-distortion ratio at this time; then the λ value is adjusted, and the steps of iteratively training the coding path are repeated until the rate-distortion ratio no longer increases and the bit rate no longer decreases, and the λ value and parameter settings of the coding path are determined; then training and adjustment are performed on other coding paths.

[0124] It is understandable that training on the training set and testing on the validation set ensures that the model performs well on unseen data. The ultimate goal is to minimize distortion while meeting the bitrate constraint, thereby improving the efficiency of video encoding.

[0125] In this embodiment, by introducing preset trade-off factors and adjusting the encoding path according to the values ​​of these factors, and by optimizing the encoding path parameters, the distortion and bit rate of each encoding path are balanced, thereby improving the efficiency and flexibility of the encoding process and ensuring the high quality of the final video encoding.

[0126] In a feasible implementation, step S01 may further include:

[0127] Step S03: determining the rate-distortion ratio and bit rate of the verification video set in the encoding path according to a preset trade-off factor;

[0128] Step S04: Adjust the trade-off factor until the rate-distortion ratio stops increasing and the bit rate stops decreasing.

[0129] Exemplarily, according to the bandwidth limit of the platform and the user demand, the user or the system sets a proper trade-off factor. According to the preset trade-off factor, the rate-distortion ratio and the bit rate of each possible encoding path are calculated, and the parameters of each encoding path are updated. Finally, in the case that the rate-distortion ratio no longer improves and the bit rate no longer decreases, the final trade-off factor and the parameter setting of the encoding path are determined to ensure that the user can watch high-quality video content.

[0130] In this embodiment, by adjusting and optimizing each encoding path before the task starts, the video quality is optimized while the encoding efficiency is ensured.

[0131] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned first embodiment and second embodiment can be referred to the above description, and will not be described hereinafter. On this basis, after step S40, the video processing method can further include steps A10-A40:

[0132] Step A10, decompressing the compressed bit stream to generate a second feature set;

[0133] It should be noted that the second feature set refers to a group of feature data extracted from the decompressed compressed bit stream, which can represent the key information (i.e., the reconstruction information of the video frame) of the video frame, for subsequent processing or analysis.

[0134] Step A20, determining the bit rate of the second feature set in each decoding path;

[0135] It should be noted that the decoding path refers to a series of processing steps or algorithm processes for recovering the original video data from the compressed bit stream, and different decoding paths can use different decoding strategies or optimization methods.

[0136] Step A30, determining a target decoding path according to the bit rate of the second feature set;

[0137] It should be noted that the target code rate of the second feature set can be determined according to the preset code rate of the second feature set and the sum of the consumed bit numbers of each decoding path, and the target decoding path can be determined according to the bit rate of the second feature set in each decoding path, the target code rate of the second feature set, and the sum of the consumed bit numbers of each decoding path.

[0138] Exemplarily, there are three decoding paths A, B and C, and the bit rates of the current second feature set in the three decoding paths are 0.3 Mbps, 0.7 Mbps and 1 Mbps respectively, and the target bit rate allocated to the current second feature set is 0.5 Mbps. In the case that the allocated bit number of the current second feature set is higher than the consumed bit number, the bit rates of the decoding paths B and C are greater than the target bit rate of the current frame, wherein the bit rate of the decoding path B is less than that of the decoding path C, and therefore the decoding path B (target decoding path) is selected to decode the current second feature set.

[0139] In step A40, the second feature set is decoded based on the target decoding path to generate a restored video frame.

[0140] It should be noted that decoding refers to the process of converting compressed or encoded data back to the original format that can be used, and the visual quality of the original video is preserved as much as possible by selecting a suitable decoding path; and the restored video frame refers to video data generated by the decompression and decoding process, which is close to the original video frame.

[0141] Exemplarily, please refer to Figure 3 , Figure 3 A decoding process example is provided, which recovers the compressed bit stream corresponding to each video frame into different feature representations (second feature set) by using an entropy decoder, selects a suitable decoding path according to the bit rate allocation, processes the second feature set in the selected target decoding path by using the autoregressive model and the multiple scalable convolution layers and scalable residual layers of the adjustable decoder, decodes step by step, reconstructs each video frame through the U-Net layer, and finally splices the video frames in sequence to form a continuous restored video.

[0142] In this embodiment, the decoding path is adaptively selected according to different bit rates, the consumption of computing resources is reduced, and the overall decoding efficiency is improved; at the same time, the quality and details of the video frame are improved through the effective decoding and reconstruction process, so that a high-quality restored video is generated.

[0143] It should be noted that the above examples are only used to understand the present application and do not limit the video processing method of the present application. More simple transformations based on this technical concept are within the protection scope of the present application.

[0144] The present application also provides a video processing device, please refer to Figure 4 , the video processing device comprises:

[0145] The acquisition module 10 is configured to acquire an input video and determine the bit rates of each video frame of the input video in each encoding path.

[0146] The determination module 20 is configured to determine the target encoding path of the video frame according to the bit rates of the video frame in each encoding path.

[0147] An encoding module 30, configured to encode the video frame based on a target encoding path and generate a first feature set corresponding to the video frame;

[0148] The compression module 40 is configured to compress the first feature set corresponding to each video frame to generate a compressed bit stream corresponding to each video frame.

[0149] The video processing device provided in the embodiments of the present application utilizes the video processing method in the above-described embodiments to improve video quality while maintaining compression quality. Compared to the prior art, the beneficial effects of the video processing device provided in the present application are the same as those of the video processing method provided in the above-described embodiments. Other technical features of the video processing device are the same as those disclosed in the above-described embodiments and are not further described here.

[0150] An embodiment of the present application provides a video processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the video processing method in the above-mentioned embodiment 1.

[0151] Reference below Figure 5 , which shows a schematic diagram of the structure of a video processing device suitable for implementing the embodiments of the present application. The video processing device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The video processing device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0152] like Figure 5As shown, the video processing device can include a processing apparatus 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 1002 or a program loaded into a random access memory (RAM) 1004 from a storage apparatus 1003. Various programs and data required for the operation of the video processing device are also stored in the RAM 1004. The processing apparatus 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input apparatus 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output apparatus 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage apparatus 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication apparatus 1009. The communication apparatus 1009 can allow the video processing device to communicate with other devices wirelessly or by wire to exchange data. Although the video processing device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0153] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication apparatus, or installed from the storage apparatus 1003, or installed from the ROM 1002. When the computer program is executed by the processing apparatus 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0154] The video processing device provided by the embodiments of the present application adopts the video processing method in the above-mentioned embodiments, and can improve the video quality while ensuring the compression quality. Compared with the prior art, the video processing device provided by the embodiments of the present application has the same beneficial effects as the video processing method provided by the above-mentioned embodiments, and other technical features in the video processing device are the same as the features disclosed in the previous embodiment method, which will not be described here.

[0155] It should be understood that parts of the application disclosed can be implemented in hardware, software, firmware or combinations thereof. In the description of the above-described embodiments, specific features, structures, materials or characteristics can be combined in any appropriate manner in any one or more embodiments or examples.

[0156] The above description is merely illustrative of the application, and the scope of the application is not limited thereto. Any variations and modifications of the application, which fall within the scope of the application, are to be considered as within the scope of the application. Therefore, the scope of the application is to be determined by the scope of the claims.

[0157] The embodiment of the application provides a computer readable storage medium having computer readable program instructions (i.e., computer programs) stored thereon, the computer readable program instructions being used to execute the video processing method in the above-described embodiment.

[0158] The computer readable storage medium provided by the embodiment of the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection having one or more conductive lines, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination of the above. In the embodiment, the computer readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system or device. The program code contained on the computer readable storage medium can be transmitted by any appropriate medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), and the like, or any appropriate combination of the above.

[0159] The above-described computer readable storage medium can be contained in a video processing device, or can exist separately and not be assembled into the video processing device.

[0160] The computer readable storage medium described above carries one or more programs, when the one or more programs are executed by the video processing device, cause the video processing device to: acquire an input video, determine bit rates of each video frame of the input video in each encoding path; determine a target encoding path of the video frame according to the bit rates of the video frame in each encoding path; encode the video frame based on the target encoding path to generate a first feature set corresponding to the video frame; compress the first feature set corresponding to each video frame to generate a compressed bit stream corresponding to each video frame.

[0161] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0162] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0163] The modules involved in the embodiments of the present application can be implemented in the manner of software or in the manner of hardware. Among them, the name of the module does not constitute a limitation to the unit itself in some cases.

[0164] The readable storage medium provided by the embodiments of the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer program) for executing the video processing method described above, and can solve the technical problem of how to adaptively adjust the code rate and improve the video quality. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the video processing method provided by the above embodiments, which will not be repeated here.

[0165] The embodiments of the present application also provide a computer program product, which comprises a computer program. When the computer program is executed by a processor, the steps of the video processing method described above are implemented.

[0166] The computer program product provided by the embodiments of the present application can improve the video quality while ensuring the compression quality. Compared with the prior art, the computer program product provided by the embodiments of the present application has the same beneficial effects as the video processing method provided by the above embodiments, which will not be repeated here.

[0167] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation, direct / indirect application in other related technical fields based on the technical concept of the present application, and the contents of the specification and drawings are included in the patent protection scope of the present application.

Claims

1. A video processing method, characterized in that: The video processing method comprises: Obtaining an input video and determining a bit rate for each video frame of the input video in each encoding path, wherein an overall neural network composed of multiple convolutional layers, pooling layers, and fully connected layers forms the encoding path; Determining a target encoding path for the video frame according to the bit rate of the video frame in each encoding path; Encoding the video frame based on the target encoding path to generate a first feature set corresponding to the video frame; The first feature set corresponding to each of the video frames is compressed to generate a compressed bit stream corresponding to each of the video frames.

2. The video processing method according to claim 1, wherein: The step of determining a target encoding path for the video frame according to the bit rate of the video frame in each encoding path includes: Determining a target bit rate for the video frame according to a preset bit rate for the video frame and a sum of the number of bits consumed by each encoding path; A target encoding path for the video frame is determined according to the bit rate of the video frame in each encoding path, the number of bits consumed by each encoding path, and the target bit rate.

3. The video processing method according to claim 2, wherein: The step of determining the target encoding path of the video frame according to the bit rate of the video frame in each encoding path, the number of bits consumed by each encoding path, and the target bit rate includes: Determining the number of allocated bits for the video frame according to the number of encoded frames of the input video and the preset bit rate of the video frame; In a case where the number of allocated bits is higher than the sum of the number of consumed bits, and there is at least one coding path with a bit rate higher than the target bit rate, determining the coding path with the lowest bit rate from the coding paths with bit rates higher than the target bit rate as the target coding path for the video frame; or In a case where the number of allocated bits is higher than the sum of the number of consumed bits and the bit rates generated based on the encoding paths are not higher than the target bit rate, determining the encoding path with the highest bit rate as the target encoding path for the video frame; or In a case where the number of allocated bits is not higher than the sum of the number of consumed bits and there is at least one coding path whose bit rate is not higher than the target bit rate, determining the coding path with the highest bit rate from the coding paths whose bit rates are not higher than the target bit rate as the target coding path for the video frame; or When the number of allocated bits is not higher than the sum of the consumed bits and the bit rates generated based on each encoding path are higher than the target bit rate, the encoding path with the smallest bit rate is determined as the target encoding path for the video frame.

4. The video processing method according to claim 1, wherein: The target coding path includes a plurality of sub-coding paths, the first feature set includes sub-features of the video frame generated based on each of the sub-coding paths, and before the step of compressing the first feature set corresponding to each of the video frames, the method further includes: splicing the sub-features; The first feature set is updated according to the spliced ​​sub-features.

5. The video processing method according to claim 4, wherein: The step of splicing the sub-features comprises: Determining a first path from each of the sub-coding paths; determining a sub-feature corresponding to the first path as a first feature, and modulating the first feature to obtain a second feature; determining a second path from the sub-encoded paths of the unmodulated features; Determine the sub-feature corresponding to the second path as the third feature, and concatenate the second feature and the third feature to obtain a fourth feature; The fourth feature is re-determined as the first feature, and the step of modulating the first feature to obtain the second feature, and subsequent steps are performed until the sub-features corresponding to the sub-coding paths are all spliced ​​together.

6. The video processing method according to claim 1, wherein: Before the step of determining the bit rate of each video frame of the input video in each encoding path, the method further includes: Adjust the trade-off factors of each encoding path, and iteratively train the parameters of each encoding path according to a preset number of training times until the rate-distortion ratio of the verification video set in each encoding path reaches a maximum value, and the bit rate of the verification video set in each encoding path reaches a minimum value.

7. The video processing method according to claim 1, wherein: After the step of generating a compressed bit stream, the method further comprises: decompressing the compressed bit stream to generate a second feature set; determining a bit rate of the second feature set in each decoding path; determining a target decoding path according to the bit rate of each decoding path of the second feature set; The second feature set is decoded based on the target decoding path to generate a restored video frame.

8. A video processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video processing method according to any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video processing method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the video processing method according to any one of claims 1 to 7 are implemented.

Images