Video coding method, storage medium and computer program product

Abstract

The invention discloses a video coding method, a storage medium and a computer program product, and relates to the technical field of video processing, and the method comprises the steps: obtaining a to-be-adjusted video, carrying out the feature extraction of the to-be-adjusted video, and obtaining a target video feature, the target video feature being used for describing the feature of the image content of the to-be-adjusted video; inputting the target video feature and the to-be-adjusted video into a preset parameter generation network to obtain a target coding parameter, the parameter generation network being obtained by training a generative adversarial network by using a historical video and a video feature of the historical video as input data and using a coding parameter corresponding to the historical video as a label; and coding the to-be-adjusted video according to the target coding parameter to obtain a coded video, sending the coded video to a display, and receiving and displaying the coded video by the display. The video coding parameter can be optimized, and the video coding effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video encoding method, a storage medium, and a computer program product. Background Art

[0002] Video coding technology is a key technology in the field of digital video. With the rise of digital technology, video coding has gradually turned to digitalization. Its main goal is to reduce the redundancy of video data to achieve efficient storage and transmission. Video coding parameters refer to a series of settings used in the process of video compression and encoding. These settings determine the compression efficiency, quality, file size, transmission bandwidth requirements, etc. of the video data.

[0003] At present, the determination of video encoding parameters mainly depends on the following methods:

[0004] Fixed parameter settings rely on the default parameter configurations provided by the encoder. These parameters are optimized based on a wide range of application scenarios. However, fixed parameters cannot be customized for specific video content, which affects the video encoding effect of fixed parameters in specific scenarios.

[0005] Manual parameter adjustment means that experienced users manually adjust encoding parameters according to video content and requirements to achieve better compression effect or visual quality. However, manual adjustment relies on user experience and intuition, and it is difficult to cover all possible video content and encoding scenarios, which affects the accuracy of encoding parameters in specific scenarios and thus affects the video encoding effect.

[0006] Rule-based optimization, that is, adjusting encoding parameters based on empirical rules and regulations, however, rules and regulations are usually formulated based on experience and it is difficult to cover all possible video content and encoding scenarios, which affects the accuracy of encoding parameters in specific scenarios and thus affects the video encoding effect.

[0007] Machine learning models predict optimal encoding parameters through training models. However, the effectiveness of machine models is limited by the quality and diversity of training data. The training data may not cover all video types, so the model may not be able to accurately predict all unknown or rare video content, which affects the accuracy of encoding parameters in specific scenarios and thus affects the video encoding effect.

[0008] In summary, the existing method of determining video encoding parameters lacks accurate adaptability to specific video content and encoding requirements, which affects the video encoding effect.

[0009] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0010] The main purpose of this application is to provide a video encoding method, storage medium and computer program product, aiming to solve the technical problem that the existing method of determining video encoding parameters lacks precise adaptability to specific video content and encoding requirements, which affects the video encoding effect.

[0011] To achieve the above object, the present application proposes a video encoding method, the method comprising:

[0012] Acquire a video to be adjusted, and perform feature extraction on the video to be adjusted to obtain target video features, wherein the target video features are used to describe characteristics of the picture content of the video to be adjusted;

[0013] Input the target video features and the video to be adjusted into a preset parameter generation network to obtain target encoding parameters, wherein the parameter generation network is obtained by training a generative adversarial network using a historical video and the video features of the historical video as input data and the encoding parameters corresponding to the historical video as labels;

[0014] The video to be adjusted is encoded according to the target encoding parameters to obtain an encoded video, and the encoded video is sent to a display, wherein the display receives and displays the encoded video.

[0015] In one embodiment, the step of extracting features from the video to be adjusted to obtain target video features includes:

[0016] Extracting a foreground object in the video to be adjusted, and performing optical flow calculation on the foreground object based on the video to be adjusted to obtain motion information of the foreground object;

[0017] Determine the motion amplitude of each of the foreground objects based on the motion information, determine a target object whose motion amplitude is greater than a preset amplitude threshold from among the foreground objects, and use the area where the target object is located as an area of ​​interest;

[0018] Performing image block processing on each video frame of the video to be adjusted to obtain a plurality of image blocks, and extracting texture features of each video frame based on each of the image blocks;

[0019] Determine inter-frame difference information and scene change information of each video frame, and use the inter-frame difference information and the scene change information as temporal dynamic information;

[0020] The motion information, the region of interest, the texture features and the temporal dynamic information are used as the target video features.

[0021] In one embodiment, the step of extracting the foreground object in the video to be adjusted and performing optical flow calculation on the foreground object based on the video to be adjusted to obtain motion information of the foreground object includes:

[0022] Determine an initial video frame set from each video frame of the video to be adjusted, and perform background modeling based on the initial video frame set to obtain a background model, wherein the initial video frame set includes a preset number of video frames that are chronologically preceding in each video frame;

[0023] For any target frame in each of the video frames, a pixel difference between each pixel of the target frame and the background model is calculated, and pixels whose pixel difference is greater than a preset difference threshold among the pixels are determined as foreground pixels, and a foreground object is determined based on each of the foreground pixels;

[0024] Determine a pixel displacement of each of the foreground pixels between adjacent video frames, and determine an optical flow vector of each of the foreground pixels based on the pixel displacement;

[0025] The motion amplitude and motion direction of the foreground object are determined based on the optical flow vectors of each foreground pixel, and the motion amplitude and the motion method are used as motion information.

[0026] In one embodiment, the step of performing image block processing on each video frame of the video to be adjusted to obtain a plurality of image blocks, and extracting texture features of each video frame based on each image block includes:

[0027] For any target frame among the video frames of the video to be adjusted, convert the target frame into a grayscale image;

[0028] Dividing the grayscale image into a plurality of non-overlapping image blocks according to a preset block size;

[0029] The grayscale co-occurrence matrix of each of the image blocks is calculated, and the texture features of the target frame are extracted based on each of the grayscale co-occurrence matrices.

[0030] In one embodiment, the step of determining the inter-frame difference information and the scene change information of each video frame includes:

[0031] For any two adjacent frames in each video frame, calculate the pixel difference between each pixel in the adjacent frames, and use the pixel difference as inter-frame difference information of the adjacent frames;

[0032] Determine the key points in each video frame by a feature detector, and match the key points in adjacent frames to obtain key point pairs, wherein the key points are unique and repeatedly detectable position points in the video frame;

[0033] Position transformation information of the key point pair is determined, and scene change information of the adjacent frames is determined based on the position transformation information of the key point pair.

[0034] In one embodiment, the parameter generation network includes an input layer, a generator and a discriminator;

[0035] The step of inputting the target video features and the video to be adjusted into a preset parameter generation network to obtain target encoding parameters includes:

[0036] Inputting the video to be adjusted and the target video parameters into the input layer to obtain a random noise vector and conditional input data;

[0037] Inputting the random noise vector and the conditional input data into the generator to obtain generated coding parameters;

[0038] Performing video encoding on the video to be adjusted according to the generated encoding parameters to obtain a generated encoded video;

[0039] Inputting the generated coded video and the video to be adjusted into the discriminator to obtain a discrimination result;

[0040] When the discrimination result is true, the generated encoding parameters are outputted by the generator, and the generated encoding parameters are used as target encoding parameters.

[0041] In one embodiment, the generator includes a first conditional embedding layer, a fusion layer, a temporal convolution layer, a three-dimensional convolution layer, a first attention mechanism, and a first output layer;

[0042] The step of inputting the random noise vector and the conditional input data into the generator to generate encoding parameters comprises:

[0043] Inputting the conditional input data into the first conditional embedding layer to obtain a first embedding vector;

[0044] Inputting the first embedding vector and the random noise vector into the fusion layer to obtain a fusion vector;

[0045] Inputting the fusion vector into the time series convolution layer to obtain time series features;

[0046] Inputting the temporal feature into the three-dimensional convolutional layer to obtain a first spatiotemporal feature;

[0047] Inputting the spatiotemporal feature into the first attention mechanism to obtain a first weighted feature;

[0048] The first weighted feature is input into the output layer to generate encoding parameters.

[0049] In one embodiment, the discriminator includes a second conditional embedding layer, a sequence feature extraction layer, a convolutional layer, a second attention mechanism, and a second output layer;

[0050] The step of inputting the generated coded video and the video to be adjusted into the discriminator to obtain a discrimination result comprises:

[0051] Inputting the conditional input data into the second conditional embedding layer to obtain a second embedding vector;

[0052] Fusing the second embedding vector and the video to be adjusted to obtain a fusion feature;

[0053] Inputting the fusion feature into the sequence feature extraction layer to obtain video features;

[0054] Inputting the video feature into the convolutional layer to obtain a second spatiotemporal feature;

[0055] Inputting the second spatiotemporal feature into the second attention mechanism to obtain a second weighted feature;

[0056] The second weighted feature is input into the second output layer to obtain a discrimination result.

[0057] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video encoding method described above are implemented.

[0058] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the video encoding method described above are implemented.

[0059] In the present application, a video to be adjusted is obtained, and features of the video to be adjusted are extracted to obtain target video features, wherein the target video features are used to describe the characteristics of the picture content of the video to be adjusted; the target video features and the video to be adjusted are input into a preset parameter generation network to obtain target encoding parameters, wherein the parameter generation network is obtained by training a generative adversarial network using historical videos and video features of the historical videos as input data and encoding parameters corresponding to the historical videos as labels; the video to be adjusted is encoded according to the target encoding parameters to obtain an encoded video, and the encoded video is sent to a display, wherein the display receives and displays the encoded video.

[0060] Compared with setting fixed video encoding parameters, this application extracts the features of the video to be adjusted and uses a parameter generation network to generate customized encoding parameters, thereby dynamically adjusting parameters according to the content characteristics of the video. This application can more accurately match the specific needs of the video, thereby improving the encoding effect.

[0061] Compared with manually adjusting video encoding parameters, this application reduces the reliance on user experience and intuition through an automated feature extraction and parameter generation process, making the determination of encoding parameters more objective and systematic, and being able to more accurately match the specific needs of the video, thereby improving the encoding effect.

[0062] Compared with determining video encoding parameters based on rules, the parameter generation network trained by the generative adversarial network in this application can learn more complex patterns from a large amount of historical video data, so that the video encoding parameters can handle a wider range of content types and encoding scenarios, and can more accurately match the specific needs of the video, thereby improving the encoding effect.

[0063] Compared with the existing machine learning model to determine the video encoding parameters, this application generates video encoding parameters based on the generative adversarial network. The generator in the generative adversarial network is responsible for generating the encoding parameters, and the discriminator evaluates the quality of the generated parameters. Through this adversarial process, the video encoding parameters are continuously optimized, so that the video encoding parameters can more accurately match the specific needs of the video, thereby improving the encoding effect. In addition, since the generative adversarial network can analyze video features in real time and dynamically generate parameters, it reduces the dependence on a large amount of historical training data, so that the parameter generation network can better generalize to unseen video data, thereby improving the adaptability to unknown or rare video content, so that the video encoding parameters can more accurately match the specific needs of the video, thereby improving the encoding effect.

[0064] In summary, the present application dynamically generates video encoding parameters by generating a parameter generation network based on the content characteristics of the video and based on the generative adversarial network structure, so that the video encoding parameters can more accurately match the specific requirements of the video, thereby improving the encoding effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1 A schematic diagram of a video encoding process involved in the video encoding method of the present application;

[0068] Figure 2 A schematic diagram of a flow chart provided for the first embodiment of the video encoding method of the present application;

[0069] Figure 3 A schematic diagram of a flow chart provided for Embodiment 2 of the video encoding method of the present application;

[0070] Figure 4 A schematic diagram of a brief flow of a video encoding method provided in one embodiment of the present application;

[0071] Figure 5 A schematic diagram of a brief process of a video encoding block provided in an embodiment of the present application;

[0072] Figure 6 A schematic diagram of the structure of a parameter generation network provided in an embodiment of the present application;

[0073] Figure 7 A schematic diagram of a training process of a parameter generation network provided in an embodiment of the present application;

[0074] Figure 8 Schematic diagram of the device structure of the hardware operating environment involved in the video encoding method in the embodiment of the present application.

[0075] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0076] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0077] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0078] The main solution of the embodiment of the present application is: obtaining a video to be adjusted, and performing feature extraction on the video to be adjusted to obtain target video features, wherein the target video features are used to describe the characteristics of the picture content of the video to be adjusted; inputting the target video features and the video to be adjusted into a preset parameter generation network to obtain target encoding parameters, wherein the parameter generation network is obtained by training a generative adversarial network using historical videos and video features of the historical videos as input data and encoding parameters corresponding to the historical videos as labels; encoding the video to be adjusted according to the target encoding parameters to obtain an encoded video, and sending the encoded video to a display, wherein the display receives and displays the encoded video.

[0079] In this embodiment, for ease of description, the following description is made with a video encoding device as the execution subject.

[0080] Video coding technology is a key technology in the field of digital video. With the rise of digital technology, video coding has gradually turned to digitalization. Its main goal is to reduce the redundancy of video data to achieve efficient storage and transmission. Please refer to Figure 1 ,The video encoding process is as follows: video input, then video encoding parameters are set based on the input video file, and then the video file is encoded based on the video encoding parameters.

[0081] Video encoding parameters refer to a series of settings used in the process of video compression and encoding. These settings determine the compression efficiency, quality, file size, transmission bandwidth requirements, etc. of the video data.

[0082] At present, the determination of video encoding parameters mainly depends on the following methods:

[0083] Fixed parameter settings rely on the default parameter configurations provided by the encoder. These parameters are optimized based on a wide range of application scenarios. However, fixed parameters cannot be customized for specific video content, which affects the video encoding effect of fixed parameters in specific scenarios.

[0084] Manual parameter adjustment means that experienced users manually adjust encoding parameters according to video content and requirements to achieve better compression effect or visual quality. However, manual adjustment relies on user experience and intuition, and it is difficult to cover all possible video content and encoding scenarios, which affects the accuracy of encoding parameters in specific scenarios and thus affects the video encoding effect.

[0085] Rule-based optimization, that is, adjusting encoding parameters based on empirical rules and regulations, however, rules and regulations are usually formulated based on experience and it is difficult to cover all possible video content and encoding scenarios, which affects the accuracy of encoding parameters in specific scenarios and thus affects the video encoding effect.

[0086] Machine learning models predict optimal encoding parameters through training models. However, the effectiveness of machine models is limited by the quality and diversity of training data. The training data may not cover all video types, so the model may not be able to accurately predict all unknown or rare video content, which affects the accuracy of encoding parameters in specific scenarios and thus affects the video encoding effect.

[0087] In summary, the existing method of determining video encoding parameters lacks accurate adaptability to specific video content and encoding requirements, which affects the video encoding effect.

[0088] In addition, the prior art has the following defects in video coding parameter optimization:

[0089] Lack of adaptability: Most encoding schemes have fixed parameters and are difficult to dynamically adjust according to video content.

[0090] Manual parameter setting is cumbersome: the adjustment process is complicated and difficult for ordinary users to master.

[0091] Unstable results: Traditional optimization methods are targeted at specific conditions, resulting in uneven encoding quality.

[0092] High data requirements: Machine learning methods rely on large amounts of labeled data, which limits their availability.

[0093] Low coding efficiency: Fixed parameter schemes perform poorly in high-dynamic scenes.

[0094] The present application provides a solution. Compared with setting fixed video encoding parameters, the present application generates customized encoding parameters by extracting the features of the video to be adjusted and using a parameter generation network, thereby dynamically adjusting the parameters according to the content characteristics of the video. The present application can more accurately match the specific needs of the video, thereby improving the encoding effect.

[0095] Compared with manually adjusting video encoding parameters, this application reduces the reliance on user experience and intuition through an automated feature extraction and parameter generation process, making the determination of encoding parameters more objective and systematic, and being able to more accurately match the specific needs of the video, thereby improving the encoding effect.

[0096] Compared with determining video encoding parameters based on rules, the parameter generation network trained by the generative adversarial network in this application can learn more complex patterns from a large amount of historical video data, so that the video encoding parameters can handle a wider range of content types and encoding scenarios, and can more accurately match the specific needs of the video, thereby improving the encoding effect.

[0097] Compared with the existing machine learning model to determine the video encoding parameters, this application generates video encoding parameters based on the generative adversarial network. The generator in the generative adversarial network is responsible for generating the encoding parameters, and the discriminator evaluates the quality of the generated parameters. Through this adversarial process, the video encoding parameters are continuously optimized, so that the video encoding parameters can more accurately match the specific needs of the video, thereby improving the encoding effect. In addition, since the generative adversarial network can analyze video features in real time and dynamically generate parameters, it reduces the dependence on a large amount of historical training data, so that the parameter generation network can better generalize to unseen video data, thereby improving the adaptability to unknown or rare video content, so that the video encoding parameters can more accurately match the specific needs of the video, thereby improving the encoding effect.

[0098] In summary, compared with fixed parameters, this application can capture the dynamic characteristics of video content by extracting target video features in real time, and input these features into the parameter generation network to generate encoding parameters adapted to the current video content, thereby enhancing the adaptability of video encoding parameters.

[0099] Compared with manually setting parameters, the present application reduces the complexity of manually adjusting parameters by utilizing a preset parameter generation network to automatically obtain encoding parameters, so that even ordinary users can easily perform video encoding and simplify the parameter setting process.

[0100] This application uses a parameter generation network to perform training based on historical video data and corresponding encoding parameters. The learned pattern can adapt to different video content and conditions, thereby improving the consistency and stability of the encoding effect.

[0101] Compared with determining parameters by machine learning models, this application continuously optimizes video encoding parameters through the adversarial process of generative adversarial networks, reducing dependence on large amounts of historical training data, allowing the parameter generation network to better generalize to unseen video data and reduce dependence on training data.

[0102] By dynamically generating encoding parameters, the present application can adapt to highly dynamic scenes, optimize the encoding process, and improve encoding efficiency.

[0103] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a video encoding device, etc. The following takes a video encoding device as an example to illustrate this embodiment and the following embodiments.

[0104] Based on this, the embodiment of the present application provides a video encoding method, referring to Figure 2 , Figure 2 This is a flowchart of the first embodiment of the video encoding method of the present application.

[0105] In this embodiment, the video encoding method includes steps S10 to S30:

[0106] Step S10, obtaining a video to be adjusted, and extracting features of the video to be adjusted to obtain target video features, wherein the target video features are used to describe characteristics of the picture content of the video to be adjusted;

[0107] It should be noted that the video to be adjusted refers to the video source that needs to be encoded. The video to be adjusted is obtained, and then a preset algorithm is used to analyze the video content and extract features that can describe the characteristics of the video. These features may include color distribution, texture, motion information, scene complexity, etc. Through feature extraction, the video content can be understood more accurately, providing a basis for subsequent encoding parameter adjustment.

[0108] Step S20, inputting the target video features and the video to be adjusted into a preset parameter generation network to obtain target encoding parameters, wherein the parameter generation network is obtained by training a generative adversarial network using historical videos and the video features of the historical videos as input data and the encoding parameters corresponding to the historical videos as labels;

[0109] It should be noted that the parameter generation network is trained by using the historical video and the video features of the historical video as input data and the encoding parameters corresponding to the historical video as labels to generate adversarial network. When new video features are extracted, these features and the video are input into the network together to predict the optimal encoding parameters.

[0110] By using a network trained with historical data to generate encoding parameters, human intervention can be reduced, encoding efficiency can be improved, and encoding quality can be potentially improved because the network is able to learn which parameters are optimal for specific types of content.

[0111] In a feasible implementation manner, the parameter generation network includes an input layer, a generator, and a discriminator; the process of inputting the target video features and the video to be adjusted into a preset parameter generation network to obtain the target encoding parameters may include:

[0112] Step S201, inputting the video to be adjusted and the target video parameters into the input layer to obtain a random noise vector and conditional input data;

[0113] The video to be adjusted and the target video parameters are input to the input layer of the network. The function of the input layer is to generate random noise vectors and conditional input data. The random noise vector is a random input used by the generator to generate new data, while the conditional input data contains the feature information of the video, which is used to guide the generator to generate encoding parameters that match the input video features.

[0114] By introducing random noise, the diversity of encoding parameters generated by the generator can be increased to avoid overfitting. At the same time, the conditional input data ensures that the generated encoding parameters are related to the video content, improving the adaptability and accuracy of the encoding parameters.

[0115] Step S202, inputting the random noise vector and the conditional input data into the generator to obtain generated coding parameters;

[0116] The generator receives a random noise vector and conditional input data, and then generates encoding parameters. The goal of the generator is to learn how to generate effective encoding parameters based on the input data, which can be used to efficiently encode the video.

[0117] By learning from historical data, the generator can generate parameters similar to the actual encoding parameters, which helps reduce trial and error in the encoding process and improve encoding efficiency.

[0118] Step S203, encoding the video to be adjusted according to the generated encoding parameters to obtain a generated encoded video;

[0119] The video is encoded using the encoding parameters generated by the generator to obtain the generated encoded video. This step is the process of actually applying the generated parameters, and the purpose is to verify the validity of the generated parameters.

[0120] Through the actual encoding process, the performance of the generated parameters can be evaluated to provide feedback for subsequent optimization.

[0121] Step S204, inputting the generated encoded video and the video to be adjusted into the discriminator to obtain a discrimination result;

[0122] The generated encoded video and the original video to be adjusted are input to the discriminator. The task of the discriminator is to distinguish whether the generated video is close enough to the original video in quality, that is, whether it is authentic.

[0123] The discriminator provides a quality control mechanism that ensures that only high-quality encoded videos are accepted, which helps maintain the overall quality of the video encoding.

[0124] Step S205: when the judgment result is true, the generated coding parameters are outputted by the generator, and the generated coding parameters are used as target coding parameters.

[0125] If the discriminator determines that the generated encoded video is authentic, that is, close enough to the original video in quality, the encoding parameters generated by the generator are accepted and used as target encoding parameters for subsequent video encoding.

[0126] This step ensures that only high-quality encoding parameters are used, thereby improving the overall quality and efficiency of video encoding.

[0127] It is understandable that through the quality control of the discriminator, only high-quality encoding parameters will be adopted. Through automation and optimization of encoding parameters, resource consumption in the encoding process can be reduced and costs can be reduced. High-quality encoding parameters ensure the smoothness and clarity of video playback and enhance the user's viewing experience.

[0128] Specifically, the generator includes a first conditional embedding layer, a fusion layer, a temporal convolution layer, a three-dimensional convolution layer, a first attention mechanism and a first output layer; the process of inputting the random noise vector and the conditional input data into the generator to obtain the generated encoding parameters may include:

[0129] Step S2021, inputting the conditional input data into the first conditional embedding layer to obtain a first embedding vector;

[0130] The function of the conditional embedding layer is to map the conditional input data (such as video features) into a high-dimensional space and generate the first embedding vector. This embedding vector contains the key information of the input data and provides a basis for subsequent feature extraction.

[0131] Through the embedding layer, the input data can be effectively converted into a form suitable for subsequent processing, improving the efficiency and accuracy of feature extraction.

[0132] Step S2022, inputting the first embedding vector and the random noise vector into the fusion layer to obtain a fusion vector;

[0133] The fusion layer combines the first embedding vector and the random noise vector to generate a fused vector. This process introduces randomness, which helps to generate diverse encoding parameters and avoid overfitting. The fusion layer increases the diversity of generated parameters and improves the generalization ability of the model by combining conditional information and randomness.

[0134] Step S2023, inputting the fusion vector into the time series convolution layer to obtain time series features;

[0135] The temporal convolution layer processes the fused vectors and extracts temporal features. This layer can capture the temporal information in the input data, such as the temporal relationship between video frames. The temporal convolution layer helps understand the temporal dynamics of the video content and provides support for generating encoding parameters that match the video content.

[0136] Step S2024, inputting the temporal features into the three-dimensional convolutional layer to obtain a first spatiotemporal feature;

[0137] The 3D convolution layer further processes the temporal features and extracts spatiotemporal features. This layer can capture both spatial and temporal information, providing rich contextual information for generating encoding parameters. The 3D convolution layer can more comprehensively understand the video content and improve the accuracy and applicability of the encoding parameters.

[0138] Step S2025, inputting the spatiotemporal feature into the first attention mechanism to obtain a first weighted feature;

[0139] The attention mechanism layer weights the spatiotemporal features to generate the first weighted features. The attention mechanism can identify the most important features and give them higher weights, thereby increasing the model's sensitivity to key information. The attention mechanism increases the model's focus on the key parts of the video content, making the generated encoding parameters more accurate and effective.

[0140] Step S2026: input the first weighted feature into the output layer to generate coding parameters.

[0141] The output layer receives the first weighted features and generates the final encoding parameters. This layer combines all the extracted and weighted features to generate parameters for video encoding. The output layer integrates all processed features to generate encoding parameters suitable for the video content, improving the efficiency and quality of encoding.

[0142] It is understandable that through the automated feature extraction and parameter generation process, human intervention is reduced, encoding efficiency is improved, and efficient and high-quality encoding parameters ensure the smoothness and clarity of video playback, improving the user's viewing experience.

[0143] Specifically, the discriminator includes a second conditional embedding layer, a sequence feature extraction layer, a convolution layer, a second attention mechanism and a second output layer; the process of inputting the generated encoded video and the video to be adjusted into the discriminator to obtain a discrimination result may include:

[0144] Step S2041, inputting the conditional input data into the second conditional embedding layer to obtain a second embedding vector;

[0145] The conditional embedding layer maps the conditional input data (such as video features) to a high-dimensional space and generates a second embedding vector. This embedding vector contains the key information of the input data and provides a basis for subsequent feature extraction. Through the embedding layer, the input data can be effectively converted into a form suitable for subsequent processing, improving the efficiency and accuracy of feature extraction.

[0146] Step S2042, fusing the second embedding vector and the video to be adjusted to obtain a fusion feature;

[0147] The fusion layer combines the second embedding vector and the video to be adjusted to generate fusion features. This process introduces the original information of the video, which helps the discriminator to more accurately evaluate the quality of the generated encoded video. By combining conditional information and video content, the fusion layer increases the discriminator's understanding of the video content and improves the accuracy of the discrimination results.

[0148] Step S2043, inputting the fusion feature into the sequence feature extraction layer to obtain video features;

[0149] The sequence feature extraction layer processes the fused features and extracts video features. This layer can capture key features of the video content, such as motion, texture, etc. The sequence feature extraction layer helps to understand the complexity of the video content and provides rich feature information for the discriminator.

[0150] Step S2044, inputting the video feature into the convolutional layer to obtain a second spatiotemporal feature;

[0151] The convolutional layer further processes the video features and extracts spatiotemporal features. This layer can capture both spatial and temporal information and provide comprehensive contextual information for the discriminator. The convolutional layer can understand the video content more comprehensively and improve the accuracy and reliability of the discrimination results.

[0152] Step S2045, inputting the second spatiotemporal feature into the second attention mechanism to obtain a second weighted feature;

[0153] The attention mechanism layer weights the second spatiotemporal features to generate the second weighted features. The attention mechanism can identify the most important features and give them higher weights, thereby increasing the model's sensitivity to key information. The attention mechanism increases the model's focus on the key parts of the video content, making the discrimination results more accurate and effective.

[0154] Step S2046: input the second weighted feature into the second output layer to obtain a discrimination result.

[0155] The output layer receives the second weighted feature and generates the final discrimination result. This layer combines all processed features to generate a discrimination result for judging the quality of the generated encoded video. The output layer integrates all processed features to generate a discrimination result suitable for the video content, which improves the efficiency and accuracy of the discrimination process.

[0156] It can be understood that through multi-level feature extraction and attention mechanism, the discriminator is able to more accurately evaluate the quality of generated encoded videos, which helps to maintain the overall quality of video encoding.

[0157] Step S30, encoding the video to be adjusted according to the target encoding parameters to obtain an encoded video, and sending the encoded video to a display, wherein the display receives and displays the encoded video.

[0158] The target encoding parameters are obtained and the video is encoded using these parameters. The encoding process involves compressing the video data to reduce the file size while maintaining the video quality as much as possible. The encoding parameters generated using machine learning may be more accurate than traditional encoding parameters, thereby reducing the file size while maintaining video quality and improving storage and transmission efficiency.

[0159] Compared with setting fixed video encoding parameters, this embodiment extracts the features of the video to be adjusted and uses a parameter generation network to generate customized encoding parameters, thereby dynamically adjusting the parameters according to the content characteristics of the video. This embodiment can more accurately match the specific needs of the video, thereby improving the encoding effect.

[0160] Compared with manually adjusting video encoding parameters, this embodiment reduces the reliance on user experience and intuition through an automated feature extraction and parameter generation process, making the determination of encoding parameters more objective and systematic, and being able to more accurately match the specific needs of the video, thereby improving the encoding effect.

[0161] Compared with determining video encoding parameters based on rules, the parameter generation network trained by the generative adversarial network in this embodiment can learn more complex patterns from a large amount of historical video data, so that the video encoding parameters can handle a wider range of content types and encoding scenarios, and can more accurately match the specific needs of the video, thereby improving the encoding effect.

[0162] Compared with the existing machine learning model to determine the video encoding parameters, this embodiment generates video encoding parameters based on a generative adversarial network. The generator in the generative adversarial network is responsible for generating encoding parameters, and the discriminator evaluates the quality of the generated parameters. Through this adversarial process, the video encoding parameters are continuously optimized, so that the video encoding parameters can more accurately match the specific needs of the video, thereby improving the encoding effect. In addition, since the generative adversarial network can analyze video features in real time and dynamically generate parameters, it reduces the dependence on a large amount of historical training data, so that the parameter generation network can better generalize to unseen video data, thereby improving the adaptability to unknown or rare video content, so that the video encoding parameters can more accurately match the specific needs of the video, thereby improving the encoding effect.

[0163] In summary, compared with fixed parameters, this embodiment can capture the dynamic characteristics of video content by extracting target video features in real time, and input these features into the parameter generation network, so as to generate encoding parameters adapted to the current video content and enhance the adaptability of video encoding parameters.

[0164] Compared with manually setting parameters, this embodiment uses a preset parameter generation network to automatically obtain encoding parameters, which reduces the complexity of manually adjusting parameters, allowing even ordinary users to easily perform video encoding and simplifying the parameter setting process.

[0165] In this embodiment, a parameter generation network is trained based on historical video data and corresponding encoding parameters, and the learned pattern can adapt to different video contents and conditions, thereby improving the consistency and stability of the encoding effect.

[0166] Compared with determining parameters by machine learning models, this embodiment continuously optimizes video encoding parameters through the adversarial process of generative adversarial networks, reducing dependence on a large amount of historical training data, so that the parameter generation network can better generalize to unseen video data and reduce dependence on training data.

[0167] This embodiment uses dynamically generated encoding parameters to adapt to high dynamic scenes, optimize the encoding process, and improve encoding efficiency.

[0168] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 Step S10: extracting features of the video to be adjusted to obtain target video features, including:

[0169] Step S101, extracting a foreground object in the video to be adjusted, and performing optical flow calculation on the foreground object based on the video to be adjusted to obtain motion information of the foreground object;

[0170] First, the foreground objects in the video are identified, which is usually achieved through background subtraction, edge detection or other segmentation techniques. Then, the motion information of these foreground objects is calculated using an optical flow algorithm, which estimates the motion of objects by analyzing the changes in their positions in consecutive video frames. By extracting the motion information of foreground objects, the main dynamic elements in the video can be identified, which is crucial for understanding the video content and subsequent video analysis (such as motion detection and behavior recognition).

[0171] Step S102, determining the motion amplitude of each of the foreground objects based on the motion information, determining a target object whose motion amplitude is greater than a preset amplitude threshold from each of the foreground objects, and taking the area where the target object is located as an area of ​​interest;

[0172] The motion amplitude of each foreground object is analyzed and compared with a preset threshold. If the motion amplitude of an object exceeds this threshold, it is considered a target object and its area is marked as a region of interest. Identifying objects with large motion amplitudes can help focus on key activity areas in the video, which is very useful for applications such as surveillance, security analysis, or behavior recognition.

[0173] Step S103, performing image block processing on each video frame of the video to be adjusted to obtain a plurality of image blocks, and extracting texture features of each video frame based on each image block;

[0174] The video frame is divided into multiple image blocks, and then texture features are extracted from each image block. Texture features can describe the texture pattern of the image block, such as roughness, directionality, etc. Texture features help to identify static content in the video, such as the surface and material of objects in the scene, which is very important for scene analysis and object recognition.

[0175] Step S104, determining inter-frame difference information and scene change information of each video frame, and using the inter-frame difference information and the scene change information as temporal dynamic information;

[0176] Compute the difference between consecutive video frames and identify scene changes. The inter-frame difference information can help identify new objects that appear or disappear in the video, while the scene change information can indicate broader environmental changes. By analyzing inter-frame differences and scene changes, we can better understand the temporal dynamics of video content, which is very valuable for applications such as activity detection, event recognition, and video summary generation.

[0177] Step S105: taking the motion information, the region of interest, the texture features and the temporal dynamic information as the target video features.

[0178] All extracted information (motion information, regions of interest, texture features, and temporal dynamic information) are integrated as target video features, which will be used for subsequent video processing tasks such as video encoding, content analysis, or training of machine learning models.

[0179] In a feasible implementation manner, the step S101: extracting the foreground object in the video to be adjusted, and performing optical flow calculation on the foreground object based on the video to be adjusted to obtain motion information of the foreground object, includes:

[0180] Step S1011, determining an initial video frame set from each video frame of the video to be adjusted, and performing background modeling based on the initial video frame set to obtain a background model, wherein the initial video frame set includes a preset number of video frames that are chronologically preceding in each video frame;

[0181] First, a series of initial frames are selected from the video, which usually include several consecutive frames at the beginning of the video. Then, these frames are used to build a background model, which usually involves calculating the average or other statistical information of the pixel values ​​of these frames. Through background modeling, the static background and dynamic foreground in the video can be distinguished, providing a basis for subsequent foreground object extraction and motion analysis.

[0182] Step S1012, for any target frame in each of the video frames, calculate the pixel difference between each pixel of the target frame and the background model, and determine the pixels whose pixel difference is greater than a preset difference threshold among the pixels as foreground pixels, and determine the foreground object based on each of the foreground pixels;

[0183] For each frame (target frame) in the video, the difference between each pixel and the background model is calculated. If the difference exceeds the preset threshold, the pixel is considered to be a foreground pixel. By calculating the pixel difference and setting the threshold, the foreground object can be effectively identified, which is very important for subsequent motion analysis and object tracking.

[0184] Step S1013, determining a pixel displacement of each of the foreground pixels between adjacent video frames, and determining an optical flow vector of each of the foreground pixels based on the pixel displacement;

[0185] Optical flow, i.e. the direction and distance of pixel movement, is calculated by comparing the position changes of foreground pixels in consecutive frames. Optical flow is a vector field that describes the movement of pixels. Optical flow calculation provides pixel-level motion information, which is very useful for understanding the movement pattern and speed of objects.

[0186] Step S1014: determining the motion amplitude and motion direction of the foreground object based on the optical flow vector of each foreground pixel, and using the motion amplitude and the motion method as motion information.

[0187] By analyzing the optical flow vectors of all foreground pixels, the overall motion magnitude and direction of each foreground object can be calculated. This usually involves aggregating and averaging the optical flows of all relevant pixels. By determining the motion magnitude and direction, the motion characteristics of the object can be more accurately described, which is critical for applications such as motion detection, behavior analysis, and video surveillance.

[0188] In a feasible implementation manner, the step S103: performing image block processing on each video frame of the video to be adjusted to obtain a plurality of image blocks, and extracting texture features of each video frame based on each image block, comprises:

[0189] Step S1031, for any target frame among the video frames of the video to be adjusted, convert the target frame into a grayscale image;

[0190] Select any frame in the video as the target frame and convert it from a color image to a grayscale image. Grayscale images only contain brightness information, not color information, which helps to simplify the subsequent texture analysis process. Converting to grayscale can reduce the complexity of processing data while retaining important texture information of the image, providing a basis for texture feature extraction.

[0191] Step S1032, dividing the grayscale image into a plurality of non-overlapping image blocks according to a preset block size;

[0192] The grayscale image is divided into multiple non-overlapping image blocks according to a pre-set block size (such as 8x8, 16x16, etc.). This block processing helps to extract local texture features, because different areas of the image may have different texture characteristics. Through block processing, the local texture features of the image can be analyzed more finely, which is very important for subsequent video encoding and analysis, especially when different encoding strategies need to be applied to different areas.

[0193] Step S1033, calculating the gray level co-occurrence matrix of each of the image blocks, and extracting the texture features of the target frame based on each of the gray level co-occurrence matrices.

[0194] For each image block, its gray-level co-occurrence matrix (GLCM) is calculated. GLCM (Gray-Level Co-occurrence Matrix) is a technique that describes the distribution of pixel grayscale in an image and its spatial relationship. It quantifies the frequency of a pair of grayscale values ​​that co-occur in a specific direction and distance. It can be used to extract image features and analyze image texture to help identify and classify image content. Then, texture features such as contrast, uniformity, energy, and entropy are extracted from GLCM. By calculating GLCM and extracting texture features, the texture properties of the image block can be quantified, which is very useful for understanding image content and optimizing coding strategies. For example, high-texture areas may require higher encoding bit rates to maintain visual quality.

[0195] It can be understood that through precise texture feature extraction, detail-rich areas and flat areas in the image can be identified, so that different processing strategies can be adopted for these areas during encoding to improve encoding efficiency. For areas with rich texture features, more bit rate can be allocated to maintain details, while for areas with fewer texture features, the bit rate can be reduced, thereby reducing the file size while maintaining the overall video quality.

[0196] In a feasible implementation manner, the step S104: determining the inter-frame difference information and scene change information of each video frame includes:

[0197] Step S1041, for any two adjacent frames in each video frame, calculate the pixel difference between each pixel in the adjacent frames, and use the pixel difference as inter-frame difference information of the adjacent frames;

[0198] For any two consecutive frames (adjacent frames) in a video, the difference between each corresponding pixel is calculated. This difference is usually obtained through a simple subtraction operation, that is, the pixel value of the current frame is subtracted from the pixel value of the previous frame. The calculated pixel difference can be expressed as inter-frame difference information, which reflects the changes between the two frames. Inter-frame difference information is key data in video compression. It helps to identify static and dynamic areas in the video, so that different strategies can be adopted for these areas during encoding to improve compression efficiency.

[0199] Step S1042, determining key points in each video frame by a feature detector, and matching key points in adjacent frames to obtain key point pairs, wherein the key points are unique and repeatedly detectable position points in the video frame;

[0200] Key points are identified in each frame using feature detection algorithms such as SIFT, SURF, or ORB. Key points are unique and reproducibly detected points in an image, such as corner points, edge points, etc. Then, key points detected in adjacent frames are matched to determine the correspondence between them. Key point matching helps understand object motion and scene changes in the video, which is very important for applications such as video analysis, target tracking, and 3D reconstruction. It should be noted that SIFT (Scale-Invariant Feature Transform) is a classic feature point detection algorithm that can automatically detect key points in an image at multiple scales and describe them so that they have good invariance and are suitable for image matching and object recognition. SURF (Speeded Up Robust Features): The SURF algorithm is an improvement on the SIFT algorithm. Compared with the SIFT algorithm, the SURF algorithm has a higher calculation speed and is suitable for real-time application scenarios. The SURF algorithm uses the determinant value of the Hessian matrix as the basis for feature point detection and uses integral graphs to speed up the calculation process. ORB (Oriented FAST and Rotated BRIEF) is an algorithm that combines feature detection and description. It is fast and invariant to rotation, suitable for mobile devices or low computing power environments.

[0201] Step S1043: determine the position transformation information of the key point pair, and determine the scene change information of the adjacent frames based on the position transformation information of the key point pair.

[0202] Analyze the position changes of matched keypoint pairs in adjacent frames. This change can be translation, rotation or scaling. Through this position change information, the degree and type of scene change can be determined. Scene change information is crucial for applications such as video content understanding, event detection and dynamic scene analysis. It can help identify important events in the video, such as scene switching or objects entering / leaving the screen.

[0203] It is understandable that by analyzing the inter-frame differences and scene changes, encoding resources can be allocated more efficiently, using more bitrate for areas with large changes and less bitrate for areas with small changes, thereby improving compression efficiency.

[0204] For example, to help understand the implementation process of the video encoding method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 4 , Figure 4 A brief flow chart of a video encoding method is provided. Specifically, the overall design block diagram of this embodiment is as follows, which is mainly divided into four key modules:

[0205] 1. Video feature extraction: Extract necessary features from the input video to provide basic data for subsequent encoding. For details, please refer to Figure 5 ,Video feature extraction includes: motion vector extraction: extracting motion information through background modeling and optical flow calculation; texture complexity analysis: analyzing the texture features of the video through image segmentation and feature extraction; temporal dynamic analysis: calculating the difference between frames and scene changes to analyze temporal dynamics; scene activity evaluation: identifying active frames and activity levels to evaluate the activity of video scenes.

[0206] 2. Training GAN: Generate encoding parameters and optimize them through generative adversarial networks (GAN). Generator: Generate encoding parameters based on input features. Discriminator: Evaluate the difference between the generated parameters and the true parameters to improve the generator.

[0207] 3. Real-time encoding process: Apply feature extraction and generated parameters to actual video encoding. Input video stream: Receive input video and extract features. Coding parameter generation: Get parameters from GAN and apply them to encoding.

[0208] 4. Real-time feedback and adjustment: Evaluate the encoding results and make feedback adjustments to optimize the generator. Evaluate the output video: Evaluate the quality and efficiency of the encoded video. Optimize the generator: Adjust the generation parameters based on the evaluation feedback to improve the subsequent encoding quality.

[0209] The relationship between modules is as follows Figure 4As shown in Figure 1, each module is connected through data flow. Specifically, the video feature extraction module passes the extracted feature data to GAN to generate encoding parameters. The real-time feedback mechanism ensures that the system can continuously optimize its performance based on the evaluation results, forming a closed-loop system.

[0210] Please refer to Figure 6 The structure of the conditional adaptive video generation adversarial model GAVGAN++ is as follows Figure 6 As shown in Figure 1, the GAVGAN++ model includes an input layer, a generator, a discriminator, a loss function, a training strategy, regularization, and a post-processing stage.

[0211] 1. Input Layer includes: random noise (z), condition input (Condition Input);

[0212] 2. The generator includes: conditional embedding layer, fusion layer, temporal convolution layer, 3D convolution layer, attention mechanism (Attention LayerGen), and output layer.

[0213] 3. The discriminator includes: condition embedding layer (Condition Embedding Disc), sequence feature extraction layer (Sequence Feature), convolution layer (Conv Layer), attention mechanism (Attention Layer Disc), and output layer (Output Layer Disc).

[0214] 4. Loss Functions, which can specifically include multiple losses: adversarial loss, which is used to evaluate the gap between the video generated by the generator and the real video, and encourage the generator to generate more realistic videos; conditional consistency loss, which ensures the consistency between the generated content and the conditional input, so that the generated video is more in line with user needs; temporal consistency loss, which promotes the consistency of the generated video in the time dimension and ensures a smooth transition between frames; perceptual loss, which measures the quality of the generated video through high-level feature representation and enhances the visual effect of the generated video.

[0215] 5. Training Strategies, which can include multiple strategies: Step-by-step training strategy: Initially train the discriminator to enhance its ability to distinguish between real and generated videos; then focus on training the generator to optimize its generation performance. Balanced training strategy: Optimize the training of the generator and the discriminator at the same time within a certain stage to maintain the training balance of both parties and prevent mode collapse or performance imbalance. Regularization: Apply various regularization techniques, including but not limited to: spectral normalization to limit the weight changes of the discriminator, prevent instability during training, and improve the performance of the generator; instance normalization: used to improve the consistency of generated content, especially in style generation tasks, to ensure that each generated frame has visual coherence; post-processing stage: further process the generated video to enhance its overall quality and user experience.

[0216] 6. Post-processing steps include: Video enhancement: Apply image processing techniques (such as denoising, sharpening, color correction) to improve the visual quality of the generated video, making it more realistic and attractive. Frame interpolation: Use interpolation technology to increase the frame rate of the video, improve visual fluidity, and make the video motion smoother. Quality assessment: Use quantitative indicators such as structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) to evaluate the quality of the generated video and compare it with the real video to ensure that the generated content meets the set quality standards.

[0217] 7. Model Evaluation: Systematically evaluate the performance of the GAVGAN++ model to ensure that the model can meet the needs of actual applications: Diversity of generated videos: Statistically analyze the diversity of generated videos to ensure that the generator has a high level of diversity and can generate rich content. Generation speed: Evaluate the computational efficiency of the model to ensure that the speed of generating videos meets the requirements of real-time applications. User feedback collection: Collect subjective evaluations of generated videos through user experiments and satisfaction surveys to further optimize the model.

[0218] For details, please refer to Figure 7 , the data processing flow of the input layer, generator, discriminator and post-processing stage in the conditional adaptive video generation adversarial model GAVGAN++ can be:

[0219] The input layer's input layer outputs random noise z and conditional input. The conditional input is sent to the embedding layer of the conditional embedding layer in the generator. The output of the embedding layer is embedded in the conditional vector. The conditional vector is fused with the input video sequence to obtain a fusion result. The fusion result is fused with the random noise z in the fusion layer to obtain a fusion feature. The fusion feature is input into the temporal convolution layer for temporal feature extraction to generate video features. The video features are input into the 3D convolution layer to obtain spatial features. The spatial features are input into the attention mechanism to allocate time attention and space attention to obtain attention features. The attention features are output to the post-processing stage. Video enhancement and video quality assessment are performed in the post-processing stage. The conditional input is sent to the embedding layer of the conditional embedding layer in the discriminator. The output of the embedding layer is embedded in the conditional vector. The conditional vector is fused with the input video sequence to obtain a fusion result. The fusion result is fused with the random noise z in the fusion layer to obtain a fusion feature. The fusion feature is input into the temporal convolution layer for temporal feature extraction to generate video features. The video features are input into the 3D convolution layer to obtain spatial features. The spatial features are input into the attention mechanism to obtain the focus area and distinguish the true or false of the focus area. In this embodiment, a public dataset with rich video content is selected for training. This embodiment uses the YouTube-8M dataset to ensure that the model can access diverse input data. A random sampling method is used to select training samples to avoid overfitting, and amplification technology is used to increase the diversity of training data. A reasonable learning rate and optimizer are set to accelerate convergence. At the same time, the loss value changes are monitored and the learning rate is adjusted in time to ensure stable model training.

[0220] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the video encoding method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0221] The present application also provides a video encoding device, which adopts the video encoding method in the above embodiment to solve the technical problem that the existing method of determining video encoding parameters lacks accurate adaptability to specific video content and encoding requirements, resulting in affecting the video encoding effect. Compared with the prior art, the beneficial effects of the video encoding device provided by the present application are the same as the beneficial effects of the video encoding method provided by the above embodiment, and other technical features in the video encoding device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0222] The present application provides a video encoding device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video encoding method in the above-mentioned embodiment 1.

[0223] Reference below Figure 8 , which shows a schematic diagram of the structure of a video encoding device suitable for implementing an embodiment of the present application. The video encoding device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The video encoding device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0224] like Figure 8 As shown, the video encoding device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the video encoding device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the video encoding device to communicate with other devices wirelessly or by wire to exchange data. Although the video encoding device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0225] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0226] The video encoding device provided by the present application adopts the video encoding method in the above embodiment, which can solve the technical problem that the existing method of determining video encoding parameters lacks accurate adaptability to specific video content and encoding requirements, resulting in affecting the video encoding effect. Compared with the prior art, the beneficial effects of the video encoding device provided by the present application are the same as the beneficial effects of the video encoding method provided by the above embodiment, and other technical features in the video encoding device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0227] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0228] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0229] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the video encoding method in the above-mentioned embodiment.

[0230] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0231] The computer-readable storage medium may be included in the video encoding device; or may exist independently without being assembled into the video encoding device.

[0232] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the video encoding device, the video encoding device implements the video encoding method in the above embodiment.

[0233] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0234] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0235] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0236] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned video encoding method, and can solve the technical problem that the existing method of determining video encoding parameters lacks accurate adaptability to specific video content and encoding requirements, resulting in affecting the video encoding effect. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the video encoding method provided by the above-mentioned embodiment, and will not be repeated here.

[0237] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned video encoding method when executed by a processor.

[0238] The computer program product provided by the present application can solve the technical problem that the existing method of determining video encoding parameters lacks accurate adaptability to specific video content and encoding requirements, which affects the video encoding effect. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the video encoding method provided by the above embodiment, and will not be repeated here.

[0239] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A video encoding method, characterized in that: The video encoding method comprises: Acquire a video to be adjusted, and perform feature extraction on the video to be adjusted to obtain target video features, wherein the target video features are used to describe characteristics of the picture content of the video to be adjusted; Input the target video features and the video to be adjusted into a preset parameter generation network to obtain target encoding parameters, wherein the parameter generation network is obtained by training a generative adversarial network using a historical video and the video features of the historical video as input data and the encoding parameters corresponding to the historical video as labels; The video to be adjusted is encoded according to the target encoding parameters to obtain an encoded video, and the encoded video is sent to a display, wherein the display receives and displays the encoded video.

2. The video encoding method according to claim 1, characterized in that: The step of extracting features from the video to be adjusted to obtain target video features includes: Extracting a foreground object in the video to be adjusted, and performing optical flow calculation on the foreground object based on the video to be adjusted to obtain motion information of the foreground object; Determine the motion amplitude of each of the foreground objects based on the motion information, determine a target object whose motion amplitude is greater than a preset amplitude threshold from among the foreground objects, and use the area where the target object is located as an area of ​​interest; Performing image block processing on each video frame of the video to be adjusted to obtain a plurality of image blocks, and extracting texture features of each video frame based on each of the image blocks; Determine inter-frame difference information and scene change information of each video frame, and use the inter-frame difference information and the scene change information as temporal dynamic information; The motion information, the region of interest, the texture features and the temporal dynamic information are used as the target video features.

3. The video encoding method according to claim 2, characterized in that: The step of extracting the foreground object in the video to be adjusted, and performing optical flow calculation on the foreground object based on the video to be adjusted to obtain motion information of the foreground object includes: Determine an initial video frame set from each video frame of the video to be adjusted, and perform background modeling based on the initial video frame set to obtain a background model, wherein the initial video frame set includes a preset number of video frames that are chronologically preceding in each video frame; For any target frame in each of the video frames, a pixel difference between each pixel of the target frame and the background model is calculated, and pixels whose pixel difference is greater than a preset difference threshold among the pixels are determined as foreground pixels, and a foreground object is determined based on each of the foreground pixels; Determine a pixel displacement of each of the foreground pixels between adjacent video frames, and determine an optical flow vector of each of the foreground pixels based on the pixel displacement; The motion amplitude and motion direction of the foreground object are determined based on the optical flow vectors of each foreground pixel, and the motion amplitude and the motion method are used as motion information.

4. The video encoding method according to claim 2, wherein: The step of performing image block processing on each video frame of the video to be adjusted to obtain a plurality of image blocks, and extracting texture features of each video frame based on each image block comprises: For any target frame among the video frames of the video to be adjusted, convert the target frame into a grayscale image; Dividing the grayscale image into a plurality of non-overlapping image blocks according to a preset block size; The grayscale co-occurrence matrix of each of the image blocks is calculated, and the texture features of the target frame are extracted based on each of the grayscale co-occurrence matrices.

5. The video encoding method according to claim 2, wherein: The step of determining the inter-frame difference information and scene change information of each video frame includes: For any two adjacent frames in each video frame, calculate the pixel difference between each pixel in the adjacent frames, and use the pixel difference as inter-frame difference information of the adjacent frames; Determine the key points in each video frame by a feature detector, and match the key points in adjacent frames to obtain key point pairs, wherein the key points are unique and repeatedly detectable position points in the video frame; Position transformation information of the key point pair is determined, and scene change information of the adjacent frames is determined based on the position transformation information of the key point pair.

6. The video encoding method according to any one of claims 1 to 5, characterized in that: The parameter generation network includes an input layer, a generator and a discriminator; The step of inputting the target video features and the video to be adjusted into a preset parameter generation network to obtain target encoding parameters includes: Inputting the video to be adjusted and the target video parameters into the input layer to obtain a random noise vector and conditional input data; Inputting the random noise vector and the conditional input data into the generator to obtain generated coding parameters; Performing video encoding on the video to be adjusted according to the generated encoding parameters to obtain a generated encoded video; Inputting the generated coded video and the video to be adjusted into the discriminator to obtain a discrimination result; When the discrimination result is true, the generated encoding parameters are outputted by the generator, and the generated encoding parameters are used as target encoding parameters.

7. The video encoding method according to claim 6, characterized in that: The generator includes a first conditional embedding layer, a fusion layer, a temporal convolution layer, a three-dimensional convolution layer, a first attention mechanism and a first output layer; The step of inputting the random noise vector and the conditional input data into the generator to generate encoding parameters comprises: Inputting the conditional input data into the first conditional embedding layer to obtain a first embedding vector; Inputting the first embedding vector and the random noise vector into the fusion layer to obtain a fusion vector; Inputting the fusion vector into the time series convolution layer to obtain time series features; Inputting the temporal feature into the three-dimensional convolutional layer to obtain a first spatiotemporal feature; Inputting the spatiotemporal feature into the first attention mechanism to obtain a first weighted feature; The first weighted feature is input into the output layer to generate encoding parameters.

8. The video encoding method according to claim 7, characterized in that: The discriminator includes a second conditional embedding layer, a sequence feature extraction layer, a convolution layer, a second attention mechanism, and a second output layer; The step of inputting the generated coded video and the video to be adjusted into the discriminator to obtain a discrimination result comprises: Inputting the conditional input data into the second conditional embedding layer to obtain a second embedding vector; Fusing the second embedding vector and the video to be adjusted to obtain a fusion feature; Inputting the fusion feature into the sequence feature extraction layer to obtain video features; Inputting the video feature into the convolutional layer to obtain a second spatiotemporal feature; Inputting the second spatiotemporal feature into the second attention mechanism to obtain a second weighted feature; The second weighted feature is input into the second output layer to obtain a discrimination result.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video encoding method according to any one of claims 1 to 8 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the video encoding method according to any one of claims 1 to 8 are implemented.

Images