Intelligent video coding optimization method and optimization system
The intelligent video coding method enhances encoding efficiency and quality by segmenting frames, applying deep learning for differentiated quantization, addressing inefficiencies in traditional coding methods.
Patent Information
- Application Number
- CN202411073357.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-06
AI Technical Summary
Traditional video coding methods face inefficiencies and image quality loss when processing high dynamic range and complex scene videos due to uniform processing of frames without considering dynamically changing and static areas, leading to suboptimal utilization of spatial and temporal redundancy.
An intelligent video coding method utilizing motion detection, frame segmentation, multi-scale processing, and deep learning prediction to differentiate between foreground and background areas, applying distinct quantization parameters for efficient encoding.
Improves compression ratio and coding efficiency while maintaining high-quality video encoding by separately encoding moving and static areas with tailored quantization parameters.
Smart Images

Figure CN119071489A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video coding, and in particular to an intelligent video coding optimization method and optimization system. Background Art
[0002] With the rapid development of video technology, the explosive growth of video data has put forward higher requirements for video coding technology. Traditional video coding methods often face problems such as low coding efficiency and large image quality loss when processing high dynamic range videos, complex scene videos and real-time video transmission. In order to solve these problems, intelligent video coding technology came into being. It combines advanced image processing technology and deep learning algorithms to achieve efficient compression and high-quality reconstruction of video data.
[0003] In the prior art, video coding usually relies on uniform processing of video frames, using the same coding parameters for coding regardless of how the video content changes. However, video content often contains a large number of dynamically changing areas (such as moving objects) and relatively static background areas. Using the same coding parameters for these two types of areas not only fails to fully utilize the spatial and temporal redundancy of the video content, but may also lead to low coding efficiency and reduced image quality.
[0004] Therefore, it is necessary to provide an intelligent video coding optimization method and optimization system to solve the above technical problems. Summary of the invention
[0005] In order to solve the above technical problems, the present invention provides an intelligent video coding optimization method and optimization system, which realizes intelligent processing and efficient encoding of video content by combining preliminary motion detection, video frame segmentation, multi-scale processing, deep learning prediction and differential coding, which not only improves the compression ratio and coding efficiency of video coding, but also ensures the high quality and low distortion of the encoded video.
[0006] The present invention provides an intelligent video coding optimization method, the optimization method comprising the following steps:
[0007] S1: Based on the adjacent video frames of the current video frame in the video to be encoded, preliminary motion detection is performed on the current video frame by a motion vector estimation method to obtain a motion region and a non-motion region belonging to the current video frame;
[0008] S2: Segment the current video frame based on a selected coding standard to obtain a plurality of sub-blocks;
[0009] S3: constructing a resolution pyramid of the current video frame, and performing multi-scale processing on the multiple sub-blocks belonging to the motion area based on the resolution pyramid to obtain the motion area after multi-scale processing;
[0010] S4: taking the motion region after multi-scale processing as input, and using the pre-trained deep learning model to predict the current video frame, to obtain the foreground and background of the motion region and the first quantization parameter and the second quantization parameter corresponding to the foreground and the background, respectively, wherein the first quantization parameter is less than the second quantization parameter;
[0011] S5: Encode the foreground and background areas in the motion area respectively using the first quantization parameter and the second quantization parameter, and encode the non-motion area using the second quantization parameter.
[0012] Preferably, step S1 comprises the following steps:
[0013] S101: using a feature point detection algorithm to identify feature points in a current frame and adjacent video frames of a video to be encoded, determining a correspondence between the identified feature points between frames, and calculating a motion vector of each feature point between the current frame and adjacent video frames;
[0014] S102: identifying and locating a motion region in the current video frame by analyzing the direction and size change of the motion vector, and obtaining a non-motion region by elimination method;
[0015] S103: Marking the obtained motion area and non-motion area with visual markers, wherein the visual markers include boundary markers and color markers.
[0016] Preferably, step S2 comprises the following steps:
[0017] S201: Initialize parameters according to the selected coding standard;
[0018] S202: Divide the current video frame into a plurality of sub-blocks using a block division mechanism corresponding to the selected coding standard.
[0019] Preferably, step S3 comprises the following steps:
[0020] S301: Starting from the original resolution of the current video frame, a resolution pyramid with different resolution levels is constructed by gradually reducing the resolution;
[0021] S302: In each layer of the resolution pyramid, select a sub-block belonging to the motion area;
[0022] S303: performing multi-scale processing on all sub-blocks belonging to the motion area in each selected pyramid layer;
[0023] S304: Fusing all sub-blocks of the motion region in each layer of the pyramid after multi-scale processing to obtain the motion region after multi-scale processing.
[0024] Preferably, step S4 comprises the following steps:
[0025] S401: extracting features from the input multi-scale processed motion region using a pre-trained deep learning model to obtain foreground features and background features in the motion region, wherein the deep learning model is a neural network model trained based on a data set of annotated video frames;
[0026] S402: Generate a probability map according to the foreground feature and the background feature, and obtain the foreground and the background based on the probability map, wherein the probability map is used to represent the probability that the sub-block in the motion area belongs to the foreground and the background;
[0027] S403: Differentiately allocate quantization parameters to the obtained foreground and background to obtain a first quantization parameter and a second quantization parameter corresponding to the foreground and background respectively.
[0028] Preferably, step S5 comprises the following steps:
[0029] S501: configuring corresponding encoding parameters according to a first quantization parameter and a second quantization parameter;
[0030] S502: using the configured first quantization parameter and the corresponding encoding parameter, encoding the foreground of the motion area through an encoder, and using the configured second quantization parameter and the corresponding encoding parameter, encoding the background of the non-motion area and the motion area through an encoder;
[0031] S503: Perform fusion processing on the encoded motion area and non-motion area to obtain video frame encoding data.
[0032] The present invention also provides an intelligent video coding optimization system for executing an intelligent video coding optimization method, wherein the optimization system comprises:
[0033] A motion region acquisition module is used to perform preliminary motion detection on the current video frame by a motion vector estimation method based on adjacent video frames of the current video frame in the video to be encoded, and obtain a motion region and a non-motion region belonging to the current video frame;
[0034] A segmentation module, used for segmenting the current video frame based on a selected coding standard to obtain a plurality of sub-blocks;
[0035] A multi-scale processing module, used for constructing a resolution pyramid of the current video frame, and performing multi-scale processing on the multiple sub-blocks belonging to the motion region based on the resolution pyramid to obtain the motion region after the multi-scale processing;
[0036] A prediction module, configured to use the motion region after multi-scale processing as input, and predict the current video frame using a pre-trained deep learning model to obtain a foreground and a background of the motion region and a first quantization parameter and a second quantization parameter corresponding to the foreground and the background, respectively, wherein the first quantization parameter is less than the second quantization parameter;
[0037] The encoding module is used to encode the foreground and background areas in the motion area using the first quantization parameter and the second quantization parameter respectively, and to encode the non-motion area using the second quantization parameter.
[0038] Compared with the related art, the intelligent video coding optimization method and optimization system provided by the present invention have the following beneficial effects:
[0039] The present invention identifies moving areas and non-moving areas by utilizing a motion vector estimation method; secondly, segments video frames based on a coding standard, and constructs a resolution pyramid for multi-scale processing; then, uses a pre-trained deep learning model to predict the foreground and background and corresponding quantization parameters; finally, encodes the foreground and background of the moving area and the non-moving area separately according to differentiated quantization parameters, uses low quantization parameters to preserve details in the foreground, and uses high quantization parameters to improve the compression ratio in the background and non-moving areas, thereby realizing intelligent processing and efficient encoding of video content, which not only improves the compression ratio and encoding efficiency of video encoding, but also ensures the high quality and low distortion of the encoded video. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 A flowchart of an intelligent video coding optimization method provided by the present invention;
[0041] Figure 2 A module structure diagram of an intelligent video encoding optimization system provided by the present invention. DETAILED DESCRIPTION
[0042] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.
[0043] It should also be noted that, for ease of description, only the part relevant to the present invention but not all the content is shown in the accompanying drawings. It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as the processing or method described as a flow chart. Although the flow chart describes each operation (or step) as being processed sequentially, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.
[0044] Embodiment 1
[0045] The present invention provides an intelligent video coding optimization method, referring to Figure 1 As shown, the optimization method includes the following steps:
[0046] S1: Based on the adjacent video frames of the current video frame in the video to be encoded, preliminary motion detection is performed on the current video frame by a motion vector estimation method to obtain a motion region and a non-motion region belonging to the current video frame.
[0047] In this embodiment, the specific process of preliminary motion detection is as follows: first, the previous frame and the next frame of the current video frame are selected as adjacent frames, and then, the pixel or block level changes between the current frame and the adjacent frames are analyzed using a motion vector estimation method including but not limited to an optical flow method or a block matching method, so as to identify the motion area in the current frame, and the non-motion area is obtained by an exclusion method (i.e., the part of the current video frame excluding the motion area).
[0048] S2: Segment the current video frame based on the selected coding standard to obtain a plurality of sub-blocks.
[0049] In this embodiment, based on the selected coding standard, the current video frame is evenly or non-evenly segmented to obtain multiple sub-blocks. The size and shape of the segmentation are adjusted according to the requirements of the coding standard and the characteristics of the video content, aiming to divide the complex video frame into small units that are easier to process, so as to facilitate subsequent coding optimization.
[0050] S3: constructing a resolution pyramid of the current video frame, and performing multi-scale processing on the multiple sub-blocks belonging to the motion region based on the resolution pyramid to obtain the motion region after the multi-scale processing.
[0051] In this embodiment, a resolution pyramid of the current video frame is first constructed, that is, starting from the original resolution, the resolution of the image is gradually reduced to generate a series of images of different scales. Then, for the sub-blocks in the motion area obtained by preliminary motion detection, multi-scale processing is performed on the resolution pyramid to capture the feature information of the motion area at different scales.
[0052] S4: Taking the motion area after multi-scale processing as input, using the pre-trained deep learning model to predict the current video frame, to obtain the foreground and background of the motion area and the first quantization parameter and the second quantization parameter corresponding to the foreground and background, respectively, wherein the first quantization parameter is less than the second quantization parameter.
[0053] In this embodiment, the motion region after multi-scale processing is used as input and sent to a pre-trained deep learning model (such as a neural network CNN). The model has been trained with a large amount of labeled data and has learned how to identify the foreground and background from the input data and obtain the first quantization parameter and the second quantization parameter corresponding to them respectively.
[0054] S5: Encode the foreground and background areas in the motion area respectively using the first quantization parameter and the second quantization parameter, and encode the non-motion area using the second quantization parameter.
[0055] In this embodiment, according to the first quantization parameter and the second quantization parameter predicted by deep learning, the encoder encodes the foreground and background areas in the non-moving area and the moving area respectively. The foreground area is finely encoded using a smaller first quantization parameter to maintain image quality and details; the non-moving area and the background area are coarsely encoded using a larger second quantization parameter to improve the compression ratio. This step ultimately achieves efficient and high-quality encoding of the video.
[0056] Specifically, step S1 includes the following steps:
[0057] S101: using a feature point detection algorithm to identify feature points in a current frame and adjacent video frames of a video to be encoded, determining the correspondence between the identified feature points between frames, and calculating motion vectors of each feature point between the current frame and adjacent video frames.
[0058] In this embodiment, the specific implementation process of step S101 is as follows:
[0059] Feature point detection: First, a feature point detection algorithm is applied to the current frame of the video to be encoded and its adjacent frames (i.e., the previous frame and the next frame), including but not limited to SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), or ORB (Oriented FAST and Rotated BRIEF). These algorithms can identify points with significant features (such as corner points, edge points, etc.) in the image as feature points.
[0060] Determine the correspondence: Use feature matching algorithms, including but not limited to FLANN matcher and BF matcher, to find the correspondence between feature points in the current frame and adjacent frames. These correspondences reflect the motion trajectory of feature points between frames.
[0061] Motion vector calculation: Based on the determined correspondence between feature points, the motion vector of each feature point between the current frame and the adjacent frame is calculated. The motion vector contains the direction and distance information of the feature point moving between frames.
[0062] S102: identifying and locating the motion region in the current video frame by analyzing the direction and size change of the motion vector, and obtaining the non-motion region by elimination method.
[0063] In this embodiment, the specific implementation process of step S102 is as follows:
[0064] Analyze motion vectors: Perform statistical analysis on the calculated motion vectors, including direction, size, and rate of change. By comparing the motion vectors of feature points between adjacent frames, areas with significant motion characteristics can be identified.
[0065] Identify the motion region: According to the analysis result of the motion vector, the motion region in the current frame is identified by a preset threshold. For example, the motion region can be divided based on the size and direction consistency of the motion vector.
[0066] Locating the moving area: After identifying the moving area, the moving area is accurately located in the image space of the current frame through coordinate mapping, and the non-moving area is obtained by the exclusion method (that is, the part of the current video frame except the moving area).
[0067] S103: Marking the obtained motion area and non-motion area with visual markers, wherein the visual markers include boundary markers and color markers.
[0068] In this embodiment, the specific implementation process of step S103 is as follows:
[0069] Boundary marking: Use boundary lines or boundary boxes to mark the identified motion areas and non-motion areas. The boundary lines or boundary boxes should be drawn closely to the edges of the motion areas and non-motion areas so that the range of the motion areas can be clearly identified in subsequent processing.
[0070] Color marking: In order to more intuitively display the motion area, specific colors or patterns can be filled in the motion area and non-motion area. The color or pattern should be selected to ensure a sharp contrast with the background for easy observation and identification.
[0071] Visual marker integration: Integrate boundary markers and color markers into the current video frame to form a video frame image containing clear motion area markers and non-motion area markers.
[0072] Specifically, step S2 includes the following steps:
[0073] S201: Initialize parameters according to the selected coding standard.
[0074] In this embodiment, the specific implementation process of step S201 is as follows:
[0075] Select the coding standard: First, select a suitable video coding standard, such as H.264 / AVC, H.265 / HEVC, based on application requirements and environmental conditions.
[0076] Initialize encoding parameters: Initialize relevant encoding parameters according to the selected encoding standard. These parameters may include but are not limited to: encoding rate, frame rate, resolution, quantization step, prediction mode (such as intra-frame prediction, inter-frame prediction), number of reference frames, etc. The settings of these parameters will directly affect the quality and compression ratio of the encoded video.
[0077] Configure the block mechanism: According to the requirements of the coding standard, configure the corresponding block mechanism, which includes determining the size, shape and segmentation method of the sub-blocks (such as uniform segmentation and content-based segmentation).
[0078] S202: Divide the current video frame into a plurality of sub-blocks using a block division mechanism corresponding to the selected coding standard.
[0079] In this embodiment, the specific implementation process of step S202 is as follows:
[0080] Applying a block mechanism: Using the block mechanism corresponding to the coding standard configured in step S201, the current video frame is divided into multiple sub-blocks. These sub-blocks can be rectangular blocks of fixed size (such as 16x16, 32x32, etc.), or blocks that are adaptively divided based on content (such as blocks divided according to texture complexity, motion characteristics, etc.).
[0081] Optimized block division: In the process of dividing sub-blocks, optimization and adjustment can be made according to the characteristics of the video content. For example, in areas with complex textures or intense motion, smaller sub-blocks can be divided to improve encoding accuracy; while in areas with smooth textures or static areas, larger sub-blocks can be divided to reduce encoding complexity.
[0082] Record block information: record the position, size and relationship of each sub-block with other sub-blocks. This information will be used in the subsequent encoding process.
[0083] Specifically, step S3 includes the following steps:
[0084] S301: Starting from the original resolution of the current video frame, a resolution pyramid with different resolution levels is constructed by gradually reducing the resolution.
[0085] In this embodiment, the specific implementation process of step S301 is as follows:
[0086] Determine the resolution level: First, determine the number of layers of the resolution pyramid and the resolution of each layer based on the original resolution of the video frame and the requirements of subsequent processing. Generally, the more layers of the pyramid, the richer the scale information that can be captured, but the computational complexity will also increase accordingly.
[0087] Gradually reduce the resolution: Starting from the original resolution of the current video frame, the image resolution is gradually reduced through image downsampling techniques, including but not limited to bilinear interpolation, bicubic interpolation, and Gaussian filtering and downsampling, to generate a series of images with different resolution levels. These images are arranged from high to low resolution to form a resolution pyramid.
[0088] S302: In each layer of the resolution pyramid, a sub-block belonging to the motion area is selected.
[0089] In this embodiment, the specific implementation process of step S302 is as follows:
[0090] Locating the motion area: Based on the result of the preliminary motion detection in step S1, locate the motion area in the current video frame.
[0091] Mapping to each level of the pyramid: Map the position information of the motion area to each level of the resolution pyramid to determine which sub-blocks belong to the motion area in each level.
[0092] Select sub-blocks: In each layer of the pyramid, select sub-blocks belonging to the motion area according to the mapping results. These sub-blocks will be used as input for subsequent multi-scale processing.
[0093] S303: Perform multi-scale processing on all sub-blocks belonging to the motion area in each selected pyramid layer.
[0094] In this embodiment, the specific implementation process of step S303 is as follows:
[0095] Apply multi-scale analysis method: Apply pyramid decomposition method to perform multi-scale processing on all sub-blocks belonging to the motion area in each layer of the selected pyramid, which can decompose the sub-blocks into components of different scales, thereby capturing the feature information of the sub-blocks at different scales.
[0096] Feature extraction: In the process of multi-scale processing, feature information of sub-blocks at different scales is extracted, such as edges, textures, motion vectors, etc. This feature information will be used for subsequent coding optimization and deep learning prediction.
[0097] S304: Fusing all sub-blocks of the motion region in each layer of the pyramid after multi-scale processing to obtain the motion region after multi-scale processing.
[0098] In this embodiment, the specific implementation process of step S304 is as follows:
[0099] Merge processing results: Merge the processing results of all sub-blocks of the motion area in each layer of the pyramid that have undergone multi-scale processing, which includes integrating and fusing feature information at different scales to form a more comprehensive and in-depth understanding of the motion area.
[0100] Generate multi-scale processed motion region: Based on the merged processing results, generate a multi-scale processed motion region image or feature map, which will serve as the input for subsequent deep learning prediction and coding optimization.
[0101] Specifically, step S4 includes the following steps:
[0102] S401: Using a pre-trained deep learning model to perform feature extraction on the input multi-scale processed motion area to obtain foreground features and background features in the motion area, wherein the deep learning model is a neural network model trained based on a data set of labeled video frames.
[0103] In this embodiment, the specific implementation process of step S401 is as follows:
[0104] Input the motion region after multi-scale processing: The motion region after multi-scale processing in step S3 is used as input and sent to the pre-trained deep learning model.
[0105] Pre-trained deep learning model: This model is a neural network model trained based on a dataset of labeled video frames (including foreground and background labeling information).
[0106] Feature extraction: Use a deep learning model to extract features from the input multi-scale motion area. The model will automatically capture and learn the difference between the foreground and background, and generate corresponding feature representations.
[0107] Output foreground features and background features: After being processed by the deep learning model, the foreground features and background features in the motion area are output, which will be used for subsequent foreground and background separation.
[0108] S402: Generate a probability map according to the foreground features and the background features, and obtain the foreground and the background based on the probability map, wherein the probability map is used to represent the probability that the sub-block in the motion region belongs to the foreground and the background.
[0109] In this embodiment, the specific implementation process of step S402 is as follows:
[0110] Generate a probability map: Generate a probability map based on the foreground features and background features extracted in step S401. The probability map is used to characterize the probability that each sub-block in the motion area belongs to the foreground or background. The closer the probability value is to 1, the higher the confidence that the pixel belongs to the corresponding category.
[0111] Threshold processing: Set a threshold (such as 0.5) and perform threshold processing on the probability map. Pixels with probability values greater than the threshold are classified as foreground, and pixels with probability values less than the threshold are classified as background.
[0112] Obtaining foreground and background: After threshold processing, the foreground and background in the motion area are obtained, and these areas will be used for subsequent differential allocation of quantization parameters.
[0113] S403: Differentiately allocate quantization parameters to the obtained foreground and background to obtain a first quantization parameter and a second quantization parameter corresponding to the foreground and background respectively.
[0114] In this embodiment, the specific implementation process of step S403 is as follows:
[0115] Analyze foreground and background characteristics: Analyze the foreground and background obtained in step S402 to obtain characteristics of the foreground and background, which include motion complexity, texture complexity, and brightness contrast.
[0116] Determine the quantization parameters: Based on the characteristics of the foreground and background, as well as the goals of video encoding (such as compression ratio, video quality, etc.), determine the corresponding first quantization parameter and second quantization parameter respectively. For foreground areas with high motion complexity and rich texture details, a lower quantization parameter can be used to maintain a high encoding quality; while for relatively static background areas with simple textures, a higher quantization parameter can be used to save encoding bit rate.
[0117] Differential allocation: The determined first quantization parameter and the second quantization parameter are applied to the encoding process of the foreground and background areas respectively, so that the encoding efficiency can be further optimized while maintaining the overall video quality.
[0118] Specifically, step S5 includes the following steps:
[0119] S501: Configure corresponding encoding parameters according to the first quantization parameter and the second quantization parameter.
[0120] In this embodiment, the specific implementation process of step S501 is as follows:
[0121] Based on quantization parameter configuration: According to the first quantization parameter and the second quantization parameter determined in step S403, the corresponding encoding parameters are configured for the foreground and background areas respectively. These encoding parameters may include but are not limited to prediction mode (intra-frame prediction, inter-frame prediction), motion vector, number of reference frames, quantization step size of transform coefficients, etc.
[0122] Optimization and adjustment: During the process of configuring encoding parameters, optimization and adjustment are performed according to the characteristics of the video content and encoding requirements. For example, for the foreground area with intense motion, more reference frames and a more sophisticated motion search algorithm can be configured; for the background area with simple texture, a larger quantization step size and fewer reference frames can be used to reduce the encoding complexity.
[0123] S502: Using the configured first quantization parameter and the corresponding encoding parameter, the encoder encodes the foreground of the moving area, and using the configured second quantization parameter and the corresponding encoding parameter, the encoder encodes the background of the non-moving area and the moving area.
[0124] In this embodiment, the specific implementation process of step S502 is as follows:
[0125] Foreground coding: Use the configured first quantization parameter and corresponding coding parameters to encode the foreground area through the encoder. The encoder will compress the pixel values, motion information, etc. of the foreground area according to the configured parameters to generate the coded data of the foreground area.
[0126] Non-motion area and background encoding: Similarly, the non-motion area and the background of the motion area are encoded by the encoder using the configured second quantization parameter and the corresponding encoding parameter. Since the characteristics and encoding parameters of the background area are different from those of the foreground area, the encoding process will also be different.
[0127] Parallel processing: To improve coding efficiency, the foreground and background areas can be encoded in parallel.
[0128] S503: Perform fusion processing on the encoded motion area and non-motion area to obtain video frame encoding data.
[0129] In this embodiment, the specific implementation process of step S503 is as follows:
[0130] Data integration: Integrate the encoded data of the motion area and the non-motion area, which includes merging the encoded bit streams of the two into a complete video frame encoded data.
[0131] Synchronous processing: During the integration process, it is necessary to ensure that the encoded data of the motion area and the non-motion area remain synchronized in time and space.
[0132] Output coded data: After integration is completed, the coded data of the video frame is output. This data can be used for subsequent application scenarios such as storage, transmission, or decoding and playback.
[0133] The working principle of an intelligent video coding optimization method provided by the present invention is as follows:
[0134] First, the technical solution uses the motion vector estimation method to use the pixel changes between the current video frame and its adjacent frames to preliminarily detect the moving area and non-moving area in the current frame. The key to this step is to capture the dynamic changes between video frames and provide the basis for motion information for subsequent processing.
[0135] Next, the current video frame is segmented into multiple smaller sub-blocks according to the selected video coding standard. This segmentation helps to process the video content more finely, especially for complex scenes or high dynamic range videos, and can make more efficient use of coding resources.
[0136] Then, a resolution pyramid of the current video frame is constructed. Resolution pyramid is a multi-scale representation method that creates a series of images of different scales by gradually reducing the image resolution. In this step, special attention is paid to the sub-blocks of the motion area and they are processed at multiple scales. Multi-scale processing can reveal motion features at different scales and enhance the accuracy and robustness of motion detection.
[0137] Subsequently, the multi-scale processed motion region is used as input to a pre-trained deep learning model. After being trained on a large number of annotated video frames, the deep learning model has learned how to identify the foreground (such as moving objects) and background in the motion region, and outputs corresponding quantization parameters based on their characteristics. Specifically, the model outputs two quantization parameters: the first quantization parameter is for the foreground region, and its value is small, aiming to maintain image quality and details; the second quantization parameter is for the background region, and its value is large, in order to improve the compression ratio within an acceptable range.
[0138] Finally, based on the quantization parameters output by the deep learning model, the encoder performs differential encoding on the foreground and background in the moving area and the non-moving area. The foreground area is finely encoded using a smaller quantization parameter to retain key information and details; while the background area and non-moving area are coarsely encoded using a larger quantization parameter to reduce data redundancy and improve compression efficiency. This differential encoding strategy not only ensures the overall quality of the video, but also significantly improves the encoding efficiency, making video data more efficient during storage and transmission.
[0139] Embodiment 2
[0140] The present invention also provides an intelligent video coding optimization system for executing an intelligent video coding optimization method, referring to Figure 2 As shown, the optimization system comprises:
[0141] The motion region acquisition module 100 is used to perform preliminary motion detection on the current video frame by a motion vector estimation method based on adjacent video frames of the current video frame in the video to be encoded, and obtain a motion region and a non-motion region belonging to the current video frame.
[0142] The segmentation module 200 is used to segment the current video frame based on the selected coding standard to obtain a plurality of sub-blocks.
[0143] The multi-scale processing module 300 is used to construct a resolution pyramid of the current video frame, and perform multi-scale processing on the multiple sub-blocks belonging to the motion region based on the resolution pyramid to obtain the motion region after multi-scale processing.
[0144] The prediction module 400 is used to take the motion area after multi-scale processing as input, use the pre-trained deep learning model to predict the current video frame, and obtain the foreground and background of the motion area and the first quantization parameter and the second quantization parameter corresponding to the foreground and background, respectively, wherein the first quantization parameter is less than the second quantization parameter.
[0145] The encoding module 500 is used to encode the foreground and background areas in the motion area using the first quantization parameter and the second quantization parameter, respectively, and to encode the non-motion area using the second quantization parameter.
[0146] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0147] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.
[0148] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or includes elements that are inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.
Claims
1. An intelligent video coding optimization method, characterized in that: The optimization method includes the following steps: S1: Based on the adjacent video frames of the current video frame in the video to be encoded, preliminary motion detection is performed on the current video frame by a motion vector estimation method to obtain a motion region and a non-motion region belonging to the current video frame; S2: Segment the current video frame based on a selected coding standard to obtain a plurality of sub-blocks; S3: constructing a resolution pyramid of the current video frame, and performing multi-scale processing on the multiple sub-blocks belonging to the motion area based on the resolution pyramid to obtain the motion area after multi-scale processing; S4: taking the motion region after multi-scale processing as input, and using the pre-trained deep learning model to predict the current video frame, to obtain the foreground and background of the motion region and the first quantization parameter and the second quantization parameter corresponding to the foreground and the background, respectively, wherein the first quantization parameter is less than the second quantization parameter; S5: Encode the foreground and background areas in the motion area respectively using the first quantization parameter and the second quantization parameter, and encode the non-motion area using the second quantization parameter.
2. The intelligent video coding optimization method according to claim 1, characterized in that: Step S1 includes the following steps: S101: using a feature point detection algorithm to identify feature points in a current frame and adjacent video frames of a video to be encoded, determining a correspondence between the identified feature points between frames, and calculating a motion vector of each feature point between the current frame and adjacent video frames; S102: identifying and locating a motion region in the current video frame by analyzing the direction and size change of the motion vector, and obtaining a non-motion region by elimination method; S103: Marking the obtained motion area and non-motion area with visual markers, wherein the visual markers include boundary markers and color markers.
3. The intelligent video coding optimization method according to claim 2, characterized in that: Step S2 includes the following steps: S201: Initialize parameters according to the selected coding standard; S202: Divide the current video frame into a plurality of sub-blocks using a block division mechanism corresponding to the selected coding standard.
4. The intelligent video coding optimization method according to claim 3, characterized in that: Step S3 includes the following steps: S301: Starting from the original resolution of the current video frame, a resolution pyramid with different resolution levels is constructed by gradually reducing the resolution; S302: In each layer of the resolution pyramid, select a sub-block belonging to the motion area; S303: performing multi-scale processing on all sub-blocks belonging to the motion area in each selected pyramid layer; S304: Fusing all sub-blocks of the motion region in each layer of the pyramid after multi-scale processing to obtain the motion region after multi-scale processing.
5. The intelligent video coding optimization method according to claim 4, characterized in that: Step S4 includes the following steps: S401: extracting features from the input multi-scale processed motion region using a pre-trained deep learning model to obtain foreground features and background features in the motion region, wherein the deep learning model is a neural network model trained based on a data set of annotated video frames; S402: Generate a probability map according to the foreground feature and the background feature, and obtain the foreground and the background based on the probability map, wherein the probability map is used to represent the probability that the sub-block in the motion area belongs to the foreground and the background; S403: Differentiately allocate quantization parameters to the obtained foreground and background to obtain a first quantization parameter and a second quantization parameter corresponding to the foreground and background respectively.
6. The intelligent video coding optimization method according to claim 5, characterized in that: Step S5 includes the following steps: S501: configuring corresponding encoding parameters according to a first quantization parameter and a second quantization parameter; S502: using the configured first quantization parameter and the corresponding encoding parameter, encoding the foreground of the motion area through an encoder, and using the configured second quantization parameter and the corresponding encoding parameter, encoding the background of the non-motion area and the motion area through an encoder; S503: Perform fusion processing on the encoded motion area and non-motion area to obtain video frame encoding data.
7. An intelligent video coding optimization system, used to execute an intelligent video coding optimization method according to any one of claims 1 to 6, characterized in that: The optimization system comprises: A motion region acquisition module is used to perform preliminary motion detection on the current video frame by a motion vector estimation method based on adjacent video frames of the current video frame in the video to be encoded, and obtain a motion region and a non-motion region belonging to the current video frame; A segmentation module, configured to segment the current video frame based on a selected coding standard to obtain a plurality of sub-blocks; A multi-scale processing module, used for constructing a resolution pyramid of the current video frame, and performing multi-scale processing on the multiple sub-blocks belonging to the motion region based on the resolution pyramid to obtain the motion region after the multi-scale processing; A prediction module, configured to use the motion region after multi-scale processing as input, and predict the current video frame using a pre-trained deep learning model to obtain a foreground and a background of the motion region and a first quantization parameter and a second quantization parameter corresponding to the foreground and the background, respectively, wherein the first quantization parameter is less than the second quantization parameter; The encoding module is used to encode the foreground and background areas in the motion area using the first quantization parameter and the second quantization parameter respectively, and to encode the non-motion area using the second quantization parameter.
Citation Information
Patent Citations
Method, device and system for regional division / coding of image
CN101882316A
Method and system for detecting picture offset of camera device
CN102609957A
Method and device of motion feature extraction and video processing device
CN103067646A
Target tracking method based on multi-time-step pyramid codec
CN112288776A
Motion blur generation method and device and terminal equipment
CN114419073A
Cited By
Video content adaptive optimization system based on deep learning
CN119363994A
Decoupling type recording and broadcasting method and system based on key frame index and difference additional recording
CN121691793A
