Method and apparatus for hevc coding based on semantic enhancement

Abstract

The application provides a HEVC coding method and device based on semantic enhancement, comprising: obtaining original coding and a video to be coded; inputting image frames of the video to be coded into a pre-trained CTU segmentation result prediction model in sequence to obtain a CTU segmentation prediction result; coding the video to be coded according to the original coding; performing a rate-distortion optimization process in the coding process according to the CTU segmentation prediction result to obtain target coding; wherein the CTU segmentation result prediction model is obtained by training a large amount of video file samples based on a deep neural network. The application uses a deep neural network to extract semantic information to predict a CTU segmentation result, and optimizes a rate-distortion optimization traversal process according to a prediction result of the neural network, thereby optimizing a coding method, saving coding time while not changing coding performance. Meanwhile, the application is combined with an existing code, thereby saving deployment cost.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a HEVC encoding method and device based on semantic enhancement. Background Art

[0002] High Efficiency Video Coding (HEVC) is the latest video coding and decoding standard, released in 2013 as the successor to the H.264 / AVC standard. With the increasing demand for high-resolution video, HEVC introduced several new coding techniques, such as the Coding Tree Unit (CTU). This more complex coding technique and larger coding units (CTUs) enabled HEVC to achieve a bitrate reduction of approximately 50% compared to its predecessor, but the computational complexity of the encoding process increased significantly. Within the CTU, rate-distortion optimization must be performed to determine the optimal coding unit depth and prediction unit mode, resulting in high encoding time and computational overhead, making the deployment of HEVC coding expensive in real-time applications.

[0003] Since the release of HEVC, there has been a lot of work on optimizing different aspects of encoding to reduce encoding complexity. For example, this involves dynamically adjusting encoding settings for each Group of Pictures (GOP) during the encoding process, and reducing the depth of the Largest Coding Unit (LCU) by solving a distortion-complexity equation based on the concept of subjective weights to control encoding complexity. In HEVC, CTU segmentation technology was introduced, which significantly reduces the bitrate through finer image segmentation, but also significantly increases encoding complexity. In the HEVC standard reference software (HEVC test model, HM), CTU segmentation accounts for approximately 80% of the encoding time, so there is a lot of work aimed at simplifying the CTU segmentation process to reduce encoding complexity.

[0004] Generally speaking, the work of optimizing CTU segmentation can be divided into two categories: heuristic methods and learning-based methods. Heuristic methods refer to determining the CTU segmentation in advance based on the intermediate features of the encoding process to avoid the encoder traversing all possible segmentation.

[0005] Because heuristic methods often suffer from randomness in feature selection, learning-based methods have been developed to address these issues. This method can learn from CTU segmentation data from a large amount of video and form an optimal model for predicting CTU segmentation results in HEVC. While learning-based CTU segmentation prediction methods increase computational and storage resource consumption due to the model and neural network components compared to heuristic methods, they also achieve higher prediction accuracy and better encoding performance.

[0006] However, existing technologies cannot simultaneously take into account computational complexity, deployment costs, and encoding performance. More complex neural networks can better extract features from different types of videos, but this will also lead to corresponding increases in deployment costs and scale, thereby increasing training and computational complexity. Simpler neural networks have lower computational complexity themselves, but may find it difficult to accurately extract features from videos. Summary of the Invention

[0007] The present invention provides a HEVC encoding method and device based on semantic enhancement, which is used to solve the defect in the prior art that it is impossible to take into account the computational complexity, deployment cost and coding performance at the same time, and realize HEVC encoding with low computational complexity, low deployment cost and good performance.

[0008] The present invention provides a HEVC encoding method based on semantic enhancement, comprising:

[0009] Get the original encoded video and the video to be encoded;

[0010] Inputting the image frames of the video to be encoded into a pre-trained CTU segmentation result prediction model in sequence to obtain a CTU segmentation prediction result; encoding the video to be encoded according to the original code;

[0011] Performing a rate-distortion optimization process in the encoding process according to the CTU segmentation prediction result to obtain a target encoding;

[0012] The CTU segmentation result prediction model is obtained by training a deep neural network using massive video file samples.

[0013] According to a semantic enhancement-based HEVC encoding method provided by the present invention, the image frames of the video to be encoded are sequentially input into a pre-trained CTU segmentation result prediction model to obtain a CTU segmentation prediction result, specifically comprising:

[0014] Divide the CTU of the image frame into a preset number of CUs;

[0015] Convolution and pooling are performed on the CTU to obtain a first tensor; convolution and pooling are performed on the CU to be predicted to obtain a second tensor;

[0016] Concatenate the first tensor and the second tensor to obtain a target tensor;

[0017] The target tensor is input into the CTU segmentation result prediction model to obtain the CU segmentation prediction result of the current CU to be predicted, and the CTU segmentation prediction result is obtained according to all the CU segmentation prediction results.

[0018] According to a semantic enhancement-based HEVC encoding method provided by the present invention, performing the rate-distortion optimization process in the encoding process according to the CTU segmentation prediction result to obtain the target encoding specifically includes:

[0019] When encoding the current frame of the video to be encoded according to the original encoding, reading the CTU segmentation prediction result of the current frame to obtain a preset number of CU segmentation prediction results;

[0020] During encoding of each CTU of the current frame, recording a current segmentation depth;

[0021] When the current segmentation depth is less than the CU segmentation prediction result, skipping the CU rate-distortion optimization of the current segmentation depth;

[0022] When the current segmentation depth is equal to the CU segmentation prediction result, calculating the CU rate-distortion optimization of the current segmentation depth and stopping searching for the next segmentation depth;

[0023] When the current segmentation depth is greater than the CU segmentation prediction result, the current CU is jumped out.

[0024] According to a semantic enhancement-based HEVC encoding method provided by the present invention, a CTU segmentation result prediction model is obtained by training a deep neural network with massive video file samples, specifically comprising:

[0025] Get a large number of video file samples,

[0026] Modifying the original code, running the modified original code to encode the video file sample, and obtaining optimal segmentation information of a current CTU, as well as a current frame number and a current CTU number during the encoding process;

[0027] Randomly extracting approximately a preset percentage of frame images from the video file sample as frame data samples, and extracting CTU optimal segmentation information of the frame data samples from a storage location of the optimal segmentation information of the current CTU as a label of the frame data sample;

[0028] Obtaining a training data sample according to all the current frame numbers, the current CTU numbers, the optimal segmentation information of the current CTU, the frame data samples, and the labels of the frame data samples of each video file sample, thereby obtaining a sample data set;

[0029] A prediction model is constructed based on a deep neural network, and the prediction model is trained using the sample data set until a preset end condition is reached to obtain the CTU segmentation result prediction model.

[0030] According to a semantic enhancement-based HEVC encoding method provided by the present invention, the prediction model includes 2 maximum pooling layers and 3 fully connected layers.

[0031] According to the HEVC encoding method based on semantic enhancement provided by the present invention, the preset number is 4.

[0032] The present invention also provides a HEVC encoding device based on semantic enhancement, comprising:

[0033] An acquisition unit, used to acquire the original encoded video and the video to be encoded;

[0034] The encoding and prediction unit is configured to sequentially input the image frames of the video to be encoded into a pre-trained CTU segmentation result prediction model to obtain a CTU segmentation prediction result; and encode the video to be encoded according to the original encoding;

[0035] an optimization unit, configured to perform a rate-distortion optimization process in the encoding process according to the CTU segmentation prediction result to obtain a target encoding;

[0036] The CTU segmentation result prediction model is obtained by training a deep neural network using massive video file samples.

[0037] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the HEVC encoding method based on semantic enhancement as described above is implemented.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described HEVC encoding methods based on semantic enhancement.

[0039] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the HEVC encoding methods based on semantic enhancement as described above.

[0040] The HEVC encoding method and device based on semantic enhancement provided by the present invention obtain the original code and the video to be encoded; input the image frames of the video to be encoded into a pre-trained CTU segmentation result prediction model in sequence to obtain a CTU segmentation prediction result; encode the video to be encoded according to the original code; execute the rate-distortion optimization process in the encoding process according to the CTU segmentation prediction result to obtain the target code; wherein the CTU segmentation result prediction model is obtained by training a large number of video file samples based on a deep neural network. The present invention uses a deep neural network to extract semantic information to predict the CTU segmentation result, optimizes the rate-distortion optimization traversal process according to the prediction result of the neural network, thereby optimizing the encoding method and saving encoding time without changing the encoding performance. At the same time, because the present invention is combined with the existing code, it can be easily deployed and applied, saving deployment costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 This is one of the flow charts of the HEVC encoding method based on semantic enhancement provided by the present invention;

[0043] Figure 2 Schematic diagram of the architecture of the CTU segmentation result prediction model in the HEVC encoding method based on semantic enhancement provided by the present invention;

[0044] Figure 3 This is the second flow chart of the HEVC encoding method based on semantic enhancement provided by the present invention;

[0045] Figure 4 It is a structural diagram of the HEVC encoding device based on semantic enhancement provided by the present invention;

[0046] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention.

[0047] Reference numerals:

[0048] 410: Acquisition unit; 420: Encoding and prediction unit; 430: Optimization unit. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0050] The following combination Figure 1-Figure 3 Describe the HEVC encoding method based on semantic enhancement of the present invention, Figure 1 This is one of the flow charts of the HEVC encoding method based on semantic enhancement provided by the present invention, such as Figure 1 As shown, the method includes:

[0051] Step 110: Obtain the original encoded video and the video to be encoded.

[0052] It should be noted that the original code obtained is the original code that can be executed by the HEVC (High Efficiency Video Coding) standard reference software (HEVC test model, HM). The obtained video to be encoded includes video in a video conferencing scene.

[0053] The present invention is combined with the code of HEVC standard software to facilitate deployment and application, thereby reducing deployment costs.

[0054] Step 120: inputting the image frames of the video to be encoded into a pre-trained CTU segmentation result prediction model in sequence to obtain a CTU segmentation prediction result; and encoding the video to be encoded according to the original encoding.

[0055] It should be noted that the CTU segmentation result prediction model extracts features from each CTU (Coding Tree Unit) of each image frame of the input video (video to be encoded) and outputs a vector as the CTU segmentation prediction result. In some embodiments, the length of the output vector is 16. The basic architecture of the CTU segmentation result prediction model is as follows: Figure 2 shown.

[0056] In the process of performing segmentation prediction, in some embodiments, sequentially inputting the image frames of the to-be-encoded video into a pre-trained CTU segmentation result prediction model to obtain a CTU segmentation prediction result specifically includes:

[0057] Divide the CTU of the image frame into a preset number of CUs;

[0058] Convolution and pooling are performed on the CTU to obtain a first tensor; convolution and pooling are performed on the CU to be predicted to obtain a second tensor;

[0059] Concatenate the first tensor and the second tensor to obtain a target tensor;

[0060] The target tensor is input into the CTU segmentation result prediction model to obtain the CU segmentation prediction result of the current CU to be predicted, and the CTU segmentation prediction result is obtained according to all the CU segmentation prediction results.

[0061] Specifically, in the existing technology, in each 64×64 CTU, the HEVC encoder will continuously divide the CTU into four parts to the minimum 8×8 coding unit (CU). During this process, the rate-distortion cost is calculated for each possible division. This traversal process will take a lot of time.

[0062] Based on this, this embodiment uses a CTU segmentation prediction model to divide each CTU into a preset number of CUs, which are larger than those used in the prior art. This model then generates a preset number of label values ​​for each prediction, each of which represents a CU segmentation prediction result. In the vector output by the CTU segmentation prediction model, the CU segmentation prediction result is part of the CTU segmentation prediction result.

[0063] In a specific embodiment, the preset number is 4, that is, each CTU is divided into 4 32×32 CUs.

[0064] It should be noted that only predicting CU segmentation is insufficient, because the HM performs rate-distortion optimization on all 64×64 CTUs. If the prediction of whether the 64×64 CTU is segmented is skipped, the coding complexity calculated by the HM remains high. To address this issue, the present invention performs convolution and pooling on the entire 64×64 CTU to obtain a first tensor, and simultaneously performs convolution and pooling on the current CU to be predicted to obtain a second tensor. The two tensors are then concatenated for subsequent neural network processing to obtain the CU segmentation prediction result for each CU. The CTU segmentation prediction result is obtained based on all CU segmentation prediction results. In a specific embodiment, each CTU is divided into four 32×32 CUs, resulting in a tensor dimension of 16×16.

[0065] Specifically, the 64×64 CTU is first divided into four 32×32 CTUs. One of these is then convolved and pooled with the original CTU, resulting in 4×4, 128-channel data. This 4×4, 128-channel data is then fed into three fully-connected layers to produce a 1×4 prediction output. Next, the four 32×32 CTUs obtained from the segmentation are subjected to the aforementioned operations, resulting in four 1×4 outputs concatenated as the final 1×16 CTU segmentation prediction result.

[0066] The above processing of the input CTU not only ensures the prediction of CUs of all depths, but also makes the neural network only need to output a preset number of prediction values ​​at a time, reducing network complexity and training costs.

[0067] It should be noted that the CTU segmentation result prediction model is based on a deep neural network trained using massive video file samples.

[0068] In some embodiments, the CTU segmentation result prediction model is obtained by training a large number of video file samples based on a deep neural network, specifically including:

[0069] Get a large number of video file samples,

[0070] Modifying the original code, running the modified original code to encode the video file sample, and obtaining optimal segmentation information of a current CTU, as well as a current frame number and a current CTU number during the encoding process;

[0071] Randomly extracting approximately a preset percentage of frame images from the video file sample as frame data samples, and extracting CTU optimal segmentation information of the frame data samples from a storage location of the optimal segmentation information of the current CTU as a label of the frame data sample;

[0072] Obtaining a training data sample according to all the current frame numbers, the current CTU numbers, the optimal segmentation information of the current CTU, the frame data samples, and the labels of the frame data samples of each video file sample, thereby obtaining a sample data set;

[0073] A prediction model is constructed based on a deep neural network, and the prediction model is trained using the sample data set until a preset end condition is reached to obtain the CTU segmentation result prediction model.

[0074] Specifically, a sample dataset is first constructed. It is understood that the compressCtu() function in the HM code calculates the optimal CTU segmentation. During this process, the optimal CTU segmentation is stored in a variable. Outputting this variable to a file obtains the optimal segmentation information for the current CTU, represented by a segmentation matrix. In one specific embodiment, this information is represented by a 16×16 segmentation matrix. Similarly, the current frame number and current CTU number can be output to a text file. After obtaining the optimal CTU segmentation file, the segmentation matrix is ​​processed to generate label values ​​for neural network training. In one specific embodiment, this data processing includes converting the 16×16 matrix into a 1×16 vector.

[0075] During the specific implementation process, a video file sample is first obtained. The video file sample can be video file data collected on-site according to the training target, or it can be video file data obtained from a database, and the present invention does not limit this. Accordingly, the format of the video file sample can also be selected according to actual needs. In one specific embodiment, the format of the video file sample is a YUV video file.

[0076] Furthermore, in some embodiments, the massive video file samples are divided into three folders: training, testing, and verification, so as to construct the training set, test set, and verification set data sets. In some embodiments, the file name can be named [video name]_[video width] [video height]_[frame rate].yuv.

[0077] After obtaining a video file sample, the original encoding is modified to extract information. Specifically, the HM software library functions are modified to output the current frame number, current CTU number, and optimal segmentation information for the current CTU, which are saved during the encoding process. A video file sample is encoded using the modified HM to generate an optimal segmentation information file. Frame images are then randomly sampled from the video file sample as data, and the CTU segmentation information for the corresponding frame is extracted from the optimal segmentation information file and saved as a label.

[0078] Specifically, the modified HM is run, video file samples are sequentially input, and the input video file samples are encoded. The optimal CTU partition matrix generated during the process is saved to Partitiontxt. Each frame of the current video file sample is then cropped and numbered to obtain a frame number. In some embodiments, ffmpeg is called to crop each frame of the current video file sample.

[0079] Next, a preset percentage of frames from all cropped frames is randomly selected as frame data samples and saved, along with the frame number and number of CTUs contained in the frame. In one specific embodiment, the sample is saved as v_[video number]_[number]_[number of CTUs].jpg to the corresponding subfolder in the image folder for final output. In one specific embodiment, the preset percentage is 10%.

[0080] Furthermore, the frame image name saved in is read, and the corresponding CTU partition information in Partition.txt is searched accordingly, the CTU partition matrix is ​​converted into a vector, and is saved as a label of the frame data sample. In some embodiments, the CTU partition matrix is ​​16 16 matrix, convert it to 1 The vector of 16 is saved to the corresponding subfolder in the pkl folder for final output.

[0081] The final output is the saved frame data sample and the label of the frame data sample. The frame data sample contains the frame number and the number of CTUs contained in the frame, and the label of the frame data sample contains the CTU optimal segmentation information.

[0082] As encoding proceeds, the current frame numbers of all frames of each video file sample, the current CTU numbers of all CTUs in each frame, and the optimal segmentation information of the current CTU are obtained.

[0083] The constructed sample dataset is input into a CNN neural network for training, thereby obtaining a model capable of predicting CTU segmentation results in HEVC video encoding, until a preset termination condition is met. In one embodiment, the preset termination condition includes stable convergence of the training loss.

[0084] In some embodiments, the prediction model includes two maximum pooling layers and three fully connected layers. In addition, it should be noted that the present invention does not limit the training and parameter optimization process of the CTU segmentation result prediction model, and training can be performed according to actual conditions.

[0085] Furthermore, it should be emphasized that, while the CTU segmentation result prediction model is performing CTU segmentation result prediction, the to-be-encoded video is encoded according to the original encoding.

[0086] Step 130: performing a rate-distortion optimization process in the encoding process according to the CTU segmentation prediction result to obtain a target encoding;

[0087] After obtaining the CTU segmentation prediction result, it is still necessary to use the CTU segmentation prediction result to optimize and replace the rate-distortion optimization process in HEVC, so as to achieve the effect of skipping traversal and saving encoding time.

[0088] Specifically, since the CTU segmentation prediction model is encoding the video to be encoded based on the original encoding while predicting the CTU segmentation results, during the encoding process, when running the modified original encoding, a Python file will first be called to crop each frame of the input video to be encoded, save it in a temporary folder, and then input each frame image into the CTU segmentation prediction model. The trained model will be called for prediction, and the prediction results will also be saved in the temporary folder as a txt file.

[0089] At the same time, the original codec begins segmenting the current CTU, obtaining the current frame number through the added member variables and member functions. The original codec also stores the current CTU number. Using the current frame and CTU numbers, it attempts to read the CTU segmentation prediction results, which are stored in a temporary file and output by the CTU segmentation prediction model. If the read is successful, the next segmentation step begins. If the read is unsuccessful, the codec waits until the read is successful. Note that this waiting period is due to the fact that the original codec and the CTU segmentation prediction model run in parallel.

[0090] After successfully reading the CTU segmentation prediction result, the CTU segmentation rate-distortion optimization function in the original code compares the current CU depth with the predicted depth in the CU segmentation prediction result of the current CU included in the CTU segmentation prediction result. Optimization and replacement are performed based on the comparison result to obtain the target code.

[0091] In some embodiments, performing the rate-distortion optimization process in the encoding process according to the CTU segmentation prediction result to obtain the target encoding specifically includes:

[0092] When encoding the current frame of the video to be encoded according to the original encoding, reading the CTU segmentation prediction result of the current frame to obtain a preset number of CU segmentation prediction results;

[0093] During encoding of each CTU of the current frame, recording a current segmentation depth;

[0094] When the current segmentation depth is less than the CU segmentation prediction result, skipping the CU rate-distortion optimization of the current segmentation depth;

[0095] When the current segmentation depth is equal to the CU segmentation prediction result, calculating the CU rate-distortion optimization of the current segmentation depth and stopping searching for the next segmentation depth;

[0096] When the current segmentation depth is greater than the CU segmentation prediction result, the current CU is jumped out.

[0097] Specifically, after the CTU segmentation prediction result is successfully read, each CTU segmentation prediction result includes a preset number of CU segmentation prediction results, and the current CU segmentation depth is compared with the prediction depth in the CU segmentation prediction result corresponding to the current CU.

[0098] If the current depth is less than the predicted depth, the distortion calculation for the current depth is skipped and segmentation is performed directly. If the current depth is equal to the predicted depth, the distortion measure for the current depth is directly calculated and the search for the next depth is stopped. If the current depth is greater than the predicted depth, the calculation for the current depth is skipped and the search for the next depth is stopped (i.e., segmentation is considered complete).

[0099] It can be understood that when the current depth ≤ the predicted depth, distortion calculation is still performed. This is because after the CTU is segmented and the CU mode is determined, rate-distortion optimization is still required for each CU to determine the optimal PU and TU modes.

[0100] In some embodiments, a horizontal comparison was conducted between the encoding results of the present invention and the prior art based on the same quantization parameter QP on different video sequences. The results showed that the method proposed in the present invention significantly shortened the encoding time in HEVC encoding of video conferencing type videos at the cost of a slight increase in bit rate and a negligible decrease in PSNR.

[0101] The HEVC encoding method based on semantic enhancement provided by the present invention obtains the original code and the video to be encoded; inputs the image frames of the video to be encoded into a pre-trained CTU segmentation result prediction model in sequence to obtain a CTU segmentation prediction result; encodes the video to be encoded according to the original code; executes the rate-distortion optimization process in the encoding process according to the CTU segmentation prediction result to obtain the target code; wherein the CTU segmentation result prediction model is obtained by training a large number of video file samples based on a deep neural network. The present invention uses a deep neural network to extract semantic information to predict the CTU segmentation result, optimizes the rate-distortion optimization traversal process according to the prediction result of the neural network, thereby optimizing the encoding method and saving encoding time without changing the encoding performance. At the same time, because the present invention is combined with the existing code, it can be easily deployed and applied, saving deployment costs.

[0102] The HEVC encoding device based on semantic enhancement provided by the present invention is described below. The HEVC encoding device based on semantic enhancement described below and the HEVC encoding method based on semantic enhancement described above can refer to each other. Figure 4 : is a structural diagram of the HEVC encoding device based on semantic enhancement provided by the present invention, such as Figure 4 As shown, the device includes:

[0103] An acquisition unit 410 is configured to acquire the original encoded video and the video to be encoded;

[0104] The encoding and prediction unit 420 is configured to sequentially input the image frames of the video to be encoded into a pre-trained CTU segmentation result prediction model to obtain a CTU segmentation prediction result; and encode the video to be encoded according to the original code;

[0105] an optimization unit 430, configured to perform a rate-distortion optimization process in the encoding process according to the CTU segmentation prediction result to obtain a target encoding;

[0106] The CTU segmentation result prediction model is obtained by training a deep neural network using massive video file samples.

[0107] According to a semantic enhancement-based HEVC encoding device provided by the present invention, the image frames of the video to be encoded are sequentially input into a pre-trained CTU segmentation result prediction model to obtain a CTU segmentation prediction result, specifically including:

[0108] Divide the CTU of the image frame into a preset number of CUs;

[0109] Convolution and pooling are performed on the CTU to obtain a first tensor; convolution and pooling are performed on the CU to be predicted to obtain a second tensor;

[0110] Concatenate the first tensor and the second tensor to obtain a target tensor;

[0111] The target tensor is input into the CTU segmentation result prediction model to obtain the CU segmentation prediction result of the current CU to be predicted, and the CTU segmentation prediction result is obtained according to all the CU segmentation prediction results.

[0112] According to a semantic enhancement-based HEVC encoding device provided by the present invention, performing the rate-distortion optimization process in the encoding process according to the CTU segmentation prediction result to obtain the target encoding specifically includes:

[0113] When encoding the current frame of the video to be encoded according to the original encoding, reading the CTU segmentation prediction result of the current frame to obtain a preset number of CU segmentation prediction results;

[0114] During encoding of each CTU of the current frame, recording a current segmentation depth;

[0115] When the current segmentation depth is less than the CU segmentation prediction result, skipping the CU rate-distortion optimization of the current segmentation depth;

[0116] When the current segmentation depth is equal to the CU segmentation prediction result, calculating the CU rate-distortion optimization of the current segmentation depth and stopping searching for the next segmentation depth;

[0117] When the current segmentation depth is greater than the CU segmentation prediction result, the current CU is jumped out.

[0118] According to the HEVC encoding device based on semantic enhancement provided by the present invention, the CTU segmentation result prediction model is obtained by training a large number of video file samples based on a deep neural network, specifically comprising:

[0119] Get a large number of video file samples,

[0120] Modifying the original code, running the modified original code to encode the video file sample, and obtaining optimal segmentation information of a current CTU, as well as a current frame number and a current CTU number during the encoding process;

[0121] Randomly extracting approximately a preset percentage of frame images from the video file sample as frame data samples, and extracting CTU optimal segmentation information of the frame data samples from a storage location of the optimal segmentation information of the current CTU as a label of the frame data sample;

[0122] Obtaining a training data sample according to all the current frame numbers, the current CTU numbers, the optimal segmentation information of the current CTU, the frame data samples, and the labels of the frame data samples of each video file sample, thereby obtaining a sample data set;

[0123] A prediction model is constructed based on a deep neural network, and the prediction model is trained using the sample data set until a preset end condition is reached to obtain the CTU segmentation result prediction model.

[0124] According to a semantic enhancement-based HEVC encoding device provided by the present invention, the prediction model includes 2 maximum pooling layers and 3 fully connected layers.

[0125] According to a semantic enhancement-based HEVC encoding device provided by the present invention, the preset number is 4.

[0126] The HEVC encoding device based on semantic enhancement provided by the present invention obtains the original code and the video to be encoded; inputs the image frames of the video to be encoded into a pre-trained CTU segmentation result prediction model in sequence to obtain a CTU segmentation prediction result; encodes the video to be encoded according to the original code; executes the rate-distortion optimization process in the encoding process according to the CTU segmentation prediction result to obtain the target code; wherein the CTU segmentation result prediction model is obtained by training a large number of video file samples based on a deep neural network. The present invention uses a deep neural network to extract semantic information to predict the CTU segmentation result, optimizes the rate-distortion optimization traversal process according to the prediction result of the neural network, thereby optimizing the encoding method and saving encoding time without changing the encoding performance. At the same time, because the present invention is combined with the existing code, it can be easily deployed and applied, saving deployment costs.

[0127] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the HEVC encoding method based on semantic enhancement, which includes: obtaining the original code and the video to be encoded; inputting the image frames of the video to be encoded into a pre-trained CTU segmentation result prediction model in sequence to obtain a CTU segmentation prediction result; encoding the video to be encoded according to the original code; performing the rate-distortion optimization process in the encoding process according to the CTU segmentation prediction result to obtain the target code; wherein the CTU segmentation result prediction model is obtained by training a deep neural network using massive video file samples.

[0128] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0129] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the semantically enhanced HEVC encoding method provided by the above methods, the method including: obtaining the original code and the video to be encoded; inputting the image frames of the video to be encoded into a pre-trained CTU segmentation result prediction model in sequence to obtain a CTU segmentation prediction result; encoding the video to be encoded according to the original code; executing the rate-distortion optimization process in the encoding process according to the CTU segmentation prediction result to obtain the target code; wherein, the CTU segmentation result prediction model is obtained by training a deep neural network using massive video file samples.

[0130] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the semantically enhanced HEVC encoding method provided by the above-mentioned methods, the method comprising: obtaining the original code and the video to be encoded; inputting the image frames of the video to be encoded into a pre-trained CTU segmentation result prediction model in sequence to obtain a CTU segmentation prediction result; encoding the video to be encoded according to the original code; executing the rate-distortion optimization process in the encoding process according to the CTU segmentation prediction result to obtain the target code; wherein, the CTU segmentation result prediction model is obtained by training a deep neural network using massive video file samples.

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0132] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A HEVC encoding method based on semantic enhancement, characterized in that: include: Get the original encoded video and the video to be encoded; Inputting the image frames of the video to be encoded into a pre-trained CTU segmentation result prediction model in sequence to obtain a CTU segmentation prediction result; Encoding the video to be encoded according to the original encoding; Performing a rate-distortion optimization process in the encoding process according to the CTU segmentation prediction result to obtain a target encoding specifically includes: When encoding the current frame of the video to be encoded according to the original encoding, reading the CTU segmentation prediction result of the current frame to obtain a preset number of CU segmentation prediction results; During encoding of each CTU of the current frame, recording a current segmentation depth; When the current segmentation depth is less than the CU segmentation prediction result, skipping the CU rate-distortion optimization of the current segmentation depth; When the current segmentation depth is equal to the CU segmentation prediction result, calculating the CU rate-distortion optimization of the current segmentation depth and stopping searching for the next segmentation depth; When the current segmentation depth is greater than the CU segmentation prediction result, jump out of the current CU; The CTU segmentation result prediction model is obtained by training a deep neural network using massive video file samples.

2. The HEVC encoding method based on semantic enhancement according to claim 1, characterized in that The step of sequentially inputting the image frames of the video to be encoded into a pre-trained CTU segmentation result prediction model to obtain a CTU segmentation prediction result specifically includes: Divide the CTU of the image frame into a preset number of CUs; Convolution and pooling are performed on the CTU to obtain a first tensor; convolution and pooling are performed on the CU to be predicted to obtain a second tensor; Concatenate the first tensor and the second tensor to obtain a target tensor; The target tensor is input into the CTU segmentation result prediction model to obtain the CU segmentation prediction result of the current CU to be predicted, and the CTU segmentation prediction result is obtained according to all the CU segmentation prediction results.

3. The HEVC encoding method based on semantic enhancement according to claim 1, characterized in that The CTU segmentation result prediction model is obtained by training a large number of video file samples based on a deep neural network, specifically including: Get a large number of video file samples, Modifying the original code, running the modified original code to encode the video file sample, and obtaining optimal segmentation information of a current CTU, as well as a current frame number and a current CTU number during the encoding process; Randomly extracting approximately a preset percentage of frame images from the video file sample as frame data samples, and extracting CTU optimal segmentation information of the frame data samples from a storage location of the optimal segmentation information of the current CTU as a label of the frame data sample; Obtaining a training data sample according to all the current frame numbers, the current CTU numbers, the optimal segmentation information of the current CTU, the frame data samples, and the labels of the frame data samples of each video file sample, thereby obtaining a sample data set; A prediction model is constructed based on a deep neural network, and the prediction model is trained using the sample data set until a preset end condition is reached to obtain the CTU segmentation result prediction model.

4. The HEVC encoding method based on semantic enhancement according to claim 1, wherein The prediction model includes 2 maximum pooling layers and 3 fully connected layers.

5. The HEVC encoding method based on semantic enhancement according to claim 2, wherein The preset number is 4.

6. A HEVC encoding device based on semantic enhancement, characterized in that include: An acquisition unit, used to acquire the original encoded video and the video to be encoded; The encoding and prediction unit is configured to sequentially input the image frames of the video to be encoded into a pre-trained CTU segmentation result prediction model to obtain a CTU segmentation prediction result; and encode the video to be encoded according to the original encoding; An optimization unit is configured to perform a rate-distortion optimization process in the encoding process according to the CTU segmentation prediction result to obtain a target encoding, specifically comprising: When encoding the current frame of the video to be encoded according to the original encoding, reading the CTU segmentation prediction result of the current frame to obtain a preset number of CU segmentation prediction results; During encoding of each CTU of the current frame, recording a current segmentation depth; When the current segmentation depth is less than the CU segmentation prediction result, skipping the CU rate-distortion optimization of the current segmentation depth; When the current segmentation depth is equal to the CU segmentation prediction result, calculating the CU rate-distortion optimization of the current segmentation depth and stopping searching for the next segmentation depth; When the current segmentation depth is greater than the CU segmentation prediction result, jump out of the current CU; The CTU segmentation result prediction model is obtained by training a deep neural network using massive video file samples.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the HEVC encoding method based on semantic enhancement according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the HEVC encoding method based on semantic enhancement according to any one of claims 1 to 5 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the HEVC encoding method based on semantic enhancement according to any one of claims 1 to 5 is implemented.

Images