Coding method, terminal and storage medium

Abstract

The embodiment of the present application discloses a coding method, a terminal and a storage medium, the method comprising: determining a saliency map corresponding to a video sequence to be coded; determining a first coding block and a second coding block corresponding to the saliency map according to a preset pixel threshold; wherein the preset pixel threshold is used to divide the foreground and the background; the first coding block is a coding block corresponding to a foreground area in the saliency map; the second coding block is a coding block corresponding to a background area in the saliency map; calculating a first QP compensation value of the first coding block and a second QP compensation value of the second coding block; encoding the video sequence to be coded according to the first QP compensation value and the second QP compensation value to generate a bitstream.

Description

Technical Field

[0001] The present invention relates to the field of coding technology, and in particular to a coding method, a terminal and a storage medium. Background Art

[0002] At present, in the video encoding scenario with fixed bit rate or low bit rate, the region of interest (ROI) is mainly determined through face detection and body detection, and after the quantization parameter (QP) is set, the ROI area is encoded according to the allocated bit rate.

[0003] However, since the generalization of ROI region detection technology is not high and the QP value setting method does not fit the ROI region, the common encoding method cannot guarantee the encoding quality of the ROI region and has the defect of unclear boundary between the ROI region and the non-ROI region. Summary of the invention

[0004] The embodiments of the present application provide a coding method, a terminal, and a storage medium, which can improve the coding quality of an ROI area and make the boundary between the ROI area and the non-ROI area clearer.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an encoding method, the method comprising:

[0007] Determining a saliency map corresponding to a video sequence to be encoded;

[0008] Determine a first coding block and a second coding block corresponding to the saliency map according to a preset pixel threshold; wherein the preset pixel threshold is used to divide the foreground and the background; the first coding block is a coding block corresponding to the foreground area in the saliency map; and the second coding block is a coding block corresponding to the background area in the saliency map;

[0009] Calculating a first QP compensation value of the first coding block and a second QP compensation value of the second coding block;

[0010] The to-be-encoded video sequence is encoded according to the first QP compensation value and the second QP compensation value to generate a bitstream.

[0011] In a second aspect, an embodiment of the present application provides a terminal, the terminal comprising: a determination unit, a calculation unit, and a coding unit.

[0012] The determination unit is used to determine a saliency map corresponding to a video sequence to be encoded; determine a first coding block and a second coding block corresponding to the saliency map according to a preset pixel threshold; wherein the preset pixel threshold is used to divide the foreground and the background; the first coding block is a coding block corresponding to a foreground area in the saliency map; and the second coding block is a coding block corresponding to a background area in the saliency map;

[0013] The calculation unit is used to calculate a first QP compensation value of the first coding block and a second QP compensation value of the second coding block;

[0014] The encoding unit is used to encode the to-be-encoded video sequence according to the first QP compensation value and the second QP compensation value to generate a bit stream.

[0015] In a third aspect, an embodiment of the present application provides a terminal, comprising a processor and a memory storing instructions executable by the processor, wherein when the instructions are executed by the processor, the encoding method described in the first aspect is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the encoding method as described in the first aspect is implemented.

[0017] The embodiment of the present application provides a coding method, a terminal and a storage medium, wherein the terminal determines a saliency map corresponding to a video sequence to be coded; determines a first coding block and a second coding block corresponding to the saliency map according to a preset pixel threshold; wherein the preset pixel threshold is used to divide the foreground and the background; the first coding block is a coding block corresponding to the foreground area in the saliency map; the second coding block is a coding block corresponding to the background area in the saliency map; calculates a first QP compensation value of the first coding block and a second QP compensation value of the second coding block; and encodes the video sequence to be coded according to the first QP compensation value and the second QP compensation value to generate a bitstream. That is, in the embodiment of the present application, the terminal can divide the foreground area and the background area corresponding to the video sequence to be coded based on the preset pixel threshold, and calculates the QP compensation value corresponding to the coding block in the foreground area and the QP compensation value corresponding to the coding block in the background area respectively using different methods, and then can use the corresponding QP compensation value to reduce the QP value of the foreground area and increase the QP value of the background area, so that when the bit rate is fixed or low, a higher bit rate is allocated to the foreground area and a lower bit rate is allocated to the background area, thereby greatly improving the coding quality of the ROI area, and making the boundary between the ROI area and the non-ROI area clearer. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Illustration of visual saliency detection Figure 1 ;

[0019] Figure 2 Illustration of visual saliency detection Figure 2 ;

[0020] Figure 3 Schematic diagram of the implementation process of the encoding method proposed in the embodiment of the present application Figure 1 ;

[0021] Figure 4 Schematic diagram of the implementation process of the encoding method proposed in the embodiment of the present application Figure 2 ;

[0022] Figure 5 Schematic diagram of the implementation process of the encoding method proposed in the embodiment of the present application Figure 3 ;

[0023] Figure 6 Schematic diagram of the implementation process of the encoding method proposed in the embodiment of the present application Figure 4 ;

[0024] Figure 7 Schematic diagram of the implementation process of the encoding method proposed in the embodiment of the present application Figure 5 ;

[0025] Figure 8 Schematic diagram of the implementation process of the encoding method proposed in the embodiment of the present application Figure 6 ;

[0026] Fig. 9 The structure of the terminal proposed in the embodiment of the present application is shown in FIG. Figure 1 ;

[0027] Fig.10 The structure of the terminal proposed in the embodiment of the present application is shown in FIG. Figure 2 . DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It is understood that the specific embodiments described herein are only used to explain the related applications, rather than to limit the applications. It should also be noted that, for ease of description, only the parts related to the related applications are shown in the drawings.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0030] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. It should also be noted that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0031] In machine vision and image processing, the area to be processed is outlined in the form of a box, circle, ellipse, irregular polygon, etc. from the processed image, which is called the Region Of Interest (ROI). In machine vision software such as Halcon, OpenCV, and Matlab, various operators and functions are often used to obtain the ROI of the region of interest and perform the next step of image processing.

[0032] ROI video coding is mainly used in scenarios with limited video bandwidth. ROI coding is to encode the ROI area in high definition at the expense of the background video image quality. Currently, ROI intelligent coding technology mainly relies on manual determination of ROI area and target detection technology, and its application scenarios mainly include video calls, video live broadcasts, etc.

[0033] However, there is no clear and unified way to determine the ROI area. The conventional ones are face detection and human body detection. However, the ROI area that the human eye actually focuses on may not be common people and animals. In order to quickly obtain important information from massive image and video data, the concept of visual saliency is introduced. Among them, visual saliency means that the human eye automatically focuses on the area of ​​interest and ignores the area of ​​no interest. This salient area is more in line with the concept of ROI area. Even if such objects are not included in the training data set, the salient area can be detected in the video image.

[0034] Visual Saliency Detection (VSD) simulates human visual characteristics through intelligent algorithms and extracts salient areas (i.e., areas of human interest) in images. Specifically, VSD can allocate limited computing resources to more important information in images and videos. Since it better meets human visual cognitive needs, VSD has important application value in target recognition, image and video compression, image retrieval, image redirection, etc.

[0035] Figure 1 Illustration of visual saliency detection Figure 1 , Figure 2 Illustration of visual saliency detection Figure 2 , in the case of Figure 1 In the process of saliency detection of the ECSSD dataset 997-natural scene house shown in FIG, it can be identified that the house is the area where the human eye focuses, so the final result is as follows Figure 2 The results of the significance test are shown.

[0036] Currently, ROI coding is the trend of video coding technology. Determining the ROI area is a problem. How to make the ROI area detection technology more generalized? Even when such images have not been trained, the ROI area can be determined based on some brightness, texture details and other information. Saliency detection is a trend.

[0037] However, since the generalization of ROI region detection technology is not high and the QP value setting method does not fit the ROI region, the common encoding method cannot guarantee the encoding quality of the ROI region and has the defect of unclear boundary between the ROI region and the non-ROI region.

[0038] In order to solve the above problems, in an embodiment of the present application, the terminal can divide the foreground area and the background area corresponding to the video sequence to be encoded based on a preset pixel threshold, and use different methods to respectively calculate the QP compensation value corresponding to the coding block in the foreground area and the QP compensation value corresponding to the coding block in the background area, and then use the corresponding QP compensation value to reduce the QP value of the foreground area and increase the QP value of the background area, so that when the bit rate is fixed or low, a higher bit rate is allocated to the foreground area and a lower bit rate is allocated to the background area, thereby greatly improving the encoding quality of the ROI area and making the boundary between the ROI area and the non-ROI area clearer.

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0040] An embodiment of the present application provides an encoding method, Figure 3 Schematic diagram of the implementation process of the encoding method proposed in the embodiment of the present application Figure 1 ,like Figure 3 As shown, in an embodiment of the present application, the method for encoding by a terminal may include the following steps:

[0041] Step 101: Determine a saliency map corresponding to a video sequence to be encoded.

[0042] In an embodiment of the present application, the terminal may first determine a saliency map corresponding to the video sequence to be encoded.

[0043] It should be noted that, in the embodiment of the present application, the saliency map of the video sequence to be encoded can be an image generated after the video sequence to be encoded is processed by saliency detection and can represent the ROI region in the video sequence to be encoded. In other words, after the saliency detection process, the saliency map corresponding to the video sequence to be encoded can be obtained.

[0044] It is understandable that, in the embodiment of the present application, the terminal may obtain a salient map (such as a Score Map) corresponding to the video sequence to be encoded through salient object detection (Salient Object Detection, SOD) analysis.

[0045] Further, in the embodiment of the present application, since image data in RGB format is input when performing SOD analysis, the video sequence to be encoded may be an image in RGB format corresponding to the video to be encoded.

[0046] It should be noted that, in the embodiments of the present application, Figure 4 Schematic diagram of the implementation process of the encoding method proposed in the embodiment of the present application Figure 2 ,like Figure 4 As shown, in the embodiment of the present application, before the terminal determines the saliency map corresponding to the video sequence to be encoded, that is, before step 101, the method for encoding by the terminal may further include the following steps:

[0047] Step 105: Perform format conversion processing on the images in the video to be encoded to obtain a video sequence to be encoded.

[0048] In an embodiment of the present application, for a video to be encoded, the terminal may first perform format conversion processing on each frame of the video to be encoded, thereby obtaining a video sequence to be encoded in an RGB format.

[0049] It can be understood that in the embodiments of the present application, common videos to be encoded mainly include video sequences in YUV format. Considering that the input data for subsequent SOD analysis is image data in RGB format, it is necessary to first perform format conversion processing on the video frames to obtain a video sequence to be encoded whose data format meets the SOD analysis.

[0050] Exemplarily, in an embodiment of the present application, common videos to be encoded mainly include video sequences in the YUV420 format. For images in the video to be encoded, the image data in the YUV420 format can be first converted into image data in the YUV444 format through a bilinear interpolation algorithm, and then the image data in the YUV444 format can be converted into RGB888 image data through the following formula, so as to obtain the video sequence to be encoded.

[0051]

[0052] Furthermore, in an embodiment of the present application, due to the redundancy between video frames, it will inevitably be time-consuming to perform saliency detection (SOD analysis) on each frame of the video sequence to be encoded corresponding to the video to be encoded. Therefore, when determining the saliency map corresponding to the video sequence to be encoded, the terminal can choose to directly use the saliency map of other images that have a high similarity with the video sequence to be encoded, thereby greatly reducing the encoding overhead.

[0053] Optionally, in the present application, when determining the saliency map corresponding to the video sequence to be encoded, the terminal can choose to turn on the similarity measurement function, and perform video frame similarity analysis on the video sequence to be encoded and other images corresponding to the video to be encoded (such as the detected image of the previous frame of the video sequence to be encoded, and the previous frame of the detected image is the image that has been saliency detected before the video sequence to be encoded). If the similarity between the video sequence to be encoded and the other images is high, the saliency map of the other image can be directly used. If the similarity between the video sequence to be encoded and the other images is low, it is necessary to perform SOD analysis on the video to be encoded.

[0054] It should be noted that, in an embodiment of the present application, the terminal may first determine whether the video sequence to be encoded needs to be measured for similarity, that is, determine whether the video sequence to be encoded meets the preset conditions for the similarity measurement, and then further determine whether to perform similarity analysis on the video sequence to be encoded based on the judgment result.

[0055] Exemplarily, in an embodiment of the present application, if the video sequence to be encoded does not meet the preset conditions of the similarity measurement, then the terminal may not perform similarity analysis on the video sequence to be encoded, but directly perform saliency detection on the video sequence to be encoded to obtain the saliency map.

[0056] Exemplarily, in an embodiment of the present application, if the video sequence to be encoded meets the preset conditions of the similarity measurement, the terminal can perform a similarity analysis on the video sequence to be encoded to obtain a similarity result, and then determine the corresponding saliency map according to the similarity result.

[0057] Specifically, in an embodiment of the present application, after the terminal performs a similarity analysis on the video sequence to be encoded and obtains a similarity result, if the similarity result is that the video sequence to be encoded is similar to a detected image of a previous frame, then the terminal can determine the saliency map of the detected image of the previous frame as the saliency map of the video sequence to be encoded; if the similarity result is that the video sequence to be encoded is not similar to the detected image of the previous frame, then the terminal needs to perform a saliency detection on the video sequence to be encoded to obtain the saliency map.

[0058] It can be understood that in an embodiment of the present application, when the terminal performs a similarity analysis on the video sequence to be encoded, it can first determine the structural similarity parameters and frame interval parameters between the video sequence to be encoded and the detected image of the previous frame; and then the similarity result between the video sequence to be encoded and the detected image of the previous frame can be further determined based on the structural similarity parameters and the frame interval parameters.

[0059] Exemplarily, in the present application, if the structural similarity parameter is greater than or equal to a preset similarity threshold, and the frame interval parameter is less than or equal to the preset interval threshold, then the similarity result can be determined as that the video sequence to be encoded is similar to the detected image of the previous frame; if the structural similarity parameter is less than the preset similarity threshold, or the frame interval parameter is greater than the preset interval threshold, then the similarity result can be determined as that the video sequence to be encoded is not similar to the detected image of the previous frame.

[0060] Furthermore, in an embodiment of the present application, when the video sequence to be encoded does not meet the preset conditions of the similarity measurement, and when it is determined after the similarity analysis that the video sequence to be encoded is not similar to the detected image of the previous frame, the terminal needs to perform a saliency detection on the video sequence to be encoded, so as to obtain a saliency map corresponding to the video sequence to be encoded.

[0061] It should be noted that, in an embodiment of the present application, when the terminal performs saliency detection on the video sequence to be encoded, the video sequence to be encoded can be input into a preset saliency detection model, so that the saliency map can be output. The preset saliency detection model is used to perform saliency detection on the image, for example, the preset saliency detection model can be a MINet algorithm model. Other saliency detection algorithms that are more suitable for video encoding can also be selected for video encoding scenarios, and this application does not specifically limit this.

[0062] Optionally, in an embodiment of the present application, the terminal can choose to use the MINet algorithm model for saliency detection. Among them, the MINet structure is similar to U-Net, and an aggregate interaction strategy (AIM) and a self-interaction module (SIM) are designed for saliency detection. AIM can better utilize multi-level features to avoid interference between feature fusions caused by resolution differences. SIM is to obtain richer multi-scale features. The training process combines binary cross entropy loss (BCEL) and consistency enhanced loss function (CEL). BCEL is a commonly used loss function for semantic segmentation pixel classification, and CEL can better handle the problem of spatial consistency.

[0063] It should be noted that, in the embodiment of the present application, after the saliency detection is performed on the video sequence to be encoded, the detection result obtained is the saliency map corresponding to the video sequence to be encoded. For example, after the saliency object detection SOD analysis is performed on the video sequence to be encoded, the corresponding Score Map is obtained. Specifically, the prediction result of SOD is a Score Map with pixel values ​​of 0-255.

[0064] That is to say, in the embodiment of the present application, the saliency map corresponding to the video sequence to be encoded obtained by saliency detection is a grayscale map with a frame pixel value of 0-255.

[0065] It should be noted that in the embodiments of the present application, the terminal can be any terminal device with video encoding and decoding functions and storage functions, such as: tablet computers, mobile phones, e-readers, remote controllers, personal computers (PCs), laptops, vehicle-mounted devices, network televisions, wearable devices, personal digital assistants (PDAs), portable media players (PMPs), navigation devices, and other terminal devices. Among them, the functions implemented by the encoding method proposed in the embodiments of the present application can be implemented by calling program codes by the processor in the terminal, and of course the program codes can be stored in a computer storage medium. It can be seen that the terminal includes at least a processor and a storage medium.

[0066] Step 102: determine the first coding block and the second coding block corresponding to the saliency map according to a preset pixel threshold; wherein the preset pixel threshold is used to divide the foreground and the background; the first coding block is the coding block corresponding to the foreground area in the saliency map; and the second coding block is the coding block corresponding to the background area in the saliency map.

[0067] In an embodiment of the present application, after determining the saliency map corresponding to the video sequence to be encoded, the terminal can determine the foreground area and background area corresponding to the saliency map according to a preset pixel threshold, and further determine the first coding block and the second coding block corresponding to the saliency map.

[0068] It can be understood that, in the embodiments of the present application, the preset pixel threshold can be used to divide the foreground and background in a frame of image, that is, the corresponding foreground area and background area can be determined in the video sequence to be encoded by the preset pixel value.

[0069] Optionally, in an embodiment of the present application, the terminal may pre-set a preset pixel threshold, wherein the value of the preset pixel threshold may be any value between 0 and 255, for example, the preset pixel threshold is 30.

[0070] Further, in an embodiment of the present application, the first coding block may be a coding block corresponding to a foreground area in the saliency map; and the second coding block may be a coding block corresponding to a background area in the saliency map. The first coding block and the second coding block may be macroblocks (MB).

[0071] Further, in an embodiment of the present application, when determining the first coding block and the second coding block corresponding to the saliency map according to a preset pixel threshold, the terminal may first perform downsampling processing on the saliency map to obtain a sampled saliency map; then, the coding block in the sampled saliency map whose pixel value is greater than or equal to the preset pixel threshold may be determined as the first coding block; and at the same time, the coding block in the sampled saliency map whose pixel value is less than the preset pixel threshold may be determined as the second coding block.

[0072] Specifically, in an embodiment of the present application, after obtaining a saliency map (ScoreMap) corresponding to a video sequence to be encoded, the terminal may first perform a 16-fold downsampling process on the saliency map, thereby obtaining a sampled saliency map (NewScore Map). This is because any value in the video frame macroblock quantization parameter QP array (QPMAP) represents a 16×16 pixel area of ​​the original image, and therefore it is necessary to reduce the width and height of the saliency map to 1 / 16 of the original width and height of the saliency map through downsampling, thereby obtaining a new saliency map, i.e., a sampled saliency map.

[0073] It can be understood that in the embodiment of the present application, the terminal can select a downsampling scale according to the coding block size, so that after the downsampling process, each pixel corresponds to a coding block.

[0074] That is to say, in the present application, the sampling multiple selected for downsampling the saliency map can be set in advance, wherein the terminal can set the sampling multiple according to the condition that any coding block of the sampled saliency map corresponds to one pixel, so that each pixel in the sampled saliency map corresponds to one coding block.

[0075] It should be noted that in an embodiment of the present application, after obtaining a sampled saliency map through downsampling processing, the terminal can compare the pixel value of the sampled saliency map with a preset pixel threshold, thereby completing the division of the foreground area and the background area of ​​the saliency map according to the comparison result, and determining the first coding block corresponding to the foreground area and the second coding block corresponding to the background area in the sampled saliency map.

[0076] Optionally, in an embodiment of the present application, if the pixel value of a coding block in the sampled saliency map is greater than or equal to a preset pixel threshold, the terminal may consider that the coding block belongs to the foreground area of ​​the saliency map, and further determine the coding block as the first coding block corresponding to the foreground area.

[0077] Optionally, in an embodiment of the present application, if the pixel value of a coding block in the sampled saliency map is less than a preset pixel threshold, the terminal may consider that the coding block belongs to the background area of ​​the saliency map, and further determine the coding block as the second coding block corresponding to the background area.

[0078] It can be seen that in the embodiments of the present application, the terminal can complete the division process of the foreground area and the background area of ​​the saliency map by presetting the pixel threshold, so that the coding block corresponding to the foreground area and the coding block corresponding to the background area can be determined in the sampled saliency map, and then the QP value of the coding block in the different areas can be adjusted in different ways.

[0079] It should be noted that in the embodiments of the present application, the foreground area of ​​the saliency map obtained by saliency detection and division of foreground and background of the video sequence to be encoded can be considered as the ROI area corresponding to the video sequence to be encoded, wherein the foreground area can include portraits and other contents such as object images.

[0080] Step 103: Calculate a first QP compensation value of the first coding block and a second QP compensation value of the second coding block.

[0081] In an embodiment of the present application, after determining the first coding block and the second coding block corresponding to the saliency map according to a preset pixel threshold, the terminal may calculate a first QP compensation value for the first coding block and a second QP compensation value for the second coding block.

[0082] It should be noted that, in an embodiment of the present application, the first QP compensation value can be used to adjust the first QP initial value of the first coding block corresponding to the foreground area, and the second QP compensation value can be used to adjust the second QP initial value of the second coding block corresponding to the background area.

[0083] It can be understood that in an embodiment of the present application, in order to improve the video quality of the ROI area during the decoding process at the same bit rate or a low bit rate, the terminal not only needs to distinguish and encode the foreground area and the background area according to different bit rates, but also needs to increase the bit rate corresponding to the foreground area as much as possible. Therefore, the terminal needs to adjust the first QP initial value of the first coding block corresponding to the foreground area and the second QP initial value of the second coding block corresponding to the background area, respectively.

[0084] Specifically, in an embodiment of the present application, the terminal may first determine different QP compensation values ​​for the foreground area and the background area, and then use the different QP compensation values ​​to adjust the first QP initial value and the second QP initial value.

[0085] Further, in an embodiment of the present application, for the foreground area and the background area, the terminal may use different calculation models to calculate the QP compensation value, so that different QP compensation values ​​may be obtained for different areas.

[0086] Optionally, in the present application, when calculating the first QP compensation value of the first coding block and the second QP compensation value of the second coding block, the terminal may calculate the first QP compensation value according to a preset QP reference value, a first calculation model, and a pixel value of the first coding block; at the same time, the second QP compensation value may be calculated according to the preset QP reference value, the second calculation model, and the pixel value of the second coding block.

[0087] It should be noted that, in the embodiment of the present application, the first calculation model is used to determine the QP compensation value corresponding to the foreground area. Specifically, the first calculation model is used to reduce the initial QP value of the foreground area.

[0088] Exemplarily, in the present application, the first calculation model can be expressed as the following formula, and the terminal can calculate and obtain the first QP compensation value qp-val1 corresponding to the foreground area by using the first calculation model:

[0089] qp-val1=pix-val1 / 255×Δqp (2)

[0090] Wherein, Δqp is a preset QP reference value, and pix-val1 is a pixel value of the first coding block.

[0091] It should be noted that, in the embodiment of the present application, the second calculation model is used to determine the QP compensation value corresponding to the background area. Specifically, the second calculation model is used to increase the initial QP value of the background area.

[0092] Exemplarily, in the present application, the second calculation model can be expressed as the following formula, and the terminal can calculate and obtain the second QP compensation value qp-val2 corresponding to the background area by using the second calculation model:

[0093] qp-val2=(255-pix-val2) / 255×(-Δqp) (3)

[0094] Wherein, Δqp is a preset QP reference value, and pix-val2 is a pixel value of the second coding block.

[0095] It can be understood that in the embodiment of the present application, the preset QP reference value can be any value pre-set by the terminal for calculating the QP compensation value. The terminal can set the preset QP reference value to a fixed value, or use an algorithm to analyze and classify the video scene, and then automatically set parameters for different scenes, that is, different scenes correspond to different preset QP reference values.

[0096] Exemplarily, in an embodiment of the present application, the terminal may set the preset QP reference value Δqp to -15, and the first QP compensation value qp-val1 calculated by the above formula (2) is a negative number, then the first QP initial value is adjusted by the first QP compensation value, and the QP value of the first coding block can be reduced; accordingly, the second QP compensation value qp-val2 calculated by the above formula (3) is a positive number, then the second QP initial value is adjusted by the second QP compensation value, and the QP value of the second coding block can be increased.

[0097] Step 104: Encode the to-be-encoded video sequence according to the first QP compensation value and the second QP compensation value to generate a bitstream.

[0098] In an embodiment of the present application, after calculating the first QP compensation value of the first coding block and the second QP compensation value of the second coding block, the terminal can encode the video sequence to be encoded according to the first QP compensation value corresponding to the first coding block and the second QP compensation value corresponding to the second coding block, thereby generating a bitstream.

[0099] Furthermore, in the embodiments of the present application, Figure 5 Schematic diagram of the implementation process of the encoding method proposed in the embodiment of the present application Figure 3 ,like Figure 5 As shown, before encoding the to-be-encoded video sequence according to the first QP compensation value and the second QP compensation value to generate a bitstream, that is, before step 104, the encoding method performed by the terminal may further include the following steps:

[0100] Step 106: Determine a first QP initial value for the first coding block and a second QP initial value for the second coding block.

[0101] In an embodiment of the present application, after completing the division of the foreground area and the background area of ​​the video sequence to be encoded, the terminal can first determine the first QP initial value of the first encoding block corresponding to the foreground area and the second QP initial value of the second encoding block corresponding to the background area.

[0102] It should be noted that, in the implementation of the present application, for the foreground area, the first QP initial values ​​corresponding to different first coding blocks may be the same or different; correspondingly, for the background area, the second QP initial values ​​corresponding to different second coding blocks may be the same or different.

[0103] Further, in an embodiment of the present application, for a first coding block corresponding to the foreground area and a second coding block corresponding to the background area, a first QP initial value and a second QP initial value are different.

[0104] It can be understood that in the embodiment of the present application, after the first QP compensation value is calculated according to the first calculation model for reducing the QP initial value of the foreground area, and the second QP compensation value is calculated according to the second calculation model for increasing the QP initial value of the background area, the terminal can adjust the first QP initial value of the foreground area and the second QP initial value of the background area according to the first QP compensation value and the second QP compensation value, respectively, so as to obtain the reduced first QP value and the increased second QP value.

[0105] Optionally, in an embodiment of the present application, when encoding the video sequence to be encoded according to the first QP compensation value and the second QP compensation value, a first QP value of the first encoding block can be determined according to the first QP compensation value, and a second QP value of the second encoding block can be determined according to the second QP compensation value; and then encoding can be performed according to the first QP value and the second QP value to generate the code stream.

[0106] It should be noted that, in the present application, since the smaller the QP value, the higher the allocated bit rate, and the larger the QP value, the lower the allocated bit rate, after obtaining the first QP value and the second QP value based on the first QP compensation value and the second QP compensation value, respectively encoding the first coding block and the second coding block according to the first QP value and the second QP value can greatly improve the encoding quality of the foreground area, that is, improve the video quality of the ROI area during the decoding process. At the same time, increasing the bit rate difference allocated to the first coding block and the second coding block can make the boundary between the foreground area and the background area clearer, thereby more clearly highlighting the ROI area.

[0107] Optionally, in an embodiment of the present application, when encoding the to-be-encoded video sequence according to the first QP compensation value and the second QP compensation value, the terminal may perform a sum operation on the first QP compensation value and the first QP initial value of the first coding block to obtain the first QP value of the first coding block; and at the same time, perform a sum operation on the second QP compensation value and the second QP initial value of the second coding block to obtain the second QP value of the second coding block; and then, the first coding unit may be encoded according to the first QP value, and the second coding unit may be encoded according to the second QP value to generate the code stream.

[0108] In summary, through the encoding method proposed in the above steps 101 to 106, based on the saliency map corresponding to the video sequence to be encoded, the terminal can divide and obtain the foreground area and the background area, and calculate the QP compensation value of each encoding block corresponding to the foreground area and the background area in different ways, so that the QP initial value of the encoding block can be reduced according to the QP compensation value corresponding to the foreground area, so that the encoding block corresponding to the foreground area is allocated a higher bit rate during encoding, and at the same time, the QP initial value of the encoding block can be increased according to the QP compensation value corresponding to the background area, so that the encoding block corresponding to the background area is allocated a lower bit rate during encoding, thereby greatly improving the encoding quality of the foreground area and making the boundary between the foreground area and the background area clearer.

[0109] Specifically, the encoding method proposed in the embodiment of the present application determines the ROI area by introducing a deep learning saliency detection algorithm. The saliency detection has stronger generalization performance for salient areas in different scenes, and can also extract ROI areas for irregular objects such as houses and rocks in natural scenes.

[0110] Furthermore, in order to overcome the defect of increased coding overhead caused by redundancy between video frames, the encoding method proposed in the embodiment of the present application can perform a similarity measurement before performing SOD analysis, so that the video sequence to be encoded can directly use the saliency map of other images similar to it, thereby reducing the large amount of calculation caused by the redundancy between video frames.

[0111] It can be understood that, in the encoding method proposed in the embodiment of the present application, the SOD prediction result obtains the Score Map, and the sampled saliency map (New Score Map) obtained by downsampling the saliency map (Score Map) by 16 times just meets the QPMAP size. Then, the foreground area and the background area are distinguished based on the pixel values ​​predicted in the sampled saliency map, and the QP compensation values ​​of the coding blocks in the foreground area and the background area are set respectively in different ways, so that the QPMAP is superimposed on the original coding block (original macroblock) bit rate control process, and the QP value of the original coding block (original macroblock) is adjusted, so that the QP value of the foreground area (ROI area) is reduced, so that the encoding process will allocate more bit rate; at the same time, the QP value of the background area is increased, so that the encoding process will allocate less bit rate.

[0112] It should be noted that the encoding method proposed in the embodiment of the present application can adjust the QP value of the ROI area through the saliency detection algorithm to optimize the x264 encoding framework and improve the video quality of the ROI area during the decoding process at the same bit rate or low bit rate. Among them, the highly compressed digital video codec standard H.264 is one of the mainstream video encoding methods, and the open source H.264 video encoding function library x264 is the application version of H.264.

[0113] Furthermore, the encoding method proposed in the embodiment of the present application can also be applied to other encoding frameworks, such as x265, VP8, VP9, ​​etc.

[0114] The embodiment of the present application provides a coding method, wherein the terminal determines a saliency map corresponding to a video sequence to be coded; determines a first coding block and a second coding block corresponding to the saliency map according to a preset pixel threshold; wherein the preset pixel threshold is used to divide the foreground and the background; the first coding block is a coding block corresponding to the foreground area in the saliency map; the second coding block is a coding block corresponding to the background area in the saliency map; calculates a first QP compensation value of the first coding block and a second QP compensation value of the second coding block; and encodes the video sequence to be coded according to the first QP compensation value and the second QP compensation value to generate a bitstream. That is, in the embodiment of the present application, the terminal can divide the foreground area and the background area corresponding to the video sequence to be coded based on the preset pixel threshold, and calculates the QP compensation value corresponding to the coding block in the foreground area and the QP compensation value corresponding to the coding block in the background area respectively using different methods, and then can use the corresponding QP compensation value to reduce the QP value of the foreground area and increase the QP value of the background area, so that when the bit rate is fixed or low, a higher bit rate is allocated to the foreground area and a lower bit rate is allocated to the background area, thereby greatly improving the coding quality of the ROI area, and making the boundary between the ROI area and the non-ROI area clearer.

[0115] Based on the above embodiment, another embodiment of the present application proposes an encoding method: Figure 6 Schematic diagram of the implementation process of the encoding method proposed in the embodiment of the present application Figure 4 ,like Figure 6 As shown, the method for a terminal to determine a saliency map corresponding to a video sequence to be encoded may include the following steps:

[0116] Step 101a: If the video sequence to be encoded meets the preset conditions of the similarity measurement, a similarity analysis is performed on the video sequence to be encoded to obtain a similarity result.

[0117] Step 101b: If the similarity result is that the video sequence to be encoded is similar to the previous detected image, the salient map of the previous detected image is determined as the salient map of the video sequence to be encoded.

[0118] Step 101c: If the similarity result is that the video sequence to be encoded is not similar to the detected image of the previous frame, a saliency detection is performed on the video sequence to be encoded to obtain a saliency map.

[0119] In an embodiment of the present application, after obtaining the video sequence to be encoded corresponding to the video to be encoded, the terminal can determine whether the video sequence to be encoded needs to be measured for similarity. Specifically, the terminal can first determine whether the video sequence to be encoded meets the preset conditions of the similarity measurement, and if so, choose to perform similarity analysis on the video sequence to be encoded to obtain a similarity result.

[0120] It can be understood that, in the embodiments of the present application, the preset conditions of the similarity measurement can be used to determine the importance of the image in the video. Specifically, if the video sequence to be encoded is an important video frame in the video to be encoded, then it can be considered that it is necessary to perform a saliency detection on the video sequence to be encoded, rather than directly using the saliency map of other images with high similarity thereto, and then it can be determined that the video sequence to be encoded does not meet the preset conditions of the similarity measurement; accordingly, if the video sequence to be encoded is not an important video frame in the video to be encoded, then it can be considered that it is not necessary to perform a saliency detection on the video sequence to be encoded, and the saliency map of other images with high similarity thereto can be directly used, and then it can be determined that the video sequence to be encoded meets the preset conditions of the similarity measurement.

[0121] Exemplarily, in the present application, if the video sequence to be encoded is an I frame in the video to be encoded, then the terminal can determine that the video sequence to be encoded does not meet the preset conditions of the similarity measurement; if the video sequence to be encoded is not an I frame in the video to be encoded (but a P frame or B frame in the video to be encoded), then the terminal can determine that the video sequence to be encoded meets the preset conditions of the similarity measurement.

[0122] Exemplarily, in the present application, if the video sequence to be encoded is the first frame image in the video to be encoded, then the terminal can determine that the video sequence to be encoded does not meet the preset conditions of the similarity measurement; if the video sequence to be encoded is not the first frame image in the video to be encoded, then the terminal can determine that the video sequence to be encoded meets the preset conditions of the similarity measurement.

[0123] Among them, I frame means key frame, which can be understood as the complete preservation of this frame; decoding only requires the data of this frame, because I frame contains the complete picture. P frame means the difference between this frame and the previous key frame (or P frame). When decoding, it is necessary to superimpose the difference defined by this frame with the previously cached picture to generate the final picture. P frame is also called difference frame. P frame does not have complete picture data, but only the picture difference data with the previous frame. B frame is a bidirectional difference frame, that is, B frame records the difference between this frame and the previous and next frames. In other words, to decode B frame, not only the previous cached picture is obtained, but also the picture after decoding, and the final picture is obtained by superimposing the previous and next pictures with the data of this frame.

[0124] Furthermore, in an embodiment of the present application, when the terminal performs a similarity analysis on the video sequence to be encoded and obtains a similarity result, it can first determine the structural similarity parameters and frame interval parameters between the video sequence to be encoded and the detected image of the previous frame; if the structural similarity parameter is greater than or equal to a preset similarity threshold, and the frame interval parameter is less than or equal to a preset interval threshold, then the terminal can determine that the similarity result is that the video sequence to be encoded is similar to the detected image of the previous frame; if the structural similarity parameter is less than the preset similarity threshold, or the frame interval parameter is greater than the preset interval threshold, then the terminal can determine that the similarity result is that the video sequence to be encoded is not similar to the detected image of the previous frame.

[0125] It can be understood that in an embodiment of the present application, the structural similarity parameter can be used to determine the degree of similarity between the video sequence to be encoded and the detected image of the previous frame. The metric for similarity measurement can adopt structural similarity (SSIM), where SSIM is used to compare the brightness, contrast, and structure of two samples x and y. Specifically, the structural similarity parameter between the video sequence to be encoded (x) and the detected image of the previous frame (y) can be expressed as SSIM (x, y), and the value of SSIM (x, y) belongs to the interval [0, 1]. The larger the value, the higher the similarity between the two samples x and y.

[0126] Accordingly, in an embodiment of the present application, a preset similarity threshold can determine whether the image contents of two frames of images are similar. Based on the SSIM metric, the terminal can set the preset similarity threshold to a value greater than or equal to 0 and less than or equal to 1, for example, setting the preset similarity threshold to 0.9.

[0127] It should be noted that, in the embodiment of the present application, the frame interval parameter can be used to determine the time interval between the video sequence to be encoded and the previous detected image, wherein the frame interval parameter can limit the similarity result in terms of time dimension.

[0128] Optionally, in an embodiment of the present application, the terminal may set the preset interval threshold to a specific value, for example, setting the preset interval threshold to 10 frames; the preset interval threshold may also be set according to the distance between IDR frames (Group of picture, GOP), for example, setting the preset interval threshold to 1 / 3 of the GOP; different preset interval thresholds may also be set according to different scenarios, for example, corresponding to scenario 1, the preset interval threshold may be set to 10 frames, and corresponding to scenario 1, the preset interval threshold may be set to 8 frames.

[0129] It can be understood that in the embodiments of the present application, although the similarity between two frames of images is very high, the actual content of the two frames of images is very different. Therefore, it is necessary to use a preset interval threshold to assist in the similarity analysis of the video sequence to be encoded, so as to make the obtained similarity result more accurate and avoid the shortcomings of SSIM.

[0130] Exemplarily, in the present application, only when the structural similarity parameters and frame interval parameters between the video sequence to be encoded and the detected image of the previous frame are satisfied (such as the structural similarity parameter is greater than or equal to the preset similarity threshold, and the frame interval parameter is greater than the preset interval threshold), the terminal can determine that the similarity result is that the video sequence to be encoded is similar to the detected image of the previous frame, and then the SOD analysis of the video sequence to be encoded can be omitted, and the saliency detection result of the detected image of the previous frame can be directly used.

[0131] Furthermore, in the embodiments of the present application, Figure 7 Schematic diagram of the implementation process of the encoding method proposed in the embodiment of the present application Figure 5 ,like Figure 7 As shown, the method for the terminal to determine the saliency map corresponding to the video sequence to be encoded may also include the following steps:

[0132] Step 101d: If the video sequence to be encoded does not meet the preset conditions of the similarity measurement, a saliency detection is performed on the video sequence to be encoded to obtain a saliency map.

[0133] In an embodiment of the present application, after obtaining the video sequence to be encoded corresponding to the video to be encoded, the terminal can determine whether the video sequence to be encoded needs to be measured for similarity. Specifically, the terminal can first determine whether the video sequence to be encoded meets the preset conditions of the similarity measurement. If not, the terminal does not need to perform similarity analysis on the video sequence to be encoded, but can choose to directly perform saliency detection on the video sequence to be encoded, thereby obtaining the corresponding saliency map.

[0134] The embodiment of the present application provides a coding method, wherein the terminal determines a saliency map corresponding to a video sequence to be coded; determines a first coding block and a second coding block corresponding to the saliency map according to a preset pixel threshold; wherein the preset pixel threshold is used to divide the foreground and the background; the first coding block is a coding block corresponding to the foreground area in the saliency map; the second coding block is a coding block corresponding to the background area in the saliency map; calculates a first QP compensation value of the first coding block and a second QP compensation value of the second coding block; and encodes the video sequence to be coded according to the first QP compensation value and the second QP compensation value to generate a bitstream. That is, in the embodiment of the present application, the terminal can divide the foreground area and the background area corresponding to the video sequence to be coded based on the preset pixel threshold, and calculates the QP compensation value corresponding to the coding block in the foreground area and the QP compensation value corresponding to the coding block in the background area respectively using different methods, and then can use the corresponding QP compensation value to reduce the QP value of the foreground area and increase the QP value of the background area, so that when the bit rate is fixed or low, a higher bit rate is allocated to the foreground area and a lower bit rate is allocated to the background area, thereby greatly improving the coding quality of the ROI area, and making the boundary between the ROI area and the non-ROI area clearer.

[0135] Based on the above embodiment, another embodiment of the present application proposes an encoding method: Figure 8 Schematic diagram of the implementation process of the encoding method proposed in the embodiment of the present application Figure 6 ,like Figure 8 As shown, the encoding method proposed in the embodiment of the present application can be understood as a ROI encoding method, which adds a visual saliency detection ROI area coding block (such as macroblock MB level) QP adjustment process on the basis of the original rate control (such as the original x264 rate control) process. The QP-adjusted coding block (such as MB) can be 16×16 pixels in size. When the bit rate is fixed, the smaller the QP value of the MB, the higher the bit rate will be allocated to the MB area, and vice versa, the larger the QP value, the lower the allocated bit rate. Among them, the purpose of ROI encoding is to allocate a higher bit rate to the ROI area, so that the decoded ROI area video image is clearer.

[0136] Specifically, in an embodiment of the present application, the terminal can first measure the similarity of the video frame, and then adjust the QP through the result of SOD prediction, downsample the detection result Score Map (the pixel value of the SOD prediction result is 0-255) by 16 times to obtain a New Score Map of the QPMAP size, and then obtain the QPMAP of the video frame through the QPMAP calculation module, and superimpose it to the macroblock rate control module to adjust the MB-level QP value. In this way, the QP value of the ROI area becomes smaller, and the QP value of the non-ROI area remains unchanged or increases.

[0137] It should be noted that in the embodiment of the present application, a YUV video sequence is input, and after determining whether to perform ROI encoding (step 201), one branch is not to perform ROI encoding, but to directly enter the original x264 rate control process; after the YUV video frame to be encoded is input, it enters the frame level rate control (step 202) and macroblock rate control process, and the frame level rate control will adjust the QP level through complexity calculation, and will also adjust the QP level through feedback after macroblock rate control encoding. The adaptive quantization (step 203) process will be carried out in advance, and the MB-level QP adjustment (step 205) will be achieved by adjusting the MB energy analysis combined with the adaptive quantization strength factor.

[0138] Further, in an embodiment of the present application, a YUV video sequence is input, and after determining whether to perform ROI encoding (step 201), another branch is to perform ROI encoding, and the YUV video frame to be encoded needs to be converted into an RGB image (video sequence to be encoded) (step 206); then, after determining whether the preset conditions of the similarity measurement are met (step 207), if it is determined that the preset conditions of the similarity measurement are met, a similarity detection is performed on the video sequence to be encoded (step 208), and it is determined whether the detection result of the similarity detection is similar (step 209). If the video sequence to be encoded is similar to the detected image of the previous frame, then the saliency map (Score Map) of the detected image of the previous frame can be directly used; if it is determined that the preset conditions of the similarity measurement are not met, or if the video sequence to be encoded is not similar to the detected image of the previous frame, then a saliency detection is required (step 210) to obtain the corresponding Score Map. Among them, the Score Map is a grayscale image with a pixel value of 0-255. The Score Map is reduced by 16 times in width and height to obtain a sampled saliency map (New Score Map) of the size of QPMAP (step 211). Based on the New Score Map, the foreground area (ROI area) and the background area (non-ROI area) can be distinguished, and then different calculation models can be used to calculate different QP compensation values ​​corresponding to the coding blocks in the foreground area and the coding blocks in the background area (step 212).

[0139] It should be noted that, in the embodiment of the present application, for the ROI area and the non-ROI area, the terminal can use different QP compensation values ​​to adjust the set QP initial value, which is equivalent to adding the QP compensation value on the basis of the QP value of the MB corresponding to the foreground area calculated by the original adaptive quantization and MBTree process (step 204), and obtaining the adjusted QP values ​​corresponding to different areas. Wherein, when the calculated first QP compensation value corresponding to the foreground area is a negative number and the second QP compensation value corresponding to the background area is a positive number, the QP value corresponding to the ROI area becomes smaller and the QP value corresponding to the non-ROI area becomes larger, and the adjusted QP value is used for encoding processing (step 213) to generate a bitstream, thereby greatly improving the encoding quality of the foreground area, that is, improving the video quality of the ROI area during the decoding process, and at the same time, increasing the code rate difference allocated to the first coding block and the second coding block, so that the boundary between the foreground area and the background area can be clearer, thereby more clearly highlighting the ROI area.

[0140] Among them, New Score Map is recorded as score_map_new in the code. Based on the above formulas (2) and (3), the key code for setting the saliency detection QPMAP is as follows:

[0141]

[0142] It can be seen that the encoding method proposed in this application performs salient area detection on video frames through the deep learning saliency detection algorithm MINet, thereby adjusting the QP value of x264 video frame encoding. During the encoding process, a higher bit rate will be allocated to the foreground area, and a lower bit rate will be allocated to the background area. The ROI area can also restore high-quality video images during the decoding process at the same bit rate or a lower bit rate.

[0143] The embodiment of the present application provides a coding method, wherein the terminal determines a saliency map corresponding to a video sequence to be coded; determines a first coding block and a second coding block corresponding to the saliency map according to a preset pixel threshold; wherein the preset pixel threshold is used to divide the foreground and the background; the first coding block is a coding block corresponding to the foreground area in the saliency map; the second coding block is a coding block corresponding to the background area in the saliency map; calculates a first QP compensation value of the first coding block and a second QP compensation value of the second coding block; and encodes the video sequence to be coded according to the first QP compensation value and the second QP compensation value to generate a bitstream. That is, in the embodiment of the present application, the terminal can divide the foreground area and the background area corresponding to the video sequence to be coded based on the preset pixel threshold, and calculates the QP compensation value corresponding to the coding block in the foreground area and the QP compensation value corresponding to the coding block in the background area respectively using different methods, and then can use the corresponding QP compensation value to reduce the QP value of the foreground area and increase the QP value of the background area, so that when the bit rate is fixed or low, a higher bit rate is allocated to the foreground area and a lower bit rate is allocated to the background area, thereby greatly improving the coding quality of the ROI area, and making the boundary between the ROI area and the non-ROI area clearer.

[0144] Based on the above embodiment, in another embodiment of the present application, Fig. 9 The structure of the terminal proposed in the embodiment of the present application is shown in FIG. Figure 1 ,like Fig. 9 As shown, the terminal 10 proposed in the embodiment of the present application may include a determination unit 11, a calculation unit 12, and a coding unit 13.

[0145] The determination unit 11 is used to determine a saliency map corresponding to a video sequence to be encoded; determine a first coding block and a second coding block corresponding to the saliency map according to a preset pixel threshold; wherein the preset pixel threshold is used to divide the foreground and the background; the first coding block is a coding block corresponding to the foreground area in the saliency map; and the second coding block is a coding block corresponding to the background area in the saliency map;

[0146] The calculation unit 12 is used to calculate a first QP compensation value of the first coding block and a second QP compensation value of the second coding block;

[0147] The encoding unit 13 is configured to encode the to-be-encoded video sequence according to the first QP compensation value and the second QP compensation value to generate a bit stream.

[0148] In the embodiments of the present application, further, Fig.10 The structure of the terminal proposed in the embodiment of the present application is shown in FIG. Figure 2 ,like Fig.10As shown, the terminal 10 proposed in the embodiment of the present application may also include a processor 14 and a memory 15 storing executable instructions of the processor 14. Further, the terminal 10 may also include a communication interface 16 and a bus 17 for connecting the processor 14, the memory 15 and the communication interface 16.

[0149] In the embodiment of the present application, the processor 14 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic device used to implement the function of the processor may also be other, and the embodiment of the present application is not specifically limited. The terminal 10 may also include a memory 15, which may be connected to the processor 14, wherein the memory 15 is used to store executable program code, the program code includes computer operation instructions, and the memory 15 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0150] In the embodiment of the present application, the bus 17 is used to connect the communication interface 16, the processor 14 and the memory 15, and the mutual communication between these devices.

[0151] In the embodiment of the present application, the memory 15 is used to store instructions and data.

[0152] Further, in an embodiment of the present application, the processor 14 is used to determine a saliency map corresponding to the video sequence to be encoded;

[0153] A first coding block and a second coding block corresponding to the saliency map are determined according to a preset pixel threshold; wherein the preset pixel threshold is used to divide the foreground and the background; the first coding block is a coding block corresponding to the foreground area in the saliency map; the second coding block is a coding block corresponding to the background area in the saliency map; a first QP compensation value of the first coding block and a second QP compensation value of the second coding block are calculated; and the to-be-encoded video sequence is encoded according to the first QP compensation value and the second QP compensation value to generate a bitstream.

[0154] In practical applications, the memory 15 may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 14.

[0155] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0156] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk, etc., which can store program code.

[0157] The embodiment of the present application provides a terminal, which determines a saliency map corresponding to a video sequence to be encoded; determines a first coding block and a second coding block corresponding to the saliency map according to a preset pixel threshold; wherein the preset pixel threshold is used to divide the foreground and the background; the first coding block is a coding block corresponding to the foreground area in the saliency map; the second coding block is a coding block corresponding to the background area in the saliency map; calculates a first QP compensation value of the first coding block and a second QP compensation value of the second coding block; and encodes the video sequence to be encoded according to the first QP compensation value and the second QP compensation value to generate a bitstream. That is, in the embodiment of the present application, the terminal can divide the foreground area and the background area corresponding to the video sequence to be encoded based on the preset pixel threshold, and use different methods to calculate the QP compensation value corresponding to the coding block in the foreground area and the QP compensation value corresponding to the coding block in the background area, respectively, and then can use the corresponding QP compensation value to reduce the QP value of the foreground area and increase the QP value of the background area, so that when the bit rate is fixed or low, a higher bit rate is allocated to the foreground area and a lower bit rate is allocated to the background area, thereby greatly improving the encoding quality of the ROI area, and at the same time making the boundary between the ROI area and the non-ROI area clearer.

[0158] An embodiment of the present application provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the encoding method as described above is implemented.

[0159] Specifically, the program instructions corresponding to an encoding method in this embodiment can be stored on a storage medium such as a CD, a hard disk, a USB flash drive, etc. When the program instructions corresponding to an encoding method in the storage medium are read or executed by an electronic device, the following steps are included:

[0160] Determining a saliency map corresponding to a video sequence to be encoded;

[0161] Determine a first coding block and a second coding block corresponding to the saliency map according to a preset pixel threshold; wherein the preset pixel threshold is used to divide the foreground and the background; the first coding block is a coding block corresponding to the foreground area in the saliency map; and the second coding block is a coding block corresponding to the background area in the saliency map;

[0162] Calculating a first QP compensation value of the first coding block and a second QP compensation value of the second coding block;

[0163] The to-be-encoded video sequence is encoded according to the first QP compensation value and the second QP compensation value to generate a bitstream.

[0164] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0165] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0166] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0168] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A coding method, characterized in that: The method comprises: Determining a saliency map corresponding to a video sequence to be encoded; Determine a first coding block and a second coding block corresponding to the saliency map according to a preset pixel threshold; wherein the preset pixel threshold is used to divide the foreground and the background; the first coding block is a coding block corresponding to the foreground area in the saliency map; and the second coding block is a coding block corresponding to the background area in the saliency map; Calculate a first QP compensation value according to a preset QP reference value, a first calculation model, and a pixel value of the first coding block; calculate a second QP compensation value according to the preset QP reference value, a second calculation model, and a pixel value of the second coding block; the first QP compensation value is used to adjust a first QP initial value of the first coding block; the second QP compensation value is used to adjust a second QP initial value of the second coding block; The first QP compensation value and the first QP initial value of the first coding block are summed to obtain a first QP value of the first coding block; the second QP compensation value and the second QP initial value of the second coding block are summed to obtain a second QP value of the second coding block; the first coding block is encoded according to the first QP value, and the second coding block is encoded according to the second QP value to generate a bitstream.

2. The method according to claim 1, characterized in that The determining of a saliency map corresponding to the video sequence to be encoded comprises: If the video sequence to be encoded meets the preset condition of the similarity measurement, a similarity analysis is performed on the video sequence to be encoded to obtain a similarity result; If the similarity result is that the video sequence to be encoded is similar to the detected image of the previous frame, determining the saliency map of the detected image of the previous frame as the saliency map of the video sequence to be encoded; If the similarity result is that the video sequence to be encoded is not similar to the detected image of the previous frame, a saliency detection is performed on the video sequence to be encoded to obtain the saliency map.

3. The method according to claim 2, characterized in that The performing similarity analysis on the to-be-encoded video sequence to obtain a similarity result includes: Determining a structural similarity parameter and a frame interval parameter between the video sequence to be encoded and the detected image of the previous frame; If the structural similarity parameter is greater than or equal to a preset similarity threshold, and the frame interval parameter is less than or equal to a preset interval threshold, determining that the similarity result is that the video sequence to be encoded is similar to the detected image of the previous frame; If the structural similarity parameter is smaller than the preset similarity threshold, or the frame interval parameter is larger than the preset interval threshold, it is determined that the similarity result is that the video sequence to be encoded is not similar to the detected image of the previous frame.

4. The method according to claim 1, characterized in that The determining of a saliency map corresponding to the video sequence to be encoded comprises: If the to-be-encoded video sequence does not satisfy the preset condition of the similarity measurement, a saliency detection is performed on the to-be-encoded video sequence to obtain the saliency map.

5. The method according to claim 2 or 4, characterized in that: The performing saliency detection on the to-be-encoded video sequence to obtain the saliency map comprises: The video sequence to be encoded is input into a preset saliency detection model, and the saliency map is output.

6. The method according to claim 1, characterized in that The step of determining the first coding block and the second coding block corresponding to the saliency map according to a preset pixel threshold comprises: Downsampling the saliency map to obtain a sampled saliency map; Determine, in the sampled saliency map, a coding block whose pixel value is greater than or equal to the preset pixel threshold as the first coding block; A coding block in the sampled saliency map whose pixel value is less than the preset pixel threshold is determined as the second coding block.

7. The method according to claim 1, characterized in that Before determining the saliency map corresponding to the video sequence to be encoded, the method further includes: Perform format conversion processing on the images in the video to be encoded to obtain the video sequence to be encoded.

8. The method according to any one of claims 2 to 4, characterized in that: The method further comprises: If the to-be-encoded video sequence is an I frame, determining that the preset condition of the similarity measurement is not satisfied; If the to-be-encoded video sequence is not an I frame, it is determined that the preset condition of the similarity measurement is satisfied.

9. A terminal, characterized in that: The terminal comprises: a determination unit, a calculation unit, and an encoding unit. The determination unit is used to determine a saliency map corresponding to a video sequence to be encoded; determine a first coding block and a second coding block corresponding to the saliency map according to a preset pixel threshold; wherein the preset pixel threshold is used to divide the foreground and the background; the first coding block is a coding block corresponding to a foreground area in the saliency map; and the second coding block is a coding block corresponding to a background area in the saliency map; The calculation unit is used to calculate a first QP compensation value according to a preset QP reference value, a first calculation model, and a pixel value of the first coding block; calculate a second QP compensation value according to the preset QP reference value, a second calculation model, and a pixel value of the second coding block; the first QP compensation value is used to adjust a first QP initial value of the foreground area; the second QP compensation value is used to adjust a second QP initial value of the background area; The encoding unit is configured to perform a sum operation on the first QP compensation value and the first QP initial value of the first coding block to obtain a first QP value of the first coding block; perform a sum operation on the second QP compensation value and the second QP initial value of the second coding block to obtain a second QP value of the second coding block; encode the first coding block according to the first QP value, and encode the second coding block according to the second QP value to generate a bitstream.

10. A terminal, characterized in that: The terminal includes a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images