Predictive coding method and device and storage medium

Abstract

The invention discloses a predictive coding method and device and a storage medium, and the method comprises the steps: inputting the position information of each coding unit in at least a part of coding units in a current frame, and the position information of a corresponding coding unit of each coding unit in a reference frame into a position prediction filter, the position prediction filter predicts the position information of each coding unit in the backward frame by using the position information of each coding unit in the current frame and the corresponding position information of the reference frame; and coding the backward frame by using the position information of at least part of the coding units in the backward frame. Through the above mode, the coding accuracy and efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a predictive coding method, device and storage medium. Background Art

[0002] In the field of video predictive coding, the traditional predictive coding method based on optical flow affine correction has certain limitations. This method first assumes that the motion between image frames has undergone translation, rotation, enlargement or reduction or other irregular motion, and generates sub-block prediction based on affine motion compensation of sub-blocks. It then assumes that the motion of the sub-block is smooth, and corrects the motion vector difference between the affine and original sub-blocks through the optical flow gradient value, thereby taking into account motion compensation in both spatial and temporal domains, which can improve prediction accuracy, reduce distortion and bit rate to a certain extent.

[0003] However, the prediction coding method based on optical flow affine correction has reduced accuracy for scenes with large noises, and since each sub-block requires additional gradient and affine calculations, the amount of calculation increases significantly, seriously affecting the coding performance.

[0004] Therefore, there is an urgent need for a new predictive coding method that can reduce noise interference, improve prediction accuracy, reduce the amount of inter-frame prediction calculations, and further effectively improve coding performance. Summary of the invention

[0005] In order to solve the above technical problems, the technical solution adopted in this application is: to provide a prediction coding method, device and storage medium, so as to at least solve the problem that the prediction coding results in the related technology are of low accuracy and large calculation amount, resulting in reduced coding performance.

[0006] According to an embodiment of the present invention, a predictive coding method is provided, comprising: Inputting position information of each coding unit in at least part of the coding units in the current frame and position information of the corresponding coding unit of each coding unit in the reference frame into a position prediction filter, so that the position prediction filter predicts the position information of each coding unit in the backward frame by using the position information of each coding unit in the current frame and the corresponding position information of the reference frame; wherein the reference frame is a forward frame of the current frame; The backward frame is encoded using position information of at least part of the encoding units in the backward frame.

[0007] To solve the above technical problems, a technical solution adopted in the present application is: to provide an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the predictive coding method in the above technical solution.

[0008] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the predictive coding method in the above technical solution.

[0009] Through the above scheme, the beneficial effect of the present application is: the predictive coding method provided by the present application can more accurately grasp the position changes of each element in the video content by using the position information of the coding unit in the current frame and the reference frame through the position prediction filter, thereby more accurately predicting the position information of the coding unit in the backward frame, and then effectively guiding the encoding process of the backward frame, so that the encoded video is closer to the original video during reconstruction, improving the accuracy and efficiency of encoding, reducing the encoding redundancy and errors caused by inaccurate position information, and improving the accuracy and efficiency of predictive coding, thereby improving the overall quality and performance of video encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 It is a flowchart of an embodiment of a predictive coding method provided by the present application; Figure 2 is a flow chart of a Kalman filter working example provided by the present application; Figure 3 is a flowchart of another embodiment of the predictive coding method provided by the present application; Figure 4 It is a flowchart of a coding unit division method provided by the present application when no target is detected; Figure 5 It is a flowchart of a method for dividing coding units when a target is detected provided by the present application; Figure 6 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application; Figure 7 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0011] The present application is further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present application, but are not intended to limit the scope of the present application. Similarly, the following examples are only some embodiments of the present application rather than all embodiments, and all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0012] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0013] It should be noted that the terms "first", "second", etc. in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", etc. can explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0014] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of the predictive coding method provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not necessarily use Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes: S110: Inputting position information of each coding unit in at least part of the coding units in the current frame and position information of a corresponding coding unit of each coding unit in the reference frame to a position prediction filter.

[0015] In this embodiment, the current frame refers to an image frame that is being encoded during the video encoding process. The reference frame is a forward frame of the current frame, which is an encoded frame used to predict the current frame. The amount of encoded data can be reduced by the correlation between the reference frame and the current frame. The coding unit is the basic unit for dividing and processing an image in video encoding, and each coding unit contains a certain number of pixels.

[0016] In the prediction coding scheme provided in the present application, for each image frame, it is necessary to divide the image frame into image regions and determine the image division result in the image frame, thereby obtaining multiple coding units in the image frame.

[0017] The image division method includes fixed-size division, quadtree division, and hybrid division combining multiple strategies, etc., which are not limited in this application. In one example, the image frame is divided into multiple image blocks according to a preset fixed size, and the multiple image blocks are grouped according to the image content in the image block, and the image blocks with similar content are divided into an image block group, and each image block group is determined as a coding unit, and finally the image division result of the image frame is obtained. Exemplarily, the preset fixed size is 16×16 pixels, and the image frame is divided into multiple image blocks according to the fixed size, and the gradient texture value of each image block is calculated respectively. The similarity between image blocks is compared by image similarity calculation methods such as SSIM (Structural Similarity), and image blocks with similarities are divided into the same image block group. Each image block group is determined as a coding unit, and finally the image division result of the image frame is obtained.

[0018] For each coding unit, its position information in the current frame and the reference frame is obtained by coordinate calculation, etc. Exemplarily, a pixel coordinate-based method is adopted to record the horizontal and vertical coordinates of the preset pixel points of the coding unit in the image as the position information, wherein the preset pixel points of the coding unit may be diagonal pixels of the coding unit.

[0019] At least part of the acquired coding unit position information in the current frame and the corresponding coding unit position information in the reference frame are input into the filter in accordance with the input format and order specified by the position prediction filter. The position prediction filter can accurately predict the positions of these coding units in the backward frame based on the position information of the coding units in the current image frame and the reference frame. By accurately predicting the position of the coding units, it provides strong support for subsequent encoding operations, thereby improving the overall video or image encoding quality; preferably, the position prediction filter is a Kalman filter, which is an optimal estimation filter based on a linear system state space model.

[0020] After receiving the input information, the position prediction filter starts the internal algorithm module. The module analyzes and processes the input position information according to the preset algorithm rules. For example, the algorithm module can use a model trained by machine learning to match and predict the current input position information based on the relationship between position information and encoding mode learned from a large amount of existing video data, and output the predicted encoding mode and parameters of the current encoding unit.

[0021] Exemplarily, a Kalman filter is used to analyze and process the input position information, and its working process is mainly divided into two core steps: a prediction step and an update step. In the prediction step, the position of the coding unit of the current frame is predicted based on the coding unit position estimate of the previous frame and the system dynamics (such as the law of object motion). In the update step, the predicted position is corrected in combination with the coding unit position information actually observed from the current frame and the reference frame to obtain a more accurate estimation of the coding unit position. By repeatedly repeating the prediction and update steps, the Kalman filter can continuously and accurately predict the coding unit position between different frames, and can effectively track the position changes of the coding unit even in the presence of system noise and observation noise, thereby improving the accuracy and quality of encoding.

[0022] In one embodiment, see Figure 2 , Figure 2 This is a flow chart of a Kalman filter working example provided by the present application, which is as follows: the diagonal vertex coordinates of each coding unit in the reference frame are respectively input into the Kalman filter for initialization; the diagonal vertex coordinates of the corresponding coding unit in the current frame are input to predict the backward frame; the prediction distortion of the original frame and the predicted frame of the backward frame is calculated, and it is determined whether the prediction distortion exceeds a preset threshold; the coding unit whose prediction distortion does not exceed the preset threshold is encoded, the backward frame is used as the current frame, and the diagonal vertex coordinates of the corresponding coding unit in the input current frame are returned to perform the step of predicting the backward frame; the image area where the coding unit whose prediction distortion exceeds the preset threshold is located is re-divided into coding units.

[0023] After determining a reference frame and a plurality of coding units in a current frame, at least some of the coding units in the current frame and coding units (CUs) corresponding to at least some of the coding units in the reference frame are determined, and position information of each coding unit in at least some of the coding units in the current frame and position information of the corresponding coding unit of each coding unit in the reference frame are input into a position prediction filter, so that the position prediction filter predicts the position information of each coding unit in a backward frame by using the position information of each coding unit in the current frame and the corresponding position information of the reference frame, and applies the position information of each coding unit in the backward frame to the actual encoding process of the coding unit of the next frame.

[0024] S120: Encode the backward frame using the position information of at least part of the encoding units in the backward frame.

[0025] In the video coding sequence, the backward frame is the next frame of the current frame or any subsequent image frame of the current frame. The backward frame is encoded using the position information of at least part of the coding units in the current frame obtained in step S110.

[0026] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of another embodiment of the predictive coding method provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not necessarily use Figure 3 The process sequence shown is limited. Figure 3 As shown, this embodiment includes: S210: Inputting position information of a plurality of coding units in a reference frame to a position prediction filter.

[0027] Determine a plurality of coding units in a reference frame that need to be input into a position prediction filter, input the position information of the plurality of coding units into the position prediction filter for initialization or combine the prediction information in the position prediction filter to predict the position information of each coding unit in a backward frame of the reference frame (i.e., the current frame described later), wherein the prediction information is information based on which the position prediction filter predicts the position of the coding unit, including the position information of the coding unit in historical frames, the motion vector of the coding unit and / or the model parameters of the filter itself, and the position prediction is completed by analyzing this information and combining it with the currently input coding unit.

[0028] In one embodiment, a first coding unit and a second coding unit in a reference frame are determined, wherein the first coding unit is at least part of the coding units in the reference frame, and at least part of the coding units in the reference frame are coding units in the reference frame whose prediction distortion is less than a preset threshold; the second coding unit is a coding unit in the reference frame other than the first coding unit, and may include a coding unit in the reference frame whose prediction distortion is not less than a preset threshold and / or a coding unit without prediction information. By dividing the coding units in the reference frame into the first coding unit and the second coding unit, different processing methods can be used for coding units with different characteristics, so as to improve the accuracy of predictive coding.

[0029] If the first coding unit exists in the reference frame, the position information of the first coding unit in the reference frame is input into the position prediction filter, so as to combine the prediction information in the filter to predict the position information of each coding unit in the backward frame.

[0030] If there is a second coding unit in the reference frame, the position information of at least part of the second coding unit in the reference frame is input into the position prediction filter to combine the prediction information in the filter to predict the position information of each coding unit in the backward frame; and / or, the position information of at least part of the second coding unit in the reference frame is input into the position prediction filter for initialization.

[0031] S220: Inputting position information of coding units in the current frame and the reference frame that have a corresponding relationship to the position prediction filter.

[0032] If there is a correspondence between the coding units in the current frame and the coding units in the reference frame, the position information of the coding units in the current frame and the coding units in the reference frame that have a corresponding relationship is input into the position prediction filter, so that the position prediction filter uses the position information of each coding unit in the current frame and the corresponding position information of the reference frame to predict the position information of each coding unit in the backward frame.

[0033] Among them, the correspondence relationship indicates that the coding units in the current frame and the reference frame have similarities or associations in content, structure or position, and a mapping relationship determined by a specific algorithm. In one example, an algorithm based on feature matching can be used to extract features of the coding units of the current frame and the reference frame, such as extracting texture features, edge features, etc. By comparing these features, the correspondence between the coding units in the two frames is determined. For example, for coding units with similar texture distribution and edge contours, it is determined that they have a correspondence. In another example, the correspondence between the coding units in the current frame and the reference frame can also be determined based on the position information of each coding unit in the reference frame predicted from the position information of the current frame. There are many methods for determining the correspondence between the coding units in the current frame and the coding units in the reference frame, which are not limited here.

[0034] In one embodiment, the coding units in the current frame and the coding units in the reference frame that have a corresponding relationship include coding units in the current frame that correspond to the first coding unit in the reference frame. At least some of the coding units in the current frame that correspond to the first coding unit in the reference frame are determined to be at least some of the coding units of the current frame, and at least some of the coding units of the current frame are input into the position prediction filter.

[0035] If the position prediction filter predicts the position information of the first coding unit in the current frame based on the position information of the first coding unit, the current frame is predicted using the reference frame and the position information of the first coding unit in the current frame to obtain a predicted frame of the current frame, wherein the predicted frame includes a prediction block of the coding unit corresponding to the first coding unit in the current frame.

[0036] The coding units whose predicted position information in the current frame is relatively accurate are determined to be at least part of the coding units of the current frame, and the position information of these coding units is input into the position prediction filter to predict the position information in the backward frame. Specifically, a first difference threshold can be preset to calculate the degree of difference between each coding unit in the original frame of the current frame and the corresponding coding unit in the predicted frame. If the degree of difference is greater than the first difference threshold, it means that the predicted position information of the coding unit is inaccurate; if the degree of difference is not greater than the first difference threshold, it means that the predicted position information of the coding unit is relatively accurate, and the coding units whose predicted position information in the current frame is relatively accurate are determined to be at least part of the coding units of the current frame.

[0037] In one example, the original frame of the current frame is matched with the predicted frame to determine the prediction distortion of each coding unit corresponding to the first coding unit in the current frame. The coding units at the same position of the original frame and the predicted frame of the current frame are matched, and the prediction distortion between the coding units is calculated using the mean square error (MSE). The difference between each pixel position of the coding unit in each pair of matched original frames and predicted frames is calculated to obtain the MSE value of the coding unit. The MSE value reflects the difference of the coding unit at the pixel level. The smaller the value, the higher the prediction accuracy. In other examples, other methods such as structural similarity (SSIM) can also be used to calculate the prediction distortion of each coding unit in the original frame and the predicted frame, which is not limited here.

[0038] After obtaining the prediction distortion of each coding unit corresponding to the first coding unit in the current frame, a preset threshold is set, and the relationship between the prediction distortion of each coding unit and the preset threshold is compared. The position information of the coding unit whose prediction distortion is less than the preset threshold in the current frame is input into the position prediction filter, so that the position prediction filter uses the position information of the coding unit whose prediction distortion is less than the preset distortion threshold and the position information of the corresponding coding unit in the reference frame to predict the position information of the coding unit whose prediction distortion is less than the preset distortion threshold in the backward frame.

[0039] The position information of the coding units in the current frame whose prediction distortion is not less than a preset threshold is cleared, and the prediction information of these coding units stored in the position prediction filter is cleared, so as to re-determine at least one coding unit and its position information in the image area where the coding units in the current frame whose prediction distortion is greater than the preset threshold are located based on the original frame of the current frame.

[0040] In one embodiment, the coding unit in the current frame and the coding unit in the reference frame that have a corresponding relationship also includes the coding unit in the current frame that corresponds to the second coding unit. The coding unit in the current frame that corresponds to the second coding unit is determined, and its position information is input into the position prediction filter. The second coding unit includes a third coding unit and / or a fourth coding unit, the third coding unit is a coding unit in the reference frame whose prediction distortion is greater than a preset threshold, and the fourth coding unit is a coding unit in the reference frame that inputs position information into the filter for initialization.

[0041] If a coding unit corresponding to the third coding unit exists in the current frame, the coding unit corresponding to the third coding unit in the current frame is determined, and its position information is input into the position prediction filter for initialization.

[0042] If there is a coding unit corresponding to the fourth coding unit in the current frame, the coding unit corresponding to the fourth coding unit in the current frame is determined, and its position information is input into the position prediction filter, so that the position prediction filter uses the position information of the fourth coding unit and the position information of the coding unit corresponding to the fourth coding unit in the current frame for prediction.

[0043] Before calculating the degree of difference between each coding unit in the original frame of the current frame and the corresponding coding unit in the predicted frame, it is necessary to first obtain the original frame of the current frame and the coding unit division method in the original frame. The original frame may have some areas with a large degree of change compared to the reference frame, and it is necessary to redetermine the coding unit division method of these areas to obtain the actual coding unit division result of the current frame.

[0044] In one embodiment, a deep learning method such as a target detection algorithm may be used to determine the background area and the target area in the original frame. If there are multiple target areas, each target area may be numbered in advance. The coding units in the background area and the target area in the original frame and the corresponding areas in the reference frame are matched respectively, and the area to be updated in the original frame of the current frame is determined. The division method of the coding units in the area to be updated is updated to obtain at least one coding unit in the area to be updated.

[0045] In one example, a target detection algorithm is run to divide the image area where each target is located into a target area, and the remaining image area where no target is detected is classified as a background area. If there is a target area corresponding to a newly appeared target in the original frame, the target area can be divided into images using a preset image division method to determine at least one coding unit in the target area. If there is a target area in the original frame that has the same target number as that in the reference frame, the degree of difference between each coding unit in the target area in the reference frame and the original frame is further determined. If the degree of difference is greater than a second difference threshold, the area where the coding unit is located is designated as an area to be updated; if the degree of difference is less than the second difference threshold, the coding unit division method for the area corresponding to the coding unit in the reference frame is used.

[0046] Exemplarily, the frame difference method is used to determine the texture change degree of each coding unit in the original frame and the reference frame, and the area to be updated in the original frame is determined based on the texture change degree of each coding unit. A texture change degree threshold is preset, and for the area where all coding units whose texture change degree does not exceed the texture change degree threshold are located, the division method of the corresponding image area in the reference frame is used; for the area where all coding units whose texture change degree exceeds the texture change degree threshold are located in the original frame, the area to be updated is divided into multiple image blocks according to a preset fixed size, and the gradient texture value of each image block is calculated respectively. The similarity between image blocks is compared by using image similarity calculation methods such as SSIM (Structural Similarity), and image blocks with similarity are divided into the same image block group, and each image block group is determined as a coding unit, and finally the coding unit division method of the area to be updated is obtained.

[0047] In some embodiments, if the prediction distortion of the third coding unit in the reference frame is greater than a preset threshold or the position information of the coding unit in the current frame whose prediction distortion is not less than the preset threshold is cleared, the image area in the current frame corresponding to the area where the third coding unit in the reference frame is located and the area where the coding unit in the current frame whose prediction distortion is not less than the preset threshold is located have no image division results. In this case, if there is an area without an image division result in the prediction frame of the current frame, the area is determined to be the image area to be updated, and the coding unit division is performed again on the image area to be updated to determine at least one coding unit in the image area to be updated.

[0048] In one embodiment, a first image region to be updated in the current frame is determined based on a target detection result of the current frame. The first image region to be updated is divided into coding units to determine at least one coding unit in the first image region to be updated. For a first region other than the first image region to be updated in the current frame, the coding unit division method of the corresponding region in the reference frame is used to determine at least one coding unit in the first region, wherein the coding unit of the corresponding region in the reference frame includes at least one of the third coding unit and the fourth coding unit, and at least one coding unit in the first region corresponds one-to-one to the coding unit of the corresponding region in the reference frame.

[0049] In another embodiment, after determining a prediction block whose prediction distortion is less than a preset threshold, a second image area to be updated in the second area other than the prediction block whose prediction distortion is less than the preset threshold is determined based on the target detection result of the current frame. The second image area to be updated is divided into coding units to determine at least one coding unit in the second image area to be updated. For a third area in the second area other than the second image area to be updated, the coding unit division method of the corresponding area in the reference frame is used to determine at least one coding unit in the third area, wherein the coding unit of the corresponding area in the reference frame includes at least one of the third coding unit and the fourth coding unit, and at least one coding unit in the third area corresponds one-to-one to the coding unit of the corresponding area in the reference frame.

[0050] In other embodiments, a first image region to be updated in the current frame is determined based on the target detection result of the current frame, and the first image region to be updated is divided into coding units to determine at least one coding unit in the first image region to be updated. For the first region other than the image region to be updated in the current frame, the coding unit division method of the corresponding region in the reference frame is used to determine at least one coding unit in the first region, and at least one coding unit in the first region corresponds one-to-one to the coding unit of the corresponding region in the reference frame. The position information of at least one coding unit in the first region is input into the position prediction filter, so that the position prediction filter uses the position information of at least part of the coding units in the first region and the position information of the corresponding coding units in the reference frame to predict the position information of at least part of the coding units except the first region in the backward frame.

[0051] S230: Encode the backward frame using the position information of at least part of the encoding units in the backward frame.

[0052] The predicted frame of the backward frame is obtained by using the position information of the coding units in the current frame and the reference frame that have a corresponding relationship in the backward frame. The predicted frame includes the predicted blocks of the coding units in the backward frame that correspond to at least part of the coding units in the current frame.

[0053] The original frame and the predicted frame of the backward frame are matched to determine the difference between at least part of the coding units in the backward frame and the current frame, so as to predict subsequent frames using the coding units whose difference is not greater than a first difference threshold.

[0054] In one embodiment, the original frame and the predicted frame of the backward frame are matched to determine the prediction distortion of each coding unit in the backward frame corresponding to at least part of the coding units in the current frame. The coding unit corresponding to the prediction block is encoded using the prediction block in the backward frame whose prediction distortion is less than a preset threshold.

[0055] Assume that the coding unit in the backward frame whose prediction distortion is less than a preset threshold is the fifth coding unit. If the fifth coding unit exists in the backward frame, the current frame is used as the reference frame, the backward frame is used as the current frame, and at least some of the coding units of the current frame include the fifth coding unit. Return to execute the step of inputting the position information of the coding units that have a corresponding relationship between the current frame and the coding units in the reference frame into the position prediction filter.

[0056] In one embodiment, if there is a fifth coding unit in the backward frame, the current frame is used as a reference frame, and the backward frame is used as the current frame. It is assumed that the sixth coding unit is a coding unit in the current frame whose prediction distortion is greater than a preset threshold. For the sixth coding unit in the current frame, a coding unit division method for the area where the sixth coding unit is located is determined based on the target detection result of the current frame to determine at least one coding unit in the area where the sixth coding unit is located, and at least one coding unit in the area where the sixth coding unit is located is encoded.

[0057] In one embodiment, if there is a fifth coding unit in the backward frame, the current frame is used as a reference frame, and the backward frame is used as the current frame. The coding unit division method of the area other than the area where the fifth coding unit and the sixth coding unit are located in the current frame is determined based on the target detection result of the current frame to determine at least one coding unit in the area other than the area where the fifth coding unit and the sixth coding unit are located in the current frame, and encode at least one coding unit in the area other than the area where the fifth coding unit and the sixth coding unit are located in the current frame.

[0058] In one embodiment, if at least one coding unit in an area other than the area where the fifth coding unit and the sixth coding unit are located in the current frame includes a coding unit in the seventh coding unit that has a corresponding relationship with the reference frame and whose position information has been input once into the position prediction filter, at least part of the coding units of the current frame include coding units in the seventh coding unit and whose position information has been input once into the position prediction filter.

[0059] In one embodiment, if at least one coding unit in an area other than the area where the fifth coding unit and the sixth coding unit are located in the current frame includes a coding unit in the seventh coding unit that has a corresponding relationship with the reference frame and whose position information is not input into the position prediction filter, the position information of the coding unit in the seventh coding unit whose position information is not input into the position prediction filter is input into the position prediction filter for initialization.

[0060] In order to more clearly illustrate the predictive coding method provided by the present application, this embodiment takes the natural frame order in the video sequence frame by frame as an example, and provides the following specific embodiments of the predictive coding method for exemplary illustration: 1. Processing of the first frame: The CNN algorithm is used to detect foreground objects in the first frame. Figure 4 For example, Figure 4 This is a flow chart of the coding unit division method provided by the present application when no target is detected. The specific process is as follows: If no target is detected, the first frame is the background frame, and the image area of ​​the current frame is the background area. This background frame is used as the starting frame of the encoder, and its YUV data is processed. The background area is divided into multiple image blocks according to the preset image block size (such as 16×16 pixels), and the gradient texture of each image block is calculated respectively. A first image texture similarity threshold is set, and the texture similarity between image blocks is calculated by the image similarity calculation method. The image blocks with texture similarity greater than the first image texture similarity threshold are grouped together, and the image blocks in the same group are determined to be the same coding unit (CU), and finally all the coding units in the background area are determined.

[0061] It should be noted that, in the present embodiment, the background frame is used as the starting frame of the encoder, so that a stable background reference model can be established for subsequent video processing. In the absence of target interference, the texture features of the background frame are relatively simple, and it is easier to group the minimum macroblocks through the image similarity calculation method, thereby reducing the complexity of subsequent processing. In addition, the coding unit determined based on the background frame can continue to be used in subsequent frames as long as the background does not change significantly, and only the target area and the changed background area need to be adjusted, thereby improving the overall efficiency of coding.

[0062] 2. Processing of the second frame: Before the first detection of a target, any background frame in which no target is detected is considered the first frame, and the image frame in which the target is first detected is considered the second frame. Figure 5 , Figure 5 This is a flow chart of the method for dividing the coding unit when a target is detected provided by the present application. In the processing of the second frame in which the target is detected, the target detection algorithm is run, foreground target detection is performed on the second frame, and the image area where each target is located is divided into a target area, and an ID identifier is set for each target area. The remaining image area where no target is detected is divided into a background area, and the division method of the coding units of each target area and background area in the second frame is determined in the following way.

[0063] A. In each target area, the target area is divided according to a preset image block size in the target detection frame to obtain multiple image blocks in the target area, a second image texture similarity threshold is set, and the texture similarity between the image blocks is calculated by an image similarity calculation method. The image blocks with texture similarity greater than the second image texture similarity threshold are grouped together, and the image blocks in the same group are determined to be the same coding unit, and finally all the coding units in each target area are determined.

[0064] B. In the background area, based on the degree of texture change of each coding unit in the first frame and the second frame, first determine whether the image area where each coding unit is located needs to be updated, determine the area to be updated in the background area, and re-divide the coding units in the image area that needs to be updated.

[0065] A texture change degree threshold is preset, and for the regions where all coding units whose texture change degrees do not exceed the texture change degree threshold are located, the coding unit division method of the corresponding image region in the first frame (i.e., the reference frame of the second frame) is used; For the area where all coding units whose texture change degree exceeds the texture change degree threshold in the second frame are located, the area to be updated is divided into multiple image blocks according to a preset fixed size, and the gradient texture value of each image block is calculated respectively. The similarity between image blocks is compared by image similarity calculation methods such as SSIM (Structural Similarity), and image blocks with similar similarity are divided into the same image block group. Each image block group is determined as a coding unit, and finally the coding unit division method of the area to be updated is obtained.

[0066] At this point, after obtaining the coding units of all image areas in the second frame, the position information of each coding unit, such as the diagonal vertex coordinates of the coding unit, is obtained. The diagonal vertex coordinates of each coding unit are input into the Kalman filter for initialization. The Kalman filter is used to predict the position change of the coding unit in the future. The initialization operation is to enable the filter to perform subsequent prediction calculations based on the position information of each coding unit in the second frame, laying the foundation for subsequent dynamic tracking and prediction.

[0067] 3. Processing of the third frame: Run the target detection algorithm, perform foreground target detection on the third frame, and update the coding unit division method in the image frame based on the target detection result of the third frame to obtain the coding unit division method in the third frame and the corresponding relationship between each coding unit in the third frame and the coding unit in the second frame. The updating method of the coding unit division method is as follows: (1) For the target area corresponding to the newly appeared target in the third frame: the image blocks in each target area are grouped again according to texture similarity. Image blocks with similar texture similarity are grouped together. The image blocks in the same group are determined to be the same coding unit. The coding unit division method of the newly appeared target area in the third frame is determined.

[0068] (2) For the target that appeared in the second frame (consistent with the target ID in the second frame), it corresponds to the target area in the third frame and the background area in the third frame: based on the degree of texture change of each coding unit in the second frame and the third frame, first determine whether the image area where each coding unit is located needs to be updated, determine the area to be updated in the target area and / or background area, and re-divide the coding units in the image area that needs to be updated.

[0069] A texture change degree threshold is preset, and for the regions where all coding units whose texture change degrees do not exceed the texture change degree threshold are located, the coding unit division method of the corresponding image region in the second frame (i.e., the reference frame of the third frame) is used; For the area where all coding units whose texture change degree exceeds the texture change degree threshold in the second frame are located, the area to be updated is divided into multiple image blocks according to a preset fixed size, and the gradient texture value of each image block is calculated respectively. The similarity between image blocks is compared by image similarity calculation methods such as SSIM (Structural Similarity), and image blocks with similar similarity are divided into the same image block group. Each image block group is determined as a coding unit, and finally the coding unit division method of the area to be updated is obtained.

[0070] At this point, the coding unit division method of all image areas in the third frame and the correspondence between each coding unit in the third frame and the second frame are obtained, and the diagonal vertex coordinates of each coding unit are input into the initialized Kalman filter. The Kalman filter predicts the position information of each coding unit in the fourth frame based on the position information of each coding unit in the second frame and the third frame.

[0071] 4. Processing of the fourth frame: The fourth frame is encoded based on the predicted position information of each coding unit in the fourth frame to obtain a predicted frame of the fourth frame. The target detection algorithm is run to obtain the coding unit division method of the original frame of the fourth frame based on the target detection result and the coding unit division method of the third frame, so as to evaluate the accuracy of the prediction based on the degree of difference between the predicted frame and the original frame of the fourth frame. The updating of the coding unit division method of the original frame of the fourth frame is consistent with the processing logic of the updating method of the coding unit division method in the third frame, which will not be repeated here.

[0072] After determining the coding unit division method in the original frame of the fourth frame, the prediction distortion of the coding units at the same position in the original frame of the fourth frame and the predicted frame is calculated to evaluate the accuracy of the prediction of the coding units in the predicted frame. For the area where the coding unit prediction distortion is less than the preset threshold, it means that the prediction of the predicted frame in the area (background or target) is relatively accurate; for the area where the coding unit prediction distortion is not less than the preset threshold, it means that there is a large error in the prediction of the predicted frame in the area, and the encoding process may need to be adjusted, such as re-dividing the CU, adjusting the motion vector, or changing the quantization parameter, etc., to improve the quality of video encoding.

[0073] Exemplarily, for the first image area where the coding units whose prediction distortion is less than a preset threshold are located, the coding unit division method of the first image area in the prediction frame is retained, and the position information of these coding units is input into the Kalman filter so as to predict the corresponding position information of each coding unit in the next frame based on the coding unit position information in the first image area.

[0074] For the second image area where the coding unit whose prediction distortion is less than the preset threshold is located, the coding unit division method within the second image area in the prediction frame is cleared, and the image data of the second image area in the prediction frame can be updated based on the image area in the original frame corresponding to the second image area of ​​the prediction frame.

[0075] 5. Processing of the fifth frame: The fifth frame is encoded based on the predicted position information of at least part of the coding units in the fifth frame to obtain a predicted frame of the fifth frame.

[0076] (1) For the coding units in the image area (the image area corresponding to the first image area) in which the coding unit division method is not re-determined, the processing method in the fifth frame is logically consistent with the processing method in the fourth frame, and the processing method in the fourth frame is repeatedly executed, which is not repeated here.

[0077] (2) The position information of each coding unit in the image region (the image region corresponding to the second image region) whose coding unit division method is newly determined is input to the Kalman filter for initialization.

[0078] 6. Processing of the sixth frame: The sixth frame is encoded based on the predicted position information of at least part of the coding units in the sixth frame to obtain a predicted frame of the sixth frame.

[0079] (1) For the coding units with prediction information in the image area (the image area corresponding to the first image area in the predicted frame of the fifth frame) in which the coding unit division method is not re-determined, the processing method of the fourth frame is repeatedly executed; for the coding units without prediction information in the image area, the processing method of the third frame is repeatedly executed, which will not be repeated here.

[0080] (2) The processing method of the fifth frame is repeatedly executed for the coding units in the image area (the image area corresponding to the second image area in the predicted frame of the fifth frame) in which the coding unit division method is re-determined, and no further details are given here.

[0081] 7. Processing of subsequent frames: The processing methods of subsequent frames after the sixth frame are consistent with the processing logic of the sixth frame, thereby classifying and processing the coding units or image areas in different situations to ensure that the processing of each frame can make full use of previous information and appropriate methods, thereby improving the coding efficiency. I will not go into details here.

[0082] Thus, in this embodiment, by continuously looping the above-mentioned predictive coding process, it is possible to fully utilize inter-frame information, thereby reducing data redundancy and improving coding efficiency and video quality.

[0083] See also Figure 6 , Figure 6 It is a structural diagram of an embodiment of an electronic device provided in the present application. The electronic device 60 includes a memory 61 and a processor 62 connected to each other. The memory 61 is used to store a computer program. When the computer program is executed by the processor 62, it is used to implement the predictive coding method in the above embodiment.

[0084] The method of the above embodiment may exist in the form of a computer program, so the present application proposes a computer readable storage medium, see Figure 7 , Figure 7 It is a schematic diagram of the structure of an embodiment of a computer-readable storage medium provided in the present application. The computer-readable storage medium 80 is used to store a computer program 81, which can be executed to implement the predictive coding method in the above embodiment.

[0085] The computer-readable storage medium 80 may be a server, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which may store program codes.

[0086] The above description is only an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A predictive coding method, characterized in that: The method comprises: Inputting the position information of each coding unit in at least part of the coding units in the current frame and the position information of the corresponding coding unit of each coding unit in the reference frame into the position prediction filter, so that the position prediction filter predicts the position information of each coding unit in the backward frame by using the position information of each coding unit in the current frame and the corresponding position information of the reference frame; wherein the reference frame is the forward frame of the current frame; The backward frame is encoded using position information of at least part of the encoding units in the backward frame.

2. The predictive coding method according to claim 1, characterized in that: The step of inputting the position information of each coding unit in at least part of the coding units in the current frame and the position information of the corresponding coding unit of each coding unit in the reference frame into the position prediction filter comprises: Inputting position information of a plurality of coding units in the reference frame into the position prediction filter; The position information of the coding units in the current frame and the reference frame that have a corresponding relationship is input to the position prediction filter.

3. The predictive coding method according to claim 2, characterized in that: The step of inputting the position information of the coding units in the current frame and the reference frame corresponding to each other into the position prediction filter comprises: If the position prediction filter predicts the position information of the first coding unit in the current frame based on the position information of the first coding unit, the current frame is predicted using the reference frame and the position information of the first coding unit in the current frame to obtain a predicted frame of the current frame, wherein the first coding unit is at least part of the coding units in the reference frame, and the predicted frame includes a predicted block of the coding unit corresponding to the first coding unit in the current frame; Matching an original frame of the current frame with the predicted frame to determine prediction distortions of respective coding units in the current frame corresponding to the first coding unit; The step of inputting the position information of the coding units in the current frame and the reference frame that have a corresponding relationship to the position prediction filter comprises: The position information of the coding unit in the current frame whose prediction distortion is less than a preset threshold is input into the position prediction filter, so that the position prediction filter uses the position information of the coding unit in the current frame whose prediction distortion is less than the preset distortion threshold and the position information of the corresponding coding unit in the reference frame to predict the position information of the coding unit in the backward frame whose prediction distortion is less than the preset distortion threshold.

4. The predictive coding method according to claim 3, characterized in that: The step of inputting the position information of the coding units in the current frame and the reference frame that have a corresponding relationship to the position prediction filter further includes: A coding unit in the current frame corresponding to a second coding unit in the reference frame other than the first coding unit is determined, and position information thereof is input into the position prediction filter.

5. The predictive coding method according to claim 4, characterized in that: The determining a coding unit in the current frame corresponding to a second coding unit other than the first coding unit in the reference frame, and inputting position information thereof into the position prediction filter, comprises: The second coding unit includes a third coding unit, the third coding unit is a coding unit in the reference frame whose prediction distortion is greater than a preset threshold; determining a coding unit in the current frame corresponding to the third coding unit, and inputting its position information into the position prediction filter for initialization; and / or, The second encoding unit includes a fourth encoding unit, which is an encoding unit that inputs position information in the reference frame into the position prediction filter for initialization; determines the encoding unit corresponding to the fourth encoding unit in the current frame, and inputs its position information into the position prediction filter, so that the position prediction filter uses the position information of the fourth encoding unit and the position information of the encoding unit corresponding to the fourth encoding unit in the current frame for prediction.

6. The predictive coding method according to claim 5, characterized in that: The determining a coding unit in the current frame corresponding to a second coding unit other than the first coding unit in the reference frame, and inputting position information thereof into the position prediction filter, comprises: Determine a first image region to be updated in the current frame based on the target detection result of the current frame; divide the first image region to be updated into coding units to determine at least one coding unit in the first image region to be updated; For a first area other than the first image area to be updated in the current frame, the coding unit division method of the corresponding area in the reference frame is used to determine at least one coding unit in the first area, wherein the coding unit of the corresponding area in the reference frame includes at least one of the third coding unit and the fourth coding unit, and the at least one coding unit in the first area corresponds one-to-one to the coding unit of the corresponding area in the reference frame; or, After determining the prediction block whose prediction distortion is less than a preset threshold, determine, based on the target detection result of the current frame, a second image area to be updated in a second area other than the prediction block whose prediction distortion is less than the preset threshold; divide the second image area to be updated into coding units to determine at least one coding unit in the second image area to be updated; For a third area other than the second image area to be updated in the second area, the coding unit division method of the corresponding area in the reference frame is used to determine at least one coding unit in the third area, wherein the coding unit of the corresponding area in the reference frame includes at least one of the third coding unit and the fourth coding unit, and at least one coding unit in the third area corresponds one-to-one to the coding unit of the corresponding area in the reference frame.

7. The predictive coding method according to claim 2, characterized in that: The step of inputting position information of coding units in the current frame and the reference frame that have a corresponding relationship to each other into the position prediction filter comprises: determining a first image region to be updated in the current frame based on the target detection result of the current frame; dividing the first image region to be updated into coding units to determine at least one coding unit in the first image region to be updated; for a first region other than the image region to be updated in the current frame, dividing the coding units of the corresponding region in the reference frame to determine at least one coding unit in the first region, wherein the at least one coding unit in the first region corresponds one-to-one to the coding unit of the corresponding region in the reference frame; The position information of at least one coding unit in the first area is input into the position prediction filter, so that the position prediction filter uses the position information of at least part of the coding units in the first area and the position information of the corresponding coding units in the reference frame to predict the position information of at least part of the coding units except the first area in the backward frame.

8. The predictive coding method according to claim 2, characterized in that: The encoding of the backward frame by using the position information of at least part of the encoding units in the backward frame comprises: Obtaining a prediction frame of the backward frame using the position information of the coding units in the current frame and the reference frame that have a corresponding relationship with each other, and the position information of the backward frame, wherein the prediction frame includes prediction blocks of the coding units in the backward frame that correspond to at least part of the coding units in the current frame; matching an original frame of the backward frame with the predicted frame to determine prediction distortions of respective coding units in the backward frame corresponding to at least some of the coding units in the current frame; Encoding a coding unit corresponding to a prediction block using a prediction block in the backward frame whose prediction distortion is less than a preset threshold; If there is a fifth coding unit in the backward frame, the fifth coding unit is a coding unit in the backward frame whose prediction distortion is less than a preset threshold, the current frame is used as a reference frame, the backward frame is used as the current frame, and at least some of the coding units of the current frame include the fifth coding unit, return to the step of inputting the position information of the coding units in the current frame that correspond to the coding units in the reference frame into the position prediction filter.

9. The predictive coding method according to claim 8, characterized in that: If there is a fifth coding unit in the backward frame, the fifth coding unit is a coding unit in the backward frame whose prediction distortion is less than a preset threshold, the current frame is used as a reference frame, the backward frame is used as a current frame, and at least some coding units of the current frame include the fifth coding unit, returning to the step of inputting the position information of the coding units in the current frame and the reference frame that have a corresponding relationship into the position prediction filter, including: If the fifth coding unit exists in the backward frame, the current frame is used as a reference frame, and the backward frame is used as the current frame. For the sixth coding unit in the current frame, the sixth coding unit is a coding unit in the current frame whose prediction distortion is greater than a preset threshold. Based on the target detection result of the current frame, a coding unit division method for the area where the sixth coding unit is located is determined to determine at least one coding unit in the area where the sixth coding unit is located, and encode at least one coding unit in the area where the sixth coding unit is located.

10. The predictive coding method according to claim 8, characterized in that: If there is a fifth coding unit in the backward frame, the fifth coding unit is a coding unit in the backward frame whose prediction distortion is less than a preset threshold, the current frame is used as a reference frame, the backward frame is used as a current frame, and at least some coding units of the current frame include the fifth coding unit, returning to the step of inputting the position information of the coding units in the current frame and the reference frame that have a corresponding relationship into the position prediction filter, including: If the fifth coding unit exists in the backward frame, the current frame is used as a reference frame, and the backward frame is used as a current frame, a coding unit division method of an area other than an area where the fifth coding unit and the sixth coding unit are located in the current frame is determined based on a target detection result of the current frame, so as to determine at least one coding unit of an area other than an area where the fifth coding unit and the sixth coding unit are located in the current frame, and at least one coding unit of an area other than an area where the fifth coding unit and the sixth coding unit are located in the current frame is encoded, wherein the sixth coding unit is a coding unit in the current frame whose prediction distortion is greater than a preset threshold; If at least one coding unit in an area other than the area where the fifth coding unit and the sixth coding unit are located in the current frame includes a coding unit in the seventh coding unit that has a corresponding relationship with the reference frame and whose position information has been input once into the position prediction filter, at least some of the coding units of the current frame include coding units in the seventh coding unit whose position information has been input once into the position prediction filter; and / or, if at least one coding unit in an area other than the area where the fifth coding unit and the sixth coding unit are located in the current frame includes a coding unit in the seventh coding unit that has a corresponding relationship with the reference frame and whose position information has not been input into the position prediction filter, the position information of the coding unit in the seventh coding unit that has not been input into the position prediction filter is input into the position prediction filter for initialization.

11. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the processor is coupled to the memory, and the processor is configured to execute one or more steps of the predictive encoding method according to any one of claims 1 to 10 based on instructions stored in the memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the prediction encoding method according to any one of claims 1 to 10.

Images