Video intra-frame coding method, apparatus and device, storage medium, and program product

Abstract

Embodiments of the present application provide a video intra-frame coding method, apparatus and device, a storage medium, and a program product. The method comprises: acquiring a video frame to be processed, and performing human pose detection on said video frame to obtain human body feature points; on the basis of the human body feature points, dividing said video frame into human body regions corresponding to different human body parts, the human body regions including a non-torso region; on the basis of connecting line angle information corresponding to human body feature points in the non-torso region, determining a target angle prediction mode from among candidate angle prediction modes, the connecting line angle information being determined on the basis of an included angle between the line connecting the human body feature points at set positions and a horizontal reference line; and on the basis of the target angle prediction mode, performing intra-frame predictive coding on the non-torso region. The present solution combines human pose recognition with intra-frame prediction mode selection for video coding, improving the hit rate of predicted angles without the need for traversing all intra-frame prediction modes, thereby reducing the amount of calculation and improving the coding efficiency.

Description

Video frame intra coding method, device, equipment, storage medium and program product

[0001] This application claims priority to the Chinese patent application No. 202411072155.6 filed on August 7, 2024 with the China National Intellectual Property Office, the content of which is incorporated herein in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the field of video processing technology, in particular to a video frame intra coding method, device, equipment, storage medium and program product. BACKGROUND

[0003] In the video coding process, the encoder will automatically select a suitable prediction mode according to the characteristics of the video content to adapt to different coding scenarios and needs. The related prediction modes include intra prediction mode and inter prediction mode. Among them, the intra prediction mode includes angle prediction mode and non-angle prediction mode. Taking the VVC (Versatile Video Coding, multi-functional video coding) standard as an example, the intra prediction mode includes 65 ordinary angle prediction modes, planar mode, DC mode and 28 wide angle prediction modes, wherein each angle prediction mode corresponds to a prediction angle. The encoder needs to select a suitable prediction mode from multiple intra prediction modes for video coding.

[0004] In related technologies, video coding needs to traverse all intra prediction modes, and select the mode with the smallest prediction error as the final prediction mode, which has a large amount of calculation, low coding efficiency, and cannot adapt to real-time video live scenes, and needs to be improved. SUMMARY

[0005] Embodiments of the present application provide a video frame intra coding method, device, equipment, storage medium and program product, which solve the problem of large amount of calculation, low coding efficiency and inability to adapt to real-time video live scenes in related technologies. The embodiments of the present application combine human pose recognition with intra prediction mode selection of video coding, realize target-based content adaptive intra coding, effectively utilize the angle information of human skeleton to indicate the target direction of intra prediction, improve the angle hit rate of prediction, do not need to traverse all intra prediction modes, reduce the amount of calculation, improve the coding efficiency, and can adapt to real-time video live scenes.

[0006] In a first aspect, the embodiments of the present application provide a video frame intra coding method, which comprises:

[0007] Obtaining a to-be-processed video frame, performing human pose detection on the to-be-processed video frame to obtain human feature points;

[0008] The human body feature points are used to divide human body regions corresponding to different body parts from the video frame to be processed, and the human body regions include a non-somatic region;

[0009] A target angle prediction mode is determined from candidate angle prediction modes according to angle information of lines connecting the human body feature points in the non-somatic region, the angle information being determined based on an included angle between a connecting line of a human body feature point at a set position and a horizontal reference line;

[0010] The non-somatic region is intra-predictively encoded based on the target angle prediction mode.

[0011] In a second aspect, an embodiment of the present application further provides a video intra-coding device, which comprises:

[0012] An acquisition module configured to acquire a video frame to be processed;

[0013] A feature detection module configured to perform human posture detection on the video frame to be processed to obtain human body feature points;

[0014] A region division module configured to divide human body regions corresponding to different body parts from the video frame to be processed according to the human body feature points, and the human body regions include a non-somatic region;

[0015] A mode determination module configured to determine a target angle prediction mode from candidate angle prediction modes according to angle information of lines connecting the human body feature points in the non-somatic region, the angle information being determined based on an included angle between a connecting line of a human body feature point at a set position and a horizontal reference line;

[0016] An encoding module configured to intra-predictively encode the non-somatic region based on the target angle prediction mode.

[0017] In a third aspect, an embodiment of the present application further provides a video intra-coding device, which comprises:

[0018] One or more processors;

[0019] A storage device configured to store one or more programs,

[0020] When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the video intra-coding method provided by the embodiments of the present application.

[0021] In a fourth aspect, an embodiment of the present application further provides a non-volatile storage medium storing computer executable instructions, which, when executed by a computer processor, are configured to perform the video intra-coding method provided by the embodiments of the present application.

[0022] In a fifth aspect, an embodiment of the present application further provides a computer program product, which comprises a computer program stored in a computer readable storage medium, and at least one processor of a device reads and executes the computer program from the computer readable storage medium, so that the device executes the video frame intra-coding method provided by the embodiment of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0023] FIG. 1 is a flowchart of a video frame intra-coding method provided by an embodiment of the present application;

[0024] FIG. 2 is a distribution diagram of a human feature point provided by an embodiment of the present application;

[0025] FIG. 3 is a result diagram of human feature points obtained by performing human posture detection on a to-be-processed video frame provided by an embodiment of the present application;

[0026] FIG. 4 is a diagram for determining line angle information corresponding to human feature points in a non-somatic region provided by an embodiment of the present application;

[0027] FIG. 5 is a flowchart of a video frame intra-coding method provided by an embodiment of the present application, which comprises a frame intra-prediction code process of a somatic region;

[0028] FIG. 6 is a flowchart of a video frame intra-coding method provided by an embodiment of the present application, which comprises a process of determining a target angle prediction mode of a face region;

[0029] FIG. 7 is a flowchart of another video frame intra-coding method provided by an embodiment of the present application, which comprises a process of determining a target angle prediction mode of a face region;

[0030] FIG. 8 is a flowchart of a video frame intra-coding method provided by an embodiment of the present application, which comprises a process of determining a target angle prediction mode of a limb region;

[0031] FIG. 9 is a flowchart of another video frame intra-coding method provided by an embodiment of the present application, which comprises a process of determining a target angle prediction mode of a limb region;

[0032] FIG. 10 is a structural block diagram of a video frame intra-coding device provided by an embodiment of the present application;

[0033] FIG. 11 is a structural diagram of a video frame intra-coding device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0034] The application examples are further described in detail below with reference to the drawings and examples. It can be understood that the specific examples described herein are only intended to explain the application examples, and not to limit the application examples. In addition, it should be noted that, for the convenience of description, only the parts related to the application examples are shown in the drawings, rather than all the structures.

[0035] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not intended to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the application examples can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of a kind and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the objects before and after are in an "or" relationship.

[0036] The video intra coding method provided by the application examples can be combined with human pose recognition and intra prediction mode selection of video coding, and the angle information of the human skeleton is used to indicate the target direction of intra prediction, so as to reduce the candidate angle prediction modes that need to be traversed, and adapt to the video live scene with strong real-time performance. The related application scenarios can include video conference, video call, indoor and outdoor live broadcast, and short video, etc. The foregoing several application scenarios are only exemplary and explanatory, and in actual application, the video intra coding method can also be used in video coding in other scenarios, and the application examples are not limited thereto. The video intra coding method, device, equipment, storage medium and program product provided by the application examples aim to solve the problem that the calculation amount of video coding is large, the coding efficiency is low, and it cannot adapt to the video live scene with strong real-time performance in the related art.

[0037] The execution subject of each step of the video intra coding method provided by the application examples can be a computer device, which refers to any electronic device with data calculation, processing and storage capabilities, such as mobile phones, PC (Personal Computer), tablet computers and other terminal devices, and can also be a server or other device, and the application examples are not limited thereto.

[0038] FIG. 1 is a flowchart of a video intra coding method according to an application example. As shown in FIG. 1, the method includes the following steps:

[0039] In step S101, a video frame to be processed is obtained, and human feature points are obtained by performing human pose detection on the video frame to be processed.

[0040] The to-be-processed video frame can be a video frame sequence input from different video live scenes, for example, the to-be-processed video frame can be a key frame that needs to perform intra prediction coding in the video frame sequence. Since the person in the video live scene is the object that the user cares most about, for example, the host in the indoor live scene, it is necessary to quickly and effectively code the person, and in the embodiment of the application, the capture of the person feature information can be realized by performing human body posture detection on the to-be-processed video frame, and the human body feature points are obtained. In an embodiment, the human body posture detection can be to detect the to-be-processed video frame by using a target detection algorithm such as YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), and the like to obtain a human body target box. Then, an anchor point is set in a region corresponding to the human body target box according to a preset interval, and a deep learning model is used to extract features of the anchor point and a surrounding region of the anchor point. Finally, the coordinate offset of each anchor point to all human body feature points is predicted according to the obtained feature information, and the position coordinates of the human body feature points are determined by weighted calculation based on the prediction results of all anchor points. In an embodiment, the human body posture detection can use a KAPAO (Keypoints and Poses as Objects) model, an A2J (Anchor-to-Joint Regression Network) model, and the like, which are not limited herein. In an embodiment, FIG. 2 is a distribution diagram of human body feature points provided in the embodiment of the application, as shown in FIG. 2, a human body skeleton is obtained by human body posture detection, and the human body skeleton can include 17 human body feature points, which are a nose feature point 101, a left eye feature point 102, a right eye feature point 103, a left ear feature point 104, a right ear feature point 105, a left shoulder feature point 106, a right shoulder feature point 107, a left elbow feature point 108, a right elbow feature point 109, a left wrist feature point 110, a right wrist feature point 111, a left hip feature point 112, a right hip feature point 113, a left knee feature point 114, a right knee feature point 115, a left ankle feature point 116, and a right ankle feature point 117. Of course, the developer can also adaptively adjust the composition of the human body feature points according to the portrait characteristics of the actual application scene, which are not limited herein. In an embodiment, FIG. 3 is a result diagram of human body feature points obtained by performing human body posture detection on a to-be-processed video frame, as shown in FIG. 3, human body feature points of a portrait 2011 in a state of showing a front face can be obtained by performing human body posture detection on a to-be-processed video frame 201, and human body feature points of a portrait 2021 in a state of showing a side face can be obtained by performing human body posture detection on a to-be-processed video frame 202.

[0041] In step S102, a human body region corresponding to different human body parts is divided from the to-be-processed video frame according to the human body feature points, wherein the human body region includes a non-trunk region.

[0042] In the foregoing embodiment, the human body region includes a trunk region and a non-trunk region. The trunk region can be a correlation coding region of a polygonal region formed by connecting the left shoulder feature point, the right shoulder feature point, the left hip feature point, and the right hip feature point. The correlation coding region can be composed of coding units in the to-be-processed video frame that overlap the polygonal region. The non-trunk region can be a correlation coding region of a plurality of line segments obtained by connecting the other feature points two by two. The correlation coding region can be composed of coding units in the to-be-processed video frame that overlap the plurality of line segments. In an embodiment, the non-trunk region can include a face region and a limb region. The face region can be a correlation coding region of a polyline path formed by connecting the left ear feature point, the left eye feature point, the nose feature point, the right eye feature point, and the right ear feature point in sequence. The correlation coding region can be composed of coding units in the to-be-processed video frame that overlap the polyline path. In an embodiment, the limb region can be further divided into a plurality of sub-regions, and an intra prediction mode suitable for each sub-region is determined. For example, each two adjacent human body feature points in the limb region can determine a sub-region, such as the left shoulder feature point and the left elbow feature point, the left elbow feature point and the left wrist feature point, and the like. In the example of the human body feature points provided in FIG. 2, one of the sub-regions can be a correlation coding region of a line segment obtained by connecting the adjacent left wrist feature point and the left elbow feature point. The correlation coding region can be composed of coding units in the to-be-processed video frame that overlap the line segment. Of course, the division of the foregoing human body region is only an exemplary description, and developers can adaptively adjust the division according to the characteristics of the human body in the actual application scenario and the coding requirements, which are not limited herein.

[0043] In step S103, a target angle prediction mode is determined from the candidate angle prediction modes according to line angle information corresponding to the human body feature points in the non-trunk region, wherein the line angle information is determined based on an included angle between a connecting line of the human body feature points at a set position and a horizontal reference line.

[0044] In one embodiment, taking the non-somatic region as the facial region as an example, different human postures correspond to different face horizontal inclination angles, and thus the face horizontal inclination angle can be determined according to the line angle information corresponding to the human feature points. For example, for the case of a frontal face, the line angle information can be the included angle between the connecting line of the left eye feature point and the right eye feature point and the horizontal reference line, and the included angle can be determined as the face horizontal inclination angle. For example, for the case of a side face, the line angle information can be the included angle between the connecting line of the recognized eye feature point and the ear feature point and the horizontal reference line, and the included angle can be determined as the face horizontal inclination angle. In one embodiment, taking the non-somatic region as the limb region as an example, the line angle information can be the included angle between the connecting line of adjacent human feature points and the horizontal reference line, and the included angle can be used to determine the target angle prediction mode of the associated coding region corresponding to the adjacent human feature points. In one embodiment, each two adjacent human feature points can determine the corresponding associated coding region and the target angle prediction mode corresponding to the associated coding region. In one embodiment, FIG. 4 is a schematic diagram of determining the line angle information corresponding to the human feature points in the non-somatic region, as shown in FIG. 4, the left wrist feature point 301 and the left elbow feature point 302 in the upper limb region are connected to obtain a connecting line, and the line angle information corresponding to the associated coding region determined by the left wrist feature point and the left elbow feature point is the included angle 303 between the connecting line and the horizontal reference. In one embodiment, since the candidate angle prediction mode is multiple, in order to quickly determine the appropriate angle prediction mode, the angle prediction mode closest to the angle can be selected from the candidate angle prediction mode according to the line angle information, and the rate distortion cost of the angle prediction mode is calculated, and if the rate distortion cost satisfies the preset threshold condition, the angle prediction mode is determined as the target angle prediction mode.If the rate-distortion cost does not satisfy the preset threshold condition, in an embodiment, the prediction angle corresponding to the angle prediction mode can be set as a reference angle, and the rate-distortion cost of other angle prediction modes is calculated in order of the angle difference from the reference angle from small to large until the next angle prediction mode whose rate-distortion cost satisfies the preset threshold condition is found, and the process stops. In another embodiment, the prediction angle corresponding to the angle prediction mode can be set as a reference angle. If there are angle prediction modes to be traversed in both counterclockwise and clockwise directions corresponding to the reference angle, one of the directions can be selected to traverse and calculate other angle prediction modes in the direction until the next angle prediction mode whose rate-distortion cost satisfies the preset threshold condition is found, and the process stops. If no angle prediction mode whose rate-distortion cost satisfies the preset threshold condition is found, other angle prediction modes in the other direction are traversed and confirmed. The foregoing process of determining the target angle prediction mode based on the reference line angle information is only an exemplary description, and the developer can adaptively adjust the threshold condition and the traversal method according to the coding accuracy and coding efficiency requirements of the actual application scenario, which is not limited herein.

[0045] In step S104, the non-body region is intra-predictively encoded based on the target angle prediction mode.

[0046] In an embodiment, the non-body region is a face region, and the face region can be intra-predictively encoded according to the determined target angle prediction mode. In an embodiment, the non-body region is a limb region, and each two adjacent human feature points can be used to determine a sub-region and a corresponding target angle prediction mode, and the sub-region can be intra-predictively encoded according to the target angle prediction mode to adapt to the diversified angle information in the limb region. Optionally, the non-human region in the video frame to be processed can be intra-predictively encoded or subjected to other fast encoding strategies based on a set prediction mode, which is not limited herein.

[0047] According to the above, the human body feature points are obtained by performing human body posture detection on the to-be-processed video frame, the human body regions corresponding to different human body parts are divided from the to-be-processed video frame according to the human body feature points, the human body regions include the non-somatic region, the target angle prediction mode is determined from the candidate angle prediction modes according to the connection angle information of the human body feature points in the non-somatic region, the connection angle information is determined based on the included angle between the connecting line of the human body feature points at the set position and the horizontal reference line, and the non-somatic region is in-frame prediction coded based on the target angle prediction mode. In the above scheme, the human body feature points are obtained by performing human body posture detection on the to-be-processed video frame, the portrait in the video frame can be accurately recognized to obtain key feature information, which is beneficial to subsequent combination of human body posture recognition and selection of the in-frame prediction mode of video coding, realization of target-based content adaptive in-frame coding, improvement of the accuracy and efficiency of selection of the in-frame coding mode, partition coding of the angle features of different human body parts through division of different human body regions based on the human body feature points, effective utilization of the angle information of the human skeleton to indicate the target direction of in-frame prediction through determination of the target angle prediction mode based on the connection angle information of the non-somatic region and in-frame prediction coding of the non-somatic region based on the target angle prediction mode, improvement of the angle hit rate of prediction, reduction of the calculation amount, improvement of the coding efficiency, and adaptation to the video live streaming scenario with high real-time performance.

[0048] FIG. 5 is a flowchart of a video in-frame coding method of an in-frame prediction coding process of a video frame according to an embodiment of the present application. As shown in FIG. 5, the method comprises the following steps:

[0049] In step S201, a to-be-processed video frame is obtained, and human body feature points are obtained by performing human body posture detection on the to-be-processed video frame.

[0050] In step S202, human body regions corresponding to different human body parts are divided from the to-be-processed video frame according to the human body feature points, wherein the human body regions include a non-somatic region and a somatic region.

[0051] In step S203, a target angle prediction mode is determined from the candidate angle prediction modes according to the connection angle information of the human body feature points in the non-somatic region, wherein the connection angle information is determined based on the included angle between the connecting line of the human body feature points at the set position and the horizontal reference line.

[0052] In step S204, the non-somatic region is in-frame prediction coded based on the target angle prediction mode.

[0053] In step S205, the rate-distortion cost corresponding to the non-angle prediction mode is calculated for the somatic region.

[0054] Wherein, since the video content corresponding to the body region is usually clothes, the relative brightness and color are relatively uniform, and the overall angle feature is not obvious, the non-angle prediction mode with higher possibility in the intra prediction mode can be used to preferentially perform rate-distortion cost calculation. Taking the VVC standard as an example, the non-angle prediction mode can be the planar mode and the DC mode. For example, the related formula for calculating the rate-distortion cost is as follows: J = D + λ mode ·R

[0055] Wherein, D represents the distortion value under the current coding mode, R represents the code rate value under the current coding mode, and λ mode is the Lagrange factor.

[0056] Step S206, in the case that the rate-distortion cost is less than or equal to the preset threshold, the body region is intra-predictively encoded based on the non-angle prediction mode.

[0057] Wherein, the preset threshold can be the maximum rate-distortion cost allowed by the video coding in the current application scenario, which can guarantee less coding quality loss, and can be adaptively adjusted by the developer according to the coding accuracy and efficiency demand of the actual application scenario, which is not limited herein. In one embodiment, if multiple non-angle prediction modes are pre-set, the rate-distortion cost of each non-angle prediction mode can be calculated in a preset order, and if a non-angle prediction mode with a rate-distortion cost less than or equal to the preset threshold appears, the non-angle prediction mode can be regarded as the target angle prediction mode. In one embodiment, if multiple non-angle prediction modes pre-set satisfy the rate-distortion cost less than or equal to the preset threshold, the non-angle prediction mode with the minimum rate-distortion cost can be selected as the final target angle prediction mode.

[0058] Step S207, in the case that the rate-distortion cost is greater than the preset threshold, the body region is intra-predictively encoded based on the angle prediction mode in the candidate angle prediction mode that satisfies the preset condition.

[0059] If the rate-distortion cost is greater than the preset threshold, it can be considered that the non-angular prediction mode is not suitable for the intra prediction encoding of the body region, and an angle prediction mode that meets the preset condition can be selected from the candidate angle prediction modes for the intra prediction encoding of the body region. In an embodiment, the rate-distortion cost meeting the preset condition can be that the rate-distortion cost of the angle prediction mode is calculated in the preset sequence order until the rate-distortion cost of the angle prediction mode is found to be less than or equal to the preset threshold. In an embodiment, the rate-distortion cost meeting the preset condition can be that part of the angle prediction modes that need to be traversed is determined in combination with the texture information of the body region, and the angle prediction mode with the smallest rate-distortion cost is selected from the part of the angle prediction modes as the final target angle prediction mode. Of course, the developer can also adaptively adjust the preset condition according to the coding accuracy and efficiency requirements of the actual application scenario, which is not limited herein.

[0060] By combining the content characteristics and angle characteristics of the body region, the non-angular prediction mode is preferentially selected for the calculation of the rate-distortion cost, which can improve the hit rate and accuracy of the prediction mode of the body region and speed up the encoding speed of the body region.

[0061] FIG. 6 is a flowchart of a video intra-frame encoding method provided by an embodiment of the present application, which includes a process of determining the target angle prediction mode of the face region. As shown in FIG. 6, the process includes the following steps:

[0062] In step S301, a video frame to be processed is obtained, and human feature points are obtained by performing human posture detection on the video frame to be processed.

[0063] In step S302, human regions corresponding to different human body parts are divided from the video frame to be processed according to the human feature points, wherein the human regions include a non-body region, and the non-body region includes a face region.

[0064] In step S303, a face horizontal inclination angle is determined according to the face orientation of the face region and the angle between the connecting line of the face feature points at the preset position and the horizontal reference line.

[0065] The face orientation can be divided into two cases of frontal face and profile face, which can be distinguished by detecting the number of ears and eyes of the face region or detecting the completeness of the face, which is not limited in the present application. For example, when the number of ears and eyes is even, it can be determined that the face orientation is frontal face. For example, only a single ear and a single eye are detected, it can be determined that the face orientation is profile face. The face horizontal angle can be used to represent the angle feature of the face region. When the face orientation is frontal face, the face horizontal angle of the face region can be determined by the angle between the connecting line of the double eye feature points or the double ear feature points and the horizontal reference line. When the face orientation is profile face, the face horizontal angle of the face region can be determined by the angle between the connecting line of the eye feature points and the ear feature points and the horizontal reference line. In an embodiment, when the face orientation of the face region is frontal face, the face horizontal angle is calculated by the angle between the connecting line of the left eye feature point and the right eye feature point and the horizontal reference line, wherein the vector coordinates of the connecting line can be calculated based on the coordinates of the left eye feature point and the right eye feature point in the image coordinate system, and the angle between the vector coordinates of the connecting line and the horizontal reference line is calculated. Of course, other mathematical methods can also be used to calculate the angle, which is not limited in the present application. In an embodiment, when the face orientation of the face region is profile face, the face horizontal angle is calculated by the angle between the connecting line of the eye feature point and the ear feature point and the horizontal reference line.

[0066] In step S304, the first angle prediction mode with the closest prediction angle to the face horizontal angle is selected from the candidate angle prediction modes, and the first rate-distortion cost of the face region applying the first angle prediction mode is calculated.

[0067] The first angle prediction mode with the closest prediction angle to the face horizontal angle can be considered as the highest matching degree to the angle feature of the face region, and the first rate-distortion cost of the first angle prediction mode can be calculated preferentially.

[0068] In step S305, the first angle prediction mode is determined as the first target angle prediction mode when the first rate-distortion cost is less than or equal to a preset threshold.

[0069] When the first rate-distortion cost is less than or equal to the preset threshold, it can be considered that the first angle prediction mode can meet the coding quality requirement of the current application scene, and thus the first angle prediction mode can be determined as the first target angle prediction mode.

[0070] In step S306, the face region is intra-predictively encoded based on the first target angle prediction mode.

[0071] The angle information provided by the face region through the connection surface can be used to preferentially locate the angle prediction mode with a high matching degree, thereby improving the hit rate and accuracy of the angle prediction mode of the face region and achieving the purpose of fast encoding of the face region.

[0072] FIG. 7 is a flowchart of another video frame intra-coding method provided by an embodiment of the present application, which includes a process of determining a target angle prediction mode of a face region. As shown in FIG. 7, the process includes the following steps:

[0073] In step S401, a video frame to be processed is obtained, and human body feature points are obtained by performing human body posture detection on the video frame to be processed.

[0074] In step S402, human body regions corresponding to different human body parts are divided from the video frame to be processed according to the human body feature points, wherein the human body regions include a non-somatic region, and the non-somatic region includes a face region.

[0075] In step S403, a face horizontal inclination angle is determined according to an angle between a connection line of a face feature point at a preset position and a horizontal reference line and a face orientation of the face region.

[0076] In step S404, a first angle prediction mode with a prediction angle closest to the face horizontal inclination angle is selected from candidate angle prediction modes, and a first rate-distortion cost corresponding to the first angle prediction mode is calculated for the face region.

[0077] In step S405, the first angle prediction mode is determined as a first target angle prediction mode in a case where the first rate-distortion cost is less than or equal to a preset threshold.

[0078] In step S406, the face region is intra-predictively encoded based on the first target angle prediction mode.

[0079] In step S407, in a case where the first rate-distortion cost is greater than the preset threshold, second rate-distortion costs of the face region applying other angle prediction modes are sequentially calculated in an order from small to large according to angle differences between the prediction angles and the face horizontal inclination angle, until a second target angle prediction mode with a second rate-distortion cost less than the preset threshold is determined.

[0080] If the first rate-distortion cost is greater than the preset threshold, it can be considered that the first angle prediction mode is not suitable for intra prediction coding of the face region. However, the obtained face horizontal inclination angle can be used as a reference angle, and other angle prediction modes with a prediction angle close to the face horizontal inclination angle are preferentially traversed. The traversal order of the other angle prediction modes can be determined according to the size of the angle difference between the prediction angle and the face horizontal inclination angle. By sequentially calculating the second rate-distortion cost of the face region applying the other angle prediction modes, the second target angle prediction mode with a second rate-distortion cost less than the preset threshold is determined.

[0081] In step S408, the face region is intra-predictively coded based on the second target angle prediction mode.

[0082] In the case where the rate-distortion cost corresponding to the first angle prediction mode cannot meet the threshold condition, the face horizontal inclination angle can be used as a reference angle, and the size of the angle difference between the prediction angle of the other angle prediction mode and the reference angle can be used as a basis for determining the traversal order. This can facilitate rapid determination of the angle prediction mode matching the face region from the other angle prediction modes, and effectively improve the coding efficiency.

[0083] FIG. 8 is a flowchart of a video intra-coding method provided by an embodiment of the present application, which includes a process of determining a target angle prediction mode of a limb region. As shown in FIG. 8, the process includes the following steps:

[0084] In step S501, a video frame to be processed is obtained, and human feature points are obtained by performing human posture detection on the video frame to be processed.

[0085] In step S502, human regions corresponding to different human body parts are divided from the video frame to be processed according to the human feature points, wherein the human regions include a non-somatic region, and the non-somatic region includes a face region and a limb region.

[0086] In step S503, a face horizontal inclination angle is determined according to the face orientation of the face region and the size of the angle between the connecting line of the face feature points at the preset position and the horizontal reference line.

[0087] In step S504, a first angle prediction mode with a prediction angle closest to the face horizontal inclination angle is selected from candidate angle prediction modes, and a first rate-distortion cost corresponding to the face region applying the first angle prediction mode is calculated.

[0088] In step S505, in the case where the first rate-distortion cost is less than or equal to a preset threshold, the first angle prediction mode is determined as a first target angle prediction mode.

[0089] In step S506, the face region is intra-predictively coded based on the first target angle prediction mode.

[0090] Step S507, the size of the angle between the connecting line of the adjacent feature points in the limb region and the horizontal reference line is calculated to obtain a reference angle.

[0091] In the above, since the human body limbs involve more joints, there can be a large difference between the angle information corresponding to different parts of the limbs, so an associated coding region can be determined for each two adjacent feature points, and an angle prediction mode matching the associated coding region can be selected according to the angle between the connecting line of the adjacent feature points and the horizontal reference line.

[0092] Step S508, a third angle prediction mode with a prediction angle closest to the reference angle is selected from the candidate angle prediction modes, and a third rate-distortion cost corresponding to the third angle prediction mode is calculated for the associated coding region of the adjacent feature points.

[0093] In the above, the third angle prediction mode with the prediction angle closest to the reference angle can be considered as having the highest matching degree with the angle feature of the associated coding region, and the third rate-distortion cost of the third angle prediction mode can be calculated preferentially.

[0094] Step S509, in the case where the third rate-distortion cost is less than or equal to a preset threshold, the third angle prediction mode is determined as a third target angle prediction mode.

[0095] In the above, if the third rate-distortion cost is less than or equal to the preset threshold, it can be considered that the third angle prediction mode can meet the coding quality requirement of the current application scenario, and thus the third angle prediction mode can be determined as the third target angle prediction mode.

[0096] Step S510, the associated coding region is intra-predictively encoded based on the third target angle prediction mode.

[0097] In the above, the limb region can determine a plurality of associated coding regions according to a plurality of adjacent feature points, each associated coding region can determine a corresponding target angle prediction mode, and the coding is performed based on the corresponding target angle prediction mode.

[0098] In the above, by dividing the limb region into a plurality of associated coding regions determined based on adjacent feature points, the angle information distribution of the limb region is matched, the angle features of the limb region are effectively fitted for partition refinement coding, by combining the reference angle provided by each associated coding region, the angle prediction mode with a higher matching degree can be preferentially located in the prediction direction, the hit rate and accuracy of the angle prediction mode of the associated coding region can be improved, the content adaptive intra coding based on the target is realized, and the coding efficiency is guaranteed.

[0099] FIG. 9 is a flowchart of another video frame intra coding method provided by an embodiment of the present application, which includes a process of determining a target angle prediction mode of a limb region. As shown in FIG. 9, the process includes the following steps:

[0100] In step S601, a video frame to be processed is obtained, and human body feature points are obtained by performing human body posture detection on the video frame to be processed.

[0101] In step S602, human body regions corresponding to different human body parts are divided from the video frame to be processed according to the human body feature points, wherein the human body regions include a non-somatic region, and the non-somatic region includes a face region and a limb region.

[0102] In step S603, a face horizontal inclination angle is determined according to a face orientation of the face region and an included angle between a connecting line of face feature points at a preset position and a horizontal reference line.

[0103] In step S604, a first angle prediction mode whose prediction angle is closest to the face horizontal inclination angle is selected from candidate angle prediction modes, and a first rate-distortion cost corresponding to the first angle prediction mode is calculated for the face region.

[0104] In step S605, in a case where the first rate-distortion cost is less than or equal to a preset threshold, the first angle prediction mode is determined as a first target angle prediction mode.

[0105] In step S606, the face region is intra-predictively encoded based on the first target angle prediction mode.

[0106] In step S607, in a case where the first rate-distortion cost is greater than the preset threshold, second rate-distortion costs of the face region applying other angle prediction modes are sequentially calculated in an order from small to large according to angle differences between the prediction angles and the face horizontal inclination angle, until a second target angle prediction mode whose second rate-distortion cost is less than the preset threshold is determined.

[0107] In step S608, the face region is intra-predictively encoded based on the second target angle prediction mode.

[0108] In step S609, a reference angle is obtained by calculating an included angle between a connecting line of adjacent feature points in the limb region and a horizontal reference line.

[0109] In step S610, a third angle prediction mode whose prediction angle is closest to the reference angle is selected from the candidate angle prediction modes, and a third rate-distortion cost corresponding to the third angle prediction mode is calculated for a related coding region of the adjacent feature points.

[0110] In step S611, in a case where the third rate-distortion cost is less than or equal to the preset threshold, the third angle prediction mode is determined as a third target angle prediction mode.

[0111] Step S612, intra-predictively encode the associated coding region based on the third target angle prediction mode.

[0112] Step S613, in the case that the third rate-distortion cost is greater than the preset threshold, in order of the angle difference between the prediction angle and the reference angle from small to large, sequentially calculate fourth rate-distortion costs of the associated coding region applying other angle prediction modes, until a fourth target angle prediction mode whose fourth rate-distortion cost is less than the preset threshold is determined.

[0113] Wherein, if the third rate-distortion cost is greater than the preset threshold, it can be considered that the third angle prediction mode is not suitable for intra-predictive encoding of the associated coding region. However, the obtained reference angle can be used as a reference angle to preferentially traverse other angle prediction modes close to the reference angle. The traversal order of other angle prediction modes can be determined according to the size of the angle difference between the prediction angle and the reference angle. By sequentially calculating the fourth rate-distortion costs of the associated coding region applying other angle prediction modes, until the fourth target angle prediction mode whose fourth rate-distortion cost is less than the preset threshold is determined, the traversal is stopped.

[0114] Step S614, intra-predictively encode the associated coding region based on the fourth target angle prediction mode.

[0115] In the above, in the case that the rate-distortion cost corresponding to the third angle prediction mode cannot meet the threshold condition, by taking the reference angle as a reference angle and using the angle difference between the prediction angle of other angle prediction modes and the reference angle as the basis for determining the traversal order, it is beneficial to quickly determine the angle prediction mode matching the associated coding region from other angle prediction modes, and effectively improve the coding efficiency.

[0116] FIG. 10 is a structural block diagram of a video intra-coding device provided by an embodiment of the present application, which is configured to execute the video intra-coding method provided by the above-mentioned embodiment, and has the corresponding function modules and beneficial effects of the execution method. As shown in FIG. 10, the device includes:

[0117] The acquisition module 401 is configured to acquire a to-be-processed video frame.

[0118] The feature detection module 402 is configured to perform human posture detection on the to-be-processed video frame to obtain human feature points.

[0119] The region division module 403 is configured to divide human regions corresponding to different human body parts from the to-be-processed video frame according to the human feature points, wherein the human regions include a non-somatic region.

[0120] The mode determination module 404 is configured to determine a target angle prediction mode from the candidate angle prediction modes according to line angle information corresponding to the human body feature points in the non-body region, the line angle information being determined based on an included angle between a connecting line of the human body feature points at a set position and a horizontal reference line;

[0121] The encoding module 405 is configured to perform intra prediction encoding on the non-body region based on the target angle prediction mode.

[0122] In the above, the human body feature points are obtained by performing human body posture detection on the to-be-processed video frame; the human body regions corresponding to different human body parts are divided from the to-be-processed video frame according to the human body feature points, and the human body regions include the non-body region; the target angle prediction mode is determined from the candidate angle prediction modes according to line angle information corresponding to the human body feature points in the non-body region, the line angle information being determined based on an included angle between a connecting line of the human body feature points at a set position and a horizontal reference line; and the non-body region is encoded based on the target angle prediction mode. In the above scheme, the human body feature points are obtained by performing human body posture detection on the to-be-processed video frame, which can accurately identify the human image in the video frame to obtain key feature information, and is conducive to the combination of human body posture recognition and selection of the intra prediction mode of video encoding in the subsequent process, so as to realize target-based content adaptive intra coding and improve the accuracy and efficiency of the selection of the intra coding mode. Different human body regions are divided based on the human body feature points, which can adapt to the partition coding of the angle features of different human body parts. The target angle prediction mode is determined based on the line angle information of the non-body region, and the non-body region is intra prediction encoded based on the target angle prediction mode, which can effectively use the angle information of the human body skeleton to indicate the target direction of the intra prediction, improve the angle hit rate of the prediction, reduce the calculation amount without traversing all the intra prediction modes, and improve the coding efficiency, and can be adapted to the video live streaming scenario with high real-time requirement.

[0123] In one possible embodiment, the human body region further includes a body region, and the apparatus further includes a second encoding module configured to:

[0124] calculate a rate-distortion cost of the body region corresponding to the non-angle prediction mode;

[0125] in a case where the rate-distortion cost is less than or equal to a preset threshold, perform intra prediction encoding on the body region based on the non-angle prediction mode;

[0126] in a case where the rate-distortion cost is greater than the preset threshold, perform intra prediction encoding on the body region based on an angle prediction mode in the candidate angle prediction modes that satisfies a preset condition.

[0127] In one possible embodiment, the non-body region includes a face region, and the mode determination module 404 is further configured to:

[0128] determine the face horizontal angle according to the face orientation of the face region and the angle size between the connecting line of the face feature point at the preset position and the horizontal reference line;

[0129] select a first angle prediction mode with a prediction angle closest to the face horizontal angle from the candidate angle prediction modes, and calculate a first rate-distortion cost corresponding to the first angle prediction mode applied to the face region;

[0130] determine the first angle prediction mode as a first target angle prediction mode in a case where the first rate-distortion cost is less than or equal to a preset threshold;

[0131] Correspondingly, the encoding module 405 is further configured to:

[0132] perform intra-prediction encoding on the face region based on the first target angle prediction mode.

[0133] In one possible embodiment, the mode determination module 404 is further configured to:

[0134] in a case where the face orientation of the face region is a front face, calculate the face horizontal angle according to the angle size between the connecting line of the left eye feature point and the right eye feature point and the horizontal reference line;

[0135] in a case where the face orientation of the face region is a side face, calculate the face horizontal angle according to the angle size between the connecting line of the eye feature point and the ear feature point and the horizontal reference line.

[0136] In one possible embodiment, further comprising a third encoding module configured to:

[0137] in a case where the first rate-distortion cost is greater than the preset threshold, sequentially calculate second rate-distortion costs of the face region applying other angle prediction modes according to the angle difference between the prediction angle and the face horizontal angle from small to large until a second target angle prediction mode with a second rate-distortion cost less than the preset threshold is determined;

[0138] perform intra-prediction encoding on the face region based on the second target angle prediction mode.

[0139] In one possible embodiment, the non-somatic region further comprises a limb region, and the mode determination module 404 is further configured to:

[0140] calculate a reference angle according to the angle size between the connecting line of the adjacent feature points in the limb region and the horizontal reference line;

[0141] select a third angle prediction mode with a prediction angle closest to the reference angle from the candidate angle prediction modes, and calculate a third rate-distortion cost corresponding to the third angle prediction mode applied to the associated coding region of the adjacent feature points.

[0142] In a case that the third rate-distortion cost is less than or equal to a preset threshold, the third angle prediction mode is determined as the third target angle prediction mode.

[0143] Correspondingly, the encoding module 405 is configured to:

[0144] perform intra-prediction encoding on the associated coding region based on the third target angle prediction mode.

[0145] In one possible embodiment, the apparatus further comprises a fourth encoding module configured to:

[0146] In a case that the third rate-distortion cost is greater than the preset threshold, fourth rate-distortion costs of the associated coding region applying other angle prediction modes are sequentially calculated in an order from small to large of angle differences between the prediction angles and the reference angles, until a fourth target angle prediction mode with a fourth rate-distortion cost less than the preset threshold is determined.

[0147] perform intra-prediction encoding on the associated coding region based on the fourth target angle prediction mode.

[0148] FIG. 11 is a structural schematic diagram of a video intra-encoding device according to an embodiment of the present application. As shown in FIG. 11, the device comprises a processor 501, a memory 502, an input device 503 and an output device 504. The number of processors 501 in the device can be one or more, and one processor 501 is taken as an example in FIG. 11. The processor 501, the memory 502, the input device 503 and the output device 504 in the device can be connected through a bus or other means, and connection through a bus is taken as an example in FIG. 11. The memory 502 as a computer readable storage medium can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the video intra-encoding method in the embodiments of the present application. The processor 501 performs various function applications and data processing of the device by running the software programs, instructions and modules stored in the memory 502, that is, the video intra-encoding method is implemented. The input device 503 can be configured to receive input digital or character information, and generate key signal input related to user settings and function control of the device. The output device 504 can include a display device such as a display screen.

[0149] The embodiment of the present application further provides a nonvolatile storage medium comprising computer executable instructions, which are configured to perform a video frame intra coding method described in the above embodiment when executed by a computer processor, and the method comprises the following steps: obtaining a to-be-processed video frame, performing human body posture detection on the to-be-processed video frame to obtain human body feature points; dividing human body regions corresponding to different human body parts from the to-be-processed video frame according to the human body feature points, wherein the human body regions comprise non-somatic regions; determining a target angle prediction mode from candidate angle prediction modes according to line angle information corresponding to the human body feature points in the non-somatic regions, wherein the line angle information is determined based on an included angle between a connecting line of the human body feature points at a set position and a horizontal reference line; and performing intra prediction coding on the non-somatic regions based on the target angle prediction mode.

[0150] It is worth noting that in the above embodiment of the video frame intra coding device, each unit and module included is only divided according to functional logic, but is not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for the convenience of mutual differentiation, and do not configure the protection scope of the embodiment of the present application.

[0151] In some possible implementation manners, each aspect of the method provided by the present application can also be implemented in the form of a program product, which includes program codes. When the program product runs on a computer device, the program codes are configured to make the computer device perform the steps in the method according to various exemplary embodiments of the present application described in the specification, for example, the computer device can perform the video frame intra coding method described in the embodiment of the present application. The program product can be implemented in any combination of one or more readable media.

Claims

1. A method of intra coding of video frames, wherein, The method comprises: obtaining a to-be-processed video frame, and performing human posture detection on the to-be-processed video frame to obtain human feature points; dividing human regions corresponding to different human body parts from the to-be-processed video frame according to the human feature points, wherein the human regions comprise a non-somatic region; determining a target angle prediction mode from candidate angle prediction modes according to connection angle information of the human feature points in the non-somatic region, wherein the connection angle information is determined based on an included angle between a connecting line of a human feature point at a preset position and a horizontal reference line; performing intra-frame prediction encoding on the non-somatic region based on the target angle prediction mode.

2. The video frame-in coding method of claim 1, wherein, The human regions further comprise a somatic region, and the method further comprises: calculating a rate-distortion cost corresponding to a non-angle prediction mode applied to the somatic region; in a case where the rate-distortion cost is less than or equal to a preset threshold, performing intra-frame prediction encoding on the somatic region based on the non-angle prediction mode; in a case where the rate-distortion cost is greater than the preset threshold, performing intra-frame prediction encoding on the somatic region based on an angle prediction mode whose rate-distortion cost satisfies a preset condition among the candidate angle prediction modes.

3. The video frame-in coding method of claim 1, wherein, The non-somatic region comprises a face region, and the determining of the target angle prediction mode from the candidate angle prediction modes according to the connection angle information of the human feature points in the non-somatic region comprises: determining a face horizontal inclination angle according to a face orientation of the face region and an included angle between a connecting line of a face feature point at a preset position and a horizontal reference line; selecting a first angle prediction mode whose prediction angle is closest to the face horizontal inclination angle from the candidate angle prediction modes, and calculating a first rate-distortion cost corresponding to the first angle prediction mode applied to the face region; in a case where the first rate-distortion cost is less than or equal to a preset threshold, determining the first angle prediction mode as a first target angle prediction mode; correspondingly, the performing of the intra-frame prediction encoding on the non-somatic region based on the target angle prediction mode comprises: performing intra-frame prediction encoding on the face region based on the first target angle prediction mode.

4. The video frame-in coding method of claim 3, wherein, The determining of the face horizontal inclination angle according to the face orientation of the face region and the included angle between the connecting line of the face feature point at the preset position and the horizontal reference line comprises: in a case where the face orientation of the face region is a front face, calculating the included angle between the connecting line of the left eye feature point and the right eye feature point and the horizontal reference line to obtain the face horizontal inclination angle; in a case where the face orientation of the face region is a side face, calculating the included angle between the connecting line of the eye feature point and the ear feature point and the horizontal reference line to obtain the face horizontal inclination angle.

5. The video frame-in coding method of claim 3, wherein, The method further comprises: in a case where the first rate-distortion cost is greater than the preset threshold, sequentially calculating second rate-distortion costs of other angle prediction modes applied to the face region in order of angle difference between a prediction angle and the face horizontal inclination angle from small to large, until a second target angle prediction mode whose second rate-distortion cost is less than the preset threshold is determined; performing intra-frame prediction encoding on the face region based on the second target angle prediction mode.

6. The video frame-in coding method according to any one of claims 1 or 3, wherein, The non-somatic region further comprises a limb region, and the determining the target angle prediction mode from the candidate angle prediction modes according to the line angle information corresponding to the human body feature points in the non-somatic region further comprises: calculating an included angle between a connecting line of adjacent feature points in the limb region and a horizontal reference line to obtain a reference angle; selecting a third angle prediction mode from the candidate angle prediction modes, the third angle prediction mode having a prediction angle closest to the reference angle, and calculating a third rate-distortion cost of the associated coding region of the adjacent feature points using a third rate-distortion cost corresponding to the third angle prediction mode; in a case where the third rate-distortion cost is less than or equal to the preset threshold, determining the third angle prediction mode as a third target angle prediction mode; correspondingly, the intra-frame prediction encoding the non-somatic region based on the target angle prediction mode comprises: intra-frame prediction encoding the associated coding region based on the third target angle prediction mode.

7. The video frame-in coding method of claim 6, wherein, The method further comprises: in a case where the third rate-distortion cost is greater than the preset threshold, sequentially calculating fourth rate-distortion costs of the associated coding region using other angle prediction modes in an order of angle difference between a prediction angle and the reference angle from small to large, until a fourth target angle prediction mode having a fourth rate-distortion cost less than the preset threshold is determined; intra-frame prediction encoding the associated coding region based on the fourth target angle prediction mode.

8. A video frame-in coding apparatus, wherein, comprises: an acquisition module configured to acquire a to-be-processed video frame; a feature detection module configured to perform human body posture detection on the to-be-processed video frame to obtain human body feature points; a region division module configured to divide human body regions corresponding to different human body parts from the to-be-processed video frame according to the human body feature points, the human body regions comprising a non-somatic region; a mode determination module configured to determine a target angle prediction mode from candidate angle prediction modes according to line angle information corresponding to the human body feature points in the non-somatic region, the line angle information being determined based on an included angle between a connecting line of a human body feature point at a set position and a horizontal reference line; an encoding module configured to intra-frame prediction encode the non-somatic region based on the target angle prediction mode.

9. A video frame-in-coding device, the device comprising: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more programs cause the one or more processors to implement the video intra-frame encoding method in any one of claims 1-7.

10. A non-volatile storage medium storing computer executable instructions, which, when executed by a computer processor, are configured to perform the video intra-frame encoding method in any one of claims 1-7.

11. A computer program product comprising a computer program, wherein, The computer program is executed by the processor to implement the video intra-frame encoding method in any one of claims 1-7.

Images