Video encoding method, apparatus, electronic device, storage medium, and program

Abstract

This disclosure provides video encoding methods, apparatus, electronic devices, storage media, and programs. [Solution] The specific solution includes: obtaining a reference frame adjacent to the current frame and in which a hierarchical partitioning of the initial region of interest (ROI) has been established; reusing motion vectors generated during the motion-compensated time filtering process in the reference frame to determine the predicted ROI position of the current frame; extracting sensory feature points from the current frame and correcting the predicted ROI position based on the feature point spatial topology relationship in the initial ROI; assigning differentiated quantization parameters to the corrected primary face region, secondary face region, and background region, respectively; dynamically allocating the bitrate of the three layer regions based on real-time network bandwidth; and outputting the encoded frame of the current frame and associated ROI hierarchical metadata.

Description

[Technical Field]

[0001] This disclosure relates to the field of image processing technology, particularly to the field of video coding, and can be used in application scenarios such as live video streaming. Specifically, it relates to video coding methods, apparatus, electronic devices, storage media, and programs. [Background technology]

[0002] In the field of network live streaming technology, to improve the viewer experience, optimization is typically performed by focusing on encoding regions of interest (ROIs) within the live streamed video, ensuring high quality of important content under limited bitrates. However, existing ROI encoding technologies lack real-time capabilities, consume large amounts of resources, and have coarse bitrate allocation, failing to adequately meet the high efficiency and high quality requirements of live streaming scenarios. [Overview of the Initiative] [Problems that the invention aims to solve]

[0003] This disclosure provides video encoding methods, apparatus, electronic devices, storage media, and programs. [Means for solving the problem]

[0004] A first aspect of this disclosure provides a video encoding method, which is: Obtain a reference frame adjacent to the current frame, for which a hierarchical partitioning of the initial region of interest (ROI) has been established. The motion vectors generated during the motion compensation time filtering process in the reference frame are reused to determine the ROI position prediction for the current frame, and This involves extracting sensory feature points from the current frame and correcting the ROI position prediction value based on the feature point spatial topology relationship in the initial ROI, By placing differentiated quantization parameters in the corrected primary face region, secondary face region, and background region, Dynamically allocating bitrates across three layers based on real-time network bandwidth, Outputting the encoded frame of the current frame and associated ROI hierarchy metadata, wherein the ROI hierarchy metadata includes boundary coordinates of the primary face region, secondary face region and background region, quantization parameter offset values ​​corresponding to each region, and bitrate allocation weight values ​​for each region.

[0005] A second aspect of this disclosure provides a video encoding device, which, A reference frame acquisition module for obtaining reference frames adjacent to the current frame and for which a hierarchical partitioning of the initial region of interest (ROI) has been established, A position prediction module for determining the ROI position prediction value of the current frame by reusing motion vectors generated during the motion compensation time filtering process in the reference frame, A position correction module for extracting sensory feature points from the current frame and correcting ROI position prediction values ​​based on the feature point spatial topology relationship in the initial ROI, A parameter placement module for placing differentiated quantization parameters in the corrected main face region, secondary face region, and background region, respectively. A bitrate allocation module for dynamically allocating bitrates in the 3-layer domain based on real-time network bandwidth, A data output module for outputting the encoded frame of the current frame and associated ROI hierarchy metadata, wherein the ROI hierarchy metadata includes boundary coordinates of the primary face region, secondary face region and background region, quantization parameter offset values ​​corresponding to each region, and bitrate allocation weight values ​​for each region.

[0006] A third aspect of this disclosure provides an electronic device, which is At least one processor, The system comprises at least one processor and memory that is communicated with, The memory stores instructions that are executable by the at least one processor, and when executed by the at least one processor, the instructions cause one of the methods in the embodiments of the present disclosure to be performed.

[0007] A fourth aspect of this disclosure provides a non-temporary computer-readable storage medium that stores computer instructions for causing a computer to perform any one of the methods of the embodiments of this disclosure.

[0008] A fifth aspect of this disclosure provides a program, when executed by a processor, for performing any of the methods in the embodiments of this disclosure.

[0009] By adopting the method of this disclosure, the overhead of independent motion estimation can be eliminated by reusing motion vectors generated during the motion-compensated time-domain filtering process. Interframe tracking can be achieved by predicting ROI positions based on motion vectors. Feature points can be extracted from the first frame, and spatial topology relationships can be established as correction criteria. These topology relationships can then be reused in subsequent frames to correct the predicted ROI positions, ensuring localization accuracy. Combined with differentiated quantization parameters and dynamic adaptation to the network environment, coding efficiency and quality can be improved.

[0010] It should be understood that the information contained herein is not intended to describe any key points or important features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Further details of other features of this disclosure will be provided in the specification below.

[0011] The attached drawings are for the purpose of better understanding this solution and do not constitute a limitation of this disclosure. [Brief explanation of the drawing]

[0012] [Figure 1] It is a flowchart of a video encoding method according to an embodiment of the present disclosure. [Figure 2] It is another flowchart of a video encoding method according to an embodiment of the present disclosure. [Figure 3] It is a schematic configuration diagram of a video encoding apparatus according to an embodiment of the present disclosure. [Figure 4] It is a schematic scenario diagram of a video encoding method according to an embodiment of the present disclosure. [Figure 5] It is a block diagram of an electronic device of a video encoding method for realizing an embodiment of the present disclosure.

Embodiments for Carrying out the Invention

[0013] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the accompanying drawings. These drawings include various details of the embodiments of the present disclosure for the purpose of assisting understanding, and these should be considered to be merely exemplary. Therefore, those skilled in the art should understand that various changes and corrections can be made to the embodiments described in this specification without departing from the scope of the present disclosure. Similarly, descriptions of well-known features and structures are omitted in the following description for the sake of clarity and brevity.

[0014] In this specification, the term "and / or" simply describes a relational relationship that describes related objects, meaning, for example, that there may be three relationships, A and / or B, which can represent A existing alone, A and B existing together, and B existing alone. In this specification, the term "at least one" means any combination of at least one of a plurality or at least two of a plurality, for example, that includes at least one of A, B, and C, and may include any one or more elements selected from the set consisting of A, B, and C. In this specification, the terms "first" and "second" refer to and distinguish multiple similar technical terms, and do not mean to limit the order or limit that there are only two. For example, "first feature" and "second feature" mean that there are two kinds / two features, where the first feature may be one or more, and the second feature may be one or more.

[0015] Furthermore, in order to better illustrate this disclosure, many specific details are provided in the following specific embodiments. Those skilled in the art should understand that this disclosure can be similarly implemented without these specific details. In some examples, detailed descriptions of methods, means, elements, and circuits that are well known to those skilled in the art are omitted in order to facilitate the understanding of the essence of this disclosure.

[0016] Before describing the technical aspects of the embodiments of this disclosure, we will further explain the technical terms that may be used in this disclosure.

[0017] Video encoding refers to the process of compressing and encoding video data. Its aim is to reduce the size of video data while maintaining the highest possible video quality. This technology achieves efficient storage and transmission by removing redundant information such as temporal redundancy, spatial redundancy, and statistical redundancy from the video.

[0018] In internet live streaming technology, ROI coding optimization techniques are typically used to optimize the encoding of the streamer's face and important control areas in order to improve the viewer experience. However, conventional techniques have significant flaws, such as a lack of real-time capabilities, time-consuming detection using complex deep learning algorithms leading to increased encoding latency and stuttering in live streams, high resource consumption making them unsuitable for mobile devices and forcing reliance on cloud servers, further increasing network transmission latency, and coarse bitrate allocation that simply distinguishes between ROI and background without refining adjustments for important parts of the ROI (such as the five senses of a person's face). Furthermore, strategies that extract and detect frames ignore the dynamic characteristics of the scene, making them prone to mismatches in encoding resources during scene transitions, resulting in wasted bitrate and degraded image quality in important areas.

[0019] This disclosure proposes a video coding method that, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, eliminates redundant calculations by reusing motion vectors in motion-compensated time filtering, significantly reduces processing delays in the motion tracking module, avoids independent motion estimation operations, achieves near-zero overhead ROI inter-frame tracking, and improves the real-time performance of coding. By reusing the topological relationships of the first frame between frames, the spatial consistency of feature points is maintained, quality fluctuations due to feature points reconstructed frame by frame are eliminated, the clarity of details in important areas is ensured, thereby optimizing coding quality. Furthermore, in combination with hierarchical coding and dynamic bitrate allocation, quality and fluency are intelligently balanced during bandwidth fluctuations, enhancing bandwidth adaptability. In addition, the collaborative optimization of motion tracking and bitrate control reduces the risk of stuttering in live streaming. While ensuring visual quality in the main face area, overall bitrate efficiency is optimized, and a low-latency, high-definition coding architecture consisting of a "tracking-hierarchical-flow control" trinity is constructed.

[0020] Embodiments of this disclosure provide a video encoding method, Figure 1 is a flowchart of a video encoding method according to an embodiment of this disclosure, and this video encoding method can be applied to a video encoding device. This video encoding device is located in an electronic device. The electronic device may include, but is not limited to, a fixed device and / or a mobile device. For example, a fixed device may include, but is not limited to, a server, and the server may be a cloud server or a regular server. For example, a mobile device may include, but is not limited to, a video live streaming device, which may be a mobile phone, tablet, or in-vehicle terminal. In some possible implementations, the video encoding method can be implemented by a processor calling computer-readable instructions stored in memory. As shown in Figure 1, the video encoding method includes the following:

[0021] In S101, a reference frame adjacent to the current frame and for which the initial ROI hierarchical division has been established is obtained.

[0022] In S102, the motion vector generated during the motion compensation time filtering process in the reference frame is reused to determine the ROI position prediction value for the current frame.

[0023] In S103, sensory feature points are extracted from the current frame, and the ROI position prediction value is corrected based on the feature point spatial topology relationship in the initial ROI.

[0024] In S104, differentiated quantization parameters are placed in the corrected main face region, secondary face region, and background region, respectively.

[0025] In S105, the bitrate of the 3-layer domain is dynamically allocated based on real-time network bandwidth.

[0026] In S106, the encoded frame of the current frame and the associated ROI hierarchy metadata are output.

[0027] Here, the current frame refers to the frame in the video sequence that currently needs to be encoded. In the embodiments of this disclosure, the video to be encoded consists of a series of consecutive image frames. In particular, when processing the current frame, information from the preceding and succeeding frames can be referenced.

[0028] Here, ROI refers to an area or object of particular interest in image processing or computer vision. In embodiments of this disclosure, the ROI may be an area in a video frame that should be encoded with emphasis. Exemplaryly, the ROI may be a face area.

[0029] In embodiments of this disclosure, a reference frame adjacent to the current frame can be extracted from the acquired video sequence. Exemplarily, the reference frame may be the frame immediately preceding or immediately following the current frame, and simultaneously, the reference frame may be the Nth frame preceding the current frame, where N ≥ 1 and N is a positive integer. Similarly, the reference frame may be the Mth frame following the current frame, where M ≥ 1 and M is a positive integer. Likewise, the reference frame may be a keyframe selected by inter-frame prediction, and in particular, the reference frame may be determined by other methods in the prior art, and this disclosure is not limited thereto. Furthermore, based on the determined reference frame, a hierarchy of divided primary face regions, secondary face regions, and background regions can be obtained, and boundary information for these regions can be recorded. The above is an illustrative description and does not limit all possible ways in which reference frames can be obtained, nor is it exhaustively listed here.

[0030] Here, motion-compensated temporal filtering (MCTF) is a method that performs prediction and compensation using inter-frame information in the time dimension. In the embodiments of this disclosure, MCTF can predict the pixel value of the current frame by analyzing the motion trend of pixels between preceding and succeeding frames, thereby reducing the temporal redundancy of the video.

[0031] Here, a motion vector (MV) can describe the trajectory of motion of a pixel block between preceding and succeeding frames. In embodiments of this disclosure, the MV can be used to indicate the displacement of a target from a reference frame to the current frame.

[0032] Here, ROI position prediction refers to estimating the specific location of the ROI in the current frame based on MV.

[0033] In embodiments of this disclosure, motion vectors generated by the MCTF process are first extracted in a reference frame, and then the ROI position in the reference frame is mapped to the current frame using MV, and the predicted ROI position can be determined by operations such as translation and scaling. The above is merely an illustrative description and does not limit all possibilities for determining the predicted ROI position of the current frame, nor is it exhaustively listed here.

[0034] Here, the five sensory feature points refer to key points in the facial region, such as the contour points of the eyes, nose, and mouth. In the embodiments of this disclosure, the five sensory feature points can be automatically extracted by a face detection algorithm.

[0035] Here, the feature point space topology relationship can describe the positional relationship between feature points. In the embodiments of this disclosure, the feature point space topology relationship can describe the relative positions between five sensory feature points, such as the relative positions of the eyes and nose.

[0036] In embodiments of this disclosure, the five sensory feature points of the current frame can first be extracted using a deep learning model or a specific face detector. Subsequently, the ROI prediction can be adjusted based on the location of the feature points and the ROI location in the reference frame. The above is merely an illustrative description and does not limit, nor exhaustively list, all possibilities for correcting the ROI location prediction.

[0037] Here, the primary facial region refers to the region within the ROI that contains key feature points among the five sensory feature points. Exemplary examples include the eyes, nose, and mouth. In particular, key feature points can be subdivided depending on the actual situation, and this disclosure is not limited thereto.

[0038] Here, the secondary facial region refers to the region in the ROI that includes edge feature points among the five sensory feature points, excluding the primary facial region. Exemplary examples of edge feature points include the cheeks and chin. In particular, edge feature points can be subdivided depending on the actual situation, and this disclosure is not limited thereto.

[0039] Here, the background region refers to the areas in the ROI other than the primary and secondary facial regions.

[0040] Here, the quantization parameter is a parameter used to control the encoding quality. In the embodiments of this disclosure, a smaller quantization parameter results in a higher bitrate, and higher quality results in a larger file size.

[0041] In the embodiments of this disclosure, different quantization parameters can be assigned to the primary face region, secondary face region, and background region based on the ROI partitioning results, thereby reducing resource consumption in the background region. In particular, the minimum quantization parameter can be assigned to the primary face region, the medium quantization parameter to the secondary face region, and the maximum quantization parameter to the background region. The above is merely an illustrative description and does not limit, nor exhaustively list, all possible ways in which differentiated quantization parameters can be applied.

[0042] Exemplary, when dividing a region into three layers, feature point spatial topology relationships can be established simultaneously. Specifically, in the process of establishing feature point spatial topology relationships, a local rectangular coordinate system can first be established with the upper-left vertex of the initial ROI region as the spatial origin, then the absolute image coordinates of each feature point can be transformed into normalized coordinates relative to the initial ROI, and finally, a keypoint spatial topology structure record can be generated based on the normalized coordinate values, where the topology structure record includes the normalized coordinate values ​​of each feature point and a description of the relative positional relationship between any two feature points.

[0043] For example, the process of converting the absolute coordinates of each feature point in an image to normalized coordinates relative to the initial ROI can be expressed by the following formula.

[0044]

number

[0045] Here, x Normalization represents horizontally normalized coordinates, and y Normalization represents vertically normalized coordinates, x j represents the horizontal coordinate of the j-th feature point, and y j represents the vertical coordinate of the j-th feature point, L ROI,L represents the left boundary of the ROI, L ROI,U represents the upper boundary of the ROI, W ROI represents the width of the ROI, H ROI This represents the ROI.

[0046] Here, network bandwidth refers to the maximum amount of data that a communication channel or network can transmit in a given time period, and can be measured in bits per second. In the embodiments of this disclosure, network bandwidth can represent the network's ability to transmit data; the larger the bandwidth, the more data can be transmitted per unit time, and the higher the network's transmission efficiency.

[0047] Here, bitrate refers to the amount of data transmitted per unit time, and is usually measured in units of thousands of bits per second. In the embodiments of this disclosure, bitrate can represent the amount of data transmitted per unit time, and the higher the bitrate, the larger the amount of data transmitted and the higher the video quality. In particular, bitrate affects video quality and transmission speed.

[0048] In embodiments of this disclosure, the bitrate allocation for the primary face region, secondary face region, and background region can be dynamically adjusted based on the current network bandwidth to ensure a good user experience. The above is an illustrative description and does not limit, nor exhaustively list, all possibilities for allocating bitrates to the three layers.

[0049] Here, an encoded frame refers to a compressed image frame generated in the video encoding process. Its core function is to convert the original video data into a smaller, more easily stored and transmitted format while maintaining the visual quality of the original video as much as possible.

[0050] Here, metadata refers to additional information about an encoded frame, describing its structure, characteristics, or content, and providing guidance information to the decoder or subsequent processing.

[0051] In embodiments of this disclosure, ROI information can be added to the metadata after encoding the current frame, which is useful during decoding. The above is an illustrative description and does not limit, nor exhaustively list, all possibilities for the output of encoded frames and metadata.

[0052] The solution in the embodiments of this disclosure avoids independent motion estimation operations by reusing motion vectors generated in the MCTF process, predicts the ROI in the current frame based on the physical ROI position in the reference frame, and ensures real-time tracking. It also extracts sensory feature points from the first frame to construct spatial topology relationships, reuses these topology relationships in subsequent frames to correct the predicted ROI position, and maintains consistency between frames. Furthermore, it can prioritize the quality of critical regions by arranging differentiated quantization parameters for different hierarchical regions and combining them with a dynamic bitrate allocation strategy. The encoding end outputs ROI hierarchy metadata, and by outputting encoded frames and metadata, it can support optimized display at the decoding end, helping the decoder perform post-regionalization processing (e.g., sharpening of the main face) based on the metadata to restore the display quality of critical regions. By adopting the solution in the embodiments of this disclosure, encoding delay can be reduced while simultaneously improving the sharpness of the main face, the stuttering rate during bandwidth fluctuations can be significantly reduced, and high-definition, low-bitrate real-time live streaming push can be achieved.

[0053] In some embodiments, establishing an initial ROI involves identifying a face region using a lightweight face detection algorithm for the first frame of the video, and dividing the location of the face region into three layers based on the five sense keypoints: a primary face region, a secondary face region, and a background region, where the primary face region includes a minimum rectangular area covering the eyebrows, eyes, nose, and mouth; the secondary face region includes an annular area covering the cheeks and chin; and the background region includes the remaining area excluding the face.

[0054] Here, a lightweight face detection algorithm is an efficient, low-computational-resource-consuming algorithm designed for a face detection task. Exemplaryly, a lightweight face detection algorithm may select a fast detection architecture based on a hierarchical structure, or a conventional technique such as a combination of region extraction techniques and a specific pre-trained model, and may be selected depending on the actual situation, and is not limited to this disclosure.

[0055] In the embodiments of this disclosure, first, the first frame image can be extracted from the video stream as the first frame of the video. Then, the image can be scanned using a lightweight model to detect the position of a human face, and a rectangular bounding box for the human face can be output as a positioning result. In particular, if multiple human faces are detected in the first frame, the primary face can be screened according to rules such as the size and position of the area, and the results can be verified and filtered to eliminate false positives. The above is merely an illustrative description and does not limit all possibilities for outputting encoded frames and metadata, nor is it exhaustively listed here.

[0056] In the embodiments of this disclosure, first, a facial keypoint detection algorithm can be used to identify the locations of keypoints for the five facial features, including eyebrows, eyes, nose, mouth, cheeks, and chin, and to output the coordinates of these keypoints as the basis for facial region segmentation. Next, based on the keypoints such as eyebrows, eyes, nose, and mouth, the smallest rectangular boundary covering these feature points can be calculated as the primary face region. Then, an annular region covering the cheeks and chin can be formed by extending outward from the primary face region to form a secondary face region. In particular, the extension range can be determined according to the proportion of the primary face region. Finally, the primary and secondary face regions are subtracted from the ROI, and the remaining image region is extracted as the background region. Lastly, the boundary coordinates of the primary, secondary, and background regions can be saved, the region segmentation results can be saved as metadata, and associated with the encoding information of the first frame. The above is merely an illustrative description and does not limit all possibilities for segmenting the three-layer region, nor is it exhaustively listed here.

[0057] Thus, the lightweight algorithm allows for the rapid identification of the face region in the first frame of a video, meeting the demands of real-time video encoding. By extracting the five sense keypoints, the primary and secondary face regions of a person's face can be accurately divided, allowing encoding resources to be concentrated on the important areas. By using the five sense keypoints to divide the area into primary, secondary, and background regions, the encoding quality of important content is increased while the encoding quality of unimportant content is reduced, decreasing resource consumption in irrelevant areas and improving the user's viewing experience. By outputting the region division results, the division results can be used as metadata, which is suitable for dynamic adjustment of bitrate allocation and quantization parameters.

[0058] In some embodiments, the video encoding method further includes calculating a central region weight and an area weight for each of the multiple face regions detected in the first frame, wherein the central region weight reflects the proximity of the face center position to the center of the image, and the area weight reflects the proportion of the face region that occupies in the total area of ​​the image; calculating an overall score for each face based on a preset central weight factor and area weight factor; selecting the face with the highest overall score as the primary tracking target and determining the primary tracking target's face region as the primary face region; and integrating the remaining face regions not selected as primary tracking targets into secondary face regions.

[0059] Here, the central region weight is an evaluation metric for measuring the proximity of a face region to the center of the image. In the embodiments of this disclosure, the central region weight reflects whether a person's face is located at the visual midpoint of the image. In particular, faces closer to the center of the image are assigned a higher weight because they align with the user's focus of interest.

[0060] Here, area weights are used to assess the importance of a face region, reflecting the proportion of the total image area occupied by that face region. In the embodiments of this disclosure, larger faces need to be more emphasized and encoded with greater emphasis. In particular, area weights are used to determine the importance of faces that occupy a larger visual proportion.

[0061] In the embodiments of the present disclosure, first, a lightweight face detection algorithm is used to identify the positions of all face regions in the first frame and record the bounding box of each face. Then, the central region weight can be calculated. Specifically, first, the center point coordinates (x c , y c ) of the image are obtained. Next, the center point coordinates (x i , y i ) of each face region are determined. Then, the distance between the center point of each face and the center point of the image can be calculated. Finally, the calculated distance is normalized and mapped to the interval [0, 1]. After that, the central region weight C i can be taken inversely so that the smaller the distance, the larger the weight. Furthermore, the area of the bounding box of each face is obtained, and the ratio of the face area to the total area of the image is calculated, and this ratio can be used as the area weight A i . The above is only an exemplary description and does not limit all possibilities of calculating the central region weight and the area weight, and is not listed comprehensively here.

[0062] As an example, the process of calculating the distance between the center point of each face and the center point of the image can be expressed by the following formula.

[0063] [[ID=Z2]]

Equation

[0064] Here, L i represents the distance between the center point of the i-th face and the center point of the image.

[0065] As an example, the process of calculating the ratio of the face area to the total area of the image and using this ratio as the area weight A i can be expressed by the following formula.

[0066] ]>

Equation

[0067] [[ID=Z9]] Here, Si represents the i-th face area, S total This represents the total area of ​​the image.

[0068] Here, the median weight factor is a pre-defined weight factor that emphasizes the importance of the median region weight in the overall score. In the embodiments of this disclosure, the median weight factor can be determined based on the actual scene requirements.

[0069] Here, the area weighting factor is a pre-defined weighting factor used to adjust the degree of influence of area weights on the overall score. In embodiments of this disclosure, the importance of area weights can be emphasized or diminished by adjusting the magnitude of the area weighting factor.

[0070] Here, the overall score is the combined score of the face regions obtained by calculating the weighting factors, which are a combination of the central region weight and the area weight. In the embodiments of this disclosure, the combined score can be used to compare the importance of multiple face regions.

[0071] In embodiments of this disclosure, a central weighting factor and an area weighting factor can be set based on actual scene requirements or predefined parameters, and a total score can be calculated for each facial region. The above is merely an illustrative description and does not limit, nor exhaustively list, all possibilities for calculating a total score for each person's face.

[0072] For example, the process of calculating the overall score can be represented by the following formula:

[0073]

number

[0074] Here, P i β represents the overall score of the i-th facial region, β represents the median weighting factor, and γ represents the area weighting factor.

[0075] Here, the primary tracking target is the face region with the highest overall score selected, representing the most important face in the image. In the embodiments of this disclosure, the primary tracking target is encoded with emphasis as the primary face region.

[0076] In embodiments of this disclosure, first, all detected facial regions can be iterated through, the overall score of each region can be compared, and the facial region with the highest overall score can be selected and marked as the primary tracking target. Next, the five sensory keypoints can be extracted from the primary tracking target facial region and further divided into primary facial regions. The above is merely an illustrative description and does not limit all possibilities for determining primary facial regions, nor is it exhaustively listed here.

[0077] In embodiments of this disclosure, the remaining facial regions can be marked as sub-facial regions by integrating their boundary frames into a single whole. The above is an illustrative description and does not limit, nor exhaustively list, all possibilities for determining sub-facial regions.

[0078] In this way, by calculating the central region weight and area weight, the importance of each face region in the image can be quantified, while simultaneously supporting multi-target detection, handling cases where multiple faces are present in the first frame, and ensuring that no face regions are overlooked. By adjusting the central weight and area weight factors, importance evaluation can be optimized for different scenes. By calculating an overall score, deviations from a single indicator can be avoided, and the results can be made to better match the needs of the actual scene. The face with the highest overall score can be selected as the primary tracking target, ensuring accurate localization of important regions, and further concentrating encoding resources on the most important face regions to improve encoding efficiency and image quality. After integrating secondary face regions, it is ensured that other unselected face regions are also encoded, preventing secondary targets from being ignored, allowing for unified processing and saving encoding resources.

[0079] In some embodiments, the video encoding method further includes obtaining the intra-frame predicted cost and inter-frame predicted cost of the current frame; determining that a scene change has occurred if the intra-frame predicted cost does not exceed the product of a predetermined coefficient and the inter-frame predicted cost; and, in response to the occurrence of a scene change, interrupting the current ROI trace flow, making the current frame a reference frame for the new scene, and re-executing the ROI establishment operation on the reference frame.

[0080] Here, the intraframe predicted cost is the estimated encoding cost using the content of the current frame itself when measuring the compression efficiency of the current frame. In the embodiments of this disclosure, the intraframe predicted cost can reflect the resource consumption required by the current frame when performing intra-coding.

[0081] Here, the inter-frame predicted cost is the coding cost predicted using a reference frame when measuring the compression efficiency of the current frame. In embodiments of this disclosure, the inter-frame predicted cost can reflect the resource demand of the current frame when intercoding.

[0082] In the embodiments of this disclosure, intra-frame predictive coding is first performed on the current frame, the optimal intra-frame prediction mode is selected, the magnitude of the prediction residual, the quantization coefficient and coding complexity are measured, and the intra-frame prediction cost can be obtained. Subsequently, inter-frame predictive coding is performed using a reference frame, the complexity of the motion vector, the reference frame matching accuracy and the processing cost of the coding residual are calculated, and the inter-frame prediction cost can be obtained. The above is merely an illustrative description and does not limit all possibilities for obtaining intra-frame and inter-frame prediction costs, nor is it exhaustively listed here.

[0083] Here, the pre-set coefficients are weighting factors set to adjust the comparison conditions between intra-frame prediction costs and inter-frame prediction costs. In embodiments of this disclosure, the pre-set coefficients may reflect the relative importance of intra-coding and inter-coding.

[0084] Here, scene switching means a significant change in video content. For example, a scene switching may be a transition from one scene to another, where the content of the reference frame becomes unrelated to the content of the current frame. In the embodiments of this disclosure, the inter-frame prediction cost increases sharply during scene switching, while the intra-frame prediction cost is relatively low.

[0085] In the embodiments of this disclosure, first, the relationship between the intra-frame prediction cost and the inter-frame prediction cost is compared, and a scene switch can be considered to have occurred if the intra-frame prediction cost does not exceed the product of a preset coefficient and the inter-frame prediction cost. Specifically, a scene switch occurs when the prediction efficiency of the reference frame for the current frame decreases sharply, and the inter-frame prediction cost at this time is clearly higher than the intra-frame prediction cost, indicating that the content of the reference frame can no longer effectively predict the current frame. Exemplarily, the range of values ​​for the preset coefficient may be [0.3, 0.6]. In particular, the preset coefficient can be selected from other ranges of values ​​depending on the actual situation, and this disclosure is not limited thereto. The above is merely an illustrative explanation and does not limit all possibilities for determining that a scene switch has occurred, nor is it exhaustively listed here.

[0086] In embodiments of this disclosure, when a scene change occurs, the correlation between the reference frame and the current frame is lost, allowing the ROI tracking process based on the reference frame, such as motion vector prediction and feature point correction, to be stopped. Furthermore, the current frame can be marked as a scene change point, the correlation flow can be reinitialized in encoding, and the current frame can be used as a new reference frame for inter-frame prediction of subsequent frames. In particular, data related to past reference frames, such as motion vectors and ROI information, can be erased. Next, face regions can be re-identified on the new reference frame using a lightweight face detection algorithm, and a new ROI hierarchical partitioning can be established to divide the primary face region, secondary face region, and background region based on feature point detection. Finally, the new ROI can be associated with the encoding process, providing an accurate prediction basis for subsequent frames. The above is merely an illustrative description and does not limit, nor exhaustively list, all possibilities for re-executing the ROI establishment operation.

[0087] For example, the process of re-executing the ROI establishment operation can treat the current frame where the scene switch occurred as the first frame of the new scene, re-execute the initial ROI detection process for face detection and hierarchical segmentation on that frame, use that frame as a reference frame for later frames chronologically, and use the feature point space topology relationship established by the ROI of the current frame as the feature point space topology relationship of later frames.

[0088] Thus, by calculating intra-frame and inter-frame prediction costs, the encoding efficiency of the current frame can be dynamically evaluated, providing a precise basis for scene transition determination. By using a comparison of intra-frame and inter-frame prediction costs, significant changes in the video scene can be effectively detected, ensuring that the encoding process adapts to the new scene. When a scene transition occurs, the difference between the content of the reference frame and the current frame becomes large, leading to a decrease in encoding efficiency due to inter-frame prediction. Scene transition determination can avoid the use of inefficient reference frames. After interrupting the current ROI trace flow, it is possible to avoid predicting the ROI position using an incorrect reference frame, improving the accuracy of ROI positioning in subsequent frames. Initializing a new reference frame can adapt to scene changes in a timely manner, avoiding a decrease in encoding performance due to reference frame errors. By re-establishing the ROI, it is possible to ensure that important areas in the new scene are encoded with high quality, improving the overall video quality.

[0089] In some embodiments, reusing motion vectors generated during the motion-compensated time filtering process in a reference frame to determine the predicted ROI position of the current frame includes: dividing the physical ROI of the reference frame into a set of macroblocks based on a standard macroblock size; obtaining motion vectors calculated during the motion-compensated time filtering process for each macroblock; and generating the predicted ROI position of the current frame using a motion vector aggregation algorithm.

[0090] Here, the physical ROI of the reference frame refers to the actual location region determined by extracting the ROI from the reference frame during the video encoding process, and then performing accurate detection and correction. In the embodiments of this disclosure, this region is the part of the reference frame that requires the most emphasis on encoding.

[0091] Here, a standard macroblock is a fixed-size rectangular block obtained by dividing an image during the video encoding process, and is used in a subsequent compression process. In the embodiments of this disclosure, a macroblock is the basic unit of video encoding. Exemplaryly, in the H.264 standard, a typical macroblock size is 16 × 16 pixels, and similarly, in the H.265 standard, a typical macroblock size range is from 8 × 8 pixels to 64 × 64 pixels.

[0092] Here, a macroblock set refers to a set formed by dividing the physical ROI of a reference frame into multiple macroblocks. In embodiments of this disclosure, the macroblocks in the macroblock set jointly cover the entire ROI.

[0093] In embodiments of this disclosure, first, a detected ROI can be extracted from a reference frame and divided into several rectangular blocks based on a preset standard macroblock size, with the divided rectangular blocks being macroblocks. In particular, if the ROI boundaries do not perfectly match the macroblock size, the division can be completed by padding or trimming. Finally, all the divided macroblocks can be recorded as a single set, each macroblock containing its position coordinates and pixel or feature data. The above is merely an illustrative description and does not limit, nor exhaustively enumerate, all possibilities for dividing a set of macroblocks.

[0094] In the embodiments of this disclosure, the motion vector (MV) is used to estimate the motion relationship between a reference frame and the current frame, and typical methods include block matching and optical flow methods. In the MCTF process, motion estimation is performed for each macroblock to find the block that best matches the current frame and generate the MV. In particular, the motion vector for each macroblock includes a horizontal component and a vertical component, where the horizontal component reflects the horizontal displacement of the macroblock, and similarly, the vertical component reflects the vertical displacement of the macroblock. Finally, by linking the motion vector of each macroblock to its spatial position, the MV for each macroblock can be obtained. The above is merely an illustrative description and does not limit all possibilities for obtaining an MV, nor is it exhaustively listed here.

[0095] Here, an aggregation algorithm means generating an overall result by integrating, statistically analyzing, or optimizing the motion data (MV) of multiple macroblocks. In embodiments of this disclosure, the aggregation algorithm is used to integrate the motion information of a set of macroblocks as a predicted ROI position for the current frame.

[0096] Here, the ROI position prediction value is the ROI position estimation result in the current frame obtained by motion vector analysis. In the embodiment of this disclosure, the ROI position prediction value is the predicted position after the ROI position of the reference frame has been adjusted by motion compensation.

[0097] In the embodiments of this disclosure, first, the horizontal and vertical components can be extracted from the motion vectors of all macroblocks, and the medians of the horizontal and vertical components can be calculated to remove the influence of abnormal motion vectors. Next, the center point coordinates of the reference frame ROI can be obtained, and the medians of the horizontal and vertical components can be superimposed on the center point coordinates of the reference frame ROI to obtain the predicted center point of the current frame ROI. Finally, the predicted center point can be used as the center point of the current frame ROI, and the width and height of the region can be matched with the reference frame ROI to reconstruct the ROI position of the current frame. In particular, the ROI position prediction value includes the center point coordinates and region boundary of the current frame ROI, the region boundary can be calculated from the center point and dimensions. The above is merely an illustrative description and does not limit all possibilities for generating ROI position prediction values, nor is it exhaustively listed here.

[0098] Thus, each prediction is based on the most recently corrected physical ROI, ensuring that the predicted position does not deviate from reality due to error transmission. Aggregation using the median of motion vectors effectively filters out anomalous or noisy motion vectors, improving prediction accuracy. Dividing the ROI into a set of macroblocks reduces the computational complexity of motion compensation. After aggregating the motion vectors, prediction results can be generated directly, eliminating the need to re-detect the ROI and significantly improving real-time processing efficiency. If the ROI in the video moves over time, the motion compensation and aggregation algorithms can dynamically adjust the ROI position, ensuring the validity of the prediction results.

[0099] In some embodiments, the video encoding method further includes: calculating the degree of agreement of the motion vectors of each macroblock in the physical ROI of a reference frame as a confidence score; determining that local occlusion has occurred in the corresponding macroblock region if the confidence score exceeds a preset occlusion threshold; and updating the positional information of the macroblock region in the current frame using a feature point matching method for the macroblock region where local occlusion has occurred.

[0100] Here, the confidence score is an index that measures the degree of agreement of the motion vectors of each macroblock in the physical ROI of the reference frame. In embodiments of this disclosure, the confidence score may reflect whether the motion vectors of macroblocks within the region tend to move in a consistent manner. In particular, the confidence score is used to determine whether there is occlusion or irregular motion within the ROI, where occlusion occurs, resulting in mismatched motion vectors of macroblocks and a decrease in the confidence score.

[0101] In the embodiments of this disclosure, first, the statistical features of the motion vectors of all macroblocks in the physical ROI of the reference frame can be calculated, and these statistical features can be used as an agreement index. For example, the statistical features can be either the mean or the variance. If the mean is used as the statistical feature, the mean direction and amplitude of the motion vectors of all macroblocks can be calculated. Similarly, if the variance is used as the statistical feature, the degree of discreteness of the motion vectors can be evaluated. Next, a confidence score can be calculated from the agreement index. In particular, a lower confidence score indicates greater dispersion and discrepancy in the motion vectors. The above is merely an illustrative explanation and does not limit all possible ways in which a confidence score can be calculated; such possibilities are not exhaustively listed here.

[0102] For example, when variance is selected as the degree of agreement, the process of calculating the confidence score can be expressed by the following formula.

[0103]

number

[0104] Here, C o represents the confidence score, and D represents the variance of all macroblock MVs.

[0105] Specifically, when the motion vectors (MVs) are very similar, the variance is small and the confidence score approaches 1. Conversely, when the difference in motion vectors is large, the variance increases and the confidence score decreases, the score approaches 0.

[0106] Here, the occlusion threshold is a preset value compared to a confidence score to determine whether occlusion is likely to occur in a macroblock in the ROI. In embodiments of this disclosure, the occlusion threshold can be a sensitivity adjustment parameter for occlusion detection. Specifically, a lower value of the occlusion threshold makes occlusion detection more sensitive, while a higher value increases tolerance to occlusion. Exemplaryly, the occlusion threshold may be taken from the range [0.2, 0.5]. In particular, the occlusion threshold may be selected from other ranges of values ​​depending on the actual situation, and this disclosure is not limited thereto.

[0107] In the embodiments of this disclosure, a confidence score can be calculated for each ROI macroblock region and compared with an occlusion threshold. If the confidence score falls below a preset occlusion threshold (e.g., 0.3), it is determined that local occlusion has occurred in that macroblock region. Specifically, occlusion indicates that there is a significant difference or irregular distribution of motion vectors in the local region between the reference frame and the current frame, increasing the mismatch of motion vectors and lowering the confidence score. The above is merely an illustrative description and does not limit all possible ways in which local occlusion can be determined, nor is it exhaustively listed here.

[0108] Here, the feature point matching method is an algorithm for detecting and tracking feature points in an ROI. In the embodiments of this disclosure, the local region's position information can be updated by matching the feature points of the current frame with those of a reference frame.

[0109] In embodiments of this disclosure, keypoints can be extracted within the occlusion macroblock region of a reference frame using a feature point detection algorithm. Then, feature point detection can be performed again within the corresponding position region of the current frame to obtain a set of feature point candidates. Next, feature points in the reference frame and the current frame can be compared to find matching pairs, and the displacement of the local region can be estimated based on the matching feature point pairs. Exemplarily, the matching method can be Euclidean distance or Hamming distance. Finally, the feature point matching results can be mapped to the macroblock region to update the position of the occlusion macroblock in the current frame. This is an illustrative description and does not limit, nor exhaustively list, all possibilities for updating positional information.

[0110] For example, if feature points are occluded, the actual straight-line distance between each pair of feature points visible in the current frame can be measured. If the relative deviation between the measured distance value of the feature point pair and the corresponding distance value stored in the relative distance matrix exceeds a preset error threshold (for example, a preset error threshold range of 20% to 40%), it is determined that the feature point location has failed. All feature points that were determined to be invalid for location are removed, and the ROI position correction amount is recalculated using only the remaining valid feature points.

[0111] In this way, by calculating the consistency of motion vectors, abnormal motion within macroblock regions can be effectively detected, and local occlusion can be accurately identified. By setting different ranges of values ​​for the occlusion detection threshold, it can be adapted to the occlusion detection needs of different scenes. If motion vectors become invalid due to occlusion, the feature point matching method matches through local features, ensuring that the positional information of the occluded region is updated and the integrity of the ROI region is ensured. Error propagation is reduced by detecting and correcting the motion vectors of the occluded region in real time.

[0112] In some embodiments, correcting ROI position predictions based on feature point spatial topology relationships in the initial ROI includes: estimating the desired position coordinates of each feature point in the current frame in combination with the ROI position predictions based on the feature point spatial topology relationships recorded in the initial ROI; obtaining the actual position coordinates of the corresponding feature point in the current frame using a five-sensory feature point detection algorithm; calculating spatial transformation parameters based on the correspondence between the desired position coordinates and the actual position coordinates; and correcting ROI position predictions using the spatial transformation parameters.

[0113] Here, the desired position coordinates are the spatial topological relationship of the feature points recorded in the initial ROI and the theoretical position of the feature points in the current frame calculated from the ROI position prediction.

[0114] In embodiments of this disclosure, first, the feature point spatial topology relationship of the initial ROI can be obtained, and the initial topology relationship can be mapped to the current frame using the predicted ROI center point, dimensions, and shape. Exemplarily, the desired position coordinates can be obtained by adding a relative offset amount to the feature points based on the center point of the predicted ROI position. Then, based on the predicted ROI position, the theoretical position of each of the five sensory feature points in the current frame can be calculated to form a desired set of feature point positions. This is merely an illustrative description and does not limit, nor exhaustively list, all possibilities for estimating the desired position coordinates of feature points.

[0115] Here, the five-sensory feature point detection algorithm is used to detect key feature points in a human face and extract the actual position coordinates of the feature points.

[0116] Here, the actual position coordinates are the true positions of the feature points detected by the sensory feature point detection algorithm in the current frame.

[0117] In the embodiments of this disclosure, a feature point detection algorithm for the five senses can be first executed in the current frame to accurately extract the actual locations of feature points such as eyes, nose, and mouth. For example, the location of keypoints can be identified using a deep learning model, or points on edges and corners can be extracted using image processing techniques. Subsequently, the detected feature point coordinates can be recorded to form the actual location coordinates of the corresponding feature points. The above is merely an illustrative description and does not limit all possibilities for obtaining the actual location coordinates of feature points, nor is it exhaustively listed here.

[0118] Here, the spatial transformation parameter is used as transformation information that describes the geometric relationship between a desired position and an actual position. In the embodiments of this disclosure, the spatial transformation parameter can be represented by a mathematical model and may include translation parameters, rotation parameters, scale parameters, and the like.

[0119] In the embodiments of this disclosure, after associating desired position coordinates with actual position coordinates one by one, the deviations between the coordinates of each corresponding group can be compared, and geometric transformation parameters can be calculated based on the deviation information. For example, to obtain translation parameters, the offset between the desired position center and the actual position center can be calculated, and rotation parameters can be obtained by analyzing the angular change of a set of feature points. Scale parameters can be obtained by calculating the rate of change of distance between feature points. In particular, the transformation relationship from the desired position to the actual position can also be fitted using a linear transformation or affine transformation model. The above is merely an illustrative description and does not limit all possibilities for calculating spatial transformation parameters, nor is it exhaustively listed here.

[0120] In the embodiments of this disclosure, spatial transformation parameters can be applied to the predicted ROI position, and the corrected ROI position can be set as the final ROI position of the current frame. The above is merely an illustrative description and does not limit, nor exhaustively list, all possibilities for correcting predicted ROI positions.

[0121] For example, the process of correcting ROI position predictions can be as follows: First, the desired position coordinates of each feature point in the current frame can be estimated based on the ROI position in the current frame predicted from the motion vector and the normalized coordinates. Next, the actual position coordinates of each feature point in the current frame can be detected. Then, the ROI position correction amount can be determined by minimizing the total distance deviation between the actual position coordinates and the desired position coordinates of all feature points. Finally, the correction amount can be applied to adjust the position of the predicted ROI region.

[0122] In this way, by correcting the ROI position prediction value using spatial transformation parameters, deviations due to scene changes and prediction errors are reduced. The five-sensory feature point detection algorithm can adapt to various facial expression changes, head movements, or occlusion situations, ensuring accuracy in detecting the actual position. At the same time, by dynamically adjusting the ROI position using spatial transformation parameters, it can adapt to dynamically changing scenes, and the corrected ROI position is closer to the true region, avoiding resource consumption due to misjudgment.

[0123] In some embodiments, dynamically allocating the bitrate of the three-layer region based on real-time network bandwidth includes, in response to detection that the real-time network bandwidth does not fall below a preset bandwidth threshold, controlling the proportion of the bitrate allocated to the main face region to the total bitrate so as not to fall below a preset proportion threshold, and, in response to detection that the real-time network bandwidth is below a preset bandwidth threshold, ensuring that the bitrate of the main face region does not fall below a preset minimum guaranteed bitrate, and compressing the bitrate of the background region within a first compression coefficient range of the original reference value.

[0124] Here, the bandwidth threshold is a pre-set reference value. In the embodiments of this disclosure, the bandwidth threshold can be used to determine whether the network bandwidth is sufficient. In particular, if the network bandwidth is greater than or equal to the bandwidth threshold, the network condition is considered good and a high bitrate can be allocated; otherwise, the bitrate needs to be reduced to comply with bandwidth limitations.

[0125] Here, the proportional threshold is the lowest percentage of the total bitrate allocated to the main face region when sufficient bandwidth is available. In embodiments of this disclosure, the proportional threshold can ensure that the main face region receives sufficient bitrate resources to maintain high-quality encoding when sufficient network bandwidth is available. Exemplary, the proportional threshold may be in the range of [0.5, 0.7]. In particular, the proportional threshold may be selected from other ranges of values ​​depending on the actual situation, and this disclosure is not limited thereto.

[0126] In the embodiments of this disclosure, the current network bandwidth value can be obtained in real time through a network state monitoring mechanism, and the current network bandwidth value can be compared with a bandwidth threshold. For example, the process of obtaining the current network bandwidth value can be carried out by a Real-time Control Protocol (RTCP), etc. Next, if it is detected that the real-time network bandwidth does not fall below a preset bandwidth threshold, that is, if the bandwidth is considered sufficient, the product of the total bitrate value corresponding to the real-time network bandwidth and the proportional threshold can be used as the bitrate of the main face area, and the bitrate of the main face area can be allocated according to the proportional threshold. The above is merely an illustrative description and does not limit all possibilities for controlling the allocation of bitrates, nor is it exhaustively listed here.

[0127] Here, the minimum guaranteed bitrate is the minimum bitrate that must be allocated to the main face region even when bandwidth is insufficient, in order to ensure the basic visual quality of the main face region. In embodiments of this disclosure, the minimum guaranteed bitrate can prevent a significant degradation in the quality of the main face region in order to ensure the clarity of important areas even when network bandwidth is low.

[0128] Here, the original reference value is the initial bitrate allocated to the background region when bandwidth is sufficient. In embodiments of this disclosure, the original reference value is the normal encoding state of the background region. In particular, when network bandwidth is insufficient, bitrate compression can be performed on the background region based on the original reference value.

[0129] Here, the first compression factor is a range of background compression ratios in the event of bandwidth shortage, in order to conserve bandwidth by reducing the bitrate allocated to the background region. In embodiments of this disclosure, the first compression factor can control the bitrate compression width of the background region and maintain efficient use of network bandwidth. Exemplary, the range of values ​​for the first compression factor may be [0.5, 0.8]. In particular, the first compression factor may be selected from other ranges of values ​​depending on the actual situation, and this disclosure is not limited thereto.

[0130] In embodiments of this disclosure, if network bandwidth is insufficient, first, it is ensured that the bitrate of the main face region does not fall below the minimum guaranteed bitrate; that is, the minimum guaranteed bitrate is compared with the product of the total bitrate value and a proportional threshold, and the maximum value between the two is taken as the bitrate of the main face region. Subsequently, the bitrate of the background region is compressed, and the product of the original reference value and a first compression coefficient is taken as the bitrate of the background region, which can be controlled to stay within the range of the first compression coefficient of the original reference value. The above is merely an illustrative description and does not limit all possibilities for compressing the bitrate of the background region, nor is it exhaustively listed here.

[0131] In this way, by preferentially allocating bitrate to the main face region, the encoding quality of the main face region can be ensured and the user experience improved, regardless of whether the network bandwidth is sufficient or insufficient. The minimum guarantee mechanism ensures that even with insufficient network bandwidth, the main face region maintains the minimum bitrate, avoiding serious distortion in critical areas of the image. By detecting network bandwidth in real time, the system can dynamically adjust the bitrate allocation strategy, ensuring the stability of the video stream under different bandwidth conditions. By controlling the range of the first compression factor, flexible bitrate compression can be performed on the background region, improving bandwidth utilization efficiency.

[0132] In some embodiments, the video encoding method further includes: compressing the bitrate of the background region to a specific value within a first compression factor range when bandwidth shortage occurs for the first time; compressing the bitrate of the secondary face region to a second compression factor range when bandwidth continues to decrease and the rate of decrease per unit time exceeds a preset rate threshold; and ensuring in the compression process that the bitrate of the primary face region does not fall below a value obtained by multiplying a preset minimum guaranteed bitrate by a guard redundancy factor multiplier.

[0133] In embodiments of this disclosure, network bandwidth can be monitored in real time, and when it is first detected that the bandwidth has fallen below a preset bandwidth threshold, background region bitrate compression logic is triggered. Specifically, a specific compression value can be selected from a first compression coefficient range, and the background region bitrate can be adjusted to the product of the compression value and the total bitrate, freeing up bandwidth resources to be allocated to the primary and secondary face regions. The above is merely an illustrative description and does not limit, nor exhaustively list, all possibilities for compressing the background region bitrate.

[0134] Here, the rate threshold is a predefined criterion for the rate of bandwidth descent. In embodiments of this disclosure, the rate threshold can be used to determine whether network bandwidth is rapidly decreasing within a unit of time.

[0135] Here, the second compression factor is the compression range of the bitrate of the secondary face region when bandwidth drops sharply, and is used to reduce the bitrate allocation of the secondary face region. In embodiments of this disclosure, the second compression factor dynamically adjusts the bitrate allocation of the secondary face region, allowing more bandwidth resources to be reserved for the primary face region. Exemplarily, the second compression factor can take values ​​in the range of [0.8, 0.95]. In particular, the second compression factor may be selected from other ranges of values ​​depending on the actual situation, and this disclosure is not limited thereto.

[0136] In the embodiments of this disclosure, the bandwidth drop rate can be calculated in real time, and if the drop exceeds a rate threshold, the bitrate compression logic for the secondary face region is triggered. Specifically, a specific compression value can be selected from a range of second compression coefficients, and the bitrate of the secondary face region can be adjusted to the product of the compression value and the total bitrate, freeing up bandwidth resources and allocating them to the primary face region. The above is merely an illustrative description and does not limit, nor exhaustively list, all possibilities for compressing the bitrate of the secondary face region.

[0137] Here, the guard redundancy factor is a guaranteed bitrate multiplier for the main face region in the event of a rapid decrease in bandwidth, and is used to ensure the lowest possible visual quality in the main face region. In embodiments of this disclosure, the guard redundancy factor can ensure that no serious distortion occurs in the main face region even when the bandwidth decreases rapidly. Exemplary, the guard redundancy factor can take values ​​in the range of [1.1, 1.3]. In particular, the guard redundancy factor may be selected from other ranges of values ​​depending on the actual situation, and this disclosure is not limited thereto.

[0138] In embodiments of this disclosure, the minimum value of the main area bitrate can be calculated based on the minimum guaranteed bitrate and the guard redundancy factor. For example, the product of the minimum guaranteed bitrate and the guard redundancy factor can be used as the minimum value of the main area bitrate. Subsequently, even if the bandwidth rapidly decreases, it is ensured that the bitrate allocated to the main face area does not fall below the minimum value of the main area bitrate, thereby ensuring the visual quality of critical areas. The above is merely an illustrative description and does not limit, nor exhaustively list, all possibilities for protecting the bitrate of the main face area.

[0139] In this way, the primary face region always maintains a high bitrate, ensuring that even with a rapid decrease in bandwidth, the lowest visual quality requirements are met, avoiding distortion and blurring. The compression process prioritizes the quality of the primary face region, clearly defining the allocation priority of encoding resources. The background region's bitrate is compressed only when bandwidth shortage is first detected, ensuring it does not affect the primary and secondary regions. Bandwidth changes are dynamically monitored based on rate thresholds, immediately triggering secondary region bitrate compression to ensure video stream stability. If bandwidth decreases further, the secondary face region's bitrate is compressed, further saving bandwidth resources. The primary face region remains sharp at all times, and the quality of critical areas of the video stream remains unaffected even with decreased network bandwidth.

[0140] In some embodiments, assigning differentiated quantization parameters to the corrected primary face region, secondary face region, and background region includes setting the minimum quantization parameter offset value for the primary face region, enabling a first prediction mode combination, and assigning the highest priority coding resource, so that the quantization parameters for the primary face region are lower than the encoder's base quantization parameters; setting a medium quantization parameter offset value for the secondary face region, enabling a second prediction mode combination, and assigning a normal priority coding resource, so that the quantization parameters for the secondary face region are equal to or slightly higher than the encoder's base quantization parameters; and setting the highest quantization parameter offset value for the background region, enabling a third prediction mode combination, and assigning the lowest priority coding resource.

[0141] Here, the first combination of prediction modes is a set of prediction modes that the encoder assigns to the main face region. In embodiments of this disclosure, the first combination of prediction modes may include prediction modes that divide the image into 4x4 and 8x8 block sizes.

[0142] In the embodiments of this disclosure, the main face region is a critical area of ​​user interest, and therefore requires a minimum quantization parameter to reduce quantization errors. For example, the quantization parameter of the main face region may be the difference between the base quantization parameter and the minimum quantization parameter offset value. Subsequently, the first prediction mode combination can be enabled, and the main face region can be coded with high granularity by dividing it using 4x4 and 8x8 block sizes. In particular, the 4x4 size division is suitable for processing complex textures and details such as eyes and mouths, while the 8x8 size division is suitable for processing somewhat larger, uniform areas. Furthermore, the main face region gains more computational resources and bandwidth, ensuring that the details of that region are preserved with the highest quality. The above is merely an illustrative description and does not limit, nor exhaustively list, all possibilities for enabling the first prediction mode combination.

[0143] Here, the combination of second prediction modes is a set of prediction modes that the encoder assigns to the sub-face region. In embodiments of this disclosure, the combination of second prediction modes may include prediction modes that divide the region into 8x8 and 16x16 block sizes.

[0144] In the embodiments of this disclosure, the secondary face region is not a critical area of ​​user focus, but good visual quality is required, and therefore a moderate quantization parameter is needed to reduce quantization errors. For example, the quantization parameter for the secondary face region may be the difference between the base quantization parameter and a moderate quantization parameter offset value. Subsequently, the secondary face region can be encoded with moderate granularity by enabling a combination of second prediction modes and dividing it using 8x8 and 16x16 block sizes. Furthermore, the computational resources and bandwidth allocated to the secondary face region are lower than those for the primary face region but higher than those for the background region, ensuring a balance between visual quality and resource allocation. The above is merely an illustrative description and does not limit, nor exhaustively list, all possibilities for enabling a combination of second prediction modes.

[0145] Here, the third prediction mode combination is a set of prediction modes that the encoder assigns to the background region. In embodiments of this disclosure, the third prediction mode combination may include prediction modes that divide the region into blocks of 16 × 16 or larger.

[0146] In the embodiments of this disclosure, the background region, being of least user interest, can accept the highest quantization parameters, thereby reducing the need to retain background details and saving resources. Exemplarily, the quantization parameters for the background region may be the difference between the base quantization parameter and the maximum quantization parameter offset value. The background region can then be coded at a low granularity by enabling a third prediction mode combination and dividing it using a block size of 16 × 16 or larger. Furthermore, the background region receives minimal computational resources and bandwidth, prioritizing the coding quality of the main face region. This is merely an illustrative description and does not limit, nor exhaustively list, all possibilities for enabling a third prediction mode combination.

[0147] In this way, the main face region uses minimal quantization parameters and small block division to ensure the highest encoding quality in the important areas of the video that the user focuses on, while simultaneously reducing visual distortion and improving the user experience by precisely encoding complex details through a combination of first prediction modes. By concentrating resources on the main face region and balancing them with the secondary face region, resource consumption in the background region is reduced, enabling efficient use of encoding resources. Quantization parameters are dynamically adjusted based on the importance of the video content, ensuring that details in important areas are clear while reducing resource consumption in the secondary face and background regions. At the same time, different block division methods are adopted for different regions, adapting to the texture complexity of each region to achieve an optimal balance between quality and efficiency.

[0148] In some embodiments, the ROI detection step for the first frame can be performed first. Specifically, in the first frame of the live stream video, a lightweight face detection algorithm can be used to detect the face region, determine the initial ROI, and record information such as its position, size, and shape. At the same time, based on the key feature points of a person's face (e.g., eyes, nose, mouth), the face region is further divided into three layers: the primary face region (including the core parts of the five senses), the secondary face region (the contour and surrounding area of ​​the face), and the background region (the part other than the face).

[0149] Furthermore, a step of ROI tracking based on MCTF motion estimation can be performed. Specifically, the MV obtained in the MCTF process can be reused to continuously track the face ROI region. Exemplarily, in the motion estimation stage of MCTF, the MV of each macroblock between the current frame and adjacent reference frames can be calculated, and this process does not consume additional time, especially since it is usually a necessary module of the encoder. Subsequently, for the face ROI determined in the first frame, it can be divided into multiple macroblocks, aggregation analysis can be performed based on the motion vectors of adjacent macroblocks to predict the position of that ROI in the current frame, and finally, the predicted position can be corrected to match the relative positional relationships of face key feature points to obtain the accurate ROI position and update the region range in three tiers.

[0150] Furthermore, a scene transition determination step can be performed. Specifically, it is possible to determine whether a scene transition has occurred by calculating the difference in image features between the current frame and adjacent frames. For example, by calculating the cost difference between the intra-frame prediction mode and the inter-frame prediction mode of the current frame, it can be determined that a scene transition has occurred if the cost of the intra-frame prediction mode is much lower than that of the inter-frame prediction mode, and otherwise, it can be determined that a scene transition has not occurred.

[0151] Specifically, if a scene change is detected, the ROI detection step is re-executed to update the face ROI and its three hierarchical divisions, while if no scene change has occurred, the ROI and hierarchical divisions obtained by tracking based on MCTF motion estimation are continued to be used.

[0152] Furthermore, a bitrate tiered coding step can be performed. Specifically, based on the first determined ROI and its three tiers, the encoder's coding parameters can be adjusted to perform differentiated coding. For the main face area, lower quantization parameters, a higher coding quality level, and a finer prediction mode are adopted to ensure clarity of the details of the five senses. For the secondary face area, the quantization parameters are set to a medium level, and the normal prediction mode is adopted. For the background area, the quantization parameters are appropriately increased, a coarser prediction mode is adopted, the coding quality is reduced, and the bitrate allocation is decreased. At the same time, the bitrate ratio of each tier is adjusted in real time based on the network bandwidth. If the bandwidth is sufficient, the bitrate ratio of the main face area is increased, while if the bandwidth is insufficient, the basic bitrate requirement of the main face area is prioritized.

[0153] Finally, an encoded output step can be performed. Specifically, the video frames encoded using bitrate tiering can be encapsulated to generate a live streaming video output.

[0154] As shown in Figure 2, another flowchart of the video encoding method according to an embodiment of the present disclosure is shown. As shown in Figure 2, the video encoding method includes the following:

[0155] In S201, acquire the live streaming video frame.

[0156] In S202, ROI detection is performed using a lightweight face detection algorithm.

[0157] In S203, MVs undergoing MTCF processing are reused to perform ROI tracking.

[0158] In S204, it is determined whether a scene change has occurred. If a scene change has occurred, the program proceeds to S202. If no scene change has occurred, the program proceeds to S205.

[0159] Bitrate-tiered coding is performed in S205.

[0160] In S206, video frames are encapsulated and output.

[0161] This solution provides a tripartite low-latency, high-definition coding architecture combining tracking, hierarchical design, and flow control. By reusing MCTF motion vectors, it eliminates redundant calculations, significantly reduces processing delays in the motion tracking module, avoids independent motion estimation operations, achieves near-zero overhead ROI inter-frame tracking, and improves the real-time nature of coding. The mechanism for reusing the topological relationships of the first frame across frames maintains spatial consistency of feature points, eliminates quality variations caused by feature points reconstructed frame by frame, ensures clarity of details in critical areas, and optimizes coding quality. Combined with hierarchical coding and dynamic bitrate allocation, it intelligently balances quality and fluency during bandwidth fluctuations, enhances bandwidth adaptability, and reduces the risk of stuttering in live streaming through the collaborative optimization of motion tracking and bitrate control. It optimizes overall bitrate efficiency while ensuring visual quality in the main face area.

[0162] The schematic diagram shown in Figure 2 is illustrative, not restrictive, and expandable. Those skilled in the art will understand that various obvious changes and / or substitutions can be made based on the example in Figure 2, and that the resulting techniques still fall within the scope of the embodiments of this disclosure.

[0163] Embodiments of the present disclosure provide a video encoding device, as shown in Figure 3, comprising: a reference frame acquisition module 301 for acquiring a reference frame adjacent to the current frame and in which a hierarchical division of the initial region of interest (ROI) has been established; a position prediction module 302 for reusing motion vectors generated during a motion compensation time filtering process in the reference frame to determine the predicted ROI position of the current frame; a position correction module 303 for extracting sensory feature points from the current frame and correcting the predicted ROI position based on the feature point spatial topology relationship in the initial ROI; a parameter placement module 304 for placing differentiated quantization parameters in the corrected primary face region, secondary face region, and background region, respectively; a bitrate allocation module 305 for dynamically allocating the bitrate of the three layer regions based on real-time network bandwidth; and a data output module 306 for outputting the encoded frame of the current frame and associated ROI hierarchical metadata, wherein the ROI hierarchical metadata includes boundary coordinates of the primary face region, secondary face region, and background region, quantization parameter offset values ​​corresponding to each region, and bitrate allocation weight values ​​for each region.

[0164] In some embodiments, the video encoding device further includes a face identification module 307 (not shown in Figure 3) for identifying a face region using a lightweight face detection algorithm for the first frame of the video, and a hierarchical division module 308 (not shown in Figure 3) for dividing the face region into three layers: a primary face region, a secondary face region, and a background region, based on the five sense keypoints, wherein the primary face region includes a minimum rectangular area covering the eyebrows, eyes, nose, and mouth; the secondary face region includes an annular area covering the cheeks and chin; and the background region includes the remaining area other than the face.

[0165] In some embodiments, the video encoding device further includes a weight calculation module 309 (not shown in Figure 3) for calculating a central region weight and an area weight for multiple face regions detected in the first frame, wherein the central region weight reflects the proximity of the face center position to the image center, and the area weight reflects the proportion of the face region to the total area of ​​the image; an integrated score calculation module 310 (not shown in Figure 3) for calculating an integrated score for each face based on a preset central weight factor and area weight factor; a primary face determination module 311 (not shown in Figure 3) for selecting the face with the highest integrated score as the primary tracking target and determining the face region of the primary tracking target as the primary face region; and a secondary face determination module 312 (not shown in Figure 3) for integrating the positions of the remaining face regions not selected as primary tracking targets into secondary face regions.

[0166] In some embodiments, the video encoding device includes a prediction cost acquisition module 313 (not shown in Figure 3) for acquiring the intra-frame prediction cost and inter-frame prediction cost of the current frame; a scene switch determination module 314 (not shown in Figure 3) for determining that a scene switch has occurred if the intra-frame prediction cost does not exceed the product of a preset coefficient and the inter-frame prediction cost; and a region update module 315 (not shown in Figure 3) for interrupting the current ROI tracking flow, setting the current frame as the reference frame of the new scene, and re-executing the ROI establishment operation on the reference frame, in response to the occurrence of a scene switch.

[0167] In some embodiments, the position prediction module 302 includes a macroblock splitting submodule for dividing the physical ROI of a reference frame into a set of macroblocks based on a standard macroblock size; a motion vector acquisition submodule for obtaining motion vectors calculated during a motion compensation time filtering process for each macroblock; and a prediction generation submodule for generating ROI position predictions for the current frame using a motion vector aggregation algorithm.

[0168] In some embodiments, the video encoding device further includes a confidence calculation module 316 (not shown in Figure 3) for calculating the degree of agreement of each macroblock motion vector in the physical ROI of a reference frame as a confidence score; a local occlusion determination module 317 (not shown in Figure 3) for determining that local occlusion has occurred in a corresponding macroblock region when the confidence score exceeds a preset occlusion determination threshold; and a position information update module 318 (not shown in Figure 3) for updating the position information of a macroblock in the current frame using a feature point matching method for a macroblock region where local occlusion has occurred.

[0169] In some embodiments, the position correction module 303 includes: a desired position estimation submodule for estimating the desired position coordinates of each feature point in the current frame in combination with ROI position prediction values ​​based on the feature point spatial topology relationships recorded in the initial ROI; an actual position acquisition submodule for obtaining the actual position coordinates of the corresponding feature point in the current frame using a five-sensory feature point detection algorithm; a transformation parameter calculation submodule for calculating spatial transformation parameters based on the correspondence between the desired position coordinates and the actual position coordinates; and a position prediction correction submodule for correcting the ROI position prediction values ​​using the spatial transformation parameters.

[0170] In some embodiments, the bitrate allocation module 305 includes a main face bitrate allocation submodule for controlling the proportion of the bitrate allocated to the main face region to the total bitrate so as not to fall below a predetermined proportion threshold, in response to the detection that the real-time network bandwidth does not fall below a predetermined bandwidth threshold, and a background bitrate allocation submodule for ensuring that the bitrate of the main face region does not fall below a predetermined minimum guaranteed bitrate, and for compressing the bitrate of the background region within a first compression coefficient range of the original reference value, in response to the detection that the real-time network bandwidth is less than a predetermined bandwidth threshold.

[0171] In some embodiments, the video encoding device further includes a background bitrate dynamic adjustment module 319 (not shown in Figure 3) for compressing the background bitrate to a specific value within a first compression coefficient range when bandwidth shortage occurs for the first time, and for compressing the secondary face bitrate to a second compression coefficient range when bandwidth continues to decrease and the rate of decrease per unit time exceeds a preset rate threshold, and a main face bitrate dynamic adjustment module 321 (not shown in Figure 3) for ensuring that the main face bitrate does not fall below a value obtained by multiplying a preset minimum guaranteed bitrate by a guard redundancy coefficient during the compression process.

[0172] In some embodiments, the parameter placement module 304 includes a first combination submodule for setting a minimum quantization parameter offset value in the primary face region so that the quantization parameters of the primary face region are lower than the basic quantization parameters of the encoder, and for enabling a first prediction mode combination and allocating the highest priority coding resource; a second combination submodule for setting a medium quantization parameter offset value in the secondary face region so that the quantization parameters of the secondary face region are equal to or slightly higher than the basic quantization parameters of the encoder, and for enabling a second prediction mode combination and allocating a normal priority coding resource; and a third combination submodule for setting a maximum quantization parameter offset value in the background region, and for enabling a third prediction mode combination and allocating the lowest priority coding resource.

[0173] A description of the specific functions and examples of the modules and submodules of the apparatus according to the embodiments of this disclosure can be found in the relevant descriptions of the corresponding steps in the embodiments of the method described above, and will not be repeated here.

[0174] The video encoding apparatus in the embodiments of this disclosure avoids independent motion estimation calculations by reusing motion vectors generated by the MCTF process, predicts the current frame ROI based on the physical ROI position in the reference frame, and ensures real-time tracking. It extracts sensory feature points from the first frame to construct spatial topological relationships, reuses these topological relationships in subsequent frames to correct the predicted ROI position, and maintains consistency between frames. Differentiated quantization parameters can be placed for different hierarchical regions and, combined with a dynamic bitrate allocation strategy, can prioritize the quality of critical regions. The encoding end outputs ROI hierarchical metadata and outputs encoded frames and metadata to support optimized display at the decoding end, helping the decoder perform post-regionalization processing (e.g., sharpening of the main face) based on the metadata to restore the display quality of critical regions. This reduces encoding delay while simultaneously improving the sharpness of the main face, significantly reduces stuttering during bandwidth fluctuations, and enables high-definition, low-bitrate real-time live streaming push.

[0175] An embodiment of this disclosure provides a schematic scene diagram of a video encoding method, as shown in Figure 4.

[0176] As described above, the video encoding methods provided in the embodiments of this disclosure are applicable to electronic devices. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other appropriate computers.

[0177] Specifically, electronic devices can perform the following operations:

[0178] The process includes obtaining a reference frame adjacent to the current frame and in which a hierarchical partitioning of the initial region of interest (ROI) has been established; reusing motion vectors generated during the motion-compensated time filtering process in the reference frame to determine the predicted ROI position of the current frame; extracting sensory feature points from the current frame and correcting the predicted ROI position based on the feature point spatial topology relationships in the initial ROI; assigning differentiated quantization parameters to the corrected primary face region, secondary face region, and background region, respectively; dynamically allocating bitrates to the three layer regions based on real-time network bandwidth; and outputting the encoded frame of the current frame and associated ROI hierarchical metadata, wherein the ROI hierarchical metadata includes boundary coordinates of the primary face region, secondary face region, and background region, quantization parameter offset values ​​corresponding to each region, and bitrate allocation weight values ​​for each region.

[0179] The scene diagram shown in Figure 4 is for illustrative purposes only and is not restrictive. Those skilled in the art will understand that various obvious changes and / or substitutions can be made based on the example in Figure 4, and that the resulting techniques still fall within the scope of the embodiments of this disclosure.

[0180] In the technical aspects of this disclosure, the acquisition, storage, and use of relevant user personal information will comply with the provisions of applicable laws and regulations and will not violate public order and morals.

[0181] According to embodiments of the present disclosure, the present disclosure further provides electronic devices, non-temporary computer-readable storage media, and program products.

[0182] Figure 5 is a block diagram of an electronic device 500 according to an embodiment of the present disclosure. An electronic device refers to any form of digital computer, such as a laptop computer, desktop computer, workstation, personal digital assistant, server, blade server, mainframe computer, and other compatible computers. An electronic device further refers to any form of mobile device, such as a personal digital assistant, cellular phone, intelligent phone, wearable device, and other similar computer equipment. The components, their connections, and functions described in this disclosure are illustrative and do not limit the realization of anything described or specified in this disclosure.

[0183] As shown in Figure 5, device 500 includes a computing unit 501 capable of performing various appropriate operations and processes based on computer program instructions stored in read-only memory (ROM) 502 or computer program instructions loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 can further store various programs and data necessary for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are connected to each other via bus 504. An input / output (I / O) interface 505 is also connected to bus 504.

[0184] Multiple components in device 500 are connected to an I / O interface 505, which includes an input unit 506 such as a keyboard and mouse, an output unit 507 such as various displays and speakers, a storage unit 508 such as a magnetic disk or optical disk, and a communication unit 509 such as a network card, modem, or wireless communication transceiver. The communication unit 509 allows device 500 to exchange information / data with other devices via computer networks such as the Internet and / or various carrier networks.

[0185] The computing unit 501 may be a variety of general-purpose and / or dedicated processing components having processing and computing capabilities. Some exemplary components of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, a computing unit that executes various machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs each of the methods and processes described above, for example, the video encoding method. For example, in some embodiments, the video encoding method can be implemented as a computer software program tangibly contained in a machine-readable medium such as a storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed into the device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the video encoding method described above can be performed. Additionally, in other embodiments, the computing unit 501 may be configured to execute the video encoding method by any other suitable method (e.g., firmware).

[0186] Various embodiments of the systems or technologies described in this disclosure can be implemented by digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standards (ASSPs), systems-on-a-chip (SOCs), complex-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. Each of these embodiments may be implemented by one or more computer programs that run and / or interpret on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0187] Program code for performing the methods of this disclosure can be written in any combination of one or more programming languages. These program codes are provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programming data processing device, so that when the program code is executed by the processor or controller, it can perform the functions / operations defined in the flowcharts and / or block diagrams. The program code may run entirely in a mainscan, partially in a mainscan, partially as an independent soft encapsulation and partially in a remote mainscan, or entirely in a remote mainscan or server.

[0188] In this disclosure, machine-readable media may be tangible media containing or storing programs used by or in conjunction with instruction execution systems, devices, or equipment. Machine-readable media may be machine-readable signal media or machine-readable storage media. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any suitable combination of the contents described above. Further exemplary examples of machine-readable storage media include one or more wired electrical connections, portable computer disk cartridges, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any combination of the contents described above.

[0189] To provide user interaction, a computer may implement the systems and technologies described herein, which may include a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor), a keyboard and pointing device for the user to provide input to the computer (e.g., a mouse or trackball). Other types of devices may also be used to provide user interaction; for example, the feedback provided to the user may be any form of sensor feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user may be accepted in any form (e.g., acoustic input, voice input, haptic input).

[0190] The systems and technologies described herein can be implemented in computing systems that include background components (e.g., as data servers), computing systems that include middleware components (e.g., application servers), computing systems that include front-end components (e.g., user computers having a graphics user interface or network browser, through which users can interact with embodiments of the systems and technologies described herein), or in any combination of such background components, middleware components, or front-end components. Components of the system can be connected to one another via digital data communication in any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0191] A computer system can include a client and a server. Typically, the client and server are geographically separated and interact via a communication network. The client-server relationship is created by a computer program that operates on the corresponding computer. The server may be a cloud server, a server in a distributed system, or a server incorporating blockchain technology, etc.

[0192] It should be understood that steps can be newly ranked, added, or deleted using the various forms of flows shown above. For example, each step described in this disclosure may be executed in parallel, sequentially, or in a different order. This disclosure is not limited to this, as long as the technical aspects disclosed in this disclosure can achieve the desired results.

[0193] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions are possible due to design considerations and other factors. Any changes, equivalent substitutions, and improvements within the gist and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video encoding method, Obtain a reference frame adjacent to the current frame and for which a hierarchical partitioning of the initial region of interest (ROI) has been established. The motion vector generated during the motion compensation time filtering process in the aforementioned reference frame is reused to determine the ROI position prediction value for the current frame, The process involves extracting sensory feature points from the current frame and correcting the ROI position prediction value based on the feature point spatial topology relationship in the initial ROI. By placing differentiated quantization parameters in the corrected primary face region, secondary face region, and background region, Dynamically allocating bitrates across three layers based on real-time network bandwidth, Outputting the encoded frame of the current frame and associated ROI hierarchy metadata, wherein the ROI hierarchy metadata includes boundary coordinates of the primary face region, secondary face region and background region, quantization parameter offset values ​​corresponding to each region, and bitrate allocation weight values ​​for each region. Video encoding method.

2. The establishment of the initial ROI is Identifying the face region using a lightweight face detection algorithm on the first frame of the video, The method involves dividing the facial region into three layers based on the five key points of the five senses: a primary facial region, a secondary facial region, and a background region, wherein the primary facial region includes a minimum rectangular area covering the eyebrows, eyes, nose, and mouth; the secondary facial region includes an annular area covering the cheeks and chin; and the background region includes the remaining area other than the face. The video encoding method according to claim 1.

3. The aforementioned video encoding method further, For the multiple face regions detected in the first frame, the central region weight and area weight are calculated for each face, wherein the central region weight reflects the proximity of the face's center position to the image center, and the area weight reflects the proportion of the face region that occupies the total area of ​​the image. The overall score for each face is calculated based on pre-set median weighting factors and area weighting factors. The face with the highest overall score is selected as the primary tracking target, and the facial region of the primary tracking target is determined to be the primary facial region. This includes integrating the remaining facial regions not selected as the primary tracking target into the secondary facial regions, The video encoding method according to claim 2.

4. The aforementioned video encoding method further, The process involves obtaining the intra-frame predicted cost and inter-frame predicted cost for the current frame, A scene change is determined to have occurred if the predicted cost within a frame does not exceed the product of a predetermined coefficient and the predicted cost between frames. In response to the aforementioned scene switch, the current ROI trace flow is interrupted, the current frame is made the reference frame of the new scene, and the ROI establishment operation is re-executed on the reference frame, including: The video encoding method according to claim 1.

5. Reusing the motion vector generated during the motion compensation time filtering process in the aforementioned reference frame to determine the ROI position prediction value for the current frame is: The physical ROI of the aforementioned reference frame is divided into sets of macroblocks based on the standard macroblock size, For each macroblock, obtain the motion vector calculated during the motion-compensated time filtering process, This includes generating ROI position prediction values ​​for the current frame using a motion vector aggregation algorithm, The video encoding method according to claim 1.

6. The aforementioned video encoding method further, The degree of agreement of the motion vectors of each macroblock in the physical ROI of the aforementioned reference frame is calculated as a confidence score, When the aforementioned confidence score exceeds a pre-set occlusion threshold, it is determined that local occlusion has occurred in the corresponding macroblock region. This includes updating the positional information of a macroblock region in the current frame using a feature point matching method for a macroblock region where local occlusion has occurred. The video encoding method according to claim 5.

7. Correcting the ROI position prediction value based on the feature point space topology relationship in the initial ROI is, Based on the feature point spatial topology relationship recorded in the initial ROI, the desired position coordinates of each feature point in the current frame are estimated by combining this with the ROI position prediction values. The five-sensory feature point detection algorithm obtains the actual position coordinates of the corresponding feature point in the current frame, Calculating spatial transformation parameters based on the correspondence between desired position coordinates and actual position coordinates, This includes correcting the ROI position prediction value using the spatial transformation parameter, The video encoding method according to claim 1.

8. Dynamically allocating the bitrate of the three layers based on the aforementioned real-time network bandwidth is, In response to the detection that the real-time network bandwidth does not fall below a predetermined bandwidth threshold, the bitrate allocated to the main face area is controlled so that its proportion of the total bitrate does not fall below a predetermined percentage threshold. This includes, in response to the detection that the real-time network bandwidth is below a preset bandwidth threshold, ensuring that the bitrate of the main face area does not fall below a preset minimum guaranteed bitrate, and compressing the bitrate of the background area within the range of the first compression factor of the original reference value, The video encoding method according to claim 1.

9. The aforementioned video encoding method further, If a bandwidth shortage occurs for the first time, the bitrate of the background area is compressed to a specific value within the first compression coefficient range, If the bandwidth continues to decrease and the rate of decrease per unit time exceeds a preset rate threshold, the bitrate of the secondary region will be compressed to within the range of the second compression factor. The compression process includes ensuring that the bitrate of the main face region does not fall below a value obtained by multiplying the preset minimum guaranteed bitrate by a guard redundancy factor multiplier, The video encoding method according to claim 8.

10. Placing differentiated quantization parameters in the corrected primary face region, secondary face region, and background region, respectively, The process involves setting the minimum quantization parameter offset value for the main face region so that the quantization parameters of the main face region are lower than the encoder's base quantization parameters, enabling the first prediction mode combination, and allocating the highest priority coding resource. Set a moderate quantization parameter offset value in the sub-face region so that the quantization parameters of the sub-face region are equal to or slightly higher than the encoder's base quantization parameters, enable the second prediction mode combination, and allocate coding resources with normal priority. This includes setting the highest quantization parameter offset value in the background region, enabling the third prediction mode combination, and allocating the lowest priority coding resource. The video encoding method according to claim 1.

11. A video encoding device, A reference frame acquisition module for obtaining reference frames adjacent to the current frame and for which a hierarchical partitioning of the initial region of interest (ROI) has been established, A position prediction module for determining the ROI position prediction value of the current frame by reusing motion vectors generated during the motion compensation time filtering process in the aforementioned reference frame, A position correction module for extracting sensory feature points from the current frame and correcting the ROI position prediction value based on the feature point spatial topology relationship in the initial ROI, A parameter placement module for placing differentiated quantization parameters in the corrected main face region, secondary face region, and background region, respectively. A bitrate allocation module for dynamically allocating bitrates in the three-tier domain based on real-time network bandwidth, A data output module for outputting the encoded frame of the current frame and associated ROI hierarchy metadata, wherein the ROI hierarchy metadata includes boundary coordinates of the primary face region, secondary face region and background region, quantization parameter offset values ​​corresponding to each region, and bitrate allocation weight values ​​for each region. Video encoding device.

12. At least one processor, The system comprises at least one processor and a memory that is communicated with by it, The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor causes the at least one processor to perform the method according to any one of claims 1 to 10. Electronic devices.

13. A non-temporary computer-readable storage medium storing computer instructions that cause a computer to perform the method described in any one of claims 1 to 10.

14. A program for implementing the method described in any one of claims 1 to 10, which is executed by a processor in a computer.

Images