Key point smoothing method and apparatus

Abstract

The application provides a key point smoothing method and device. The method comprises the following steps: acquiring a key point of a target object on a to-be-processed image; constructing a key point vector corresponding to the target object according to the key point of the target object; acquiring a reconstruction matrix; acquiring a reconstruction coefficient according to the key point vector and the reconstruction matrix; and determining a smoothed key point vector according to the reconstruction coefficient and the reconstruction matrix. The method uses principal component analysis to analyze the face shape, and then uses several main components obtained by solving to reconstruct the key point position, so that the coordinates of the corrected key point are closer to the real face, and the key point position accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to a key point smoothing method and apparatus. Background Technology

[0002] Facial landmark detection can locate key facial regions, including eyebrows, eyes, nose, mouth, and facial contours. It has applications in many scenarios, such as automatic face recognition, expression analysis, 3D face reconstruction, and 3D animation. Facial landmark detection methods include parametric and non-parametric methods. Parametric methods include local and global methods. Non-parametric methods include graph model-based methods, cascade regression-based methods, and deep learning-based methods. However, regardless of the method, the detected landmark locations will always exhibit some degree of deviation.

[0003] In existing technologies, for critical points that deviate significantly, the position of the critical point is corrected using various spline curves by combining the positions of all critical points within the area where the critical point is located. However, the accuracy of the corrected position is still not high. Summary of the Invention

[0004] This application provides a key point smoothing method and apparatus for improving the accuracy of facial key point positions.

[0005] In a first aspect, this application provides a smoothing processing method, comprising: acquiring key points of a target object on an image to be processed; constructing a key point vector corresponding to the target object based on the key points of the target object; acquiring a reconstruction matrix; acquiring reconstruction coefficients based on the key point vector and the reconstruction matrix; and determining the smoothed key point vector based on the reconstruction coefficients and the reconstruction matrix.

[0006] Secondly, this application provides a smoothing processing apparatus, comprising: a construction module for acquiring key points of a target object on an image to be processed; constructing a key point vector corresponding to the target object based on the key points of the target object; acquiring a reconstruction matrix; acquiring reconstruction coefficients based on the key point vector and the reconstruction matrix; and a processing module for determining the smoothed key point vector based on the reconstruction coefficients and the reconstruction matrix.

[0007] The smoothing processing method and apparatus provided in this application acquire key points of a target object on an image to be processed; construct a key point vector corresponding to the target object based on the key points; obtain a reconstruction matrix; obtain reconstruction coefficients based on the key point vector and the reconstruction matrix; and determine the smoothed key point vector based on the reconstruction coefficients and the reconstruction matrix. The method of this application utilizes principal component analysis to analyze the face shape, and then uses the solved principal components to reconstruct the key point positions, making the coordinates of the corrected key points closer to the real face, thus improving the accuracy of the key point positions. Attached Figure Description

[0008] Figure 1 A schematic diagram of facial landmarks provided for this application;

[0009] Figure 2 A schematic flowchart of an embodiment of the smoothing method provided in this application;

[0010] Figure 3 A diagram illustrating the sorting of facial landmarks provided for this application;

[0011] Figure 4 A flowchart illustrating Embodiment 2 of the smoothing processing method provided in this application;

[0012] Figure 5 A schematic diagram of the smoothing processing apparatus provided in this application;

[0013] Figure 6 A schematic diagram of the hardware structure of the chip provided in this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] In the present application, it should be noted that the terms "first" and "second" are used for descriptive purposes only, and cannot be construed as indicating or implying relative importance. In addition, "at least one" means one or more, and "a plurality of" means two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three kinds of relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects are in an "or" relationship. "The following at least one (item)" or similar expressions refers to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can mean: a alone, b alone, c alone, a combination of a and b, a combination of a and c, a combination of b and c, or a combination of a, b and c, where a, b and c can be single or multiple.

[0016] Face key point detection can be applied in many scenarios, such as automatic face recognition, expression analysis, 3D face reconstruction and 3D animation. At present, there are various methods for face key point detection. However, no matter what method is adopted, the positions of key points detected will have deviations of varying degrees. In the prior art, for key points that are seriously deviated, the positions of the key points are corrected through various spline curves by combining the positions of all key points in the region where the key point is located.

[0017] For example, see Figure 1 As shown in the figure, the position of the key point numbered 18 on the face contour is seriously deviated. The positions of all key points on the face contour can be combined, and the position of the key point numbered 18 can be corrected through various spline curves. However, the correction method based on spline curves does not take into account the prior information of the face shape, and the accuracy of the corrected position is still not high.

[0018] In order to solve the above technical problem, the present application introduces principal component analysis into key point position correction. Specifically, principal component analysis is used to analyze the face shape, and then several main components obtained through calculation are used to reconstruct the key point positions, thereby correcting the key point positions. Since this method is based on the analysis of face shape, compared with the aforementioned correction method based on spline curves, the corrected position is more accurate.

[0019] The technical solution of the present application and how the technical solution of the present application solves the above technical problem will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0020] Example 1

[0021] Figure 2 This is a flowchart illustrating an embodiment of the smoothing method provided in this application. Figure 2 As shown, the smoothing method provided in this embodiment includes:

[0022] S201. Obtain the key points of the target object on the image to be processed.

[0023] Optionally, the target object in this embodiment can be a human face or other parts of the human body; this application is not limited in this regard. This embodiment uses a human face as an example to illustrate the solution.

[0024] Optionally, local-based methods, global-based methods, graph model-based methods, cascade regression-based methods, and deep learning-based methods can be used to obtain the key points of the target object. For detailed processes, please refer to the existing technologies, which will not be repeated here.

[0025] S202. Based on the key points of the target object, construct the key point vector corresponding to the target object.

[0026] Specifically, all keypoints can be sorted, and a keypoint vector can be constructed based on the sorting result. For example, the keypoints of the facial contour can be sorted first, then the keypoints of the eyebrows, then the keypoints of the nose, then the keypoints of the eyes, and then the keypoints of the mouth. For each part, the keypoints can be sorted according to a certain direction, such as starting from the left, starting from the right, starting from the top, or starting from the bottom. Figure 3 An example of sorting is shown.

[0027] It should be noted that each keypoint has an x-coordinate and a y-coordinate. In the sorted result, the x-coordinate of the first keypoint is used as the first element of the keypoint vector, and the y-coordinate of the first keypoint is used as the second element of the keypoint vector. In the sorted result, the x-coordinate of the second keypoint is used as the third element of the keypoint vector, the y-coordinate of the second keypoint is used as the fourth element of the keypoint vector, and so on.

[0028] Combination Figure 3 In the example shown, the x-coordinate of the keypoint with index 0 is used as the first element of the keypoint vector, the y-coordinate of the keypoint with index 0 is used as the second element, the x-coordinate of the keypoint with index 1 is used as the third element, the y-coordinate of the keypoint with index 1 is used as the fourth element, and so on. Because... Figure 3 The face shown includes 83 key points, each with an x-coordinate and a y-coordinate. Therefore, the constructed key point vector has a total of 83×2 elements.

[0029] S203, Obtain the reconstruction matrix.

[0030] In one possible implementation, the reconstruction matrix can be obtained as follows:

[0031] First, a keypoint matrix is ​​constructed based on the image set, where the keypoints of the target object in each image in the image set form one row of the keypoint matrix. Then, the target matrix is ​​obtained based on the keypoint matrix. Next, the covariance matrix is ​​obtained based on the target matrix and the number of images in the image set. Finally, the reconstruction matrix is ​​obtained based on the covariance matrix.

[0032] For ease of explanation, the key point matrix in this embodiment uses X. n×k This indicates that the target matrix uses X′. n×k This indicates that the covariance matrix uses C k×k This indicates that the reconstructed matrix uses A. k′×k This indicates that the reconstruction coefficients use c. k′ express.

[0033] It should be noted that: obtaining the reconstruction matrix A k′×k This step can be performed before S201, after obtaining the reconstruction matrix A. k′×k Then, the reconstructed matrix A can be... k′×k Save it in a file, and you can read it directly from the file when needed.

[0034] It should be noted that the images in the image collection include images of different people.

[0035] The following describes how to construct a keypoint matrix X from an image set. n×k Possible ways to achieve this:

[0036] For each image in the image set, the coordinates of the key points of the face in that image are obtained. Then, the key point vector corresponding to the face is constructed using the same method as in S202. This process is repeated for all images to obtain the key point vector corresponding to each image. The key point vector corresponding to each image is then used as the key point matrix X. n×k In the first line, n is the number of images in the image set, and k is the number of keypoints multiplied by 2. For example, if the image set has 10,000 images, each image has 83 keypoints, and each keypoint has an x-coordinate and a y-coordinate, then the constructed keypoint matrix X... n×k There are 10000 × 166 elements.

[0037] The following describes how to use the keypoint matrix X n×k Obtain the target matrix X′ n×k Possible ways to achieve this:

[0038] It can obtain the mean coordinates of keypoints on all images in the image set, and convert the keypoint matrix X... n×k Subtracting the mean from the coordinates of the key points yields the target matrix X′. n×k.

[0039] Specifically, the mean of the x-coordinates of keypoints on all images can be calculated, and the keypoint matrix X′ can be generated. n×k Subtract the mean from the x-coordinates in the matrix. Similarly, calculate the mean of the y-coordinates of all keypoints in the image, and then convert the keypoint matrix X′ to its mean. n×k Subtracting the mean from the ordinate in the matrix yields the target matrix X′. n×k Based on the example above, calculate the mean of the x-coordinates of all keypoints in 10,000 images, and then convert the keypoint matrix X′... n×k Subtract the mean from the x-coordinates in the matrix, calculate the mean of the y-coordinates of all keypoints in 10,000 images, and then convert the keypoint matrix X′ to its mean. n×k Subtracting the mean from the ordinate of the matrix gives the target matrix X′. n×k .

[0040] The following describes how to use the target matrix X′ n×k The covariance matrix C is obtained from the number of images in the image set. k×k Possible ways to achieve this:

[0041] The covariance matrix C can be obtained using the following formula. k×k :

[0042]

[0043] Where n represents the number of images in the image set.

[0044] The following describes the covariance matrix C. k×k Obtain the reconstruction matrix A k′×k Possible ways to achieve this:

[0045] First, consider the covariance vector moment C. k×k The matrix is ​​decomposed into eigenvalues ​​to obtain at least one eigenvalue; then, the at least one eigenvalue is sorted in descending order, and the eigenvectors corresponding to the first preset number of eigenvalues ​​are obtained; based on the eigenvectors corresponding to the preset number of eigenvalues, a reconstruction matrix A is constructed. k′×k .

[0046] Optionally, after sorting at least one feature value in descending order, the feature vectors corresponding to the top 50% of feature values ​​can be obtained, and the feature vectors corresponding to the top 50% of feature values ​​can be used to form the reconstruction matrix A. k′×k .

[0047] S204. Obtain the reconstruction coefficients based on the key point vector obtained in S202 and the reconstruction matrix obtained in S203.

[0048] In one possible implementation, the reconstruction coefficient c can be obtained using the following formula. k′ :

[0049] c k′ =A k′×k ·x T Where x represents the key point vector obtained by S202.

[0050] S205. Determine the key point vector after smoothing based on the reconstruction coefficients and reconstruction matrix.

[0051] In one possible implementation, the smoothed keypoint vector can be determined using the following formula:

[0052]

[0053] Here, x′ represents the smoothed keypoint vector. The smoothed keypoint vector includes the coordinates of each keypoint after smoothing.

[0054] The smoothing method provided in this embodiment uses principal component analysis to analyze the shape of the face, and then uses the solved principal components to reconstruct the key point positions, so that the coordinates of the corrected key points are closer to the real face, thus improving the accuracy of the key point positions.

[0055] In video streaming scenarios, due to the randomness of noise and the generally limited robustness of facial landmark detection algorithms, keypoint jitter is frequently observed between adjacent frames. To improve the stability of keypoints between frames, this application proposes to estimate the keypoints of the target object in the current frame by combining the coordinates of keypoints from several frames prior to the current frame, obtaining a keypoint estimation vector. Based on the keypoint estimation vector and the smoothed keypoint vector, the stabilized keypoint vector is obtained. This improves the stability of keypoints between frames. The following embodiments will detail the inter-frame stabilization process.

[0056] Example 2

[0057] Figure 4 This is a flowchart illustrating Embodiment Two of the smoothing processing method provided in this application. Any frame in the video stream can be smoothed using the smoothing processing method provided in this embodiment. For ease of explanation, the current frame is referred to as the image to be processed in this embodiment. In this embodiment, the image stabilization process needs to be combined with key points from at least two frames preceding the current frame. Therefore, if the image to be processed is the first or second frame of the video stream, it can be smoothed using steps S401-S405. If the image to be processed is not the first or second frame of the video stream, further image stabilization processing is performed using steps S406-S407 after smoothing using steps S401-S405. See also... Figure 4 As shown, the smoothing method provided in this embodiment includes:

[0058] S401. Obtain the coordinates of key points of the target object on the image to be processed.

[0059] S402. Construct the key point vector corresponding to the target object based on the coordinates of the key points of the target object.

[0060] S403, Obtain the reconstruction matrix.

[0061] S404. Obtain the reconstruction coefficients based on the key point vector obtained in S402 and the reconstruction matrix obtained in S403.

[0062] S405. Determine the key point vector after smoothing based on the reconstruction coefficients and reconstruction matrix.

[0063] The implementation process of S401-S405 can be found in S201-S205 of the above embodiment, and will not be repeated here.

[0064] S406. Based on the key point vectors corresponding to the target object in at least two previous frames of the image to be processed, obtain the key point estimation vector corresponding to the image to be processed.

[0065] In one possible implementation, a variance vector can be calculated based on the keypoint vectors corresponding to the target object in at least two previous frames of the image to be processed; a second variable corresponding to the image to be processed can be determined based on the variance vector and a first variable corresponding to the previous frame of the image to be processed; a gain corresponding to the image to be processed can be determined based on the second variable; a first variable corresponding to the image to be processed can be determined based on the gain and the second variable; and a keypoint estimation vector corresponding to the image to be processed can be determined based on the keypoint estimation vector corresponding to the previous frame of the image to be processed, the gain of the image to be processed, and the smoothed keypoint vector.

[0066] For ease of explanation, the variance vector in this embodiment uses q. k This indicates that the first variable corresponding to the previous frame of the image to be processed uses p. k-1 This indicates that the second variable corresponding to the image to be processed uses p′. k This indicates that the gain corresponding to the image to be processed is expressed using g. k This indicates that the first variable corresponding to the image to be processed uses p. k This indicates that the keypoint estimation vector corresponding to the previous frame of the image to be processed uses px. k-1 This indicates that the smoothed keypoint vector uses x k express.

[0067] The following describes how to calculate the variance vector q. k Possible ways to achieve this:

[0068] Taking the third frame as an example, based on the keypoint vectors corresponding to the target object in the first frame and the second frame, the variance of each element is calculated, and the variance vector q is formed using the variance of each element. k .

[0069] The following describes how to determine the second variable p′ corresponding to the image to be processed. k Possible ways to achieve this:

[0070] The second variable p′ corresponding to the image to be processed can be determined by the following formula. k :

[0071]

[0072] Where δ is an empirical value. The first variable in the first frame is p1, p1 = 0.

[0073] The following describes how to determine the gain g corresponding to the image to be processed. k Possible ways to achieve this:

[0074] The gain g of the image to be processed can be determined by the following formula. k :

[0075]

[0076] The following describes how to determine the first variable p corresponding to the image to be processed. k Possible ways to achieve this:

[0077] The first variable p corresponding to the image to be processed can be determined by the following formula. k :

[0078] p k =(1-g k )*p′ k

[0079] The following describes a feasible method for determining the estimated vectors of key points in an image to be processed:

[0080] The keypoint estimation vectors for the image to be processed can be determined using the following formula:

[0081] px k =px k-1 +g k *(x k -px k-1 )

[0082] Among them, px k Represents the keypoint estimation vector corresponding to the image to be processed, px k-1 x represents the keypoint estimation vector corresponding to the previous frame of the image to be processed.k This represents the smoothed keypoint vector.

[0083] S407. Based on the smoothed keypoint vector and the keypoint estimation vector, determine the stabilized keypoint vector corresponding to the image to be processed.

[0084] Specifically, the Euclidean distance between the smoothed keypoint vector and the estimated keypoint vector is obtained; if the Euclidean distance is greater than a preset threshold, the estimation is considered incorrect, and the smoothed keypoint vector is determined as the keypoint vector after image stabilization; if the Euclidean distance is less than or equal to the preset threshold, the estimated keypoint vector is determined as the keypoint vector after image stabilization, thereby improving the stability of inter-frame keypoints in video stream scenarios.

[0085] Figure 5 A schematic diagram of the smoothing processing apparatus provided in this application. Figure 5 As shown, the smoothing processing apparatus provided in this application includes:

[0086] Construction module 501 is used to obtain key points of a target object on the image to be processed; construct a key point vector corresponding to the target object based on the key points of the target object; obtain a reconstruction matrix; and obtain reconstruction coefficients based on the key point vector and the reconstruction matrix.

[0087] The processing module 502 is used to determine the smoothed key point vector based on the reconstruction coefficients and the reconstruction matrix.

[0088] Optionally, building module 501 is specifically used for:

[0089] Based on the image set, a key point matrix is ​​constructed, wherein the key points of the target object on each image in the image set constitute a row of the key point matrix;

[0090] Based on the key point matrix, obtain the target matrix;

[0091] Based on the target matrix and the number of images in the image set, obtain the covariance matrix;

[0092] The reconstruction matrix is ​​obtained based on the covariance matrix.

[0093] Optionally, building module 501 is specifically used for:

[0094] Obtain the mean coordinates of key points on all images in the image set;

[0095] The target matrix is ​​obtained by subtracting the mean value from the coordinates of the key points in the key point matrix.

[0096] Optionally, building module 501 is specifically used for:

[0097] The covariance matrix is ​​obtained using the following formula:

[0098]

[0099] Among them, C k×k Let X′ represent the covariance matrix. n×k Let n represent the target matrix, and n represent the number of images in the image set.

[0100] Optionally, building module 501 is specifically used for:

[0101] The covariance matrix is ​​decomposed into eigenvalues ​​to obtain at least one eigenvalue.

[0102] Sort the at least one feature value in descending order and obtain the feature vectors corresponding to the first preset number of feature values;

[0103] The reconstruction matrix is ​​constructed based on the eigenvectors corresponding to the preset number of eigenvalues.

[0104] Optionally, building module 501 is specifically used for:

[0105] The reconstruction coefficients are obtained using the following formula:

[0106] c k′ =A k′×k ·x T

[0107] Among them, c k′ A represents the reconstruction coefficient. k′×k Let x represent the reconstruction matrix and x represent the key point vector.

[0108] Optionally, the processing module 502 is specifically used for:

[0109] The smoothed keypoint vector is determined using the following formula:

[0110] x′=A T ·c k′

[0111] Where x′ represents the keypoint vector after smoothing, A represents the reconstruction matrix, and c k′ This represents the reconstruction coefficient.

[0112] Optionally, the processing module 502 is also used for:

[0113] Based on the key point vectors corresponding to the target object in at least two previous frames of the image to be processed, obtain the key point estimation vector corresponding to the image to be processed.

[0114] Based on the smoothed keypoint vector and the keypoint estimation vector, the stabilized keypoint vector corresponding to the image to be processed is determined.

[0115] Optionally, the processing module 502 is specifically used for:

[0116] Calculate the variance vector based on the key point vectors corresponding to the target object in the at least two frames of images;

[0117] The second variable corresponding to the image to be processed is determined based on the variance vector and the first variable corresponding to the previous frame image of the image to be processed;

[0118] The gain corresponding to the image to be processed is determined based on the second variable corresponding to the image to be processed;

[0119] The first variable corresponding to the image to be processed is determined based on the gain corresponding to the image to be processed and the second variable corresponding to the image to be processed;

[0120] The keypoint estimation vector corresponding to the image to be processed is determined based on the keypoint estimation vector corresponding to the previous frame of the image to be processed, the gain corresponding to the image to be processed, and the smoothed keypoint vector.

[0121] Optionally, the processing module 502 is specifically used for:

[0122] The second variable corresponding to the image to be processed is determined by the following formula:

[0123]

[0124] Where, p′ k q represents the second variable corresponding to the image to be processed. k Let p represent the variance vector. k-1 The first variable represents the image corresponding to the previous frame of the image to be processed, and δ is an empirical value.

[0125] Optionally, the processing module 502 is specifically used for:

[0126] The gain corresponding to the image to be processed is determined by the following formula:

[0127]

[0128] Among them, g k p′ represents the gain corresponding to the image to be processed. k The second variable represents the image to be processed.

[0129] Optionally, the processing module 502 is specifically used for:

[0130] The first variable corresponding to the image to be processed is determined by the following formula:

[0131] p k =(1-g k )*p′ k

[0132] Where, p k The first variable, g, represents the image to be processed. k p′ represents the gain corresponding to the image to be processed. k The second variable represents the image to be processed.

[0133] Optionally, the processing module 502 is specifically used for:

[0134] The keypoint estimation vector corresponding to the image to be processed is determined by the following formula:

[0135] px k =px k-1 +g k *(x k -px k-1 )

[0136] Among them, px k The keypoint estimation vector corresponding to the image to be processed is represented by px. k-1 g represents the keypoint estimation vector corresponding to the previous frame of the image to be processed. k x represents the gain corresponding to the image to be processed. k This represents the smoothed keypoint vector.

[0137] Optionally, the processing module 502 is specifically used for:

[0138] Obtain the Euclidean distance between the smoothed keypoint vector and the estimated keypoint vector;

[0139] If the Euclidean distance is greater than a preset threshold, then the smoothed key point vector is determined as the stabilized key point vector.

[0140] If the Euclidean distance is less than or equal to the preset threshold, then the key point estimation vector is determined as the key point vector after the image stabilization process.

[0141] The key point smoothing device provided in this embodiment can be used to perform the steps in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0142] Figure 6 A schematic diagram of the hardware structure of the chip provided in this application. (For example...) Figure 6 As shown, the chip in this embodiment may include:

[0143] Memory 601 is used to store program instructions.

[0144] The processor 602 is configured to implement the key point smoothing method described in any of the above embodiments when the program instructions are executed. The specific implementation principle can be found in the above embodiments, and will not be repeated here.

[0145] This application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the keypoint smoothing method described in any of the above embodiments.

[0146] This application also provides a program product comprising a computer program stored in a readable storage medium, wherein at least one processor can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause a chip to implement the keypoint smoothing method described in any of the above embodiments.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0148] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0150] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] It should be understood that the processor described in this application can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A smoothing processing method, characterized in that, include: Obtain key points of the target object on the image to be processed; Based on the key points of the target object, construct the key point vector corresponding to the target object; Obtain the reconstruction matrix; Based on the key point vector and the reconstruction matrix, the reconstruction coefficients are obtained; Based on the reconstruction coefficients and the reconstruction matrix, the smoothed keypoint vector is determined; The image to be processed is an image from a video stream; The method further includes: Calculate the variance vector based on the key point vectors corresponding to the target object in at least two frames of images; Based on the variance vector and the first variable corresponding to the previous frame of the image to be processed, determine the second variable corresponding to the image to be processed; based on the second variable corresponding to the image to be processed, determine the gain corresponding to the image to be processed; based on the gain corresponding to the image to be processed and the second variable corresponding to the image to be processed, determine the first variable corresponding to the image to be processed; based on the keypoint estimation vector corresponding to the previous frame of the image to be processed, the gain corresponding to the image to be processed, and the smoothed keypoint vector, determine the keypoint vector corresponding to the image to be processed after image stabilization.

2. The method according to claim 1, characterized in that, The process of obtaining the reconstructed matrix includes: Based on the image set, a key point matrix is ​​constructed, wherein the key points of the target object on each image in the image set constitute a row of the key point matrix; Based on the key point matrix, obtain the target matrix; Based on the target matrix and the number of images in the image set, obtain the covariance matrix; The reconstruction matrix is ​​obtained based on the covariance matrix.

3. The method according to claim 2, characterized in that, The step of obtaining the target matrix based on the key point matrix includes: Obtain the mean coordinates of key points on all images in the image set; The target matrix is ​​obtained by subtracting the mean value from the coordinates of the key points in the key point matrix.

4. The method according to claim 2, characterized in that, The step of obtaining the covariance matrix based on the target matrix and the number of images in the image set includes: The covariance matrix is ​​obtained using the following formula: in, Let the covariance matrix be represented. Let n represent the target matrix, and n represent the number of images in the image set.

5. The method according to claim 2, characterized in that, The step of obtaining the reconstruction matrix based on the covariance matrix includes: The covariance matrix is ​​decomposed into eigenvalues ​​to obtain eigenvalues; Sort the feature values ​​in descending order and obtain the feature vectors corresponding to the first preset number of feature values; The reconstruction matrix is ​​constructed based on the eigenvectors corresponding to the preset number of eigenvalues.

6. The method according to any one of claims 1-5, characterized in that, The step of obtaining the reconstruction coefficients based on the key point vector and the reconstruction matrix includes: The reconstruction coefficients are obtained using the following formula: in, Represents the reconstruction coefficients. Represents the reconstruction matrix, This represents the key point vector.

7. The method according to claim 6, characterized in that, The step of determining the smoothed keypoint vector based on the reconstruction coefficients and the reconstruction matrix includes: The smoothed keypoint vector is determined using the following formula: in, This represents the keypoint vector after the smoothing process. Represents the reconstruction matrix, This represents the reconstruction coefficient.

8. The method according to claim 1, characterized in that, The step of determining the second variable corresponding to the image to be processed based on the variance vector and the first variable corresponding to the previous frame image of the image to be processed includes: The second variable corresponding to the image to be processed is determined by the following formula: in, This represents the second variable corresponding to the image to be processed. Denotes the variance vector. The first variable represents the image corresponding to the previous frame of the image to be processed. These are experience points.

9. The method according to claim 8, characterized in that, The step of determining the gain corresponding to the image to be processed based on the second variable corresponding to the image to be processed includes: The gain corresponding to the image to be processed is determined by the following formula: in, This represents the gain corresponding to the image to be processed. The second variable represents the image to be processed.

10. The method according to claim 8, characterized in that, The step of determining the first variable corresponding to the image to be processed based on the gain corresponding to the image to be processed and the second variable corresponding to the image to be processed includes: The first variable corresponding to the image to be processed is determined by the following formula: in, The first variable represents the image to be processed. This represents the gain corresponding to the image to be processed. The second variable represents the image to be processed.

11. The method according to claim 8, characterized in that, The step of determining the keypoint estimation vector corresponding to the image to be processed based on the keypoint estimation vector corresponding to the previous frame image of the image to be processed, the gain corresponding to the image to be processed, and the smoothed keypoint vector includes: The keypoint estimation vector corresponding to the image to be processed is determined by the following formula: in, This represents the keypoint estimation vector corresponding to the image to be processed. This represents the keypoint estimation vector corresponding to the previous frame of the image to be processed. This represents the gain corresponding to the image to be processed. This represents the smoothed keypoint vector.

12. The method according to claim 1, characterized in that, The step of determining the image-stabilized keypoint vector corresponding to the image to be processed based on the smoothed keypoint vector and the keypoint estimation vector includes: Obtain the Euclidean distance between the smoothed keypoint vector and the estimated keypoint vector; If the Euclidean distance is greater than a preset threshold, then the smoothed key point vector is determined as the stabilized key point vector. If the Euclidean distance is less than or equal to the preset threshold, then the key point estimation vector is determined as the key point vector after the image stabilization process.

13. A smoothing processing apparatus, characterized in that, include: Build a module to obtain key points of the target object on the image to be processed; Based on the key points of the target object, construct the key point vector corresponding to the target object; Obtain the reconstruction matrix; The reconstruction coefficients are obtained based on the key point vectors and the reconstruction matrix; the image to be processed is an image from the video stream. The processing module is used to determine the smoothed keypoint vector based on the reconstruction coefficients and the reconstruction matrix; The processing module is further configured to: calculate a variance vector based on keypoint vectors corresponding to the target object in at least two frames of images; determine a second variable corresponding to the image to be processed based on the variance vector and a first variable corresponding to the previous frame of the image to be processed; determine a gain corresponding to the image to be processed based on the second variable corresponding to the image to be processed; determine a first variable corresponding to the image to be processed based on the gain and the second variable corresponding to the image to be processed; determine a keypoint estimation vector corresponding to the image to be processed based on the keypoint estimation vector corresponding to the previous frame of the image to be processed, the gain corresponding to the image to be processed, and the smoothed keypoint vector; and determine a stabilized keypoint vector corresponding to the image to be processed based on the smoothed keypoint vector and the keypoint estimation vector.

Images