Video processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a video processing method and device, electronic equipment and a storage medium. According to the method, a video is displayed, the video comprises at least two continuous frames of face images, and the at least two frames of face images comprise the face of the same object; based on the rotation center of the human face, performing image stabilization processing on the human face image so as to enable the human face in the at least two frames of human face images to be kept stable; wherein the rotation center is determined based on key points of the face in the face image, and the key points are pixel points representing facial features of the face. According to the method, in the head rotation process, the rotation center of the face is stabilized, very high anti-shake performance is achieved, the stability of the face in the video is guaranteed, unnatural rotation around the nose tip point caused by head rotation is eliminated, unnatural stretching or rotation does not exist, and the user experience is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a video processing method, device, electronic device and storage medium. Background Art

[0002] Face video stabilization is a special video stabilization technology that aims to maintain the stability of the face in the video. It aims to reduce the blur and shaking of the face in the image or video caused by camera shake through software or hardware. Face video stabilization technology is essential for shooting stable and clear face images.

[0003] In the related art, a specified key point in the face, such as the tip of the nose, is selected, and then the key point is kept at a fixed position in the video to achieve face stabilization in the video. However, keeping the specified key point still is contrary to the actual movement law of head rotation, resulting in unnatural head rotation in the video and poor user experience. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a video processing method, device, electronic device and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a video processing method is provided, the method comprising:

[0006] Displaying a video, wherein the video includes at least two consecutive frames of facial images, and the at least two frames of facial images include a face of the same subject;

[0007] Based on the rotation center of the human face, performing image stabilization processing on the human face image so that the human face in the at least two frames of human face images remains stable;

[0008] The rotation center is determined based on key points of the face in the face image, and the key points are pixel points that characterize facial features of the face.

[0009] The at least two frames of facial images at least include a first facial image and a second facial image, wherein the second facial image is a previous frame of facial image of the first facial image;

[0010] In some embodiments, the performing image stabilization processing on the facial image based on the rotation center of the face so as to keep the face in the at least two frames of facial images stable includes:

[0011] Acquire a first rotation center coordinate of the rotation center in the first face image and a historical rotation center coordinate of the rotation center in the second face image;

[0012] Based on the first rotation center coordinates and the historical rotation center coordinates, the first facial image is stabilized to obtain a processed first facial image; wherein the second rotation center coordinates in the processed first facial image are the same as the historical rotation center coordinates.

[0013] In some embodiments, obtaining the first rotation center coordinates of the rotation center in the first facial image includes:

[0014] Determine the three-dimensional coordinates of each of the key points based on the position information and depth information of each of the key points in the first facial image;

[0015] Based on the three-dimensional coordinates of the multiple key points, the first rotation center coordinates are determined.

[0016] In some embodiments, before determining the three-dimensional coordinates of each key point based on the position information and depth information of each key point in the plurality of key points of the first facial image, the method further includes:

[0017] Detecting the key points in the first facial image that match the preset key points;

[0018] The key point matches the preset key point, which means that the feature of the key point matches the feature of the preset key point.

[0019] In some embodiments, before determining the three-dimensional coordinates of each key point based on the position information and depth information of each key point in the plurality of key points of the first facial image, the method further includes:

[0020] For each key point, obtaining a first coordinate of the key point in a first coordinate system, where the first coordinate system is a two-dimensional coordinate system in the first facial image;

[0021] Based on the first coordinates of the key point, the position information of the key point is determined.

[0022] In some embodiments, determining the position information of the key point based on the first coordinate of the key point includes:

[0023] Performing lens distortion correction on the first coordinate to obtain a corrected second coordinate;

[0024] Based on the camera intrinsic parameter matrix, the second coordinate is normalized to obtain a normalized third coordinate, and the third coordinate is determined as the position information.

[0025] In some embodiments, the position information is a two-dimensional coordinate, and determining the three-dimensional coordinate of each key point based on the position information and depth information of each key point in the plurality of key points of the first facial image includes:

[0026] For each key point, convert the two-dimensional coordinates of the key point into homogeneous coordinates;

[0027] Each value in the homogeneous coordinate is multiplied by the depth information of the key point to obtain the three-dimensional coordinate.

[0028] In some embodiments, before determining the three-dimensional coordinates of each key point based on the position information and depth information of each key point in the plurality of key points of the first facial image, the method further includes:

[0029] Obtaining a first three-dimensional coordinate representation of a first key point and a second three-dimensional coordinate representation of a second key point, wherein the first three-dimensional coordinate representation includes a first depth identifier, which is used to indicate depth information of the first key point, and the second three-dimensional coordinate representation includes a second depth identifier, which is used to indicate depth information of the second key point, wherein the first key point and the second key point are any two key points among the multiple key points;

[0030] Determine, based on the first three-dimensional coordinate representation and the second three-dimensional coordinate representation, a vector representation pointing from the first key point to the second key point;

[0031] Determine the depth information of the first key point and the depth information of the second key point based on the vector representation and a reference vector, wherein the reference vector is a vector pointing from the first key point to the second key point.

[0032] In some embodiments, before determining the depth information of the first key point and the depth information of the second key point based on the vector representation and the reference vector, the method further includes:

[0033] Determining the reference vector based on the posture information of the head of the object, a reference coefficient and a preset unit vector;

[0034] The reference coefficient is used to characterize the relationship between the pixel distance and the spatial distance between the first key point and the second key point.

[0035] In some embodiments, before determining the reference vector based on the posture information of the head of the object, the reference coefficient and the preset unit vector, the method further includes:

[0036] Obtaining a spatial distance between the first key point and the second key point;

[0037] Determine a pixel distance between the first key point and the second key point based on the position information of the first key point and the position information of the second key point;

[0038] The ratio between the spatial distance and the pixel distance is determined as the reference coefficient.

[0039] In some embodiments, determining the first rotation center coordinates based on the three-dimensional coordinates of the plurality of key points includes:

[0040] Based on the three-dimensional coordinates of the multiple key points, fitting the three-dimensional coordinates of the rotation center;

[0041] The three-dimensional coordinates of the rotation center are converted into the first rotation center coordinates, where the first rotation center coordinates are two-dimensional coordinates.

[0042] In some embodiments, performing image stabilization processing on the first face image based on the first rotation center coordinates and the historical rotation center coordinates to obtain the processed first face image includes:

[0043] Determine the historical rotation center coordinates as the second rotation center coordinates;

[0044] Based on the variation between the first rotation center coordinates and the second rotation center coordinates, perspective transformation and cropping are performed on the first facial image to obtain the processed first facial image.

[0045] According to a second aspect of an embodiment of the present disclosure, a video processing device is provided, the device comprising:

[0046] A video display module is configured to display a video, wherein the video includes at least two consecutive frames of face images, and the at least two frames of face images include a face of the same subject;

[0047] An image stabilization processing module is configured to perform image stabilization processing on the face image based on the rotation center of the face, so as to keep the face in the at least two frames of face images stable;

[0048] The rotation center is determined based on key points of the face in the face image, and the key points are pixel points that characterize facial features of the face.

[0049] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0050] processor;

[0051] a memory for storing processor-executable instructions;

[0052] The processor is configured to execute the method as described in the first aspect of the embodiment of the present disclosure.

[0053] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in the first aspect of the embodiment of the present disclosure.

[0054] The above method of the present disclosure has the following beneficial effects:

[0055] The method provided by the embodiment of the present disclosure displays a video, wherein the video includes at least two consecutive frames of face images, and at least two frames of face images include the face of the same object; based on the rotation center of the face, the face image is stabilized to make the face in at least two frames of face images stable; wherein the rotation center is determined based on the key points of the face in the face image, and the key points are pixels that characterize the facial features of the face. Compared with the related art in which the nose tip point remains stationary, the method provided by the present disclosure stabilizes the rotation center of the face during the head rotation process, has high anti-shake performance, ensures the stability of the face in the video, and eliminates the unnatural rotation around the nose tip point caused by the head rotation, and there is no unnatural stretching or rotation, which improves the user experience.

[0056] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0058] Figure 1 is a flow chart of a video processing method according to an exemplary embodiment;

[0059] Figure 2 is a flow chart of a video processing method according to an exemplary embodiment;

[0060] Figure 3 is a flow chart of a video processing method according to an exemplary embodiment;

[0061] Figure 4 is a flow chart showing a process for determining location information according to an exemplary embodiment;

[0062] Figure 5 is a flowchart showing a depth information determination process according to an exemplary embodiment;

[0063] Figure 6is a block diagram of a video processing device according to an exemplary embodiment;

[0064] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0065] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0066] In the related art, key points are extracted from the face image, and then designated key points in the face, such as the nose tip, are selected so that the designated key points remain in a fixed position in the video to achieve face stability in the video. However, when the head rotates normally, the center of rotation of the head, i.e., the center point of the C1 vertebra of the cervical spine, remains stationary, while the designated key point is the nose tip or other key points in the face. The designated key point remains stationary, which is contrary to the actual movement law of the head rotation, resulting in unnatural head rotation in the video and poor user experience.

[0067] The disclosed embodiment proposes a new video processing method. Instead of directly taking the key points in the face image as the designated key points, the rotation center of the face is determined based on multiple key points. During the head rotation process, the rotation center is stabilized. Compared with the designated key points, the rotation center is more in line with the actual movement law of the head rotation. Therefore, the unnatural rotation around the tip of the nose caused by the head rotation is eliminated, and there is no unnatural stretching or rotation, thereby improving the user experience.

[0068] The method provided by the embodiments of the present disclosure can be applied to video preview, video recording, live broadcast, video chat and other scenarios involving videos to correct facial distortion that occurs in these scenarios; the method can also be applied to face recognition, face tracking, facial expression analysis and other scenarios that require accurate face recognition or analysis to avoid jitter or shaking in the video affecting the accuracy of recognition or analysis.

[0069] The method provided in the embodiment of the present disclosure is executed by an electronic device, which may be a mobile phone, a tablet computer, a laptop computer, a wearable device, an AR (Augmented Reality) / VR (Virtual Reality) head display, or other device.

[0070] Figure 1is a flowchart of a video processing method according to an exemplary embodiment, which is executed by an electronic device, see Figure 1 , the method comprises the following steps:

[0071] Step S101, displaying a video, wherein the video includes at least two consecutive frames of face images, and the at least two frames of face images include a face of the same subject.

[0072] The video is any video, for example, a video to be played, a live video stream, a video shot by a camera, etc. The object is any portrait in at least two frames of face images, and each frame of face image may include at least one portrait. Displaying videos includes: playing videos stored locally in the electronic device, playing videos obtained from the network by the electronic device, displaying videos shot by a camera, playing live videos, and displaying chat videos with other users.

[0073] Step S102, based on the rotation center of the face, stabilizing the face image to keep the face in at least two frames of face images stable; wherein the rotation center is determined based on the key points of the face in the face image, and the key points are pixel points that characterize the facial features of the face.

[0074] The rotation center refers to a point that remains unchanged when the head rotates normally. The rotation center of a face may also be referred to as the rotation center of the head. The face is the face in the head. For at least two frames of face images, when the at least two frames of face images include the face of the same subject, the faces in the at least two frames of face images need to be kept stable.

[0075] The number of key points of a facial image can be one or more, for example, the key points of a facial image can be 3, 4 or more, and the key points can be the left eye corner key points, right eye corner key points, nose tip key points, mouth corner key points, left eye pupil key points, right eye pupil key points and other pixel points in the human face.

[0076] Image stabilization processing refers to: adjusting each pixel in the face image so that the position of the rotation center in the processed face image remains unchanged from the rotation center in the previous frame of the face image, and other pixels in the face image are adjusted as the rotation center is adjusted.

[0077] The method provided by the embodiment of the present disclosure displays a video, wherein the video includes at least two consecutive frames of face images, and at least two frames of face images include the face of the same object; based on the rotation center of the face, the face image is stabilized to make the face in at least two frames of face images stable; wherein the rotation center is determined based on the key points of the face in the face image, and the key points are pixels that characterize the facial features of the face. Compared with the related art in which the nose tip point remains stationary, the method provided by the present disclosure stabilizes the rotation center of the face during the head rotation process, has high anti-shake performance, ensures the stability of the face in the video, and eliminates the unnatural rotation around the nose tip point caused by the head rotation, and there is no unnatural stretching or rotation, which improves the user experience.

[0078] In the embodiments of the present disclosure, when performing image stabilization processing on a facial image, it is necessary to consider the current facial image and the facial image previous frame of the facial image. Taking at least two facial image frames including at least a first facial image and a second facial image, where the second facial image is the facial image previous frame of the first facial image as an example, the image stabilization processing of the first facial image is explained through the following embodiments.

[0079] Figure 2 is a flowchart of a video processing method according to an exemplary embodiment, which is executed by an electronic device, see Figure 2 , the method comprises the following steps:

[0080] Step S202, obtaining the first rotation center coordinates of the rotation center in the first face image and the historical rotation center coordinates of the rotation center in the second face image.

[0081] The first rotation center coordinates are the coordinates of the rotation center in the first face image, and the historical rotation center coordinates are the coordinates of the rotation center in the second face image.

[0082] In some embodiments, based on the position information and depth information of each key point in the first facial image, the three-dimensional coordinates of each key point are determined; based on the three-dimensional coordinates of the multiple key points, the first rotation center is determined.

[0083] For each key point, the position information of the key point is used to characterize the position of the key point in the first face image, and the position information may be a two-dimensional coordinate or other information indicating the position, and the depth information of the key point is used to characterize the distance between the position point corresponding to the key point in the object photographed in the shooting scene of the first face image and the camera. Therefore, based on the position information and depth information of the key point, the three-dimensional coordinates of the key point can be determined, and the three-dimensional coordinates are obtained by fusing the depth information on the basis of the position information.

[0084] In the embodiment of the present disclosure, considering that the head rotation occurs in three-dimensional space and the rotation center is a certain point in the three-dimensional space, if two-dimensional coordinates are directly used without considering the depth information, the accurate coordinates of the rotation center cannot be obtained. Therefore, when determining the coordinates of the rotation center, the first rotation center coordinates are determined based on the three-dimensional coordinates of multiple key points.

[0085] It should be noted that the method for determining the historical rotation center coordinates is the same as the method for determining the first rotation center coordinates, which will not be described in detail here.

[0086] Step S202, based on the first rotation center coordinates and the historical rotation center coordinates, perform image stabilization processing on the first face image to obtain a processed first face image; wherein the second rotation center coordinates in the processed first face image are the same as the historical rotation center coordinates.

[0087] In order to achieve video stabilization, the rotation center of the head needs to be kept stable, that is, the coordinates of the rotation center need to be kept stable. Therefore, based on the first rotation center coordinates and the historical rotation center coordinates, the first face image is stabilized so that the second rotation center coordinates in the processed first face image are the same as the historical rotation center coordinates. The second rotation center coordinates are the coordinates of the rotation center in the processed first face image.

[0088] Compared with the related art in which the nose tip remains still, the method provided by the embodiment of the present invention determines the rotation center of the head based on multiple key points, stabilizes the rotation center during the rotation of the head, has high anti-shake performance, ensures the stability of the face in the video, and eliminates the unnatural rotation around the nose tip caused by head rotation. There is no unnatural stretching or rotation, thereby improving the user experience.

[0089] Figure 3 is a flowchart of a video processing method according to an exemplary embodiment, which is executed by an electronic device, see Figure 3 , the method comprises the following steps:

[0090] Step S301, detecting key points matching preset key points in a first face image.

[0091] The preset key point is a pre-set key point, and the number of the preset key points is multiple. The key point matches the preset key point means that the feature of the key point matches the feature of the preset key point. Optionally, the feature of the key point matches the feature of the preset key point means that the feature of the key point is the same as the feature of the preset key point, or the similarity between the feature of the key point and the feature of the preset key point is greater than a preset similarity, for example, the preset similarity is 90%, 95%, etc.

[0092] In some embodiments, features of each preset key point are pre-set, and based on the features of the preset key point, pixel points in the first face image that match the features of the preset key point are detected, and the detected pixel points are determined as detected key points.

[0093] Step S302: Determine the position information of each key point among the multiple key points of the first face image.

[0094] The process of determining the position information of key points can be found in Figure 4 The embodiments shown will not be described in detail here.

[0095] Step S303: Determine depth information of each key point among the multiple key points of the first face image.

[0096] The process of determining the depth information of key points can be found in Figure 5 The embodiments shown will not be described in detail here.

[0097] Step S304: determining the three-dimensional coordinates of each key point in the first facial image based on the position information and depth information of each key point.

[0098] Among them, the position information is the coordinates in the first coordinate system, the first coordinate system is a two-dimensional coordinate system in the first face image, the x-axis and y-axis in the first coordinate system respectively represent the horizontal direction and the vertical direction of the first face image, and the origin of the first coordinate system can be the point in the upper left corner, the point in the lower left corner or other position points of the first face image; the depth information is the depth value in the second coordinate system, the second coordinate system is a three-dimensional coordinate system based on the camera, the x-axis, y-axis and z-axis in the three-dimensional coordinate system respectively represent three mutually orthogonal directions in the real space, the three-dimensional coordinates of the key point are the coordinates in the second coordinate system, and the origin of the second coordinate system can be the position point where the camera was located when the first face image was taken or other position points.

[0099] In some embodiments, the position information is a two-dimensional coordinate. For each key point, the two-dimensional coordinate of the key point is converted into a homogeneous coordinate; each value in the homogeneous coordinate is multiplied by the depth information of the key point to obtain a three-dimensional coordinate. For example, the position information p of the key point L l =(x l ,y l ), the homogeneous coordinate is p l =(x l ,y l , 1), the depth information of the key point L is Z l , then the three-dimensional coordinates of the key point L are P l =(x l Z l ,y l Z l , Zl ).

[0100] Step S305 , based on the three-dimensional coordinates of the multiple key points, the three-dimensional coordinates of the rotation center are fitted and the three-dimensional coordinates of the rotation center are converted into first rotation center coordinates.

[0101] The first rotation center coordinates are two-dimensional coordinates.

[0102] In some embodiments, the three-dimensional coordinates of the rotation center are fitted based on the three-dimensional coordinates and posture information of multiple key points.

[0103] In some embodiments, the three-dimensional coordinates of the rotation center are projected onto a normalized plane, that is, the homogeneous coordinates of the rotation center are obtained, and then the homogeneous coordinates are converted into the first rotation center coordinates.

[0104] Step S306: determine the historical rotation center coordinates as the second rotation center coordinates.

[0105] In order to achieve video stabilization, the rotation center of the head needs to be kept stable, that is, the coordinates of the rotation center need to be kept stable, and the historical rotation center coordinates are determined as the second rotation center coordinates to adjust the coordinates of the rotation center in the first face image from the first rotation center coordinates to the second rotation center coordinates.

[0106] Step S307 , based on the variation between the first rotation center coordinates and the second rotation center coordinates, perspective transformation and cropping are performed on the first facial image to obtain a processed first facial image.

[0107] In the disclosed embodiment, in order to make the coordinates of the rotation center in the processed first face image the second rotation center coordinates, the first face image needs to be processed. During the processing, the change amount between the first rotation center coordinates and the second rotation center coordinates is first determined. The change amount refers to the change amount by which the rotation center in the unprocessed first face image needs to be adjusted. If the rotation center needs to be adjusted, then other areas in the first face image also need to be adjusted accordingly. Therefore, the first face image is perspective transformed and cropped based on the change amount. Optionally, perspective transformation and cropping of the first face image includes: translating and cropping each pixel in the first face image.

[0108] Compared with the related art in which the nose tip remains still, the method provided by the embodiment of the present invention determines the rotation center of the head based on multiple key points, stabilizes the rotation center during the rotation of the head, has high anti-shake performance, ensures the stability of the face in the video, and eliminates the unnatural rotation around the nose tip caused by head rotation. There is no unnatural stretching or rotation, thereby improving the user experience.

[0109] Moreover, the method provided by the embodiments of the present disclosure can be adaptively applied to a variety of scenarios, for example, in video preview scenarios, video shooting scenarios, live broadcast scenarios, etc., and the effect of no jump in the rotation center can be achieved in a variety of scenarios.

[0110] In the embodiment of the present disclosure, the process of determining the position information of each key point in the first face image is the same. Figure 4 In the illustrated embodiment, the process of determining the position information of a key point is described by taking the process of determining the position information of any key point among a plurality of key points as an example.

[0111] Figure 4 is a flowchart showing a process for determining location information according to an exemplary embodiment, which is executed by an electronic device, see Figure 4 , the process includes the following steps:

[0112] Step S401, obtaining the first coordinates of the key point in the first coordinate system.

[0113] The first coordinate system is a two-dimensional coordinate system in the first face image, and the first coordinates are the coordinates of the key point in the first face image in the first coordinate system.

[0114] Step S402, performing lens distortion correction on the first coordinate to obtain a corrected second coordinate.

[0115] In the disclosed embodiments, image content distortion may occur during the shooting process. For example, the first face image is displaced due to RS (Rolling Shutter) and OIS (Optical Image Stability). RS refers to the fact that the CMOS (Complementary Metal Oxide Semiconductor) sensors in electronic devices all adopt a row-by-row scanning method. It is also this scanning method that causes all pixels in the image to not be exposed at the same time, thereby causing image content distortion. Therefore, when performing EIS (Electric Image Stabilization), in order to avoid the influence of image content distortion, it is necessary to first perform lens distortion correction on the first coordinate. For example, in order to ensure the accuracy of EIS, the distortion introduced by the lens in the image is corrected, and LDC (lens distortion correction) is performed on the first coordinate.

[0116] Step S403: normalize the second coordinate based on the camera intrinsic parameter matrix to obtain a normalized third coordinate, and determine the third coordinate as position information.

[0117] For example, the camera intrinsic parameter matrix is:

[0118]

[0119] Among them, M represents the camera internal parameter matrix, f x and f y C is the parameter representing the focal length. x and C y The size of the camera center point offset.

[0120] The second coordinate is (u, v), then the following formula is used to normalize the second coordinate:

[0121]

[0122] Among them, (x, y) is the third coordinate.

[0123] It should be noted that steps S402 and S403 are a method of determining the position information of the key point based on the first coordinate of the key point. In another embodiment, other methods may be used to determine the position information. For example, based on the camera intrinsic parameter matrix, the first coordinate is normalized to obtain the fourth coordinate, and the fourth coordinate is determined as the position information.

[0124] In the embodiment of the present disclosure, since the coordinates of the key points in the first face image may be offset due to RS and OIS, the coordinates of the key points are first subjected to lens correction to obtain more accurate position information of the key points. In addition, the method for determining the three-dimensional coordinates of the key points provided in the present disclosure is a general method. In order to be applicable to this method, the second coordinates are normalized so that the third coordinates can be used for subsequent processing.

[0125] In the embodiment of the present disclosure, when determining the depth information of each key point in the first face image, two key points need to be used, and the depth information of the two key points is determined at the same time. Figure 5 In the embodiment shown, the process of determining the depth information of a key point is described by taking the process of determining the depth information of a first key point and the depth information of a second key point among multiple key points as an example, wherein the first key point and the second key point are any two key points among the multiple key points.

[0126] Figure 5 is a flowchart showing a depth information determination process according to an exemplary embodiment, which is executed by an electronic device, see Figure 5 , the process includes the following steps:

[0127] Step S501, obtain a first three-dimensional coordinate representation of a first key point and a second three-dimensional coordinate representation of a second key point, the first three-dimensional coordinate representation includes a first depth identifier, the first depth identifier is used to refer to the depth information of the first key point, and the second three-dimensional coordinate representation includes a second depth identifier, the second depth identifier is used to refer to the depth information of the second key point.

[0128] In the disclosed embodiment, it is necessary to first determine the depth information of the key points. Before the depth information is calculated, the depth information of the key points is an unknown quantity. To facilitate subsequent calculations, it is first assumed that the depth information of the first key point is the first depth identifier, and the depth information of the second key point is the second depth identifier, thereby determining the three-dimensional coordinate representation of the key points based on the known position information and the assumed depth information.

[0129] For example, the position information of the first key point is p l =(x l ,y l , 1), the position information of the second key point is p r =(x r ,y r , 1), assuming Z l is the first depth marker, Z r is the second depth marker, Z l and Z r is the unknown quantity, and the first three-dimensional coordinate is expressed as P l =(x l Z l ,y l Z l , Z l ), the second three-dimensional coordinate is represented by P r =(x r Z r ,y r Z r , Z r ).

[0130] Step S502: Determine a vector representation of the first key point pointing to the second key point based on the first three-dimensional coordinate representation and the second three-dimensional coordinate representation.

[0131] In order to solve the first depth identifier and the second depth identifier, in the embodiment of the present disclosure, two methods are used to represent the vector pointing from the first key point to the second key point, wherein the vector represented by one method includes the unknown first depth identifier and the second depth identifier, while the vector represented by the other method does not include the unknown first depth identifier and the second depth identifier, so that the vectors represented by the two methods are equal, thereby solving the depth information referred to by the first depth identifier and the depth information referred to by the second depth identifier.

[0132] The vector representation determined based on the first three-dimensional coordinate representation and the second three-dimensional coordinate representation is represented as the difference between the first three-dimensional coordinate representation and the second three-dimensional coordinate representation. For example, the first three-dimensional coordinate representation is represented as P l =(x l Z l ,y l Z l , v l ), the second three-dimensional coordinate is represented by P r =(x r Z r ,y r Z r , Z r ), then the vector is represented as:

[0133] V lr =(P l -P r ) T =(x l Z l -x r Z r ,y l Z l -y r Z r , Z l -Z r ) T

[0134] Among them, V lr is represented by a vector.

[0135] Step S503: determining a reference vector based on the posture information of the head of the object, a reference coefficient and a preset unit vector.

[0136] In another way, the reference vector is also a vector from the first key point to the second key point. When determining the reference vector, the posture information of the head is used to characterize the current posture of the head, and the reference coefficient is used to characterize the relationship between the pixel distance and the spatial distance between the first key point and the second key point. Among them, the posture information is equivalent to describing the direction when the first key point points to the second key point, the reference coefficient is equivalent to describing the distance between the first key point and the second key point, the preset unit vector is a unit vector, and the reference vector is equivalent to superimposing the reference coefficient and posture information on the basis of the unit vector.

[0137] Regarding posture information, in some embodiments, the posture information of the head includes the roll angle, rotation angle, and pitch angle of the head, and the posture of the head is described by the roll angle, rotation angle, and pitch angle.

[0138] For the reference coefficient, in some embodiments, the spatial distance corresponding to the first key point and the second key point is obtained; based on the position information of the first key point and the position information of the second key point, the pixel distance between the first key point and the second key point is determined; and the ratio between the spatial distance and the pixel distance is determined as the reference coefficient. The spatial distance is a preset distance, and the pixel distance is the distance presented in the first face image.

[0139] Optionally, since human faces are general, that is, the spatial distance between two key points in most faces will be within a certain numerical range, the preset distance can be determined based on the spatial distance between the first key point and the second key point in multiple faces in actual scenarios.

[0140] In one example, the first key point is the left eye corner key point, and the position information of the first key point is p l =(x l ,y l ), the second key point is the right eye corner key point, and the position information of the second key point is p r =(x r ,y r ), the spatial distance between the left eye corner key point and the right eye corner key point is d s , pixel distance Then the reference coefficient K is d s The ratio between d.

[0141] In some embodiments, the reference vector is determined using the following formula:

[0142] V lr =R*K(a, b, c) T =K(a R , b R , c R ) T

[0143] Among them, V lr is the reference vector, R is the posture information, K is the reference coefficient, (a, b, c) is the preset unit vector, and the modulus of (a, b, c) is 1.

[0144] Optionally, the posture information can be expressed as:

[0145]

[0146] Among them, roll represents the roll angle of the head, pitch represents the rotation angle of the head, and yaw represents the pitch angle of the head.

[0147] Step S504: Determine depth information of the first key point and depth information of the second key point based on the vector representation and the reference vector.

[0148] Since both the vector representation and the reference vector refer to the vector from the first key point to the second key point, the two vectors are the same. For example, they can be expressed as the following formula:

[0149] (x l Z l -x r Z r ,y l Z l -y r Z r , Z l -Z r ) T =K(a R , b R , c R ) T

[0150] Right now:

[0151] x l Z l -x r Z r -a R K=0

[0152] y l Z l -y r Z r -b R K=0

[0153] Z l -Z r -c R K=0

[0154] That is:

[0155]

[0156] Among them, Z l and Z r is an unknown quantity, and the other parameters are known quantities. Solving the above formula, we can get Z l and Z r Represents the depth information.

[0157] The method for determining depth information provided by the embodiments of the present disclosure can be directly obtained by solving and calculating based on the coordinates of key points and facial postures, without relying on a TOF (Time of Fight) camera or a depth estimation algorithm, and is more universal.

[0158] Figure 6is a block diagram of a video processing device according to an exemplary embodiment, which is configured in an electronic device, see Figure 6 , the device comprises:

[0159] The video display module 601 is configured to display a video, wherein the video includes at least two consecutive frames of face images, and the at least two frames of face images include the face of the same subject;

[0160] The image stabilization processing module 602 is configured to perform image stabilization processing on the face image based on the rotation center of the face, so that the face in at least two frames of face images remains stable;

[0161] The rotation center is determined based on the key points of the face in the face image, and the key points are pixels that characterize the facial features of the face.

[0162] In some embodiments, the image stabilization processing module 602 is configured to:

[0163] Obtaining first rotation center coordinates of a rotation center in the first face image and historical rotation center coordinates of a rotation center in the second face image;

[0164] Based on the first rotation center coordinates and the historical rotation center coordinates, the first face image is stabilized to obtain a processed first face image; wherein the second rotation center coordinates in the processed first face image are the same as the historical rotation center coordinates.

[0165] In some embodiments, the image stabilization processing module 602 is configured to:

[0166] Determine the three-dimensional coordinates of each key point based on the position information and depth information of each key point in the plurality of key points of the first facial image;

[0167] Based on the three-dimensional coordinates of the plurality of key points, the first rotation center coordinates are determined.

[0168] In some embodiments, the apparatus further comprises:

[0169] A key point detection module is configured to detect key points in the first face image that match preset key points;

[0170] The key point matches the preset key point, which means that the feature of the key point matches the feature of the preset key point.

[0171] In some embodiments, the apparatus further comprises:

[0172] A position information determination module is configured to obtain, for each key point, a first coordinate of the key point in a first coordinate system, where the first coordinate system is a two-dimensional coordinate system in the first face image;

[0173] Based on the first coordinate of the key point, the position information of the key point is determined.

[0174] In some embodiments, the location information determination module is configured to:

[0175] Perform lens distortion correction on the first coordinate to obtain a corrected second coordinate;

[0176] Based on the camera intrinsic parameter matrix, the second coordinate is normalized to obtain a normalized third coordinate, and the third coordinate is determined as position information.

[0177] In some embodiments, the position information is a two-dimensional coordinate, and the first coordinate determination module 501 is configured to:

[0178] For each key point, convert the two-dimensional coordinates of the key point into homogeneous coordinates;

[0179] Multiply each value in the homogeneous coordinates by the depth information of the key point to obtain the three-dimensional coordinates.

[0180] In some embodiments, the apparatus further comprises: a depth information determination module, the depth information determination module being configured to:

[0181] Obtain a first three-dimensional coordinate representation of a first key point and a second three-dimensional coordinate representation of a second key point, wherein the first three-dimensional coordinate representation includes a first depth identifier, and the first depth identifier is used to indicate depth information of the first key point, and the second three-dimensional coordinate representation includes a second depth identifier, and the second depth identifier is used to indicate depth information of the second key point, and the first key point and the second key point are any two key points among the plurality of key points;

[0182] Determine, based on the first three-dimensional coordinate representation and the second three-dimensional coordinate representation, a vector representation pointing from the first key point to the second key point;

[0183] Depth information of the first key point and depth information of the second key point are determined based on the vector representation and the reference vector, where the reference vector is a vector pointing from the first key point to the second key point.

[0184] In some embodiments, the apparatus further comprises:

[0185] A reference vector determination module is configured to determine a reference vector based on the posture information of the head of the object, a reference coefficient and a preset unit vector;

[0186] The reference coefficient is used to characterize the relationship between the pixel distance and the spatial distance between the first key point and the second key point.

[0187] In some embodiments, the apparatus further comprises: a reference coefficient determination module, the reference coefficient determination module being configured to:

[0188] Obtain the spatial distance corresponding to the first key point and the second key point;

[0189] Determine a pixel distance between the first key point and the second key point based on the position information of the first key point and the position information of the second key point;

[0190] The ratio between the spatial distance and the pixel distance is determined as the reference coefficient.

[0191] In some embodiments, the image stabilization processing module 602 is configured to:

[0192] Based on the three-dimensional coordinates of multiple key points, the three-dimensional coordinates of the rotation center are obtained by fitting;

[0193] The three-dimensional coordinates of the rotation center are converted into first rotation center coordinates, where the first rotation center coordinates are two-dimensional coordinates.

[0194] In some embodiments, the image stabilization processing module 602 is configured to:

[0195] Determine the historical rotation center coordinates as the second rotation center coordinates;

[0196] Based on the variation between the first rotation center coordinates and the second rotation center coordinates, perspective transformation and cropping are performed on the first facial image to obtain a processed first facial image.

[0197] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0198] An embodiment of the present disclosure further provides an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the video processing method in the above embodiment.

[0199] Figure 7 is a block diagram of an electronic device 700 according to an exemplary embodiment.

[0200] Reference Figure 7 , the electronic device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0201] The processing component 702 generally controls the overall operation of the electronic device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0202] The memory 704 is configured to store various types of data to support operations on the electronic device 700. Examples of such data include instructions for any application or method operating on the electronic device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0203] The power supply component 706 provides power to various components of the electronic device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 700.

[0204] The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0205] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the electronic device 700 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0206] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0207] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the electronic device 700. For example, the sensor assembly 714 can detect the open / closed state of the electronic device 700, the relative positioning of components, such as the display and keypad of the electronic device 700, and the sensor assembly 714 can also detect the position change of the electronic device 700 or a component of the electronic device 700, the presence or absence of user contact with the electronic device 700, the orientation or acceleration / deceleration of the electronic device 700, and the temperature change of the electronic device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0208] The communication component 716 is configured to facilitate wired or wireless communication between the electronic device 700 and other devices. The electronic device 700 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0209] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0210] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by a processor 720 of an electronic device 700 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0211] The embodiments of the present disclosure further provide a non-temporary computer-readable storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the video processing method in the above embodiments.

[0212] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0213] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A video processing method, characterized in that: The method comprises: Displaying a video, wherein the video includes at least two consecutive frames of facial images, and the at least two frames of facial images include a face of the same subject; Based on the rotation center of the human face, performing image stabilization processing on the human face image so that the human face in the at least two frames of human face images remains stable; The rotation center is determined based on key points of the face in the face image, and the key points are pixel points that characterize facial features of the face.

2. The method according to claim 1, characterized in that: The at least two frames of facial images at least include a first facial image and a second facial image, wherein the second facial image is a previous frame of facial image of the first facial image; The performing image stabilization processing on the face image based on the rotation center of the face so as to keep the face in the at least two frames of face images stable includes: Acquire a first rotation center coordinate of the rotation center in the first face image and a historical rotation center coordinate of the rotation center in the second face image; Based on the first rotation center coordinates and the historical rotation center coordinates, the first facial image is stabilized to obtain a processed first facial image; wherein the second rotation center coordinates in the processed first facial image are the same as the historical rotation center coordinates.

3. The method according to claim 2, characterized in that The obtaining the first rotation center coordinates of the rotation center in the first facial image includes: Determine the three-dimensional coordinates of each of the key points based on the position information and depth information of each of the key points in the first facial image; Based on the three-dimensional coordinates of the multiple key points, the first rotation center coordinates are determined.

4. The method according to claim 3, characterized in that Before determining the three-dimensional coordinates of each key point based on the position information and depth information of each key point in the plurality of key points of the first facial image, the method further includes: Detecting the key points in the first facial image that match the preset key points; The key point matches the preset key point, which means that the feature of the key point matches the feature of the preset key point.

5. The method according to claim 3, characterized in that: Before determining the three-dimensional coordinates of each key point based on the position information and depth information of each key point in the plurality of key points of the first facial image, the method further includes: For each key point, obtaining a first coordinate of the key point in a first coordinate system, where the first coordinate system is a two-dimensional coordinate system in the first facial image; Based on the first coordinates of the key point, the position information of the key point is determined.

6. The method according to claim 5, characterized in that The determining the position information of the key point based on the first coordinate of the key point includes: Performing lens distortion correction on the first coordinate to obtain a corrected second coordinate; Based on the camera intrinsic parameter matrix, the second coordinate is normalized to obtain a normalized third coordinate, and the third coordinate is determined as the position information.

7. The method according to claim 3, characterized in that The position information is a two-dimensional coordinate, and the determining of the three-dimensional coordinate of each key point based on the position information and depth information of each key point in the plurality of key points of the first facial image includes: For each key point, convert the two-dimensional coordinates of the key point into homogeneous coordinates; Each value in the homogeneous coordinate is multiplied by the depth information of the key point to obtain the three-dimensional coordinate.

8. The method according to claim 3, characterized in that Before determining the three-dimensional coordinates of each key point based on the position information and depth information of each key point in the plurality of key points of the first facial image, the method further includes: Obtaining a first three-dimensional coordinate representation of a first key point and a second three-dimensional coordinate representation of a second key point, wherein the first three-dimensional coordinate representation includes a first depth identifier, and the first depth identifier is used to indicate depth information of the first key point; the second three-dimensional coordinate representation includes a second depth identifier, and the second depth identifier is used to indicate depth information of the second key point; the first key point and the second key point are any two key points among the multiple key points; Determine, based on the first three-dimensional coordinate representation and the second three-dimensional coordinate representation, a vector representation pointing from the first key point to the second key point; Determine the depth information of the first key point and the depth information of the second key point based on the vector representation and a reference vector, wherein the reference vector is a vector pointing from the first key point to the second key point.

9. The method according to claim 8, characterized in that Before determining the depth information of the first key point and the depth information of the second key point based on the vector representation and the reference vector, the method further includes: Determining the reference vector based on the posture information of the head of the object, a reference coefficient and a preset unit vector; The reference coefficient is used to characterize the relationship between the pixel distance and the spatial distance between the first key point and the second key point.

10. The method according to claim 9, characterized in that Before determining the reference vector based on the posture information of the head of the object, the reference coefficient and the preset unit vector, the method further includes: Obtaining a spatial distance between the first key point and the second key point; Determine a pixel distance between the first key point and the second key point based on the position information of the first key point and the position information of the second key point; The ratio between the spatial distance and the pixel distance is determined as the reference coefficient.

11. The method according to claim 3, characterized in that The determining the first rotation center coordinates based on the three-dimensional coordinates of the plurality of key points includes: Based on the three-dimensional coordinates of the multiple key points, fitting the three-dimensional coordinates of the rotation center; The three-dimensional coordinates of the rotation center are converted into the first rotation center coordinates, where the first rotation center coordinates are two-dimensional coordinates.

12. The method according to claim 2, characterized in that: The step of performing image stabilization processing on the first face image based on the first rotation center coordinate and the historical rotation center coordinate to obtain the processed first face image includes: Determine the historical rotation center coordinates as the second rotation center coordinates; Based on the variation between the first rotation center coordinates and the second rotation center coordinates, perspective transformation and cropping are performed on the first facial image to obtain the processed first facial image.

13. A video processing device, characterized in that: The device comprises: A video display module is configured to display a video, wherein the video includes at least two consecutive frames of face images, and the at least two frames of face images include a face of the same subject; An image stabilization processing module is configured to perform image stabilization processing on the face image based on the rotation center of the face, so as to keep the face in the at least two frames of face images stable; The rotation center is determined based on key points of the face in the face image, and the key points are pixel points that characterize facial features of the face.

14. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method as claimed in any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 12.

Images