Perspective transformation method, device, electronic device and readable storage medium

Abstract

The present application provides a perspective transformation method, device, electronic device and readable storage medium. For a face image captured by a shooting device, a face area image in the face image is captured, and the face area image contains multiple face key points. A virtual perspective camera is constructed, and the posture parameters of the virtual perspective camera are obtained based on the reprojection error between the 3D coordinate information and the 2D coordinate information of each face key point in the face area image, and the rigid transformation information of the face during perspective transformation. Finally, the 3D coordinate information of each face key point is transformed into virtual 3D coordinate information in the virtual perspective camera space based on the posture parameters. In this solution, the reprojection error and rigid transformation information between 3D points and 2D are comprehensively considered in the perspective transformation, which can achieve good results in avoiding face transformation deformation and improving the accuracy of key point projection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a perspective transformation method, device, electronic device and readable storage medium. Background Art

[0002] In recent years, 3D facial key point detection has been widely used in face tracking, emotion recognition, and face VR special effects. In the field of 3D facial key point detection, it is usually necessary to crop the original image, which will cause the internal and external parameters of the camera to change. Therefore, in order to accurately restore the key point position information in the processed image, a virtual camera is generally constructed and the original 3D facial information in the world coordinate system is transformed into the virtual perspective camera space.

[0003] However, in the prior art, when performing perspective transformation, a fitting method such as the least square method is generally used for fitting transformation. This method is prone to problems such as inaccurate solution and distortion of the face. Summary of the invention

[0004] The purpose of the present application includes, for example, providing a perspective transformation method, device, electronic device and readable storage medium, which can avoid face transformation deformation and improve the accuracy of key point projection in virtual perspective transformation.

[0005] The embodiments of the present application can be implemented as follows:

[0006] In a first aspect, the present application provides a perspective transformation method, the method comprising:

[0007] Acquire a face image captured by a photographing device, and intercept and obtain a face region image in the face image, wherein the face region image includes a plurality of face key points;

[0008] A virtual perspective camera is constructed, and posture parameters of the virtual perspective camera are obtained according to the reprojection error between the 3D coordinate information and the 2D coordinate information of each facial key point in the facial area image, and the rigid transformation information of the face during perspective transformation; and the 3D coordinate information of each facial key point is transformed into virtual 3D coordinate information in the virtual perspective camera space based on the posture parameters.

[0009] In an optional implementation manner, the step of intercepting and obtaining a face region image in the face image includes:

[0010] Determine a plurality of facial key points constituting a facial contour in the facial image;

[0011] Obtaining a minimum bounding box of the plurality of facial key points;

[0012] The face image is intercepted based on the minimum bounding box to obtain a face area image.

[0013] In an optional implementation, the step of intercepting the face image based on the minimum bounding box to obtain a face area image includes:

[0014] Determining the center position of the minimum bounding box;

[0015] The interception length is calculated based on the position information of multiple corner points of the minimum bounding box;

[0016] A clipping range is determined according to the center position and the clipping length, and the face image is clipped based on the clipping range to obtain a face area image.

[0017] In an optional implementation, the method further includes a step of obtaining 2D coordinate information of each of the facial key points in the facial region image, the step comprising:

[0018] Obtaining 3D coordinate information of each facial key point in a world coordinate system;

[0019] projecting the 3D coordinate information of each of the facial key points into 2D coordinate information in a first image coordinate system according to the device parameters of the photographing device, where the first image coordinate system is constructed based on the facial image;

[0020] Based on the interception information when intercepting the face area image, the 2D coordinate information is updated to obtain the 2D coordinate information in the second image coordinate system constructed by the face area image.

[0021] In an optional embodiment, the interception information includes the interception length and the interception center position;

[0022] The step of updating the 2D coordinate information based on the interception information when intercepting the face area image comprises:

[0023] Subtract the horizontal coordinate of the interception center position from the horizontal coordinate in the 2D coordinate information, and add half of the interception length;

[0024] The vertical coordinate in the 2D coordinate information is subtracted from the vertical coordinate of the interception center position, and is added with half of the interception length.

[0025] In an optional implementation, the step of obtaining the pose parameters of the virtual perspective camera according to the reprojection error between the 3D coordinate information and the 2D coordinate information of each of the facial key points in the facial region image, and the rigid transformation information of the face during perspective transformation, comprises:

[0026] Constructing a first optimization item consisting of a reprojection error between the 3D coordinate information and the 2D coordinate information of each of the facial key points in the facial region image, and constructing a second optimization item consisting of rigid transformation information of the face during perspective transformation;

[0027] Assigning a weight value to the second optimization item based on a set weight coefficient;

[0028] The first optimization item and the second optimization item after the weighted value is added to construct an optimization model, and the optimization model is minimized to obtain the posture parameters of the virtual perspective camera.

[0029] In an optional implementation, the step of constructing a first optimization item consisting of a reprojection error between the 3D coordinate information and the 2D coordinate information of each of the facial key points in the facial region image comprises:

[0030] Constructing a projection item obtained from the 3D coordinate information of each facial key point in the facial region image and the posture parameter;

[0031] The first optimization item is obtained by subtracting the projection item from the 2D coordinate information of each facial key point in the facial region image.

[0032] In an optional implementation, the posture parameter includes a rotation matrix, and the step of constructing a second optimization item consisting of rigid transformation information of the face during perspective transformation includes:

[0033] Multiplying the rotation matrix by the transposed matrix of the rotation matrix to obtain rigid feature information;

[0034] The second optimization item is obtained by subtracting the rigid feature information from the constructed unit matrix.

[0035] In an optional embodiment, the method further comprises:

[0036] Constructing a face mesh based on the multiple face key points;

[0037] The mesh information of the face mesh is obtained according to the virtual 3D coordinate information of each of the face key points.

[0038] In a second aspect, the present application provides a perspective transformation device, the device comprising:

[0039] An acquisition module is used to acquire a face image captured by a shooting device and to intercept and obtain a face region image in the face image, wherein the face image includes a plurality of face key points;

[0040] A determination module is used to construct a virtual perspective camera, and obtain the posture parameters of the virtual perspective camera according to the reprojection error between the 3D coordinate information and the 2D coordinate information of each of the facial key points in the facial region image, and the rigid transformation information of the face during the perspective transformation;

[0041] A transformation module is used to transform the 3D coordinate information of each of the facial key points into virtual 3D coordinate information in a virtual perspective camera space based on the posture parameters.

[0042] In a third aspect, the present application provides an electronic device comprising one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine-executable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-executable instructions to perform the method steps described in any one of the aforementioned embodiments.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, the method steps described in any one of the aforementioned embodiments are implemented.

[0044] The beneficial effects of the embodiments of the present application include, for example:

[0045] The present application provides a perspective transformation method, device, electronic device and readable storage medium. For a face image captured by a shooting device, a face area image in the face image is captured, and the face area image contains multiple face key points. A virtual perspective camera is constructed, and the posture parameters of the virtual perspective camera are obtained based on the reprojection error between the 3D coordinate information and the 2D coordinate information of each face key point in the face area image, and the rigid transformation information of the face during perspective transformation. Finally, the 3D coordinate information of each face key point is transformed into virtual 3D coordinate information in the virtual perspective camera space based on the posture parameters. In this solution, the reprojection error and rigid transformation information between 3D points and 2D are comprehensively considered in the perspective transformation, which can achieve good results in avoiding face transformation deformation and improving the accuracy of key point projection. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A schematic diagram of an application scenario of the perspective transformation method provided in an embodiment of the present application;

[0048] Figure 2 A flowchart of a perspective transformation method provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of capturing a facial region image provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of a face grid provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of another perspective of a face grid provided in an embodiment of the present application;

[0052] Figure 6 A flowchart of a method for obtaining 2D coordinate information provided in an embodiment of the present application;

[0053] Figure 7 for Figure 2 Flow chart of sub-steps included in step S101;

[0054] Figure 8 A schematic diagram of the minimum external frame determined in an embodiment of the present application;

[0055] Fig. 9 for Figure 7 Flow chart of sub-steps included in step S1015;

[0056] Fig.10 for Figure 2 Flow chart of sub-steps included in step S103;

[0057] Fig.11 A flowchart of a method for obtaining grid information provided in an embodiment of the present application;

[0058] Fig.12 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0059] Fig.13 This is a functional module block diagram of the perspective transformation device provided in an embodiment of the present application.

[0060] Icons: 110 - storage medium; 120 - processor; 130 - perspective transformation device; 131 - acquisition module; 132 - determination module; 133 - transformation module; 140 - communication interface. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0062] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0063] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0064] In the description of the present application, it should be noted that the terms "first", "second", etc., if used, are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0065] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0066] See also Figure 1 , is a schematic diagram of an application scenario of the perspective transformation method provided in an embodiment of the present application, and the application scenario includes a server and a shooting device that is communicatively connected to the server. There may be one or more shooting devices, and the shooting device may include a device for capturing two-dimensional images, such as a camera. In addition, the shooting device may also include a device for capturing depth images, such as a depth camera. In this embodiment, the shooting device may send the captured image information or video information to the server, and the server may analyze and process the received image information or video information.

[0067] Taking the live broadcast application scenario as an example, the shooting device can be a shooting device set on the live broadcast provider or the live broadcast receiver. The face image of the anchor or the audience watching the anchor's live broadcast can be collected, and then the face image is sent to the server for analysis and processing.

[0068] It should be noted that the perspective transformation method provided in this embodiment can also be applied to other application scenarios that require analysis and processing of three-dimensional information of facial key points, and use of three-dimensional information for processing such as facial emotion recognition and facial special effects.

[0069] Combination Figure 2The present application embodiment provides a perspective transformation method applicable to an electronic device, and the electronic device may be the above-mentioned server. The method steps defined in the process related to the perspective transformation method may be implemented by the electronic device. Figure 2 The specific process shown is described in detail.

[0070] S101, obtaining a face image captured by a shooting device, and intercepting a face region image in the face image, wherein the face region image includes a plurality of face key points.

[0071] S103, constructing a virtual perspective camera, and obtaining the posture parameters of the virtual perspective camera according to the reprojection error between the 3D coordinate information and the 2D coordinate information of each facial key point in the facial area image, and the rigid transformation information of the face during the perspective transformation.

[0072] S105: transform the 3D coordinate information of each of the facial key points into virtual 3D coordinate information in a virtual perspective camera space based on the posture parameters.

[0073] Taking the live broadcast application scenario as an example, when the anchor is broadcasting live, the camera installed on the anchor's side can collect facial images containing the anchor's face. The facial images collected by the camera can be transmitted to the server along with the live video stream. Among them, the facial images collected by the camera can include multiple facial images of the anchor at different angles and with different expressions.

[0074] In addition to the main face area, the captured face image may also contain the image of the surrounding area at the time of shooting, that is, the background area image. In order to facilitate the analysis of the face information in the image, the face area image in the face image is often cut out, such as Figure 3 as shown in .

[0075] The facial image is intercepted to extract the facial region image contained therein, and the coordinate information of each pixel in the facial region image will be changed in the coordinate system of the original facial image. That is, the image coordinate system of the facial region image is changed relative to the image coordinate system of the facial image.

[0076] Moreover, due to the change of the image coordinate system, the viewpoint of each pixel in the face area image also changes, which means that each pixel in the face area image needs to be transformed to the camera space under the changed viewpoint. Therefore, in this embodiment, it is necessary to construct a virtual perspective camera, the viewpoint of which is established based on the changed viewpoint, that is, the virtual perspective camera is a perspective camera that can correspond to the three-dimensional camera space after the viewpoint is changed. The coordinate information of the pixel in the face area image needs to be perspective-transformed into the virtual perspective camera space. That is, the coordinate information of the pixel originally in the world coordinate needs to be transformed into the constructed virtual perspective camera space.

[0077] In this embodiment, the pixels that need to be processed by perspective transformation are the key points of the face in the face region image. The key points of the face can be key points that reflect the face contour information, the face features information, etc., such as the key points that constitute the face contour, the key points that constitute the mouth shape, the key points that constitute the eyebrow shape, etc.

[0078] If you want to accurately construct a virtual perspective camera, you need to determine the pose parameters of the virtual perspective camera. The pose parameters of the virtual perspective camera include a rotation matrix and a translation vector. When the pose parameters of the virtual perspective camera are determined, the virtual perspective camera can be fixed. When performing a perspective transformation of the coordinates of the key points of the face, the perspective transformation of the key points of the face can be performed based on the determined pose parameters. The pose parameters of the virtual perspective camera will affect the transformation effect of the key points of the face after the perspective transformation. Therefore, it is necessary to obtain pose parameters that can make the key points of the face have a better effect after the perspective transformation.

[0079] The coordinate information of each facial key point in the face area image includes 3D coordinate information and 2D coordinate information. Among them, the 3D coordinate information is the coordinate information in the world coordinate system, which can be understood as the coordinate information in the three-dimensional camera space before the viewpoint changes. And the 2D coordinate information is the coordinate information in the image coordinate system constructed by the face area image. After the perspective transformation, the 3D coordinate information and 2D coordinate information of each face pixel point should be accurately projected, that is, the reprojection error should be as small as possible. Among them, the reprojection error represents the error between the position point obtained by projecting the 2D coordinate and 3D coordinate of the pixel point under the current estimated posture parameter. Under a virtual perspective camera with a certain posture parameter, the smaller the reprojection error between the 3D coordinate information and the 2D coordinate information of the facial key point, the better the alignment effect between the 3D coordinate information and the 2D coordinate information. The better the effect of the subsequent perspective transformation based on the posture parameter.

[0080] In addition, in this embodiment, it is also considered that the face is easily distorted after perspective transformation. Therefore, in this embodiment, the posture parameters of the virtual perspective camera are optimized by combining the reprojection error between the 3D coordinate information and the 2D coordinate information of the key points of the face, and the rigid transformation information of the face during perspective transformation. This ensures that better results can be achieved in both reprojection error and rigid transformation of the face.

[0081] Among them, the rigid transformation information of the face reflects the information such as position, orientation, shape transformation, etc. between the key points of the face before and after the transformation. Among them, the rigid transformation information of the face can be controlled by the data characteristics of the posture parameters themselves. If the data characteristics of the posture parameters themselves show stronger rigid characteristics, it means that the rigid characteristics of the face after the transformation are stronger. The rigid characteristics of the posture parameters themselves, for example, if the rotation matrix in the posture parameters is closer to the orthogonal matrix, it means that the rigid characteristics of the posture parameters are stronger, and correspondingly, the rigid characteristics of the rigid transformation information of the face are stronger.

[0082] The pose parameters are optimized under the two indicators of reprojection error and rigid transformation information of the face. When the pose parameters of the virtual perspective camera are obtained, the 3D coordinate information of each key point of the face is transformed into virtual 3D coordinate information in the virtual perspective camera space based on the pose parameters. Based on the virtual 3D coordinate information of each key point of the face, the three-dimensional information of the face can be constructed, so as to realize application processing such as facial tracking and emotion recognition.

[0083] The perspective transformation method provided in this embodiment comprehensively considers the reprojection error between the 3D coordinate information and the 2D coordinate information of the facial key points, as well as the facial rigid transformation information reflected when the facial key points are transformed when performing perspective transformation, thereby avoiding deformation and distortion of the face after the transformation and improving the accuracy of the projection of the facial key points.

[0084] In this embodiment, for the face area image collected and intercepted by the shooting device, corresponding numbers can be set in advance for each face key point therein according to a unified rule. For example, 1220 face key points of the face part can be used to comprehensively reflect the face information. Then, among the multiple face key points, each face key point can correspond to one of the numbers 1-1220. The 1220 face key points can form a face grid, and the face grid can be composed of multiple triangular slices. Each triangular slice is composed of three adjacent face key points as vertices and combined with the lines connecting them. In this way, 2304 triangular slices can be formed. Figure 4 , 5 as shown in .

[0085] The 3D coordinate information of each facial key point in the world coordinate system can be recorded as (x, y, z), and each triangular slice can be recorded by the numbers corresponding to its three vertices, such as (i, j, k).

[0086] The 3D coordinate information of multiple facial key points contained in the face area image can be obtained using an N×3 matrix P w To represent, where N represents the number of facial key points.

[0087] As can be seen from the above, the 2D coordinate information of the face region image needs to be used in the perspective transformation. Therefore, the method provided in this embodiment also includes the step of obtaining the 2D coordinate information of each face key point in the face region image. Figure 6 , this step can be achieved by:

[0088] S1021, obtaining 3D coordinate information of each facial key point in the world coordinate system.

[0089] S1023: Project the 3D coordinate information of each of the facial key points into 2D coordinate information in a first image coordinate system according to the device parameters of the photographing device, where the first image coordinate system is constructed based on the facial image.

[0090] S1025, based on the interception information when intercepting the face area image, the 2D coordinate information is updated to obtain the 2D coordinate information in a second image coordinate system constructed by the face area image.

[0091] In this embodiment, when the device parameters of the shooting device are determined, the 3D coordinate information of each facial key point in the facial image captured by the shooting device in the world coordinate system can be determined.

[0092] For each key point of the face in the face image, there is a projection relationship between its 3D coordinate information and 2D coordinate information. Based on the device parameters of the shooting device, the 2D coordinate information of each key point of the face under the projection of the 3D coordinate information can be obtained. It should be noted that the 2D coordinate information obtained here is the 2D coordinate information of the first image constructed by the face image. The 2D coordinate information of the multiple key points of the face can be obtained by an N×2 matrix P uv To express.

[0093] When extracting the face region image from the face image, the image coordinate system changes from the first image coordinate system under the face image to the second image coordinate system under the face region image. Accordingly, the 2D coordinate information of each face key point will also change and needs to be changed to the coordinate information under the second image coordinate system.

[0094] In this embodiment, the 2D coordinate information of each key point of the face can be updated and revised based on the interception information when the face area image is intercepted from the face image, so as to be converted into 2D coordinate information in the second image coordinate system based on the face area image. Among them, the interception information is mainly the position information of the face area image relative to the face image, such as the interception length of the face area image, the center point position of the face area image in the first image coordinate system, etc.

[0095] In this embodiment, when the face area image is cut out from the face image and subsequent perspective transformation processing is performed based on the face area image, the 2D coordinate information under the face area image is revised, thereby ensuring the accuracy of the 2D coordinate information.

[0096] See also Figure 7 In this embodiment, when capturing a face region image from a face image, it can be achieved by:

[0097] S1011, determining a plurality of facial key points constituting a facial contour in the facial image.

[0098] S1013, obtaining the minimum bounding box of the multiple facial key points.

[0099] S1015: Capture the face image based on the minimum bounding box to obtain a face area image.

[0100] In this embodiment, the multiple facial key points that are determined to constitute the facial contour are multiple key points located outside the face in the image, for example Figure 8 As shown in , 68 facial key points numbered 1 to 68 can be determined. Based on the multiple facial key points, a minimum bounding box can be determined, and the minimum bounding box is the smallest bounding box that can enclose the multiple facial key points. The shape of the minimum bounding box can be a rectangle, a square, etc., mainly depending on the facial contour formed by the facial key points.

[0101] Based on the determined minimum bounding box, the face area framed by the minimum bounding box can be cut out from the face image to obtain a face area image.

[0102] See also Fig. 9 In this embodiment, when the face image is intercepted based on the minimum bounding box, the following method can be used:

[0103] S10151, determine the center position of the minimum circumscribed frame.

[0104] S10153: Calculate the clipping length based on the position information of multiple corner points of the minimum circumscribed frame.

[0105] S10155, determining a clipping range according to the center position and the clipping length, and clipping the face image based on the clipping range to obtain a face area image.

[0106] In this embodiment, the minimum bounding box may be a rectangular box, and the minimum bounding box may be formed by a coordinate vector (u tl ,v tl ,u br ,v br ) is used to represent the tl ,v tl ) is the coordinate value of the upper left corner of the minimum bounding box in the image coordinate system of the face image, (u br ,v br ) is the coordinate value of the lower right corner point of the minimum bounding box in the image coordinate system of the face image.

[0107] Based on the position information of the corner points of the minimum bounding box, such as the coordinate values ​​of the upper left corner point and the lower right corner point, the center position of the minimum bounding box can be obtained, which can be recorded as (u c ,v c ). The horizontal coordinate of the center position can be half of the sum of the horizontal coordinates of the upper left corner and the lower right corner: u c =0.5(u tl +u br ). The vertical coordinate of the center position can be half of the sum of the vertical coordinates of the upper left corner and the lower right corner: v c =0.5(v tl +v br ).

[0108] Furthermore, the interception length L can be calculated based on the position information of the corner points of the minimum bounding box, and the calculation formula can be as follows:

[0109]

[0110] After obtaining the center position of the minimum bounding box and the interception length, the interception range can be determined. The interception range is also a rectangular frame. The coordinate values ​​of the upper left corner and the lower right corner of the interception range are respectively and The intercept range can be recorded as a vector The face image can be intercepted based on the determined interception range to obtain a face area image.

[0111] In this embodiment, by determining the minimum bounding box constituting the face outline as described above and performing a certain deformation on the basis of the minimum bounding box to determine the interception range, it is possible to avoid intercepting too much background area while ensuring that the complete face area in the face image is intercepted, thereby interfering with the subsequent face information processing.

[0112] When the face area image is captured in the above manner, when revising and updating the 2D coordinate information of the face key points in the captured face area image, the revision can be made based on the capture center position and capture length of the minimum bounding box.

[0113] In a possible implementation, the horizontal coordinate of the 2D coordinate information of the facial key point may be subtracted from the horizontal coordinate of the interception center position and added with half of the interception length.

[0114] In addition, the vertical coordinate in the 2D coordinate information of the facial key point can be subtracted from the vertical coordinate of the interception center position and added with half of the interception length.

[0115] In this embodiment, in order to facilitate the processing of the face area image, the resolution of the cropped face area image can be unified to a set resolution, such as PxP, where P can be 120 or 160, etc.

[0116] Correspondingly, after the 2D coordinate information of the facial key points in the facial region image is revised and updated, both the horizontal coordinate and the vertical coordinate in the 2D coordinate information may be multiplied by P / L.

[0117] After the above processing, the 3D coordinate information and 2D coordinate information of each key point of the face can be obtained in the captured face area image. In order to transform the 3D coordinate information into virtual 3D coordinate information under the virtual perspective camera, in this embodiment, the reprojection error between the 3D coordinate information and the 2D coordinate information and the rigid transformation information of the face during perspective transformation are used as indicators for transformation.

[0118] See also Fig.10 In this embodiment, when determining the pose parameters of the virtual perspective camera, the above can be achieved by:

[0119] S1031, constructing a first optimization item consisting of the reprojection error of the 3D coordinate information and the 2D coordinate information of each of the facial key points in the facial area image, and constructing a second optimization item consisting of the rigid transformation information of the face during perspective transformation.

[0120] S1033: Assign a weight value to the second optimization item based on the set weight coefficient.

[0121] S1035, adding the first optimization item and the second optimization item after assigning a weighted value to construct the optimization model, and performing minimization processing on the optimization model to obtain the posture parameters of the virtual perspective camera.

[0122] In this embodiment, when optimizing the pose parameters of the virtual perspective camera, the reprojection error between the 3D coordinate information and the 2D coordinate information, as well as the rigid transformation information of the face during the perspective transformation, are considered. If the reprojection error between the 3D coordinate information and the 2D coordinate information is smaller, it indicates that the projection of the 2D key points after the transformation is more accurate. If the rigid features of the face during the perspective transformation are stronger, it indicates that the relative relationship between the key points of the face can be kept consistent with that before the transformation, that is, the face will not be deformed or distorted.

[0123] When the perspective transformation is performed by combining the above two, in order to control the inclination towards rigid transformation, a weight coefficient can be set for the second optimized weight value composed of rigid transformation information. Thus, the degree of attention paid to rigid features can be adjusted based on the adjustment of the weight coefficient while ensuring the reprojection error.

[0124] In this embodiment, when constructing the first optimization item based on the reprojection error, it can be achieved in the following manner:

[0125] Construct a projection item obtained by the 3D coordinate information of each facial key point in the facial area image and the posture parameter, and subtract the projection item from the 2D coordinate information of each facial key point in the facial area image to obtain the first optimization item.

[0126] The position parameters of the virtual perspective camera include the rotation matrix and translation vector of the virtual perspective camera. The rotation matrix may be a 3×3 matrix, and the translation vector may be a vector with a dimension of 3.

[0127] When constructing the projection items obtained by the 3D coordinate information of the facial key points and the posture parameters, it is equivalent to converting the 3D coordinate information of the facial key points into the virtual perspective camera space and then projecting it into 2D coordinate information. The error between the reprojections can be obtained by subtracting the 2D coordinate information of the projection from the 2D coordinate information of the facial key points in the face area image.

[0128] In addition, when constructing the second optimization item consisting of the rigid transformation information of the face during perspective transformation, it can be achieved in the following way:

[0129] The rotation matrix is ​​multiplied by the transposed matrix of the rotation matrix to obtain rigid feature information, and the rigid feature information after the multiplication is subtracted from the constructed unit matrix to obtain the second optimization item.

[0130] The rotation matrix is ​​a 3×3 matrix. If the rotation matrix is ​​closer to an orthogonal matrix, the transformation based on the rotation matrix will show more rigid characteristics. The orthogonal characteristics of the rotation matrix can be reflected by the product of the rotation matrix and the transpose of the rotation matrix. Therefore, the rigid feature information can be obtained based on the product of the rotation matrix and the transpose of the rotation matrix.

[0131] When the rotation matrix is ​​an orthogonal matrix, the rotation matrix is ​​multiplied by the transpose of the rotation matrix to obtain a unit matrix. Therefore, the result of the multiplication can be subtracted from the constructed unit matrix. If the subtraction result is closer to 0, it means that the rotation matrix is ​​closer to an orthogonal matrix, and the rigidity characteristics of the transformation based on the rotation matrix are better.

[0132] In this embodiment, in a possible implementation, the constructed optimization items may be as follows:

[0133]

[0134] in, is the first optimization item, is the second optimization item. uv Represents the 2D coordinate information of the facial key points in the face area image, P w Represents 3D coordinate information, R represents the rotation matrix, t represents the translation vector, and Z represents the coordinate information of each facial key point on the z axis in the virtual perspective camera space, which is a 1220-dimensional vector. K represents the intrinsic parameter matrix of the virtual perspective camera. || ||2 represents taking the L2 functional, 0:2 represents taking the first 2 columns of the matrix, and w represents the weight coefficient, for example, the value can be 0.2. I represents the unit matrix, with a dimension of 3×3.

[0135] When performing minimization processing based on the optimization model to determine the pose parameters of the virtual perspective transformation camera, a gradient descent method, a BFGS method or other optimization methods may be used for optimization processing, and this embodiment does not limit this.

[0136] On the basis of obtaining the posture parameters of the virtual perspective transformation camera, the 3D coordinate information of the key points of the face can be transformed into virtual 3D coordinate information in the virtual perspective camera space based on the posture parameters.

[0137] In this embodiment, the 3D coordinate information can be multiplied by the obtained rotation matrix, and then the translation vector can be added to achieve the transformation to the virtual 3D coordinate information. In this embodiment, the transformation can be achieved by the following calculation formula:

[0138] P cam =R·P w +t

[0139] Among them, Pcam Represents the transformed virtual 3D coordinate information, and similarly, can be a 1220×3-dimensional matrix, where each row in the matrix represents the 3D coordinate information of a facial key point.

[0140] When performing face tracking and face special effects processing based on 3D face information, it is usually necessary to implement it based on the face mesh related information. Therefore, when obtaining the virtual 3D coordinate information of the face key points in the virtual perspective transformation camera space, the following steps can also be performed. Please refer to Fig.11 :

[0141] S107: construct a face mesh based on the multiple face key points.

[0142] S109, obtaining mesh information of the face mesh according to the virtual 3D coordinate information of each of the face key points.

[0143] In this embodiment, the constructed face mesh is a face mesh composed of multiple face key points and triangular slices formed by each face key point as a vertex, such as Figure 4 , 5 as shown in .

[0144] Based on the virtual 3D coordinate information of each key point of the face, the grid information of the face grid can be obtained, so that face tracking, face special effects processing, etc. can be realized based on the grid information of the face grid.

[0145] The perspective transformation method provided in this embodiment comprehensively considers the reprojection error between 3D coordinate information and 2D coordinate information, as well as the rigid transformation information of the face during transformation, and through minimization optimization processing, can achieve the effect of reducing the reprojection error and improving the rigid transformation characteristics of the face after perspective transformation. It can avoid deformation of the face transformation and improve the accuracy of key point projection.

[0146] See also Fig.12 , is a schematic diagram of an exemplary component of an electronic device provided in an embodiment of the present application, and the electronic device may be Figure 1 The electronic device may include a storage medium 110, a processor 120, a perspective transformation device 130 and a communication interface 140. In the present embodiment, the storage medium 110 and the processor 120 are both located in the electronic device and the two are separately arranged. However, it should be understood that the storage medium 110 may also be independent of the electronic device and may be accessed by the processor 120 through a bus interface. Alternatively, the storage medium 110 may also be integrated into the processor 120, for example, it may be a cache and / or a general register.

[0147] The perspective transformation device 130 can be understood as the above-mentioned electronic device, or the processor 120 of the electronic device, or can be understood as a software function module independent of the above-mentioned electronic device or processor 120 that implements the above-mentioned perspective transformation method under the control of the electronic device.

[0148] like Fig.13 As shown, the perspective transformation device 130 may include an acquisition module 131, a determination module 132 and a transformation module 133. The functions of each functional module of the perspective transformation device 130 are described in detail below.

[0149] The acquisition module 131 acquires a face image captured by a shooting device, and captures a face region image in the face image, wherein the face image includes a plurality of face key points.

[0150] It can be understood that the acquisition module 131 can be used to execute the above step S101. The detailed implementation of the acquisition module 131 can refer to the above content related to step S101.

[0151] The determination module 132 is used to construct a virtual perspective camera, and obtain the posture parameters of the virtual perspective camera according to the reprojection error between the 3D coordinate information and the 2D coordinate information of each facial key point in the facial area image, and the rigid transformation information of the face during the perspective transformation.

[0152] It can be understood that the determination module 132 can be used to execute the above step S103. The detailed implementation of the determination module 132 can refer to the above content related to step S103.

[0153] The transformation module 133 is used to transform the 3D coordinate information of each of the facial key points into virtual 3D coordinate information in a virtual perspective camera space based on the posture parameters.

[0154] It can be understood that the transformation module 133 can be used to execute the above step S105. The detailed implementation of the transformation module 133 can refer to the above content related to step S105.

[0155] In a possible implementation, the acquisition module 131 may be used to:

[0156] Determine a plurality of facial key points constituting a facial contour in the facial image;

[0157] Obtaining a minimum bounding box of the plurality of facial key points;

[0158] The face image is intercepted based on the minimum bounding box to obtain a face area image.

[0159] In a possible implementation, the acquisition module 131 may be used to:

[0160] Determining the center position of the minimum bounding box;

[0161] The interception length is calculated based on the position information of multiple corner points of the minimum bounding box;

[0162] A clipping range is determined according to the center position and the clipping length, and the face image is clipped based on the clipping range to obtain a face area image.

[0163] In a possible implementation manner, the perspective transformation device 130 obtains a module for obtaining 2D coordinate information of each of the facial key points in the facial region image, and the module can be used to:

[0164] Obtaining 3D coordinate information of each facial key point in a world coordinate system;

[0165] projecting the 3D coordinate information of each of the facial key points into 2D coordinate information in a first image coordinate system according to the device parameters of the photographing device, where the first image coordinate system is constructed based on the facial image;

[0166] Based on the interception information when intercepting the face area image, the 2D coordinate information is updated to obtain the 2D coordinate information in the second image coordinate system constructed by the face area image.

[0167] In a possible implementation manner, the interception information includes the interception length and the interception center position, and the acquisition module can be used to:

[0168] Subtract the horizontal coordinate of the interception center position from the horizontal coordinate in the 2D coordinate information, and add half of the interception length;

[0169] The vertical coordinate in the 2D coordinate information is subtracted from the vertical coordinate of the interception center position, and is added with half of the interception length.

[0170] In a possible implementation, the transformation module 133 may be used to:

[0171] Constructing a first optimization item consisting of a reprojection error between the 3D coordinate information and the 2D coordinate information of each of the facial key points in the facial region image, and constructing a second optimization item consisting of rigid transformation information of the face during perspective transformation;

[0172] Assigning a weight value to the second optimization item based on a set weight coefficient;

[0173] The first optimization item and the second optimization item after the weighted value is added to construct an optimization model, and the optimization model is minimized to obtain the posture parameters of the virtual perspective camera.

[0174] In a possible implementation, the transformation module 133 may be used to:

[0175] Constructing a projection item obtained from the 3D coordinate information of each facial key point in the facial region image and the posture parameter;

[0176] The first optimization item is obtained by subtracting the projection item from the 2D coordinate information of each facial key point in the facial region image.

[0177] In a possible implementation, the transformation module 133 may be used to:

[0178] Multiplying the rotation matrix by the transposed matrix of the rotation matrix to obtain rigid feature information;

[0179] The second optimization item is obtained by subtracting the rigid feature information from the constructed unit matrix.

[0180] In a possible implementation, the perspective transformation device further includes a construction module, and the construction module can be used to:

[0181] Constructing a face mesh based on the multiple face key points;

[0182] The mesh information of the face mesh is obtained according to the virtual 3D coordinate information of each of the face key points.

[0183] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0184] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are executed, the perspective transformation method provided in the above embodiment is implemented.

[0185] Specifically, the computer-readable storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the computer-readable storage medium is executed, the above perspective transformation method can be executed. Regarding the process involved when the computer-readable storage medium and its executable instructions are executed, reference can be made to the relevant description in the above method embodiment, which will not be described in detail here.

[0186] In summary, the perspective transformation method, device, electronic device and readable storage medium provided in the embodiments of the present application intercept a face area image in a face image captured by a shooting device, and the face area image contains multiple face key points. A virtual perspective camera is constructed, and the posture parameters of the virtual perspective camera are obtained according to the reprojection error between the 3D coordinate information and the 2D coordinate information of each face key point in the face area image, and the rigid transformation information of the face during perspective transformation. Finally, the 3D coordinate information of each face key point is transformed into virtual 3D coordinate information in the virtual perspective camera space based on the posture parameters. In this solution, the reprojection error and rigid transformation information between 3D points and 2D are comprehensively considered in the perspective transformation, which can achieve good results in avoiding face transformation deformation and improving the accuracy of key point projection.

[0187] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0188] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0189] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0190] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A perspective transformation method, characterized in that: The method comprises: Acquire a face image captured by a photographing device, and intercept and obtain a face region image in the face image, wherein the face region image includes a plurality of face key points; Constructing a virtual perspective camera, constructing a first optimization item consisting of a reprojection error between the 3D coordinate information and the 2D coordinate information of each of the facial key points in the facial region image, and constructing a second optimization item consisting of the rigid transformation information of the face during perspective transformation, assigning a weight value to the second optimization item based on a set weight coefficient, adding the first optimization item and the second optimization item after the weight value is assigned to construct an optimization model, and performing minimization processing on the optimization model to obtain the posture parameters of the virtual perspective camera; Based on the posture parameters, the 3D coordinate information of each of the facial key points is transformed into virtual 3D coordinate information in a virtual perspective camera space; The posture parameters include a rotation matrix, and the second optimization item is obtained by: The rotation matrix is ​​multiplied by the transposed matrix of the rotation matrix to obtain rigid feature information, and the rigid feature information is subtracted from the constructed unit matrix to obtain the second optimization item.

2. The perspective transformation method according to claim 1, characterized in that: The step of intercepting and obtaining a face area image in the face image comprises: Determine a plurality of facial key points constituting a facial contour in the facial image; Obtaining a minimum bounding box of the plurality of facial key points; The face image is intercepted based on the minimum bounding box to obtain a face area image.

3. The perspective transformation method according to claim 2, characterized in that: The step of intercepting the face image based on the minimum bounding box to obtain a face area image comprises: Determining the center position of the minimum bounding box; The interception length is calculated based on the position information of multiple corner points of the minimum bounding box; A clipping range is determined according to the center position and the clipping length, and the face image is clipped based on the clipping range to obtain a face area image.

4. The perspective transformation method according to claim 2, characterized in that: The method further comprises the step of obtaining 2D coordinate information of each of the facial key points in the facial region image, the step comprising: Obtaining 3D coordinate information of each facial key point in a world coordinate system; projecting the 3D coordinate information of each of the facial key points into 2D coordinate information in a first image coordinate system according to the device parameters of the photographing device, where the first image coordinate system is constructed based on the facial image; Based on the interception information when intercepting the face area image, the 2D coordinate information is updated to obtain the 2D coordinate information in the second image coordinate system constructed by the face area image.

5. The perspective transformation method according to claim 4, characterized in that: The interception information includes the interception length and the interception center position; The step of updating the 2D coordinate information based on the interception information when intercepting the face area image comprises: Subtract the horizontal coordinate of the interception center position from the horizontal coordinate in the 2D coordinate information, and add half of the interception length; The vertical coordinate in the 2D coordinate information is subtracted from the vertical coordinate of the interception center position, and is added with half of the interception length.

6. The perspective transformation method according to claim 1, characterized in that: The step of constructing a first optimization item consisting of a reprojection error between the 3D coordinate information and the 2D coordinate information of each of the facial key points in the facial region image comprises: Constructing a projection item obtained from the 3D coordinate information of each facial key point in the facial region image and the posture parameter; The first optimization item is obtained by subtracting the projection item from the 2D coordinate information of each facial key point in the facial region image.

7. The perspective transformation method according to any one of claims 1 to 6, characterized in that: The method further comprises: Constructing a face mesh based on the multiple face key points; The mesh information of the face mesh is obtained according to the virtual 3D coordinate information of each of the face key points.

8. A perspective transformation device, characterized in that: For implementing the perspective transformation method according to any one of claims 1 to 7, the device comprises: An acquisition module is used to acquire a face image captured by a shooting device and to intercept and obtain a face region image in the face image, wherein the face image includes a plurality of face key points; A determination module, used to construct a virtual perspective camera, and obtain the posture parameters of the virtual perspective camera according to the reprojection error between the 3D coordinate information and the 2D coordinate information of each of the facial key points in the facial region image, and the rigid transformation information of the face during the perspective transformation; A transformation module is used to transform the 3D coordinate information of each of the facial key points into virtual 3D coordinate information in a virtual perspective camera space based on the posture parameters.

9. An electronic device, characterized in that: The electronic device comprises one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine executable instructions executable by the processor, and when the electronic device is running, the processor executes the machine executable instructions to perform the method steps described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, the method steps described in any one of claims 1 to 7 are implemented.

Images