A facial model calibration display method, device, equipment and medium

Abstract

The disclosed embodiments relate to a facial model calibration display method, device, equipment and medium, including: obtaining a vascular nerve model and performing scale calculation to obtain a corresponding vascular nerve key point set; obtaining a facial key point set corresponding to a facial image; calculating a rotation matrix based on the vascular nerve key point set and the facial key point set; converting the vascular nerve key point set based on the rotation matrix to obtain a target vascular nerve key point set; calculating a scale parameter based on the target vascular nerve key point set, the facial key point set and the rotation matrix; performing a second scale calculation on the adjusted vascular nerve model based on the scale parameter and then performing a rotation offset to obtain a target vascular nerve model and a facial reconstruction mesh model corresponding to the target face, and displaying the model simultaneously. By adopting the above technical solution, the vascular nerve model is corrected based on the scale registration method and displayed simultaneously with the facial reconstruction mesh model, thereby improving the effect of the indication planning operation.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent medical control technology, and in particular to a facial model calibration display method, device, equipment and medium. Background Art

[0002] In the field of robotics, three-dimensional vision applications and three-dimensional displays are becoming more and more widespread.

[0003] In the related art, when performing facial injection surgery, the doctor's experience is usually relied upon to select the injection point; or an intelligent medical robot can only provide a three-dimensional display of the face reconstruction network model and the real-time model of the surgical equipment to instruct the doctor's operation, resulting in poor results in the instruction planning operation. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a facial model calibration display method, device, equipment and medium.

[0005] The present disclosure provides a facial model calibration display method, the method comprising:

[0006] Acquire a pre-built vascular nerve model, perform a first scale calculation on the vascular nerve model to obtain an adjusted vascular nerve model, and acquire a set of vascular nerve key points corresponding to the adjusted vascular nerve model;

[0007] Collecting a facial image corresponding to the target face, and obtaining a set of facial key points corresponding to the facial image;

[0008] Calculating based on the vascular nerve key point set and the face key point set to obtain a rotation matrix;

[0009] Transforming the vascular nerve key point set based on the rotation matrix to obtain a target vascular nerve key point set;

[0010] Calculating based on the target blood vessel nerve key point set, the face key point set and the rotation matrix to obtain a scale parameter;

[0011] A second scale calculation is performed on the adjusted vascular nerve model based on the scale parameter, and the second scale calculation result is rotated and offset to obtain a target vascular nerve model, and the target vascular nerve model and the facial reconstruction mesh model corresponding to the target face are displayed simultaneously.

[0012] The present disclosure also provides a facial model calibration display device, the device comprising:

[0013] An acquisition module, used to acquire a pre-built vascular nerve model;

[0014] an adjustment acquisition module, configured to perform a first scale calculation on the vascular nerve model to obtain an adjusted vascular nerve model, and to obtain a set of vascular nerve key points corresponding to the adjusted vascular nerve model;

[0015] The acquisition module is used to acquire the face image corresponding to the target face and obtain the face key point set corresponding to the face image;

[0016] A first calculation module, configured to perform calculation based on the vascular nerve key point set and the face key point set to obtain a rotation matrix;

[0017] A conversion module, used to convert the blood vessel nerve key point set based on the rotation matrix to obtain a target blood vessel nerve key point set;

[0018] A second calculation module, configured to calculate based on the target blood vessel nerve key point set, the face key point set and the rotation matrix to obtain a scale parameter;

[0019] a processing module, configured to perform a second scale calculation on the adjusted vascular nerve model based on the scale parameter, and perform rotational shifting on the second scale calculation result to obtain a target vascular nerve model;

[0020] The display module is used to simultaneously display the target blood vessel nerve model and the face reconstruction grid model corresponding to the target face.

[0021] An embodiment of the present disclosure also provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the facial model calibration display method provided in the embodiment of the present disclosure.

[0022] The embodiment of the present disclosure also provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the facial model calibration display method provided by the embodiment of the present disclosure.

[0023] The present disclosure also provides a computer program product, including a computer program, wherein the computer program is executed by a processor as the facial model calibration display method provided in the embodiment of the present application.

[0024] The technical solution provided by the embodiments of the present disclosure has the following advantages over the prior art: the facial model calibration and display solution provided by the embodiments of the present disclosure obtains a pre-constructed vascular nerve model, performs a first scale calculation on the vascular nerve model, obtains an adjusted vascular nerve model, and obtains a vascular nerve key point set corresponding to the adjusted vascular nerve model; collects a facial image corresponding to a target face, and obtains a facial key point set corresponding to the facial image; performs calculation based on the vascular nerve key point set and the facial key point set to obtain a rotation matrix; transforms the vascular nerve key point set based on the rotation matrix to obtain a target vascular nerve key point set; performs calculation based on the target vascular nerve key point set, the facial key point set and the rotation matrix to obtain a scale parameter; performs a second scale calculation on the adjusted vascular nerve model based on the scale parameter, performs a rotation offset on the second scale calculation result to obtain a target vascular nerve model, and displays the target vascular nerve model and the facial reconstruction mesh model corresponding to the target face simultaneously. By adopting the above technical scheme, a vascular nerve model is formed based on a general artificial face model, and facial three-dimensional point cloud data is obtained. The vascular nerve model is corrected to the same scale and coordinate system as the facial three-dimensional point cloud data through a scaled registration algorithm, and then displayed simultaneously with the facial reconstructed mesh model, so as to achieve the purpose of indicating and planning operations, improve the efficiency and effect of indicating and planning operations, meet user needs, and enhance user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0026] Figure 1 A flowchart of a facial model calibration and display method provided by an embodiment of the present disclosure;

[0027] Figure 2 A schematic flow chart of another facial model calibration and display method provided by an embodiment of the present disclosure;

[0028] Figure 3 A schematic diagram of the structure of a facial model calibration display device provided by an embodiment of the present disclosure;

[0029] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0031] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0032] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0033] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0034] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0035] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0036] The present invention discloses a vascular nerve model based on a universal artificial human face model, and utilizes a scaled registration algorithm to register the vascular nerve three-dimensional model with a human face captured by, for example, a structured light camera to correct the scale and posture, thereby achieving the purpose of indicating the planned operation; for example, a medical facial injection surgery robot guidance scene is used to indicate facial injections.

[0037] Figure 1 The present invention provides a flow chart of a facial model calibration display method provided by an embodiment of the present invention. The method can be performed by a facial model calibration display device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method includes:

[0038] Step 101: obtain a pre-constructed vascular nerve model, perform a first scale calculation on the vascular nerve model, obtain an adjusted vascular nerve model, and obtain a set of vascular nerve key points corresponding to the adjusted vascular nerve model.

[0039] In the disclosed embodiment, a vascular nerve model is pre-constructed, for example, a vascular nerve model of a general artificial human face model is constructed. It is understandable that the data produced by the general vascular nerve model produced by different production tools, such as the 3D animation software "Maya" and the 3D modeling rendering and production software "3ds max" based on the computer system, are different, and are not based on metric or millimeter data, so they are far from the actual human proportions, and direct 3D display will lead to incorrect operation of the indication planning.

[0040] In the embodiment of the present disclosure, a first scale calculation is performed on the vascular nerve model to obtain an adjusted vascular nerve model, that is, a preliminary scale processing is performed on the vascular nerve model. As an example, the scale ratio is determined based on the model size of the face reconstruction mesh model, and the first scale calculation is performed on the vascular nerve model based on the scale ratio to obtain the adjusted vascular nerve model; as another example, the first scale calculation is directly performed on the vascular nerve model according to a preset empirical value to obtain the adjusted vascular nerve model, for example, the vascular nerve model is reduced according to a ratio of 100 of the empirical value to obtain the adjusted vascular nerve model. The above is only an example, and the embodiment of the present disclosure does not specifically limit the method of performing the first scale calculation on the vascular nerve model to obtain the adjusted vascular nerve model.

[0041] In the embodiment of the present disclosure, after obtaining the adjusted vascular nerve model, the set of vascular nerve key points corresponding to the adjusted vascular nerve model is obtained. As an example, the vascular nerve key points of the adjusted vascular nerve model are determined based on the face position corresponding to the face identification set to obtain the set of vascular nerve key points; as another example, all vascular nerve key points corresponding to the vascular nerve model are obtained, and the set of vascular nerve key points is obtained from all vascular nerve key points according to the preset screening conditions; wherein the screening conditions are, for example, vascular nerve key points at the target position of the face (such as around the brow bone, etc.). The above is only an example, and the embodiment of the present disclosure does not specifically limit the method of obtaining the set of vascular nerve key points corresponding to the adjusted vascular nerve model.

[0042] Step 102: collect a facial image corresponding to the target face, and obtain a set of facial key points corresponding to the facial image.

[0043] In the embodiments of the present disclosure, for example, a target face is photographed based on a structured light camera to obtain a face image; wherein the target face is selected and set according to the actual application scenario.

[0044] In an embodiment of the present disclosure, after acquiring a face image, a set of face key points corresponding to the face image is acquired. As an example, the face image is detected based on a pre-trained face feature point detector, and a candidate face key point set is acquired from the face point cloud data corresponding to the face image based on the detection result. Face key points are acquired from the candidate face key point set based on the face identifier corresponding to the face identifier set to obtain a face key point set. As another example, the face image is detected based on a pre-trained face feature point detector corresponding to the face identifier set, and a face key point set is acquired from the face point cloud data corresponding to the face image based on the detection result. The above is only an example, and the embodiment of the present disclosure does not specifically limit the method of acquiring a face key point set corresponding to a face image.

[0045] Step 103: Calculate based on the vascular nerve key point set and the face key point set to obtain a rotation matrix.

[0046] In the embodiments of the present disclosure, the coordinates of each vascular nerve key point can be obtained based on the vascular nerve key point set, and the coordinates of each facial key point can be obtained based on the facial key point set; a rotation matrix can be obtained by calculating based on the coordinates of each vascular nerve key point and the coordinates of each facial key point. As an example, the first center point coordinates are calculated based on the coordinates of each vascular nerve key point in the vascular nerve key point set, and the second center point coordinates are calculated based on the coordinates of each facial key point in the facial key point set. Based on the first center point coordinates and the second center point coordinates, each vascular nerve key point coordinate and each facial key point coordinate are aligned respectively to obtain first aligned point cloud coordinates and second aligned point cloud coordinates, and the first aligned point cloud coordinates and the second aligned point cloud coordinates are converted into a matrix to be decomposed, and the matrix to be decomposed is decomposed based on a preset matrix decomposition algorithm to obtain a rotation matrix; as another example, the coordinates of two key points in the vascular nerve key point set and the facial key point set are constructed for feature matching, an equation is constructed, and solved by the least squares method to obtain a rotation matrix. The above is only an example, and the embodiments of the present disclosure do not specifically limit the method of obtaining the rotation matrix by calculation based on the vascular nerve key point set and the face key point set.

[0047] Step 104: transform the blood vessel nerve key point set based on the rotation matrix to obtain a target blood vessel nerve key point set.

[0048] In the embodiment of the present disclosure, after the rotation matrix is ​​obtained, the rotation matrix is ​​used to calculate the coordinates of each vascular nerve key point in the vascular nerve key point set to obtain the target vascular nerve key point set.

[0049] Step 105: Calculate based on the target blood vessel nerve key point set, the face key point set and the rotation matrix to obtain a scale parameter.

[0050] In the embodiments of the present disclosure, there are many ways to obtain scale parameters based on the target vascular nerve key point set, the face key point set and the rotation matrix. As an example, an energy equation is constructed based on the target vascular nerve key point set, the face key point set and the rotation matrix, and the energy equation is differentiated to obtain the minimum value to obtain a linear equation. The linear equation is solved based on a preset solution algorithm to obtain the scale parameter. As another example, the target vascular nerve key point set, the face key point set and the rotation matrix are calculated based on a preset calculation algorithm to obtain the scale parameter. The above is only an example, and the embodiments of the present disclosure do not specifically limit the way to obtain the scale parameter based on the target vascular nerve key point set, the face key point set and the rotation matrix.

[0051] In the embodiment of the present disclosure, the scale parameter includes a displacement vector and a scale.

[0052] Step 106: Perform a second scale calculation on the adjusted vascular nerve model based on the scale parameter, and perform rotational offset on the second scale calculation result to obtain a target vascular nerve model, and simultaneously display the target vascular nerve model and the facial reconstruction mesh model corresponding to the target face.

[0053] In an embodiment of the present disclosure, after obtaining the scale parameter, a second scale calculation is performed on the adjusted vascular nerve model based on the scale parameter, and the second scale calculation result is rotated and offset to obtain a target vascular nerve model. As an example, all vascular nerve point coordinates corresponding to the adjusted vascular nerve model are obtained, and the displacement vector in the scale parameter is multiplied by each vascular nerve point coordinate and then added to the scale in the scale parameter to perform rotational offset to obtain the target vascular nerve point coordinates. Based on all target vascular nerve point coordinates, the target vascular nerve model is obtained; as another example, a calculation algorithm is set based on the scale parameter, and a second scale calculation is performed on the adjusted vascular nerve model according to the calculation algorithm, and then the second scale calculation is performed and offset to obtain the target vascular nerve model. The above is only an example, and the embodiment of the present disclosure does not specifically limit the method of performing a second scale calculation on the adjusted vascular nerve model based on the scale parameter, and performing rotational offset on the second scale calculation result to obtain the target vascular nerve model.

[0054] In the embodiments of the present disclosure, there are many ways to simultaneously display the target vascular nerve model and the facial reconstruction mesh model corresponding to the target face, and the specific settings are selected according to the application scenario. For example, the target vascular nerve model and the facial reconstruction mesh model corresponding to the target face are superimposed and displayed.

[0055] The facial model calibration and display solution provided by the embodiment of the present disclosure obtains a pre-constructed vascular nerve model, performs a first scale calculation on the vascular nerve model, obtains an adjusted vascular nerve model, and obtains a vascular nerve key point set corresponding to the adjusted vascular nerve model; collects a facial image corresponding to a target face, and obtains a facial key point set corresponding to the facial image; performs calculation based on the vascular nerve key point set and the facial key point set to obtain a rotation matrix; transforms the vascular nerve key point set based on the rotation matrix to obtain a target vascular nerve key point set; performs calculation based on the target vascular nerve key point set, the facial key point set and the rotation matrix to obtain a scale parameter; performs a second scale calculation on the adjusted vascular nerve model based on the scale parameter to obtain a target vascular nerve model, and simultaneously displays the target vascular nerve model and the facial reconstruction mesh model corresponding to the target face. By adopting the above technical scheme, a vascular nerve model is formed based on a general artificial face model, and facial three-dimensional point cloud data is obtained. The vascular nerve model is corrected to the same scale and coordinate system as the facial three-dimensional point cloud data through a scaled registration algorithm, and then displayed simultaneously with the facial reconstructed mesh model, so as to achieve the purpose of indicating and planning operations, improve the efficiency and effect of indicating and planning operations, meet user needs, and enhance user experience.

[0056] Figure 2 This is a flow chart of another facial model calibration and display method provided by an embodiment of the present disclosure. This embodiment further optimizes the facial model calibration and display method based on the above embodiment. Figure 2 As shown, the method includes:

[0057] Step 201: obtain a pre-constructed vascular nerve model, determine a scale ratio based on a model size of a face reconstruction mesh model, perform a first scale calculation on the vascular nerve model based on the scale ratio, and obtain an adjusted vascular nerve model.

[0058] Specifically, the vascular nerve model and the data obtained by the structured light camera are usually quite different in magnitude, so a preliminary scale change is required. More specifically, the vascular nerve model is initially scaled by performing a coarse scale calculation on the vascular nerve model, for example, first reducing it by a hundred times to obtain an adjusted vascular nerve model.

[0059] Step 202: Determine and adjust the vascular nerve key points of the vascular nerve model based on the face position corresponding to the face identification set to obtain a vascular nerve key point set.

[0060] In some embodiments, an initial face identification set is obtained, and target face key points are obtained from the initial face identification set based on the registration feature points of the target features to obtain a face identification set.

[0061] In the disclosed embodiment, the face identification set includes multiple face key point identifications. The face includes 68 key points, that is, as the initial face identification set, the target face key points can be obtained from the initial face identification set according to the registration feature points of the target features to obtain the face identification set.

[0062] Specifically, the disclosed embodiment is aimed at calibrating facial data and nerves, and there are errors in the data. Therefore, the registration feature points of the target features are selected from the 68 key points of the face (the initial face identification set), which usually refer to the data around the brow bone, the nose tip, and other thin skin areas to improve the accuracy of this registration method. It should be noted that the accuracy of the above method is still limited, so the calibration effect is mainly for indicative display, and no absolute accuracy positioning is performed.

[0063] For example, faceids represents a set of face identifiers, such as faceids = {18, 19, 20, 21, 36, 39, 22, 23, 24, 25, 42, 45, 30, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 41, 40, 47, 46, 7, 9}; wherein, numbers such as 18 are face identifiers, which can uniquely identify key points of a face.

[0064] Specifically, according to the face position corresponding to faceids, the vascular nerve key points corresponding to the vascular nerve model are selected and adjusted in A to obtain a set of vascular nerve key points.

[0065] Step 203: collect a facial image corresponding to the target face, and detect the facial image based on a pre-trained facial feature point detector, obtain a candidate facial key point set from the facial point cloud data corresponding to the facial image based on the detection result, obtain facial key points from the candidate facial key point set based on the facial identifier corresponding to the facial identifier set, and obtain a facial key point set.

[0066] Among them, the face feature point detector can be, for example, the "shape_predictor_68_face_landmarks.dat" model, which is a face feature point detector of a trained neural network. By loading the model (face feature point detector) using the tool "Dlib", the neural network can be used to perform key point detection on the face image.

[0067] Specifically, a face image obtained by a frame of a structured light camera is subjected to facial key point recognition using "Dlib" to obtain a candidate face key point set, such as 68 face key points, and then face key points are obtained from the candidate face key point set to obtain a face key point set. For example, the face key point set is obtained from the candidate face key point set according to the corresponding serial number of faceids.

[0068] It can be understood that the structured light camera simultaneously obtains the face image and the corresponding face point cloud data, and after identifying the key points, the point cloud data corresponding to each key point can be obtained from the point cloud data set.

[0069] Step 204: Calculate based on the coordinates of each vascular nerve key point in the vascular nerve key point set to obtain the first center point coordinates, and calculate based on the coordinates of each facial key point in the facial key point set to obtain the second center point coordinates.

[0070] Step 205: align the coordinates of each vascular nerve key point and the coordinates of each facial key point based on the first center point coordinates and the second center point coordinates to obtain first aligned point cloud coordinates and second aligned point cloud coordinates.

[0071] Step 206: convert the first aligned point cloud coordinates and the second aligned point cloud coordinates into a matrix to be decomposed, and decompose the matrix to be decomposed based on a preset matrix decomposition algorithm to obtain a rotation matrix.

[0072] Specifically, the rotation matrix of the vascular nerve key point set and the face key point set is solved by using a related solution algorithm such as SVD (Singular Value Decomposition).

[0073] Specifically, singular value decomposition is a commonly used matrix decomposition method that can decompose a matrix into the product of three matrices, namely A = U∑V T , where A is an m×n matrix, U is an m×m orthogonal matrix, ∑ is an m×n diagonal matrix, and V is an n×n orthogonal matrix.

[0074] In point cloud registration, the method of using SVD to solve the rotation matrix between two point clouds includes: aligning the two point clouds, first calculating the coordinates of the two center points of the two point clouds respectively, and overlapping them through the two center point coordinates. Usually, an initial guessed rotation matrix is ​​required to approximately align the two point clouds. The two aligned point clouds are represented as m×3 matrices P and Q, where each row represents the coordinates of a point. SVD is used to calculate P T Q is decomposed to get P T Q=U∑V T , calculate the rotation matrix VU T In this way, through SVD decomposition, the rotation matrix between point clouds can be solved.

[0075] Step 207: transform the vascular nerve key point set based on the rotation matrix to obtain the target vascular nerve key point set, construct an energy equation based on the target vascular nerve key point set, the face key point set and the rotation matrix, differentiate the energy equation to find the minimum value to obtain a linear equation, solve the linear equation based on a preset solution algorithm to obtain a scale parameter.

[0076] Specifically, the rotation matrix obtained is used to transform the set of vascular nerve key points to obtain the target vascular nerve key point set, and then the energy equation is constructed to obtain the minimum value and the linear equation is obtained. Finally, the linear equation is solved by the solution algorithm to obtain the scale parameter.

[0077] Specifically, the solving algorithm, such as "eigen", is the same as the principle of solving the linear equation Ax=B in linear algebra where A and B are known to solve x. The solve() method of the JacobiSVD class can solve the linear equation system of the form Ax=b, which can be obtained by directly calling the solve() method of the JacobiSVD class.

[0078] Step 208, obtaining the coordinates of all vascular nerve points corresponding to the adjusted vascular nerve model, multiplying the displacement vector in the scale parameter with each vascular nerve point coordinate and adding the scale in the scale parameter to perform rotation offset to obtain the target vascular nerve point coordinates.

[0079] Step 209: Based on the coordinates of all target blood vessel nerve points, a target blood vessel nerve model is obtained, and the target blood vessel nerve model and the face reconstruction mesh model corresponding to the target face are displayed simultaneously.

[0080] As an example, a represents the vascular nerve model of the human face, A represents the coarse-scale calculation result, and Key A represents the key points of blood vessels and nerves. B represents the face point cloud captured by structured light. B Represents the face image corresponding to B. Key B represents the key points of the face, R represents the rotation matrix, Key AR represents the key point after the rotation matrix changes, T represents the displacement vector, S represents the scale, faceids represents the 68-point face key point screening set, which is defined in the present disclosure as faceids = {18, 19, 20, 21, 36, 39, 22, 23, 24, 25, 42, 45, 30, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 41, 40, 47, 46, 7, 9}, minE represents the minimum energy, Mesh B Represents the face reconstruction mesh model.

[0081] Specifically, a is initially scaled. According to the data, a is first roughly scaled. In this disclosure, A is first reduced by a hundred times. According to the face position corresponding to faceids, a key point Key is selected in A. A ; A face image I obtained by a frame of structured light camera B Use the Dlib library to identify 68 key points of the face; according to the corresponding serial number of faceids, get the key points of the face from the point cloud B ; Use SVD to solve Key A With Key B The rotation matrix R of the solution is used to obtain Key AR ; According to the energy equation minE=||(S*Key AR +T)-Key B || 2 Derivation and minimization are obtained to obtain the linear equation Ax = B; finally, eigen is used to solve the linear equation to obtain S and T; finally, S*Key AR +T Get with Key B The same scale and coordinate system; similarly, calculate A to obtain a new model A with the same scale and coordinate system as B B ; A B With Mesh B At the same time, the functions of the blood vessels and nerves in the face can be indicated.

[0082] Therefore, based on the vascular nerve model of the general artificial face model, the facial three-dimensional point cloud data is acquired using a structured light camera, and the vascular nerve model is corrected to the same scale and coordinate system as the point cloud data through a scaled registration algorithm, and finally displayed simultaneously with the reconstructed face mesh model to achieve the purpose of indicating the planned operation.

[0083] Figure 3 This is a schematic diagram of the structure of a facial model calibration display device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. Figure 3 As shown, the device comprises:

[0084] The acquisition module 301 is used to acquire a pre-built blood vessel nerve model.

[0085] The adjustment acquisition module 302 is used to perform a first scale calculation on the vascular nerve model to obtain an adjusted vascular nerve model, and to obtain a set of vascular nerve key points corresponding to the adjusted vascular nerve model.

[0086] The acquisition module 303 is used to acquire a facial image corresponding to the target face and obtain a set of facial key points corresponding to the facial image.

[0087] The first calculation module 304 is used to perform calculation based on the blood vessel nerve key point set and the face key point set to obtain a rotation matrix.

[0088] The conversion module 305 is used to convert the blood vessel nerve key point set based on the rotation matrix to obtain a target blood vessel nerve key point set.

[0089] The second calculation module 306 is used to perform calculation based on the target blood vessel nerve key point set, the face key point set and the rotation matrix to obtain a scale parameter.

[0090] The processing module 307 is used to perform a second scale calculation on the adjusted blood vessel nerve model based on the scale parameter, and perform rotational shift on the second scale calculation result to obtain a target blood vessel nerve model.

[0091] The display module 308 is used to simultaneously display the target blood vessel nerve model and the face reconstruction mesh model corresponding to the target face.

[0092] Optionally, the adjustment acquisition module 302 is specifically used to:

[0093] Determining a scale ratio based on a model size of the face reconstruction mesh model;

[0094] Performing a first scale calculation on the vascular nerve model based on the scale ratio to obtain the adjusted vascular nerve model;

[0095] The vascular nerve key points of the adjusted vascular nerve model are determined based on the face position corresponding to the face identification set to obtain the vascular nerve key point set.

[0096] Optionally, the facial model calibration and display method further includes: a screening module, which is used to:

[0097] An initial face identification set is obtained, and target face key points are obtained from the initial face identification set based on the registration feature points of the target features to obtain the face identification set.

[0098] Optionally, the acquisition module is specifically used to:

[0099] Collecting a facial image corresponding to the target face, and detecting the facial image based on a pre-trained facial feature point detector, and obtaining the candidate facial key point set from facial point cloud data corresponding to the facial image based on the detection result;

[0100] Based on the face identifier corresponding to the face identifier set, facial key points are obtained from the candidate facial key point set to obtain the facial key point set.

[0101] Optionally, the conversion module is specifically used to:

[0102] Calculate based on the coordinates of each vascular nerve key point in the vascular nerve key point set to obtain the coordinates of the first center point;

[0103] Calculate based on the coordinates of each facial key point in the facial key point set to obtain the coordinates of the second center point;

[0104] Based on the first center point coordinates and the second center point coordinates, the coordinates of each vascular nerve key point and the coordinates of each face key point are respectively aligned to obtain first aligned point cloud coordinates and second aligned point cloud coordinates;

[0105] The first aligned point cloud coordinates and the second aligned point cloud coordinates are converted into a matrix to be decomposed, and the matrix to be decomposed is decomposed based on a preset matrix decomposition algorithm to obtain the rotation matrix.

[0106] Optionally, the second computing module is specifically configured to:

[0107] Constructing an energy equation based on the target blood vessel nerve key point set, the face key point set and the rotation matrix;

[0108] Derivatively minimizing the energy equation to obtain a linear equation;

[0109] The linear equation is solved based on a preset solution algorithm to obtain the scale parameter.

[0110] Optionally, the processing module is specifically used to:

[0111] Obtaining coordinates of all vascular nerve points corresponding to the adjusted vascular nerve model;

[0112] Based on the multiplication of the displacement vector in the scale parameter with each of the blood vessel nerve point coordinates and the addition of the scale in the scale parameter, a rotation offset is performed to obtain the target blood vessel nerve point coordinates;

[0113] Based on the coordinates of all the target blood vessel nerve points, the target blood vessel nerve model is obtained.

[0114] The facial model calibration and display device provided in the embodiments of the present disclosure can execute the facial model calibration and display method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0115] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the facial model calibration display method provided by any embodiment of the present disclosure.

[0116] Figure 4 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 4 , which shows a schematic diagram of the structure of an electronic device 400 suitable for implementing the embodiment of the present disclosure. The electronic device 400 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0117] like Figure 4 As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0118] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0119] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above functions defined in the face model calibration display method of the embodiment of the present disclosure are executed.

[0120] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0121] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0122] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0123] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a pre-constructed vascular nerve model, performs a first scale calculation on the vascular nerve model to obtain an adjusted vascular nerve model, and obtains a vascular nerve key point set corresponding to the adjusted vascular nerve model; collects a facial image corresponding to a target face, and obtains a facial key point set corresponding to the facial image; performs calculations based on the vascular nerve key point set and the facial key point set to obtain a rotation matrix; transforms the vascular nerve key point set based on the rotation matrix to obtain a target vascular nerve key point set; performs calculations based on the target vascular nerve key point set, the facial key point set and the rotation matrix to obtain a scale parameter; performs a second scale calculation on the adjusted vascular nerve model based on the scale parameter, and performs rotational offset on the second scale calculation result to obtain a target vascular nerve model, and simultaneously displays the target vascular nerve model and the facial reconstruction mesh model corresponding to the target face.

[0124] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0125] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0126] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0127] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0128] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0129] According to one or more embodiments of the present disclosure, the present disclosure provides an electronic device, including:

[0130] processor;

[0131] a memory for storing instructions executable by the processor;

[0132] The processor is used to read the executable instructions from the memory and execute the instructions to implement any facial model calibration and display method provided in the present disclosure.

[0133] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute any of the facial model calibration display methods provided by the present disclosure.

[0134] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0135] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0136] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A facial model calibration display method, characterized in that: include: Acquire a pre-constructed vascular nerve model, perform a first scale calculation on the pre-constructed vascular nerve model to obtain an adjusted vascular nerve model, and acquire a set of vascular nerve key points corresponding to the adjusted vascular nerve model; Collecting a facial image corresponding to the target face, and obtaining a set of facial key points corresponding to the facial image; Calculating based on the vascular nerve key point set and the face key point set to obtain a rotation matrix; Transforming the vascular nerve key point set based on the rotation matrix to obtain a target vascular nerve key point set; Calculating based on the target blood vessel nerve key point set, the face key point set and the rotation matrix to obtain a scale parameter; Performing a second scale calculation on the adjusted vascular nerve model based on the scale parameter, performing rotational shifting on the second scale calculation result to obtain a target vascular nerve model, and simultaneously displaying the target vascular nerve model and a facial reconstruction mesh model corresponding to the target face; The step of performing a second scale calculation on the adjusted vascular nerve model based on the scale parameter and performing a rotation shift on the second scale calculation result to obtain a target vascular nerve model includes: Obtaining coordinates of all vascular nerve points corresponding to the adjusted vascular nerve model; Based on the multiplication of the displacement vector in the scale parameter with each of the blood vessel nerve point coordinates and the addition of the scale in the scale parameter, a rotation offset is performed to obtain the target blood vessel nerve point coordinates; Based on the coordinates of all the target blood vessel nerve points, the target blood vessel nerve model is obtained.

2. The facial model calibration and display method according to claim 1, characterized in that: The performing a first scale calculation on the pre-constructed vascular nerve model to obtain an adjusted vascular nerve model, and acquiring a set of vascular nerve key points corresponding to the adjusted vascular nerve model, includes: Determining a scale ratio based on a model size of the face reconstruction mesh model; Performing a first scale calculation on the vascular nerve model based on the scale ratio to obtain the adjusted vascular nerve model; The vascular nerve key points of the adjusted vascular nerve model are determined based on the face position corresponding to the face identification set to obtain the vascular nerve key point set.

3. The facial model calibration and display method according to claim 2, characterized in that: Also includes: Get the initial face identification set; The target face key points are obtained from the initial face identification set based on the registration feature points of the target features to obtain the face identification set.

4. The facial model calibration and display method according to claim 1, characterized in that: The step of obtaining a set of facial key points corresponding to the facial image includes: Detecting the face image based on a pre-trained face feature point detector, and acquiring a candidate face key point set from face point cloud data corresponding to the face image based on the detection result; Based on the face identifier corresponding to the face identifier set, facial key points are obtained from the candidate facial key point set to obtain the facial key point set.

5. The facial model calibration and display method according to claim 1, characterized in that: The calculation based on the vascular nerve key point set and the face key point set to obtain a rotation matrix includes: Calculate based on the coordinates of each vascular nerve key point in the vascular nerve key point set to obtain the coordinates of the first center point; Calculate based on the coordinates of each facial key point in the facial key point set to obtain the coordinates of the second center point; Based on the first center point coordinates and the second center point coordinates, the coordinates of each vascular nerve key point and the coordinates of each face key point are respectively aligned to obtain first aligned point cloud coordinates and second aligned point cloud coordinates; The first aligned point cloud coordinates and the second aligned point cloud coordinates are converted into a matrix to be decomposed, and the matrix to be decomposed is decomposed based on a preset matrix decomposition algorithm to obtain the rotation matrix.

6. The facial model calibration and display method according to claim 1, characterized in that: The calculating based on the target blood vessel nerve key point set, the face key point set and the rotation matrix to obtain the scale parameter includes: Constructing an energy equation based on the target blood vessel nerve key point set, the face key point set and the rotation matrix; Derivatively minimizing the energy equation to obtain a linear equation; The linear equation is solved based on a preset solution algorithm to obtain the scale parameter.

7. A facial model calibration display device, characterized in that: include: An acquisition module, used to acquire a pre-built vascular nerve model; An adjustment acquisition module, configured to perform a first scale calculation on the pre-constructed vascular nerve model to obtain an adjusted vascular nerve model, and to obtain a set of vascular nerve key points corresponding to the adjusted vascular nerve model; The acquisition module is used to acquire the face image corresponding to the target face and obtain the face key point set corresponding to the face image; A first calculation module, configured to perform calculation based on the vascular nerve key point set and the face key point set to obtain a rotation matrix; A conversion module, used to convert the blood vessel nerve key point set based on the rotation matrix to obtain a target blood vessel nerve key point set; A second calculation module, configured to calculate based on the target blood vessel nerve key point set, the face key point set and the rotation matrix to obtain a scale parameter; a processing module, configured to perform a second scale calculation on the adjusted vascular nerve model based on the scale parameter, and perform rotational shifting on the second scale calculation result to obtain a target vascular nerve model; A display module, used for simultaneously displaying the target blood vessel nerve model and the face reconstruction mesh model corresponding to the target face; Wherein, the processing module is specifically used for: Obtaining coordinates of all vascular nerve points corresponding to the adjusted vascular nerve model; Based on the multiplication of the displacement vector in the scale parameter with each of the blood vessel nerve point coordinates and the addition of the scale in the scale parameter, a rotation offset is performed to obtain the target blood vessel nerve point coordinates; Based on the coordinates of all the target blood vessel nerve points, the target blood vessel nerve model is obtained.

8. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the facial model calibration and display method described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the facial model calibration and display method described in any one of claims 1 to 6.

Images