A full-reference quality evaluation method and system for 3D digital faces

Abstract

The present invention discloses a full-reference quality assessment method and system for three-dimensional digital faces, comprising: setting motion trajectories for three cameras, each capable of covering the viewpoint of the three-dimensional digital face; calculating and determining the centers of the motion trajectories for the three cameras; using the cameras to capture video sequences of distorted and undistorted reference three-dimensional faces; extracting frames from the captured video sequences at fixed intervals and splicing them to obtain frame sequences; sequentially inputting the frame sequences of the distorted face and the reference face into a trained network for feature extraction, obtaining extracted features of the distorted face and extracted features of the reference face; calculating the Euclidean distance between the extracted features of the distorted face and the reference face, and using the distance as an evaluation indicator for the quality of the three-dimensional digital face, thereby obtaining a quality assessment of the three-dimensional digital face. This method can solve the problem of the difficulty of directly processing three-dimensional digital face data and can efficiently and accurately assess the visual quality of distorted three-dimensional faces.

Description

Technical Field

[0001] The present invention relates to the technical field of image and video processing, and in particular to a full-reference quality evaluation method and system for three-dimensional digital faces. Background Art

[0002] In recent years, with the rapid development of 3D scanning equipment and social media, 3D digital faces have been widely used in 3D facial recognition, medical fields, and virtual and augmented reality. On the one hand, due to the jitter of the scanning equipment, 3D digital faces are inevitably affected by noise. On the other hand, due to limited bandwidth and storage constraints, 3D faces are inevitably compressed and simplified, resulting in visual distortion. How to accurately and effectively evaluate the impact of distortion on the visual quality of 3D faces has posed a significant challenge to the development of digital humans.

[0003] 3D facial data typically consists of complex meshes, making it difficult to process efficiently. However, while methods based on 2D projection cannot directly assess the quality of complex 3D faces, they can project 3D digital faces into 2D media, such as images and videos, and leverage image and video technologies for quality assessment. This approach effectively circumvents the complexities of 3D data processing and leverages established image and video quality assessment methods, making it highly valuable for research. Furthermore, in most digital face generation and transmission systems, reference digital faces are readily available, leading to the current emphasis on full-reference quality assessment methods for 3D digital face evaluation.

[0004] Therefore, how to design a full-reference quality evaluation method for three-dimensional faces is very important in the field of digital humans. Summary of the Invention

[0005] In view of the deficiencies of the prior art, the present invention aims to provide a full-reference quality evaluation method and system, terminal and medium for three-dimensional digital faces.

[0006] To achieve the above-mentioned objectives, the present invention provides a method for evaluating the quality of a three-dimensional digital face, comprising:

[0007] respectively setting a plurality of motion trajectories of the camera that can reduce the amount of data, wherein the plurality of motion trajectories can all cover the viewpoints of the three-dimensional digital face;

[0008] Calculating and determining the centers of the plurality of motion tracks;

[0009] Using the camera to move along a plurality of motion tracks, a video sequence of a distorted three-dimensional digital face and an undistorted reference three-dimensional face is captured;

[0010] The collected distorted 3D digital face video sequence and the undistorted reference 3D face video sequence are respectively extracted at fixed intervals and spliced ​​to obtain a frame sequence;

[0011] The frame sequences of the distorted face and the reference face are sequentially input into the trained network for feature extraction to obtain the extracted features of the distorted face and the extracted features of the reference face;

[0012] The Euclidean distance between the distorted face extraction features and the reference face extraction features is calculated and used as an evaluation index of the quality of the three-dimensional digital face, thereby obtaining a quality evaluation of the three-dimensional digital face.

[0013] Preferably, the plurality of motion tracks of the camera are three circular tracks for video capture, and the track equations are:

[0014] Circle A:

[0015] Circle B:

[0016] Circle C:

[0017] Among them, (x cam ,y cam ,z cam ) is the position coordinate of the camera, Circle A, Circle B and Circle C are the three corresponding trajectory equations, and r is the radius of the track.

[0018] Preferably, the calculation and determination of the centers of the three motion tracks, i.e., the determination of the center of the three-dimensional digital face,

[0019]

[0020] Among them, the three-dimensional digital face is composed of points, edges, and surfaces. i Represents the three-dimensional coordinates of the i-th point (x i ,y i ,z i ), N represents the total number of 3D digital face locations, C P Indicates the center coordinates of a 3D digital face.

[0021] Preferably, the camera is moved along a plurality of motion tracks to capture video sequences of the distorted three-dimensional digital face and the undistorted reference three-dimensional face, comprising:

[0022] Use the 3D visualization open source library to collect video sequences of distorted 3D digital faces and undistorted reference 3D faces;

[0023] Obtain the corresponding six video sequences FV according to the three camera tracks A_dis , FV B_dis , FV c_dis , FV A_ref , FV B _ ref and FV c _ ref .

[0024] Preferably, the collected distorted three-dimensional digital face video sequence and the undistorted reference three-dimensional face video sequence are respectively extracted at fixed intervals and spliced ​​to obtain a frame sequence, that is, the distorted three-dimensional digital face video sequence FV A_dis , FV B_dis and FV c_dis The frames are extracted at equal intervals and spliced ​​to obtain the frame sequence of distorted human face. The reference 3D digital face video sequence FV A_ref , FV B_ref and FV c_ref Frames are sampled at equal intervals and stitched together to obtain a frame sequence of the reference face.

[0025] Preferably, the pre-trained ResNet3D is used to extract features from the frame sequences of the distorted face and the reference face to obtain the feature W dis and W ref .

[0026] Preferably, the Euclidean distance of features extracted from the frame sequence of the distorted face and the reference face is calculated as an evaluation index of the quality of the three-dimensional digital face:

[0027]

[0028] Among them, W dis and W yef They represent the features extracted from the distorted face and the reference face frame sequence respectively. Loss is the Euclidean distance between the two features, which is used as an indicator for evaluating the quality of three-dimensional digital faces.

[0029] According to a second aspect of the present invention, a quality assessment system for a three-dimensional digital face is provided, comprising:

[0030] A track module, wherein the track module sets a plurality of motion tracks of the camera that can reduce the amount of data, and the plurality of motion tracks can all cover the viewpoints of the three-dimensional digital face;

[0031] a center module, the center module calculating and determining the centers of the plurality of motion tracks;

[0032] an acquisition module, wherein the acquisition module uses the camera to move along a plurality of motion tracks to acquire video sequences of the distorted three-dimensional digital face and the undistorted reference three-dimensional face;

[0033] A frame extraction module extracts frames at fixed intervals from the collected distorted three-dimensional digital face video sequence and the undistorted reference three-dimensional face video sequence, and then splices them together to obtain a frame sequence;

[0034] A feature extraction module, wherein the feature extraction module sequentially inputs the frame sequences of the distorted face and the reference face into the trained network to perform feature extraction, thereby obtaining extracted features of the distorted face and extracted features of the reference face;

[0035] An evaluation module calculates the Euclidean distance between the distorted face extraction features and the reference face extraction features, and uses the distance as an evaluation index of the quality of the three-dimensional digital face, thereby obtaining a quality evaluation of the three-dimensional digital face.

[0036] According to a third aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute the above-mentioned method or to execute the above-mentioned system when executing the program.

[0037] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform the above-mentioned method, or to run the above-mentioned system.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The full-reference quality assessment method for 3D faces provided by this embodiment of the present invention can capture a video sequence of a 3D digital face using a camera and convert the 3D model into a 2D projection, thus addressing the difficulty of directly processing the complex structure of 3D digital face data. Furthermore, this embodiment of the present invention utilizes Open3D and a pre-trained ResNet3D network to efficiently and accurately assess the visual quality of distorted 3D faces. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0041] Figure 1 This is a flow chart of a full-reference quality assessment method for three-dimensional digital faces according to an embodiment of the present invention.

[0042] Figure 2 3D digital face video sequence acquisition diagram according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several variations and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0044] Reference Figure 1 The figure shows a flow chart of a three-dimensional digital face quality assessment method according to an embodiment of the present invention. The three-dimensional digital face image quality assessment method is a full-reference quality assessment method based on two-dimensional projection. The method can be performed according to the following steps:

[0045] S100, respectively setting a plurality of motion trajectories of the camera that can reduce the amount of data while fully acquiring the viewpoints of the three-dimensional digital face, wherein the plurality of motion trajectories can all cover the viewpoints of the three-dimensional digital face;

[0046] S200, calculating and determining the centers of the plurality of motion tracks in S100;

[0047] S300, the camera moves along a plurality of motion tracks to capture video sequences of the distorted 3D digital face and the undistorted reference 3D face;

[0048] S400, extracting frames at fixed intervals from the distorted 3D digital face video sequence and the undistorted reference 3D face video sequence collected in S300, and splicing them to obtain a frame sequence;

[0049] S500, inputting the frame sequences of the distorted face and the reference face in S400 into the trained network in sequence for feature extraction, thereby obtaining extracted features of the distorted face and extracted features of the reference face;

[0050] S600 , calculating the Euclidean distance between the distorted face extraction features and the reference face extraction features in S500 , and using the distance as an evaluation index of the quality of the three-dimensional digital face, thereby obtaining a quality evaluation of the three-dimensional digital face.

[0051] In this embodiment, both the video sequence acquisition and facial feature acquisition processes are relatively fast. Euclidean distance is used to characterize the similarity between the original digital face and the distorted digital face video sequence. Higher similarity indicates higher quality, while lower similarity indicates lower quality. Therefore, this embodiment can quickly and effectively evaluate the visual quality of 3D digital faces, which is of great significance to the development of 3D digital face acquisition and evaluation technology.

[0052] In a preferred embodiment of the present invention, S100 is implemented. In order to reduce the amount of data collected by the video sequence, three motion tracks of the camera are set respectively to cover the viewpoints of the three-dimensional digital face as much as possible. Specifically, three circular tracks are selected for video capture, see Figure 2 As shown, the trajectory equation is as follows:

[0053] Circle A:

[0054] Circle B:

[0055] Circle C:

[0056] Among them, (x cam ,y cam ,z cam ) is the camera's position coordinate, Circle A, Circle B, and Circle C are the three corresponding trajectory equations, and r is the radius of the trajectory. This embodiment considers both data volume and quality evaluation, and the optimal selection of three trajectories is a trade-off that achieves optimal results for both.

[0057] In a preferred embodiment of the present invention, S200 is implemented to calculate and determine the center of the camera track in order to keep the three-dimensional digital face always at the center of the camera lens and at the same distance from the camera during the entire video sequence acquisition process. Specifically, the calculation formula is as follows:

[0058]

[0059] Among them, the three-dimensional digital face is composed of points, edges, and surfaces. i Represents the three-dimensional coordinates of the i-th point (x i ,y i ,z i ), N represents the total number of 3D digital face locations, C P Indicates the center coordinates of a 3D digital face.

[0060] This embodiment can ensure that the three-dimensional digital face is always in the center of the lens and cannot be offset; it can also ensure that the distance between the three-dimensional digital face and the lens is always equal. In a preferred embodiment of the present invention, S300 is implemented. Since Open3D is an open source library that supports rapid development and processing of 3D data and can provide GPU-accelerated core 3D operations and physically based rendering for the collected video sequences, we choose to use Open3D in Python to collect video sequences of distorted and reference three-dimensional digital faces, and obtain six corresponding video sequences FV according to the three camera tracks. A_dis , FV B_dis , FVc_dis , FV A_ref , FV B_ref and FV c_ref , which realizes the transformation of 3D digital face into 2D projection. OpenGL can also be used.

[0061] In a preferred embodiment, the camera rotation step size is selected to ensure the consistency of the video sequence, and the number of video sequence frames is calculated to achieve the conversion of three-dimensional digital faces into two-dimensional ones.

[0062] It should be noted that the present invention's research object is three-dimensional digital faces, which is novel. Currently, data sets related to digital faces are relatively scarce. Without data, it is difficult to carry out related work. The above-mentioned embodiments of the present invention primarily rely on cameras to capture multi-angle video sequences, resulting in the presence of video data formats throughout the entire data processing process of the present invention. In other words, the data format change process can be described as: three-dimensional face (polygonal mesh) - video - image (two-dimensional projection).

[0063] In a preferred embodiment of the present invention, S400 is implemented to process the distorted three-dimensional digital face video sequence FV A_dis , FV B_dis and FV c_dis The frame sequence of the distorted face is obtained by sampling frames at equal intervals and splicing them together to remove redundant video information and further reduce the amount of data. The frame sampling interval can generally be set to 7 frames. Similarly, for the reference 3D digital face video sequence FV A_ref , FV B_ref and FV c_ref Frames are sampled at equal intervals and stitched together to obtain a frame sequence of the reference face.

[0064] The 3D face model itself has a large amount of data and is not easy to study directly. The above embodiment is based on a method of 2D projection, frame extraction, frame skipping and other methods to reduce the amount of data and achieves the expected accuracy.

[0065] In a preferred embodiment of the present invention, S500 is implemented to quickly extract features from the frame sequences of the distorted face and the reference face using the pre-trained ResNet3D to obtain the feature W tar and W ref .

[0066] In this embodiment, pre-trained ResNet3D is used for feature extraction. In addition to the ResNet3D network, there are other networks that can realize feature extraction. This embodiment uses the ResNet3D network, and it also takes into account that it is a relatively classic network structure. The structural characteristics of the ResNet model allow it to train deeper networks, thereby achieving better results. ResNet3D is a ResNet with a three-dimensional spatiotemporal kernel, and it also inherits the advantage of ResNet that can train deeper networks. In addition, experiments have shown that ResNet3D has good performance in various video tasks. The digital face evaluation task in this application is also a video task to some extent. Therefore, in this embodiment, ResNet3D is used for feature extraction to obtain better results.

[0067] In a preferred embodiment of the present invention, S600 is implemented to calculate the Euclidean distance between features extracted from the distorted face and the reference face. This distance is used to characterize the similarity between the original digital face and the distorted digital face video sequence and serves as an evaluation indicator for the quality of the 3D digital face. Specifically, the calculation formula is as follows:

[0068]

[0069] Among them, W dis and W ref The similarity represents the features extracted from the distorted face and the reference face, respectively. Loss is the Euclidean distance between the two features, which is used as a 3D digital face quality evaluation indicator to obtain the 3D digital face quality evaluation quality. The higher the similarity, the higher the quality; the lower the similarity, the lower the quality.

[0070] In summary, the above-described embodiments of the present invention devise a novel method for acquiring a difficult-to-obtain digital face dataset, employing two-dimensional projection to ensure data accuracy while reducing data volume. Ultimately, quality evaluation is performed through accurate feature extraction and novel evaluation metrics. This overcomes the shortcomings of existing technologies, such as dataset scarcity, large data volumes, and the lack of quantifiable evaluation, while achieving highly accurate evaluations.

[0071] Based on the same inventive concept, in another embodiment of the present invention, a quality evaluation system for a three-dimensional digital face is provided, comprising a track module, a center module, an acquisition module, a frame extraction module, a feature extraction module, and an evaluation module;

[0072] The track module sets several motion tracks of the camera respectively, and the motion tracks can cover the viewpoint of the three-dimensional digital face; the center module calculates and determines the centers of several motion tracks; the acquisition module uses the camera to collect video sequences of the distorted three-dimensional digital face and the undistorted reference three-dimensional face; the frame extraction module extracts frames at fixed intervals from the collected distorted three-dimensional digital face video sequence and the undistorted reference three-dimensional face video sequence and splices them to obtain frame sequences; the feature extraction module inputs the frame sequences of the distorted face and the reference face into the trained ResNet3D network in sequence for feature extraction, and obtains the distorted face extraction features and the reference face extraction features; the evaluation module calculates the Euclidean distance between the distorted face extraction features and the reference face extraction features, and uses it as the evaluation index of the three-dimensional digital face quality, thereby obtaining the quality evaluation of the three-dimensional digital face.

[0073] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the terminal can be used to execute the above-mentioned method, or to run the above-mentioned system.

[0074] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute the above method or run the above system.

[0075] The preferred features of the above embodiments can be used alone in any embodiment, or in any combination without conflict. In addition, parts not described in detail in the embodiments can be implemented using existing technologies.

[0076] To better illustrate the above evaluation method, specific examples are provided below for detailed description. It should be understood that the parameters in the following examples are only partial embodiments of the present invention and are not intended to limit the present invention. The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various variations or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A quality assessment method for three-dimensional digital faces, characterized in that: include: respectively setting a plurality of motion trajectories of the camera that can reduce the amount of data, wherein the plurality of motion trajectories can all cover the viewpoints of the three-dimensional digital face; Calculating and determining the centers of the plurality of motion tracks; Using the camera to move along a plurality of motion tracks, a video sequence of a distorted three-dimensional digital face and an undistorted reference three-dimensional face is captured; The collected distorted 3D digital face video sequence and the undistorted reference 3D face video sequence are respectively extracted at fixed intervals and spliced ​​to obtain a frame sequence; The frame sequences of the distorted face and the reference face are sequentially input into the trained network for feature extraction to obtain the extracted features of the distorted face and the extracted features of the reference face; Calculating the Euclidean distance between the extracted features of the distorted human face and the extracted features of the reference face, and using the distance as an evaluation index of the quality of the three-dimensional digital face, thereby obtaining a quality evaluation of the three-dimensional digital face; The several motion tracks of the camera are three circular tracks for video capture, and the track equations are: Among them, (x cam ,y cam ,z cam ) is the position coordinate of the camera, Circle A, Circle B and Circle C are the three corresponding trajectory equations, and r is the radius of the trajectory; The calculation and determination of the centers of the three motion tracks is to determine the center of the three-dimensional digital face. Among them, the three-dimensional digital face is composed of points, edges, and surfaces. i Represents the three-dimensional coordinates of the i-th point (x i ,y i ,z i ), N represents the number of polygonal meshes that make up the three-dimensional digital face, C P Indicates the center coordinates of a 3D digital face.

2. The quality assessment method for three-dimensional digital faces according to claim 1, characterized in that: The camera is moved along a plurality of motion tracks to capture video sequences of a distorted three-dimensional digital face and an undistorted reference three-dimensional face, including: Use the 3D visualization open source library to collect video sequences of distorted 3D digital faces and undistorted reference 3D faces; Obtain the corresponding six video sequences FV according to the three camera tracks A_dis , FV B_dis , FV c_dis , FV A_ref , FV B_ref and FV c_ref .

3. The quality assessment method for three-dimensional digital faces according to claim 2, characterized in that: The collected distorted 3D digital face video sequence and the undistorted reference 3D face video sequence are respectively extracted at fixed intervals and spliced ​​to obtain a frame sequence, which is the distorted 3D digital face video sequence FV A_dis , FV B_dis and FV c_dis The frames are extracted at equal intervals and spliced ​​to obtain the frame sequence of distorted human face. The reference 3D digital face video sequence FV A_ref , FV B_ref and FV c_ref Frames are sampled at equal intervals and stitched together to obtain a frame sequence of the reference face.

4. The quality assessment method for three-dimensional digital faces according to claim 3, characterized in that: Use the pre-trained ResNet3D to extract features from the frame sequences of the distorted face and the reference face, and obtain the feature W dis and W ref .

5. The quality assessment method for three-dimensional digital faces according to claim 4, characterized in that: Calculate the Euclidean distance of features extracted from the frame sequence of the distorted face and the reference face as an evaluation indicator of the 3D digital face quality: Among them, W dis and W ref They represent the features extracted from the distorted face and the reference face frame sequence respectively. Loss is the Euclidean distance between the two features, which is used as an indicator for evaluating the quality of three-dimensional digital faces.

6. A quality evaluation system for three-dimensional digital faces, characterized in that: include: A track module, wherein the track module sets a plurality of motion tracks of the camera that can reduce the amount of data, and the plurality of motion tracks can all cover the viewpoints of the three-dimensional digital face; a center module, the center module calculating and determining the centers of the plurality of motion tracks; an acquisition module, wherein the acquisition module uses the camera to move along a plurality of motion tracks to acquire video sequences of the distorted three-dimensional digital face and the undistorted reference three-dimensional face; A frame extraction module extracts frames at fixed intervals from the collected distorted three-dimensional digital face video sequence and the undistorted reference three-dimensional face video sequence, and then splices them together to obtain a frame sequence; A feature extraction module, wherein the feature extraction module sequentially inputs the frame sequences of the distorted face and the reference face into the trained network to perform feature extraction, thereby obtaining extracted features of the distorted face and extracted features of the reference face; An evaluation module, wherein the evaluation module calculates the Euclidean distance between the extracted features of the distorted human face and the extracted features of the reference human face, and uses the distance as an evaluation index of the quality of the three-dimensional digital face, thereby obtaining a quality evaluation of the three-dimensional digital face; The several motion tracks of the camera are three circular tracks for video capture, and the track equations are: Among them, (x cam ,y cam ,z cam ) is the position coordinate of the camera, Circle A, Circle B and Circle C are the three corresponding trajectory equations, and r is the radius of the trajectory; The calculation and determination of the centers of the three motion tracks is to determine the center of the three-dimensional digital face. Among them, the three-dimensional digital face is composed of points, edges, and surfaces. i Represents the three-dimensional coordinates of the i-th point (x i ,y i ,z i ), N represents the number of polygonal meshes that make up the three-dimensional digital face, C P Indicates the center coordinates of a 3D digital face.

7. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it can be used to perform the method according to any one of claims 1 to 5, or to run the system according to claim 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to perform the method described in any one of claims 1 to 5, or to run the system described in claim 6.

Images