Pupil height and interpupillary distance measurement method, device and equipment considering prior and medium
By combining the local binary pattern histogram algorithm and the two-branch task neural network with Kalman filtering, the accuracy and consistency issues of pupil height and pupil distance measurement were solved, and high-precision pupil height and pupil distance measurement was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-23
- Publication Date
- 2026-03-13
AI Technical Summary
Existing methods for measuring pupil height and pupil distance suffer from low efficiency, inconsistent and inaccurate results. Furthermore, existing deep learning models ignore the three-dimensional structure of the face, leading to low measurement accuracy. There is no solution for simultaneously measuring pupil height and pupil distance with high precision.
By acquiring multiple frames of face images, texture features are extracted using the local binary pattern histogram algorithm. Combined with a two-branch task neural network model and Kalman filtering, parameter prediction and optimization fusion of pupil height and pupil distance are performed, taking into account the physical constraints of face symmetry and dynamic optimization.
It improves the measurement accuracy of pupil height and pupil distance, realizes simultaneous high-precision measurement, and reduces subjective errors in face image measurement and instability in model prediction.
Smart Images
Figure CN121647592A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of measuring parameters in eyeglass fitting processes, and in particular to a method, apparatus, equipment, and medium for measuring pupillary height and pupillary distance that takes into account prior knowledge. Background Technology
[0002] In the process of optometry and eyeglass fitting, the measurement of parameters such as pupillary height and pupillary distance is a core step, directly affecting the fit of the manufactured glasses to the user. Currently, traditional methods of measuring pupillary height and pupillary distance mainly rely on optometrists using tools such as pupillary distance rulers and pupillometers for manual operation. The measurement steps are cumbersome, and pupillary height and pupillary distance measurements must be performed in batches, making simultaneous measurements impossible. This method is highly dependent on the operator's skill and patience, and the measurement results are easily affected by the operator's experience, the cooperation of the person being measured, and many subjective factors, resulting in problems such as poor efficiency, inconsistency, and inaccuracy.
[0003] Existing technologies include some schemes that utilize deep learning models to measure pupillary distance (PD). These models are trained using extensive refraction data, aiming to obtain the PD directly from the subject's facial image. While this method has proven to be more accurate than traditional manual measurement methods, it still suffers from several limitations: it only considers fitting and measuring PD using two-dimensional images, neglecting the three-dimensional structure of the face and losing 3D information such as facial depth, leading to a discrepancy between the actual measurement accuracy and the expected result. Furthermore, existing technologies lack schemes for measuring pupillary height using deep learning models, and also lack schemes for simultaneously measuring both pupillary height and PD. High-precision measurement schemes for pupillary height remain scarce. Therefore, how to simultaneously perform high-precision measurement of both pupillary height and PD remains a pressing technical problem to be solved in current technologies. Summary of the Invention
[0004] This application provides a method, apparatus, device, and medium for measuring pupil height and pupil distance that takes into account prior art, in order to solve the technical problem that the prior art does not provide a technical solution for simultaneously measuring pupil height and pupil distance with high precision.
[0005] According to a first aspect of the embodiments of this application, a method for measuring pupil height and pupillary distance considering prior knowledge is provided, comprising: The system acquires multiple frames of a user's face images and extracts texture features from each frame of the face image sequentially based on the local binary mode histogram algorithm, thereby obtaining the texture features corresponding to each frame of the face image; wherein, the texture features are used to provide prior information for the corresponding face image. Each frame of face image is sequentially combined with its corresponding texture features and then input into a preset parameter measurement model for parameter prediction, outputting a set of predicted parameters for each frame of face image; wherein, the parameter measurement model is based on a two-branch task neural network model and is trained by using the physical constraints of face symmetry as loss; the set of predicted parameters includes a set of predicted pupil center and a set of predicted frame key points; Based on Kalman filtering, the prediction parameter group corresponding to each frame of face image is dynamically optimized and fused to obtain the user's first pupil height and first pupil distance.
[0006] This application first acquires multiple frames of the user's face images and extracts texture features based on the Local Binary Pattern Histogram algorithm. This provides prior information on texture for subsequent parameter prediction and measurement, thereby improving the accuracy of the predicted parameter set obtained through the parameter measurement model, and thus improving the accuracy of the first pupil height and first pupil distance. Next, each frame of face image and its corresponding texture features are combined and input into a parameter measurement model based on a two-branch task neural network model for parameter prediction. This allows for simultaneous prediction and measurement of pupil height and pupil distance. Furthermore, the parameter measurement model is trained using the physical constraints of face symmetry as a loss, which avoids making inferences that violate inherent face constraints during model training and prediction, thereby improving model stability. Finally, Kalman filtering is used for further dynamic optimization and fusion of the face, which further improves prediction accuracy, thus enhancing the accuracy of the obtained first pupil height and first pupil distance.
[0007] In some embodiments of this application, acquiring multiple frames of the user's face images specifically includes: In response to the user's action in response to the preset image acquisition prompt, the user's face is detected. When the user's face is detected to meet the preset acquisition conditions, the user is continuously photographed at high speed to obtain multiple initial face images. For each initial face image frame, sharpness and illumination uniformity are evaluated to obtain the sharpness and illumination uniformity corresponding to each initial face image frame. Based on the sharpness and illumination uniformity of each initial face image, the multiple initial face images are filtered to obtain multiple face images.
[0008] This application first responds to the user's feedback action on the image acquisition prompt by detecting the user's face. When the detection meets the acquisition conditions, multiple initial face images are acquired. Then, the clarity and illumination uniformity of the initial face images are evaluated and filtered to obtain multiple face images that meet the measurement standards. The matching degree between the face images and the current task is further improved through filtering, thereby improving the accuracy of subsequent parameter prediction measurements based on the face images.
[0009] In some embodiments of this application, the step of extracting texture features from each frame of a face image sequentially using the local binary pattern histogram algorithm to obtain the texture features corresponding to each frame of the face image specifically includes: Each frame of the face image is converted to grayscale and smoothed using Gaussian to obtain the corresponding first face image; At multiple preset scales, histogram features are extracted from the texture of the first face image in each frame based on a preset local binary mode operator, so as to obtain the texture features corresponding to the first face image in each frame.
[0010] This application first performs grayscale conversion and Gaussian smoothing on the face image to obtain the first face image, which can enhance the texture of the image and reduce noise interference. Then, based on the local binary mode operator, histogram feature extraction is performed on the texture of the first face image at multiple scales to obtain the corresponding texture features. It can simultaneously predict texture features at multiple scales. Combined with the high-precision texture feature extraction of the local binary mode operator, the accuracy of subsequent parameter prediction and measurement is improved.
[0011] In some embodiments of this application, the step of extracting histogram features from the texture of each frame of the first face image based on a preset local binary pattern operator at multiple preset scales to obtain the texture features corresponding to each frame of the first face image specifically includes: Based on a preset partitioning pattern, the first face image in each frame is divided into the same regions to obtain multiple sets of partitioned images; wherein, each set of partitioned images is divided into multiple non-overlapping partitioned grid regions based on the preset partitioning pattern. At the multiple scales, based on the local binary mode operator, local binary mode histograms are calculated sequentially for each grid region in each group of segmented images to obtain region histograms at multiple scales for each grid region in each group of segmented images. At the multiple scales, the histograms of regions of the same scale in all the grid regions of each group of segmented images are concatenated to obtain multiple single-scale features corresponding to each group of segmented images. The multiple single-scale features of each group of segmented images are then stacked to obtain the texture features corresponding to each group of segmented images.
[0012] This application first divides the first face image into multiple grid regions. Then, at multiple scales, it sequentially calculates the local binary pattern histogram for each grid region using the local binary pattern operator to obtain the region histogram for each scale of each grid region. Subsequently, it concatenates the region histograms of each grid region at the same scale to obtain single-scale features. Then, it stacks the single-scale features at multiple scales to obtain the corresponding texture features. It can improve the accuracy of subsequent parameter prediction and measurement by using the high-precision texture feature extraction through the local binary pattern operator and combining it with the multi-scale texture features to provide multiple prediction scales.
[0013] In some embodiments of this application, the parameter measurement model includes an input mapping layer, a modulation fusion layer, a dual-task decoding layer, and an output mapping layer; the step of sequentially combining each frame of face image with its corresponding texture features and inputting the result into a preset parameter measurement model for parameter prediction, and outputting a set of predicted parameters corresponding to each frame of face image, specifically includes: The input mapping layer performs image encoding mapping on each frame of face image to obtain the first image feature corresponding to each frame of face image; Through the modulation and fusion layer, each first image feature and its corresponding texture feature are modulated and fused to obtain the texture fusion feature corresponding to each frame of face image; Through the dual-task decoding layer, a two-branch task prediction is performed on each texture fusion feature to obtain the pupil center feature and the frame key point feature corresponding to each texture fusion feature. The output mapping layer decodes and maps the pupil center features and frame key point features corresponding to each frame of face image to obtain the prediction parameter set corresponding to each frame of face image.
[0014] This application first performs image encoding mapping on each frame of face image through an input mapping layer to obtain the first image feature. Then, it modulates and fuses the first image feature with the corresponding texture feature through a modulation fusion layer to obtain the texture fusion feature. Next, it performs a two-branch task prediction on the texture fusion feature through a dual-task decoding layer to obtain the corresponding pupil center feature and frame key point feature. Finally, it performs decoding mapping through an output mapping layer to obtain the prediction parameter set. It can fully utilize the prior of texture feature through modulation fusion to improve the accuracy of the subsequently obtained prediction parameter set, thereby improving the accuracy of the first pupil height and first pupil distance. Furthermore, through the two-branch task prediction of the dual-task decoding layer, it realizes the task of simultaneously measuring pupil height and pupil distance, which has not been implemented in the past.
[0015] In some embodiments of this application, the step of performing dynamic face optimization and fusion on the prediction parameter group corresponding to each frame of face image based on Kalman filtering to obtain the user's first pupil height and first pupil distance specifically includes: Based on the prediction parameter group corresponding to each frame of face image, prediction optimization is performed based on Kalman filtering to obtain the pupil center coordinates and key point coordinates of the frame of the user. Based on the user's pupil center coordinates and the coordinates of key points on the frame, the user's first pupil height and first pupillary distance are calculated.
[0016] This application first uses the prediction parameter set of each frame of face image to perform prediction optimization based on Kalman filtering to obtain the user's pupil center coordinates and frame key point coordinates, and then calculates the user's first pupil height and first pupil distance. It can reasonably predict the coordinates to be calculated through Kalman filtering, thereby providing a data basis for subsequent calculation of the first pupil height and first pupil distance and improving its calculation accuracy.
[0017] In some embodiments of this application, the step of performing prediction optimization based on Kalman filtering according to the prediction parameter set corresponding to each frame of face image to obtain the user's pupil center coordinates and frame key point coordinates specifically includes: Based on the prediction parameter group corresponding to each frame of the face image, the first prediction trajectory of the user's pupil center and the second prediction trajectory of each frame key point are obtained; The first predicted trajectory and the second predicted trajectory of each key point of the frame are optimized by using a Kalman filter algorithm based on a motion model to obtain the coordinates of the user's pupil center and the coordinates of the key points of the frame.
[0018] This application first obtains the first predicted trajectory of the user's pupil center and the second predicted trajectory of each frame key point based on the prediction parameter group of each frame of the face image. Then, it optimizes the user's pupil center coordinates and frame key point coordinates by using a motion model-based Kalman filter algorithm. This algorithm can take into account the coordinate movement error caused by facial micro-movements and correct it by using motion model-based Kalman filtering, thereby improving the accuracy of subsequent optimization predictions and thus improving the accuracy of the obtained user's pupil center coordinates and frame key point coordinates.
[0019] According to a second aspect of the embodiments of this application, a pupil height and pupil distance measurement device considering prior knowledge is provided, including an image acquisition and texture extraction module, a face image parameter prediction and measurement module, and a prediction parameter optimization and fusion module. The image acquisition and texture extraction module is used to acquire multiple frames of the user's face images, and based on the local binary mode histogram algorithm, extract texture features from each frame of the face image in turn to obtain the texture features corresponding to each frame of the face image; wherein, the texture features are used to provide prior information for the corresponding face image; The face image parameter prediction and measurement module is used to sequentially combine each frame of face image with the corresponding texture features and input them into a preset parameter measurement model for parameter prediction, and output the prediction parameter set corresponding to each frame of face image; wherein, the parameter measurement model is based on a two-branch task neural network model and is trained by using the physical constraints of face symmetry as loss; the prediction parameter set includes the prediction pupil center and the prediction frame key point set; The prediction parameter optimization and fusion module is used to perform dynamic face optimization and fusion on the prediction parameter group corresponding to each frame of face image based on Kalman filtering, so as to obtain the user's first pupil height and first pupil distance.
[0020] In some embodiments of this application, the image acquisition and texture extraction module includes an image acquisition unit, an image evaluation unit, and an image filtering unit; The image acquisition unit is used to detect the user's face in response to the user's action in response to the preset image acquisition prompt. When the user's face is detected to meet the preset acquisition conditions, the unit performs continuous high-speed photography of the user to obtain multiple initial face images. The image evaluation unit is used to evaluate the sharpness and illumination uniformity of each initial face image to obtain the sharpness and illumination uniformity of each initial face image. The image filtering unit is used to filter the multiple initial face images according to the clarity and illumination uniformity corresponding to each initial face image to obtain multiple face images.
[0021] In some embodiments of this application, the image acquisition and texture extraction module includes an image processing unit and a texture extraction unit; The image processing unit is used to perform grayscale conversion and Gaussian smoothing on each frame of face image to obtain the corresponding first face image; The texture extraction unit is used to extract histogram features of the texture of each frame of the first face image at multiple preset scales based on a preset local binary mode operator, so as to obtain the texture features corresponding to each frame of the first face image.
[0022] In some embodiments of this application, the texture extraction unit includes a region partitioning subunit, a region calculation subunit, and a texture extraction subunit; The region division subunit is used to perform the same region division on each frame of the first face image based on a preset division mode to obtain multiple sets of divided images; wherein, each set of divided images is divided into multiple non-overlapping division grid regions based on the preset division mode. The region calculation subunit is used to perform local binary mode histogram calculation on each grid region in each group of segmented images at the multiple scales based on the local binary mode operator, so as to obtain a region histogram at multiple scales for each grid region in each group of segmented images. The texture extraction subunit is used to concatenate the region histograms of the same scale in all the partitioned grid regions of each group of partitioned images at the multiple scales to obtain multiple single-scale features corresponding to each group of partitioned images, and to stack the multiple single-scale features of each group of partitioned images to obtain the texture features corresponding to each group of partitioned images.
[0023] In some embodiments of this application, the parameter measurement model is provided with an input mapping layer, a modulation fusion layer, a dual-task decoding layer, and an output mapping layer; the face image parameter prediction and measurement module includes an input mapping unit, a modulation fusion unit, a decoding prediction unit, and an output mapping unit. The input mapping unit is used to perform image encoding mapping on each frame of face image through the input mapping layer to obtain the first image feature corresponding to each frame of face image; The modulation fusion unit is used to modulate and fuse each first image feature with the corresponding texture feature through the modulation fusion layer to obtain the texture fusion feature corresponding to each frame of face image; The decoding prediction unit is used to perform a two-branch task prediction on each texture fusion feature through the dual-task decoding layer to obtain the pupil center feature and frame key point feature corresponding to each texture fusion feature. The output mapping unit is used to decode and map the pupil center features and frame key point features corresponding to each frame of face image through the output mapping layer to obtain the prediction parameter set corresponding to each frame of face image.
[0024] In some embodiments of this application, the prediction parameter optimization and fusion module includes a filtering prediction unit and a parameter calculation unit; The filtering prediction unit is used to perform prediction optimization based on Kalman filtering according to the prediction parameter group corresponding to each frame of face image, so as to obtain the pupil center coordinates and the key point coordinates of the frame of the user. The parameter calculation unit is used to calculate the user's first pupil height and first pupillary distance based on the user's pupil center coordinates and the coordinates of key points in the frame.
[0025] In some embodiments of this application, the filtering prediction unit includes a trajectory combination subunit and a filtering prediction subunit; The trajectory combination subunit is used to obtain the first predicted trajectory of the user's pupil center and the second predicted trajectory of each frame key point based on the prediction parameter group corresponding to each frame of the face image. The filtering prediction subunit is used to optimize the first predicted trajectory and the second predicted trajectory of each frame key point using a Kalman filter algorithm based on a motion model, so as to obtain the user's pupil center coordinates and frame key point coordinates.
[0026] This application first acquires multiple frames of the user's face images and extracts texture features based on the Local Binary Pattern Histogram algorithm. This provides prior information on texture for subsequent parameter prediction and measurement, thereby improving the accuracy of the predicted parameter set obtained through the parameter measurement model, and thus improving the accuracy of the first pupil height and first pupil distance. Next, each frame of face image and its corresponding texture features are combined and input into a parameter measurement model based on a two-branch task neural network model for parameter prediction. This allows for simultaneous prediction and measurement of pupil height and pupil distance. Furthermore, the parameter measurement model is trained using the physical constraints of face symmetry as a loss, which avoids making inferences that violate inherent face constraints during model training and prediction, thereby improving model stability. Finally, Kalman filtering is used for further dynamic optimization and fusion of the face, which further improves prediction accuracy, thus enhancing the accuracy of the obtained first pupil height and first pupil distance.
[0027] According to a third aspect of the embodiments of this application, a computer device is provided, comprising: a processor; a memory; and a computer program stored in the memory and configured to be executed by the processor; wherein the processor executes the computer program to implement a priori pupillary height and pupillary distance measurement method as described in this application.
[0028] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute a priori pupil height and pupillary distance measurement method of this application. Attached Figure Description
[0029] Figure 1 This is a schematic flowchart illustrating a method for measuring pupil height and pupil distance that takes into account prior knowledge, as shown in certain embodiments of this application. Figure 2 This is a network structure diagram of a parameter measurement model shown in some embodiments of this application; Figure 3 This is a block diagram of a pupil height and pupil distance measuring device that takes a priori considerations into account, as shown in some embodiments of this application. Detailed Implementation
[0030] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below in conjunction with the accompanying drawings are exemplary and are only used to explain some embodiments of this application, and should not be construed as limiting the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments shown in this application without inventive effort are within the protection scope of this application.
[0031] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, unless otherwise explicitly specified, "a plurality of" or "several" means two or more.
[0032] Existing technologies include some schemes that utilize deep learning models to measure pupillary distance (PD). These models are trained using extensive refraction data, aiming to obtain the PD directly from the subject's facial image. While this method has proven to be more accurate than traditional manual measurement methods, it still suffers from several limitations: it only considers fitting and measuring PD using two-dimensional images, neglecting the three-dimensional structure of the face and losing 3D information such as facial depth, leading to a discrepancy between the actual measurement accuracy and the expected result. Furthermore, existing technologies lack schemes for measuring pupillary height using deep learning models, and also lack schemes for simultaneously measuring both pupillary height and PD. High-precision measurement schemes for pupillary height remain scarce. Therefore, how to simultaneously perform high-precision measurement of both pupillary height and PD remains a pressing technical problem to be solved in current technologies.
[0033] Based on the above technical background, please refer to Figure 1 This application provides a method for measuring pupil height and pupil distance that takes into account prior knowledge, including steps S101 to S103, each step of which is as follows: Step S101: Acquire multiple frames of the user's face images, and extract texture features from each frame of the face image sequentially based on the local binary mode histogram algorithm to obtain the texture features corresponding to each frame of the face image; wherein, the texture features are used to provide prior information for the corresponding face image.
[0034] In some embodiments of this application, acquiring multiple frames of the user's face images specifically includes: In response to the user's action in response to the preset image acquisition prompt, the user's face is detected. When the user's face is detected to meet the preset acquisition conditions, the user is continuously photographed at high speed to obtain multiple initial face images. For each initial face image frame, sharpness and illumination uniformity are evaluated to obtain the sharpness and illumination uniformity corresponding to each initial face image frame. Based on the sharpness and illumination uniformity of each initial face image, the multiple initial face images are filtered to obtain multiple face images.
[0035] Specifically, the preset image acquisition prompt can be in the form of an audio prompt, or in the form of an image or text prompt. The prompt may include, for example, "Measurement is about to begin, please prepare." This application will simultaneously detect and acquire images of the user's face in conjunction with the preset image acquisition prompt. The user's response should be to keep both eyes open and head facing the camera. The user should wear pre-selected glasses frames without lenses before the acquisition. The preset acquisition conditions may be that both eyes are visible, the head deflection is no greater than a preset angle, and the head tilt is no greater than a preset angle.
[0036] Specifically, when evaluating the sharpness of the initial face image, the Laplacian variance, gray-level variance product, or Brenner gradient can be used for evaluation, with the Laplacian variance being preferred. When evaluating the illumination uniformity of the initial face image, the local brightness variance of each pixel can be calculated, and the global brightness variance can be obtained by averaging the local brightness variances of all pixels. When filtering the initial face image, a threshold method can be used, such that the initial face image must meet a preset threshold in both sharpness and illumination uniformity to be selected, otherwise it will not be selected.
[0037] Specifically, after the initial face image is selected, it needs to be cropped to retain the entire face as much as possible while cropping the image into a square.
[0038] This application first responds to the user's feedback action on the image acquisition prompt by detecting the user's face. When the detection meets the acquisition conditions, multiple initial face images are acquired. Then, the clarity and illumination uniformity of the initial face images are evaluated and filtered to obtain multiple face images that meet the measurement standards. The matching degree between the face images and the current task is further improved through filtering, thereby improving the accuracy of subsequent parameter prediction measurements based on the face images.
[0039] In some embodiments of this application, the step of extracting texture features from each frame of a face image sequentially using the local binary pattern histogram algorithm to obtain the texture features corresponding to each frame of the face image specifically includes: Each frame of the face image is converted to grayscale and smoothed using Gaussian to obtain the corresponding first face image; At multiple preset scales, histogram features are extracted from the texture of the first face image in each frame based on a preset local binary mode operator, so as to obtain the texture features corresponding to the first face image in each frame.
[0040] Specifically, the Local Binary Pattern (LBP) operator can capture textures at a preset scale radius. Therefore, different scales determine different scale radii, which in turn affects the granularity of the captured textures, such as fine-grained textures like eyelid pores and coarse-grained textures like eyeglass frames. The method of extracting histogram features from the texture of the first face image based on the LBP operator in this application is the LBPH (Local Binary Pattern Histogram) face recognition algorithm.
[0041] This application first performs grayscale conversion and Gaussian smoothing on the face image to obtain the first face image, which can enhance the texture of the image and reduce noise interference. Then, based on the local binary mode operator, histogram feature extraction is performed on the texture of the first face image at multiple scales to obtain the corresponding texture features. It can simultaneously predict texture features at multiple scales. Combined with the high-precision texture feature extraction of the local binary mode operator, the accuracy of subsequent parameter prediction and measurement is improved.
[0042] In some embodiments of this application, the step of extracting histogram features from the texture of each frame of the first face image based on a preset local binary pattern operator at multiple preset scales to obtain the texture features corresponding to each frame of the first face image specifically includes: Based on a preset partitioning pattern, the first face image in each frame is divided into the same regions to obtain multiple sets of partitioned images; wherein, each set of partitioned images is divided into multiple non-overlapping partitioned grid regions based on the preset partitioning pattern. At the multiple scales, based on the local binary mode operator, local binary mode histograms are calculated sequentially for each grid region in each group of segmented images to obtain region histograms at multiple scales for each grid region in each group of segmented images. At the multiple scales, the histograms of regions of the same scale in all the grid regions of each group of segmented images are concatenated to obtain multiple single-scale features corresponding to each group of segmented images. The multiple single-scale features of each group of segmented images are then stacked to obtain the texture features corresponding to each group of segmented images.
[0043] Specifically, the division mode refers to dividing the first face image into multiple division grid regions using a square of a preset size, such as a 3×3, 4×4, or 16×16 square. Each division grid region is the same size as the square, and the choice of the square size is related to the cropping of the initial face image after the initial face image is screened.
[0044] This application first divides the first face image into multiple grid regions. Then, at multiple scales, it sequentially calculates the local binary pattern histogram for each grid region using the local binary pattern operator to obtain the region histogram for each scale of each grid region. Subsequently, it concatenates the region histograms of each grid region at the same scale to obtain single-scale features. Then, it stacks the single-scale features at multiple scales to obtain the corresponding texture features. It can improve the accuracy of subsequent parameter prediction and measurement by using the high-precision texture feature extraction through the local binary pattern operator and combining it with the multi-scale texture features to provide multiple prediction scales.
[0045] Step S102: Sequentially combine each frame of face image with the corresponding texture features and input them into a preset parameter measurement model for parameter prediction, and output the prediction parameter set corresponding to each frame of face image; wherein, the parameter measurement model is based on a two-branch task neural network model and is trained by using the physical constraints of face symmetry as loss; the prediction parameter set includes the prediction pupil center and the prediction frame key point set.
[0046] Specifically, please see Figure 2 This is a network structure diagram of a parameter measurement model provided in this application; wherein, the input mapping layer is used to perform feature transformation on the face image; the modulation fusion layer is used to modulate and fuse the transformed image features with the corresponding texture features; the dual-task decoding layer is used to perform two-branch task prediction on the fused features, Task 1 is pupil center prediction, and Task 2 is frame keypoint prediction; the output mapping layer is used to decode the predicted pupil center features and frame keypoint features to obtain the corresponding prediction parameter set.
[0047] In some embodiments of this application, the parameter measurement model includes an input mapping layer, a modulation fusion layer, a dual-task decoding layer, and an output mapping layer; the step of sequentially combining each frame of face image with its corresponding texture features and inputting the result into a preset parameter measurement model for parameter prediction, and outputting a set of predicted parameters corresponding to each frame of face image, specifically includes: The input mapping layer performs image encoding mapping on each frame of face image to obtain the first image feature corresponding to each frame of face image; Through the modulation and fusion layer, each first image feature and its corresponding texture feature are modulated and fused to obtain the texture fusion feature corresponding to each frame of face image; Through the dual-task decoding layer, a two-branch task prediction is performed on each texture fusion feature to obtain the pupil center feature and the frame key point feature corresponding to each texture fusion feature. The output mapping layer decodes and maps the pupil center features and frame key point features corresponding to each frame of face image to obtain the prediction parameter set corresponding to each frame of face image.
[0048] Specifically, modulation fusion can be expressed as: ,in For texture blending features, The first image feature, For texture features, These are activation layers and convolutional layers based on the Sigmoid function, respectively.
[0049] Specifically, the dual-task decoding layer can be constructed by using two convolutional neural networks (CNNs) with the same structure but different weights, one CNN for Task 1 pupil center prediction and the other CNN for Task 2 frame key point prediction.
[0050] Specifically, the model structure of the two-branch task neural network model is the same as that of the parameter measurement model, and the optimizer used during training is the Adam optimizer; the total loss function used is the sum of the mean squared error and the physical constraint error of face symmetry, and the formula for the total loss can be expressed as: ;in, In order, the losses are: mean square error loss, pupil distance symmetry error loss, and pupil height symmetry error loss; pupil distance symmetry error loss Pupil height symmetry error loss , The coordinates of the left pupil center and the right pupil center are shown in the following figures, respectively. Let x be the x-coordinate of the midpoint of the bridge of the nose. The lowest points are, in order, the lowest points of the lower edges of the left and right frames. More specifically, by using physical constraints related to facial symmetry—specifically, the interpupillary distance symmetry error loss and the interpupillary height symmetry error loss—the model's predictions during training can be limited to common inherent constraints of the face, avoiding excessive distortion in predictions. Consequently, in actual predictions after training, excessive distortion in predictions can also be avoided, ensuring that the predicted values conform to the basic structure of the face and guaranteeing prediction stability.
[0051] This application first performs image encoding mapping on each frame of face image through an input mapping layer to obtain the first image feature. Then, it modulates and fuses the first image feature with the corresponding texture feature through a modulation fusion layer to obtain the texture fusion feature. Next, it performs a two-branch task prediction on the texture fusion feature through a dual-task decoding layer to obtain the corresponding pupil center feature and frame key point feature. Finally, it performs decoding mapping through an output mapping layer to obtain the prediction parameter set. It can fully utilize the prior of texture feature through modulation fusion to improve the accuracy of the subsequently obtained prediction parameter set, thereby improving the accuracy of the first pupil height and first pupil distance. Furthermore, through the two-branch task prediction of the dual-task decoding layer, it realizes the task of simultaneously measuring pupil height and pupil distance, which has not been implemented in the past.
[0052] Step S103: Based on Kalman filtering, perform dynamic face optimization and fusion on the prediction parameter group corresponding to each frame of face image to obtain the user's first pupil height and first pupil distance.
[0053] In some embodiments of this application, the step of performing dynamic face optimization and fusion on the prediction parameter group corresponding to each frame of face image based on Kalman filtering to obtain the user's first pupil height and first pupil distance specifically includes: Based on the prediction parameter group corresponding to each frame of face image, prediction optimization is performed based on Kalman filtering to obtain the pupil center coordinates and key point coordinates of the frame of the user. Based on the user's pupil center coordinates and the coordinates of key points on the frame, the user's first pupil height and first pupillary distance are calculated.
[0054] Specifically, pupil height refers to the distance between the center of the pupil and the lower edge of the glasses; pupil distance can be the distance between the centers of the pupils of both eyes (binocular pupil distance), or the distance between the left or right eye and the central axis of the bridge of the nose (monocular pupil distance). This application uses the former as pupil distance. Both pupil height and pupil distance can be calculated from the user's pupil center coordinates and the coordinates of key points on the frame.
[0055] This application first uses the prediction parameter set of each frame of face image to perform prediction optimization based on Kalman filtering to obtain the user's pupil center coordinates and frame key point coordinates, and then calculates the user's first pupil height and first pupil distance. It can reasonably predict the coordinates to be calculated through Kalman filtering, thereby providing a data basis for subsequent calculation of the first pupil height and first pupil distance and improving its calculation accuracy.
[0056] In some embodiments of this application, the step of performing prediction optimization based on Kalman filtering according to the prediction parameter set corresponding to each frame of face image to obtain the user's pupil center coordinates and frame key point coordinates specifically includes: Based on the prediction parameter group corresponding to each frame of the face image, the first prediction trajectory of the user's pupil center and the second prediction trajectory of each frame key point are obtained; The first predicted trajectory and the second predicted trajectory of each key point of the frame are optimized by using a Kalman filter algorithm based on a motion model to obtain the coordinates of the user's pupil center and the coordinates of the key points of the frame.
[0057] Specifically, when obtaining the predicted trajectory, the coordinates of the predicted pupil center in the prediction parameter group corresponding to each frame of face image are arranged sequentially according to the timestamp order to obtain the first predicted trajectory; similarly, the second predicted trajectory of each frame key point in the predicted frame key point group can be obtained.
[0058] This application first obtains the first predicted trajectory of the user's pupil center and the second predicted trajectory of each frame key point based on the prediction parameter group of each frame of the face image. Then, it optimizes the user's pupil center coordinates and frame key point coordinates by using a motion model-based Kalman filter algorithm. This algorithm can take into account the coordinate movement error caused by facial micro-movements and correct it by using motion model-based Kalman filtering, thereby improving the accuracy of subsequent optimization predictions and thus improving the accuracy of the obtained user's pupil center coordinates and frame key point coordinates.
[0059] Compared to existing technologies, this application first acquires multiple frames of the user's face images and extracts texture features based on the Local Binary Pattern Histogram algorithm. This provides prior information on texture for subsequent parameter prediction and measurement, thereby improving the accuracy of the predicted parameter set obtained through the parameter measurement model, and thus improving the accuracy of the first pupil height and first pupil distance. Then, each frame of face image and its corresponding texture features are combined and input into a parameter measurement model based on a two-branch task neural network model for parameter prediction. This allows for simultaneous prediction and measurement of pupil height and pupil distance. Furthermore, the parameter measurement model is trained using the physical constraints of face symmetry as a loss, which avoids making inferences that violate inherent face constraints during model training and prediction, thus improving model stability. Finally, Kalman filtering is used for further dynamic optimization and fusion of the face, which further improves prediction accuracy, thereby improving the accuracy of the first pupil height and first pupil distance.
[0060] For a method corresponding to the one described above, please refer to [link to relevant documentation]. Figure 3 The present application provides a pupil height and pupil distance measurement device that takes into account prior knowledge, including an image acquisition and texture extraction module 310, a face image parameter prediction and measurement module 320, and a prediction parameter optimization and fusion module 330. The image acquisition and texture extraction module 310 is used to acquire multiple frames of the user's face images, and extract texture features from each frame of the face image sequentially based on the local binary mode histogram algorithm to obtain the texture features corresponding to each frame of the face image; wherein, the texture features are used to provide prior information for the corresponding face image; The face image parameter prediction and measurement module 320 is used to sequentially combine each frame of face image with the corresponding texture features and input them into a preset parameter measurement model for parameter prediction, and output the prediction parameter set corresponding to each frame of face image; wherein, the parameter measurement model is based on a two-branch task neural network model and is trained by using the physical constraints of face symmetry as loss; the prediction parameter set includes the prediction pupil center and the prediction frame key point set; The prediction parameter optimization and fusion module 330 is used to perform dynamic face optimization and fusion on the prediction parameter group corresponding to each frame of face image based on Kalman filtering, so as to obtain the user's first pupil height and first pupil distance.
[0061] In some embodiments of this application, the image acquisition and texture extraction module 310 includes an image acquisition unit, an image evaluation unit, and an image filtering unit; The image acquisition unit is used to detect the user's face in response to the user's action in response to the preset image acquisition prompt. When the user's face is detected to meet the preset acquisition conditions, the unit performs continuous high-speed photography of the user to obtain multiple initial face images. The image evaluation unit is used to evaluate the sharpness and illumination uniformity of each initial face image to obtain the sharpness and illumination uniformity of each initial face image. The image filtering unit is used to filter the multiple initial face images according to the clarity and illumination uniformity corresponding to each initial face image to obtain multiple face images.
[0062] In some embodiments of this application, the image acquisition and texture extraction module 310 includes an image processing unit and a texture extraction unit; The image processing unit is used to perform grayscale conversion and Gaussian smoothing on each frame of face image to obtain the corresponding first face image; The texture extraction unit is used to extract histogram features of the texture of each frame of the first face image at multiple preset scales based on a preset local binary mode operator, so as to obtain the texture features corresponding to each frame of the first face image.
[0063] In some embodiments of this application, the texture extraction unit includes a region partitioning subunit, a region calculation subunit, and a texture extraction subunit; The region division subunit is used to perform the same region division on each frame of the first face image based on a preset division mode to obtain multiple sets of divided images; wherein, each set of divided images is divided into multiple non-overlapping division grid regions based on the preset division mode. The region calculation subunit is used to perform local binary mode histogram calculation on each grid region in each group of segmented images at the multiple scales based on the local binary mode operator, so as to obtain a region histogram at multiple scales for each grid region in each group of segmented images. The texture extraction subunit is used to concatenate the region histograms of the same scale in all the partitioned grid regions of each group of partitioned images at the multiple scales to obtain multiple single-scale features corresponding to each group of partitioned images, and to stack the multiple single-scale features of each group of partitioned images to obtain the texture features corresponding to each group of partitioned images.
[0064] In some embodiments of this application, the parameter measurement model is provided with an input mapping layer, a modulation fusion layer, a dual-task decoding layer, and an output mapping layer; the face image parameter prediction and measurement module 320 includes an input mapping unit, a modulation fusion unit, a decoding prediction unit, and an output mapping unit; The input mapping unit is used to perform image encoding mapping on each frame of face image through the input mapping layer to obtain the first image feature corresponding to each frame of face image; The modulation fusion unit is used to modulate and fuse each first image feature with the corresponding texture feature through the modulation fusion layer to obtain the texture fusion feature corresponding to each frame of face image; The decoding prediction unit is used to perform a two-branch task prediction on each texture fusion feature through the dual-task decoding layer to obtain the pupil center feature and frame key point feature corresponding to each texture fusion feature. The output mapping unit is used to decode and map the pupil center features and frame key point features corresponding to each frame of face image through the output mapping layer to obtain the prediction parameter set corresponding to each frame of face image.
[0065] In some embodiments of this application, the prediction parameter optimization and fusion module 330 includes a filtering prediction unit and a parameter calculation unit; The filtering prediction unit is used to perform prediction optimization based on Kalman filtering according to the prediction parameter group corresponding to each frame of face image, so as to obtain the pupil center coordinates and the key point coordinates of the frame of the user. The parameter calculation unit is used to calculate the user's first pupil height and first pupillary distance based on the user's pupil center coordinates and the coordinates of key points in the frame.
[0066] In some embodiments of this application, the filtering prediction unit includes a trajectory combination subunit and a filtering prediction subunit; The trajectory combination subunit is used to obtain the first predicted trajectory of the user's pupil center and the second predicted trajectory of each frame key point based on the prediction parameter group corresponding to each frame of the face image. The filtering prediction subunit is used to optimize the first predicted trajectory and the second predicted trajectory of each frame key point using a Kalman filter algorithm based on a motion model, so as to obtain the user's pupil center coordinates and frame key point coordinates.
[0067] This application first acquires multiple frames of the user's face images and extracts texture features based on the Local Binary Pattern Histogram algorithm. This provides prior information on texture for subsequent parameter prediction and measurement, thereby improving the accuracy of the predicted parameter set obtained through the parameter measurement model, and thus improving the accuracy of the first pupil height and first pupil distance. Next, each frame of face image and its corresponding texture features are combined and input into a parameter measurement model based on a two-branch task neural network model for parameter prediction. This allows for simultaneous prediction and measurement of pupil height and pupil distance. Furthermore, the parameter measurement model is trained using the physical constraints of face symmetry as a loss, which avoids making inferences that violate inherent face constraints during model training and prediction, thereby improving model stability. Finally, Kalman filtering is used for further dynamic optimization and fusion of the face, which further improves prediction accuracy, thus enhancing the accuracy of the obtained first pupil height and first pupil distance.
[0068] It should be understood that the apparatus provided in the embodiments of this application is corresponding to the aforementioned method. The pupil height and pupil distance measuring apparatus provided in the embodiments of this application, which takes into account prior knowledge, can realize the pupil height and pupil distance measuring method that takes into account prior knowledge provided in any embodiment of this application.
[0069] Adaptively, embodiments of this application also provide a computer device and a computer-readable storage medium.
[0070] The computer device includes: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor; When the processor executes the computer program, it implements a method for measuring pupil height and pupil distance that takes into account prior knowledge, as described in this application.
[0071] The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute a priori pupil height and pupillary distance measurement method of the present application.
[0072] The above description represents some embodiments of this application, providing a further detailed explanation of the purpose, technical solution, and beneficial effects of this application. It should be understood that the above-described embodiments of this application should not be construed as limiting this application. In particular, any changes, modifications, equivalent substitutions, and variations made by those skilled in the art within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A method for measuring pupil height and pupillary distance considering prior information, characterized in that, include: The system acquires multiple frames of a user's face images and extracts texture features from each frame of the face image sequentially based on the local binary mode histogram algorithm, thereby obtaining the texture features corresponding to each frame of the face image; wherein, the texture features are used to provide prior information for the corresponding face image. Each frame of face image is sequentially combined with its corresponding texture features and then input into a preset parameter measurement model for parameter prediction, outputting a set of predicted parameters for each frame of face image; wherein, the parameter measurement model is based on a two-branch task neural network model and is trained by using the physical constraints of face symmetry as loss; the set of predicted parameters includes a set of predicted pupil center and a set of predicted frame key points; Based on Kalman filtering, the prediction parameter group corresponding to each frame of face image is dynamically optimized and fused to obtain the user's first pupil height and first pupil distance.
2. The method for measuring pupil height and pupil distance considering prior information according to claim 1, characterized in that, The acquisition of multiple frames of the user's face images specifically includes: In response to the user's action in response to the preset image acquisition prompt, the user's face is detected. When the user's face is detected to meet the preset acquisition conditions, the user is continuously photographed at high speed to obtain multiple initial face images. For each initial face image frame, sharpness and illumination uniformity are evaluated to obtain the sharpness and illumination uniformity corresponding to each initial face image frame. Based on the sharpness and illumination uniformity of each initial face image, the multiple initial face images are filtered to obtain multiple face images.
3. The method for measuring pupil height and pupillary distance considering prior knowledge according to claim 1, characterized in that, The local binary pattern histogram algorithm sequentially extracts texture features from each frame of a face image to obtain the texture features corresponding to each frame of the face image, specifically including: Each frame of the face image is converted to grayscale and smoothed using Gaussian to obtain the corresponding first face image; At multiple preset scales, histogram features are extracted from the texture of the first face image in each frame based on a preset local binary mode operator, so as to obtain the texture features corresponding to the first face image in each frame.
4. The method for measuring pupil height and pupillary distance considering prior information according to claim 3, characterized in that, The process involves extracting histogram features from the texture of each frame of the first face image at multiple preset scales, based on a preset local binary mode operator, to obtain the texture features corresponding to each frame of the first face image. Specifically, this includes: Based on a preset partitioning pattern, the first face image in each frame is divided into the same regions to obtain multiple sets of partitioned images; wherein, each set of partitioned images is divided into multiple non-overlapping partitioned grid regions based on the preset partitioning pattern. At the multiple scales, based on the local binary mode operator, local binary mode histograms are calculated sequentially for each grid region in each group of segmented images to obtain region histograms at multiple scales for each grid region in each group of segmented images. At the multiple scales, the histograms of regions of the same scale in all the grid regions of each group of segmented images are concatenated to obtain multiple single-scale features corresponding to each group of segmented images. The multiple single-scale features of each group of segmented images are then stacked to obtain the texture features corresponding to each group of segmented images.
5. The method for measuring pupil height and pupillary distance considering prior information according to claim 1, characterized in that, The parameter measurement model includes an input mapping layer, a modulation fusion layer, a dual-task decoding layer, and an output mapping layer. The process involves sequentially combining each frame of the face image with its corresponding texture features and inputting the result into the preset parameter measurement model for parameter prediction, outputting a set of predicted parameters for each frame of the face image. Specifically, this includes: The input mapping layer performs image encoding mapping on each frame of face image to obtain the first image feature corresponding to each frame of face image; Through the modulation and fusion layer, each first image feature and its corresponding texture feature are modulated and fused to obtain the texture fusion feature corresponding to each frame of face image; Through the dual-task decoding layer, a two-branch task prediction is performed on each texture fusion feature to obtain the pupil center feature and the frame key point feature corresponding to each texture fusion feature. The output mapping layer decodes and maps the pupil center features and frame key point features corresponding to each frame of face image to obtain the prediction parameter set corresponding to each frame of face image.
6. The method for measuring pupil height and pupillary distance considering prior information according to claim 1, characterized in that, The method of dynamically optimizing and fusing the predicted parameter group corresponding to each frame of face image based on Kalman filtering to obtain the user's first pupil height and first pupil distance specifically includes: Based on the prediction parameter group corresponding to each frame of face image, prediction optimization is performed based on Kalman filtering to obtain the pupil center coordinates and key point coordinates of the frame of the user. Based on the user's pupil center coordinates and the coordinates of key points on the frame, the user's first pupil height and first pupillary distance are calculated.
7. The method for measuring pupil height and pupillary distance considering prior information according to claim 6, characterized in that, The step of optimizing the prediction based on the prediction parameter set corresponding to each frame of the face image, using Kalman filtering, to obtain the user's pupil center coordinates and frame key point coordinates, specifically includes: Based on the prediction parameter group corresponding to each frame of the face image, the first prediction trajectory of the user's pupil center and the second prediction trajectory of each frame key point are obtained; The first predicted trajectory and the second predicted trajectory of each key point of the frame are optimized by using a Kalman filter algorithm based on a motion model to obtain the coordinates of the user's pupil center and the coordinates of the key points of the frame.
8. A pupil height and pupil distance measuring device considering prior information, characterized in that, It includes an image acquisition and texture extraction module, a face image parameter prediction and measurement module, and a prediction parameter optimization and fusion module; The image acquisition and texture extraction module is used to acquire multiple frames of the user's face images, and based on the local binary mode histogram algorithm, extract texture features from each frame of the face image in turn to obtain the texture features corresponding to each frame of the face image; wherein, the texture features are used to provide prior information for the corresponding face image; The face image parameter prediction and measurement module is used to sequentially combine each frame of face image with the corresponding texture features and input them into a preset parameter measurement model for parameter prediction, and output the prediction parameter set corresponding to each frame of face image; wherein, the parameter measurement model is based on a two-branch task neural network model and is trained by using the physical constraints of face symmetry as loss; the prediction parameter set includes the prediction pupil center and the prediction frame key point set; The prediction parameter optimization and fusion module is used to perform dynamic face optimization and fusion on the prediction parameter group corresponding to each frame of face image based on Kalman filtering, so as to obtain the user's first pupil height and first pupil distance.
9. A computer device, characterized in that, include: processor; Memory; A computer program stored in the memory and configured to be executed by the processor; When the processor executes the computer program, it implements a priori pupil height and pupillary distance measurement method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute a method for measuring pupil height and pupillary distance considering prior information, as described in any one of claims 1 to 7.