Intelligent glasses frame recommendation method and system based on multi-dimensional facial data recognition

Abstract

The invention provides an intelligent glasses frame recommendation method and system based on multi-dimensional face data recognition, and the method comprises the steps: obtaining 3D face data obtained through 3D scanning of the face of a user, recognizing face key points from the 3D face data through a first pre-training AI model, carrying out the fitting through employing the recognized face key points as control points, and carrying out the recognition of the face key points; a 3D face coding point cloud carrying full face digital semantics is obtained; calling a second pre-training AI model to extract face features including skin color, age, face shape and / or hair style from the 3D face coding point cloud, and quantitatively calculating face features including face width and / or eyebrow and eye height dimensions based on the 3D face data; and calculating fitness scores of the multi-dimensional face features under the multi-dimensional mirror frame feature combinations by using a preset dynamic classification score matrix, and taking the mirror frame feature combination with the highest fitness score as a mirror frame style recommended to the user.

Description

Technical Field

[0001] This invention relates to the field of AI facial feature recognition technology, and in particular to a method and system for intelligent recommendation of eyeglass frames based on multi-dimensional facial data recognition. Background Technology

[0002] Currently, traditional eyeglass frames are mostly selected by trying them on in front of a mirror for self-assessment. However, this method is limited by ambient lighting, the number of times you can try them on, and subjective judgment, resulting in low selection efficiency and difficulty in accurately matching the user's facial features.

[0003] Online virtual try-on technology largely relies on 2D image processing and lacks in-depth analysis of the three-dimensional structure of the face, resulting in insufficient fidelity in the reproduction of the wearing effect. Although some high-end solutions have introduced 3D scanning, the high cost of the equipment and the difficulty in widespread adoption have prevented them from achieving intelligent recommendations. Summary of the Invention

[0004] To improve the accuracy and user satisfaction of intelligent eyeglass frame recommendations, and to reduce the threshold and cost of intelligent eyeglass frame recommendations, this invention proposes an intelligent eyeglass frame recommendation method based on AI and 3D facial data.

[0005] On one hand, this invention proposes a method for intelligent recommendation of eyeglass frames based on multi-dimensional facial data recognition, including: acquiring 3D facial data obtained by 3D scanning a user's face; using a first pre-trained AI model to identify facial key points from the 3D facial data; using the identified facial key points as control points for fitting to obtain a 3D facial coding point cloud carrying full-face digital semantics; calling a second pre-trained AI model to extract facial features including skin color, age, face shape, and / or hairstyle from the 3D facial coding point cloud; quantitatively calculating facial features including face width and / or eyebrow and eye height dimensions based on the 3D facial data; using a preset dynamic classification scoring matrix to calculate the fit score of multi-dimensional facial features under multi-dimensional eyeglass frame feature combinations; and using the eyeglass frame feature combination with the highest fit score as the recommended eyeglass frame style to the user; wherein, the dimensions of eyeglass frame features include some or all of color, shape, and size, and the dynamic classification scoring matrix contains the classification scores corresponding to each dimension of facial features and each dimension of eyeglass frame features.

[0006] On the other hand, the invention proposes a smart recommendation system for eyeglass frames based on facial data recognition, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of any of the methods described in the above embodiments.

[0007] The intelligent frame recommendation method and system based on multi-dimensional facial data recognition proposed in this invention can intelligently recommend frames that are suitable for users' facial features in terms of color, frame shape, and size by using user facial data collected through AI and 3D scanning. This reduces the difficulty of selection for users. At the same time, since the intelligently recommended frames are more suitable for the user's face size, it helps to improve the user's comfort when wearing the frames. This method helps to improve the user's experience in the process of purchasing frames. Attached Figure Description

[0008] Figure 1 This is a flowchart of a smart frame recommendation method in one embodiment of the present invention.

[0009] Figure 2 This is a schematic diagram of a 3D face coding point cloud in one embodiment of the present invention. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0011] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0012] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0013] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0014] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0015] To overcome the problems of existing technologies, this invention uses AI recognition technology to identify key information of a face, and then combines it with a 3D structured light camera module to scan the 3D structure of the face. By accurately identifying the skin color, age, face shape, hairstyle and / or 3D facial contour data, a weighted matching algorithm is used to recommend a frame that is suitable for the wearer in terms of appearance and size.

[0016] Figure 1This is a flowchart of a smart frame recommendation method according to an embodiment of the present invention. The method includes the following steps:

[0017] Step S110: Obtain 3D face data obtained by scanning the user's face with a 3D scanner, use the first pre-trained AI model to identify facial key points from the 3D face data, use the identified facial key points as control points to fit, and obtain a 3D face coding point cloud carrying full-face digital semantics.

[0018] Step S120: Call the second pre-trained AI model to extract facial features including skin color, age, face shape and / or hairstyle from the 3D face coding point cloud, and quantitatively calculate facial features including face width and / or eyebrow and eye height dimensions based on the 3D face data;

[0019] Step S130: Calculate the fit score of multi-dimensional facial features under multi-dimensional frame feature combinations using a preset dynamic classification score matrix, and use the frame feature combination with the highest fit score as the frame style recommended to the user; wherein, the dimensions of frame features include some or all of color, shape and size, and the dynamic classification score matrix contains the classification scores corresponding to each dimension of facial features and each dimension of frame features.

[0020] By employing this embodiment of the invention, user facial data collected through AI and 3D scanning can be intelligently recommended to users in terms of color, frame shape, and size that are suitable for their facial features, reducing the difficulty of selection for users. At the same time, since the intelligently recommended frames are more suitable for the user's face size, it is beneficial to improve the user's comfort when wearing the frames. This method helps to improve the user's experience in the process of purchasing frames.

[0021] In some embodiments of the present invention, the first pre-trained AI model is a 3D keypoint detection network based on deep learning. The step of identifying facial keypoints from the 3D face data using the first pre-trained AI model includes: identifying and extracting the three-dimensional coordinates of facial keypoints from the 3D face data using a point cloud neural network included in the 3D keypoint detection network. The facial keypoints include some or all of the points such as the center of the eyebrows, corners of the eyes, tip of the nose, corners of the mouth, and facial contour points. The 3D keypoint detection network is trained using a supervised deep learning method based on a 3D face dataset.

[0022] Using this embodiment of the invention, high-precision control points can be provided for subsequent 3D deformation model fitting, ensuring that the generated 3D face coding point cloud can accurately carry the digital semantic information of the whole face.

[0023] In some embodiments of the present invention, the step of fitting the identified facial key points as control points to obtain a 3D face-coded point cloud carrying full-face digital semantics includes: using the identified facial key points as control points, using the control points to maintain the natural topological structure of the face, using principal component analysis to process the facial key points to establish a face shape space, and using a statistically based three-dimensional deformation model as a fitting framework to construct a 3D face-coded point cloud carrying full-face digital semantics in the face shape space.

[0024] Furthermore, the fitting accuracy can be optimized based on the Iterative Closest Point (ICP) algorithm to ensure that the reconstructed 3D face conforms to individual characteristics while maintaining anatomical rationality.

[0025] Using this embodiment of the invention, the generated 3D face-coded point cloud not only contains the geometric coordinates of tens of thousands of vertices, but also embeds complete digital semantic labels, thereby establishing an accurate digital foundation for subsequent multi-dimensional feature extraction.

[0026] In some embodiments of the present invention, the method further includes the step of pre-training a second pre-trained AI model of a multi-label classification model type, using a 3D face-coded point cloud that has undergone face detection and alignment as raw data, pre-labeling facial features including skin color, age, face shape and / or hairstyle in the 3D face-coded point cloud, and using the Softmax function to achieve the output of multi-label classification.

[0027] By employing this embodiment of the invention, facial feature recognition can be achieved through a deep learning-based multi-label classification model, laying a technical foundation for subsequent intelligent recommendation of eyeglass frame styles.

[0028] In some embodiments of the present invention, the step of calculating the fit score of multi-dimensional facial features under multi-dimensional frame feature combinations using a preset dynamic classification score matrix includes: randomly generating multiple frame feature combinations initially; searching for the classification scores of each frame feature corresponding to each dimension of facial features in the dynamic classification score matrix and summing them to obtain the fit score of each frame feature combination corresponding to each dimension of facial features; and using a swarm intelligence optimization algorithm to iterate in the solution space in multiple rounds to find the frame feature combination that makes the fit score optimal.

[0029] Wherein, the dynamic classification score matrix W is a high-order tensor, and its element Wij represents the classification score under the i-th face feature dimension (such as face width) and the j-th frame feature dimension (such as frame size).

[0030] By employing this embodiment of the invention, the matching classification scores of the facial feature dimension and the frame feature dimension can be dynamically adjusted using a dynamic classification score matrix, thereby intelligently recommending the most suitable frame feature combination for the current user.

[0031] In some embodiments of the present invention, in the step of calculating the fit score of multi-dimensional facial features under multi-dimensional frame feature combinations using a preset dynamic classification score matrix, the method further includes: pre-setting weight values ​​for different dimensions of facial features, multiplying the preset weight values ​​of different dimensions with the corresponding classification scores, and then summing the products of the classification scores to obtain the fit score of each frame feature combination corresponding to each dimension of facial features.

[0032] By adopting this embodiment of the invention, the different degrees of influence of facial features of different dimensions on eyeglass frames can be comprehensively considered, thereby improving the matching degree between the final intelligently recommended eyeglass frames and the user's face, thus reducing the time and effort cost for the user to select eyeglass frames.

[0033] In some embodiments of the present invention, the method further includes: (1) after obtaining the frame style recommended to the user, obtaining the user's subjective feedback in the form of a questionnaire, and adjusting the dynamic classification score matrix based on the user's subjective feedback; and / or (2) periodically obtaining 3D face coding point clouds of facial features including skin color, age, face shape and / or hairstyle labeled by experts, and further training the second pre-trained AI model using the newly obtained 3D face coding point clouds.

[0034] By employing this embodiment of the invention, the quality of the dynamic classification score matrix and the second pre-trained AI model can be further improved, thereby effectively enhancing the accuracy of intelligent frame recommendations and user satisfaction.

[0035] Furthermore, this invention can construct an expert feedback system for recommending eyeglass frames based on aesthetic preferences: while the above recommendation method is running, information such as occupation and frame preferences of users experiencing this intelligent frame recommendation method can be collected through questionnaires. Experts in eyeglass frame design and optometry can also be periodically organized to score and annotate the data.

[0036] Using the above embodiments, a comprehensive intelligent recommendation method for eyeglass frames can be formed by combining 2D facial information recognition, 3D actual facial size, and expert scoring system.

[0037] In some embodiments of the present invention, the method further includes the classification and management of picture frames. Picture frame models with different combinations of frame features are classified, numbered, and stored in a picture frame model library.

[0038] Furthermore, the dynamic classification score matrix stores: (1) classification scores for different frame colors that match skin tone; (2) classification scores for different frame colors that match age; (3) classification scores for frame shapes that match face shape; (4) classification scores for frame shapes that match hairstyle; (5) classification scores for different frame sizes that match different face widths; and (6) classification scores for different frame sizes that match different eyebrow and eye heights.

[0039] Figure 2 This is a schematic diagram of a 3D face coding point cloud in one embodiment of the present invention. Figure 2 The red area represents 3D facial coding data, where the distance between point 1 and point 2 is the facial width, and the distance between point 3 and point 4 is the eyebrow and eye height.

[0040] In the specific implementation process, AI technology can be used to classify facial features in different dimensions: skin color classification (F1,F2,F3), age classification (N1,N2,N3), face shape classification (L1,L2,L3), and hairstyle classification (X1,X2,X3) for recognition.

[0041] Furthermore, 3D face data can be obtained through 3D scanning, and feature parameters such as face width classification (K1, K2, K3) and / or eyebrow and eye height classification (G1, G2, G3) can be identified and calculated based on the 3D face data.

[0042] The picture frame model library can be categorized and numbered, for example, by color (A,B,C), frame shape (a,b,c), and frame size (S,L,M), and can be expanded according to the richness of the picture frame database.

[0043] The following is an example of the classification scores stored in the dynamic classification score matrix:

[0044] (1) Classify and score different frame colors that match the skin tone. For example, in skin tone (F1, F2, F3), F1 corresponds to color A with 50 points, color B with 30 points, and color C with 20 points; F2 corresponds to color A with 20 points, color B with 50 points, and color C with 30 points; and F3 corresponds to color A with 30 points, color B with 20 points, and color C with 50 points.

[0045] (2) Classify and score the different frame colors that match the age. For example, in the age group (N1, N2, N3), N1 corresponds to color A with 50 points, color B with 30 points, and color C with 20 points; N2 corresponds to color A with 20 points, color B with 50 points, and color C with 30 points; and N3 corresponds to color A with 30 points, color B with 20 points, and color C with 50 points.

[0046] (3) Classify and score the face shape matching frame shape. For example, in face shape (L1, L2, L3), L1 corresponds to frame shape a 50 points, frame shape b 30 points, and frame shape c 20 points; L2 corresponds to frame shape a 20 points, frame shape b 50 points, and frame shape c 30 points; L3 corresponds to frame shape a 30 points, frame shape b 20 points, and frame shape c 50 points.

[0047] (4) Classify and score the hairstyles that match the frame shapes. For example, in hairstyles (X1, X2, X3), X1 corresponds to frame shape a 50 points, frame shape b 30 points, and frame shape c 20 points; X2 corresponds to frame shape a 20 points, frame shape b 50 points, and frame shape c 30 points; and X3 corresponds to frame shape a 30 points, frame shape b 20 points, and frame shape c 50 points.

[0048] (5) Different sizes of frames are suitable for different face widths. For example, for face widths (K1, K2, K3), K1 corresponds to frame size S50 points, L30 points, and M20 points; K2 corresponds to frame size S20 points, L50 points, and M30 points; and K3 corresponds to frame size S30 points, L20 points, and M50 points.

[0049] (6) Different sizes of frames are adapted for different eyebrow and eye heights. For example, in the eyebrow and eye height (G1, G2, G3), G1 corresponds to frame size S 50 points, L 30 points, and M 20 points; G2 corresponds to frame size S 20 points, L 50 points, and M 30 points; and G3 corresponds to frame size S 30 points, L 20 points, and M 50 points.

[0050] (7) Then, a suitable frame for the face is recommended based on the weighted scores of different frames. For example, if the scores of categories 4, 5, 6, 7, 8, and 9 are all set to 1, and assuming the customer's skin color F1, age N1, face shape L1, hairstyle X1, face width K1, and eyebrow and eye height G1, then the frame with the highest score (color A, frame shape a, frame size S) is considered to be the most suitable frame for the customer recommended by this patented method.

[0051] Table 1 shows an example of classification scores stored in another dynamic classification score matrix.

[0052]

[0053] As shown in Table 1, assuming the face attribute is (F2, L2, K2), the highest fit score is obtained when the frame attribute is (B, b, L). That is, the most suitable frame for the user with face attribute (F2, L2, K2) is a frame with color B, shape b and size L.

[0054] In summary, the intelligent frame recommendation method and system based on multi-dimensional facial data recognition proposed in this invention can intelligently recommend frames that are suitable for the user's facial features in terms of color, shape, and size using user facial data collected through AI and 3D scanning. This reduces the difficulty of selection for users. Furthermore, because the intelligently recommended frames are more adapted to the user's facial dimensions, it improves the user's comfort when wearing the frames, thus enhancing the user's experience during the frame purchase process. The improvement of this invention lies in combining 2D visual attributes such as skin color, age, face shape, and hairstyle with the actual 3D facial dimensions, so that the final recommended frames not only conform to popular aesthetic standards in appearance but also perfectly match the current face in actual physical size.

[0055] It should be noted that the above-mentioned facial attribute dimensions are not limited to these, and can be replaced with or added to other facial attribute dimensions that are easily conceived by those skilled in the art.

[0056] Corresponding to the above method, the present invention also provides a smart recommendation system for eyeglass frames based on facial data recognition. The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0057] Corresponding to the methods described above, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the method as described in any of the above embodiments. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0058] Corresponding to the above methods, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any of the above embodiments.

[0059] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0060] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0061] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent recommendation of eyeglass frames based on multi-dimensional facial data recognition, characterized in that, include: 3D face data obtained by scanning a user's face using 3D scanning is acquired. A first pre-trained AI model is used to identify facial key points from the 3D face data. The identified facial key points are used as control points for fitting to obtain a 3D face encoded point cloud carrying full-face digital semantics. The second pre-trained AI model is invoked to extract facial features, including skin color, age, face shape and / or hairstyle, from the 3D face coding point cloud. Based on the 3D face data, facial features, including face width and / or eyebrow and eye height dimensions, are quantitatively calculated. The system uses a preset dynamic classification score matrix to calculate the fit score of multi-dimensional facial features under multi-dimensional frame feature combinations, and uses the frame feature combination with the highest fit score as the frame style recommended to the user; wherein, the dimensions of frame features include some or all of color, shape and size, and the dynamic classification score matrix contains the classification scores corresponding to each dimension of facial features and each dimension of frame features.

2. The method according to claim 1, characterized in that, The first pre-trained AI model is a 3D keypoint detection network based on deep learning. The step of using the first pre-trained AI model to identify facial keypoints from the 3D face data includes: The point cloud neural network included in the 3D key point detection network is used to identify and extract the three-dimensional coordinates of key facial points from the 3D face data; The facial key points include some or all of the points such as the center of the eyebrows, the corners of the eyes, the tip of the nose, the corners of the mouth, and facial contour points. The 3D key point detection network is trained using a supervised deep learning method based on a 3D face dataset.

3. The method according to claim 1, characterized in that, The step of fitting the identified facial key points as control points to obtain a 3D face-coded point cloud carrying full-face digital semantics includes: Using the identified facial key points as control points, the natural topological structure of the face is maintained by the control points. Principal component analysis is used to process the facial key points to establish a face shape space. A statistically based three-dimensional deformation model is used as the fitting framework to construct a 3D face coding point cloud carrying full-face digital semantics in the face shape space.

4. The method according to claim 1, characterized in that, The method further includes: The second pre-trained AI model of the pre-trained multi-label classification model type uses 3D face-coded point clouds that have undergone face detection and alignment as raw data, pre-labels facial features including skin color, age, face shape and / or hairstyle in the 3D face-coded point cloud, and uses the Softmax function to achieve the output of multi-label classification.

5. The method according to claim 1, characterized in that, The step of calculating the fit score of multi-dimensional facial features under multi-dimensional frame feature combinations using a preset dynamic classification score matrix includes: Multiple frame feature combinations are randomly generated initially. The classification scores of each frame feature corresponding to each dimension of face features are found in the dynamic classification score matrix and summed to obtain the fit score of each frame feature combination corresponding to each dimension of face features. A swarm intelligence optimization algorithm is used to iterate in the solution space in multiple rounds to find the frame feature combination that maximizes the fit score.

6. The method according to claim 5, characterized in that, In the step of calculating the fit score of multi-dimensional facial features under multi-dimensional frame feature combinations using a preset dynamic classification score matrix, the method further includes: Weight values ​​are pre-set for different dimensions of facial features. The preset weight values ​​of different dimensions are multiplied by the corresponding classification scores. Then, the products of the classification scores are summed to obtain the fit score of each combination of frame features corresponding to each dimension of facial features.

7. The method according to claim 1, characterized in that, The method further includes: After receiving recommended eyeglass frame styles, a questionnaire is used to obtain subjective feedback from users, and the dynamic classification score matrix is ​​adjusted based on this feedback; and / or Periodically acquire 3D face coding point clouds of facial features, including skin color, age, face shape and / or hairstyle, labeled by experts, and further train the second pre-trained AI model using the newly acquired 3D face coding point clouds.

8. A smart recommendation system for eyeglass frames based on facial data recognition, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images