Automatic glasses matching method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an automatic glasses matching method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the face scanning of a user through structured light, obtaining the face three-dimensional data of the user, and generating target image data according to the face three-dimensional data; extracting a facial feature vector from the target image data through a pre-trained data model, and determining a first glasses sample from a preset digital glasses library according to the facial feature vector; and determining a second glasses sample from the first glasses samples according to the price budget of the user, and recommending the second glasses sample to the user. According to the embodiment of the invention, the method achieves the screening of the glasses sample according with the user through the facial features of the user and the price budget, thereby reducing the time for the user to select and try on glasses.

Description

Technical Field

[0001] The present invention relates to the technical field of glasses configuration, and in particular to a method and device for automatic glasses configuration, electronic equipment, and storage medium. Background Art

[0002] At present, as people use their eyes more frequently, more and more people have vision problems, and they inevitably need to wear various types of glasses.

[0003] In the prior art, people generally undergo a vision test, select lenses and frames, and then try on the glasses repeatedly to find glasses that are comfortable for them to wear.

[0004] However, due to the wide variety of glasses, the selection process not only wastes a lot of time, but also makes some people with choice anxiety unable to start. Summary of the invention

[0005] In view of the above problems, a method and device for automatic eyeglass fitting, an electronic device, and a storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0006] A method for automatically fitting glasses, the method comprising:

[0007] Scanning the face of a user by structured light to obtain three-dimensional facial data of the user, and generating target image data based on the three-dimensional facial data;

[0008] Extracting a facial feature vector from the target image data by using a pre-trained data model, and determining a first glasses sample from a preset digital glasses library according to the facial feature vector;

[0009] According to the price budget of the user, a second glasses sample is determined from the first glasses sample, and the second glasses sample is recommended to the user.

[0010] Optionally, generating target image data according to the facial three-dimensional data includes:

[0011] Converting the facial three-dimensional data into a three-dimensional voxel image to obtain target image data;

[0012] Alternatively, the three-dimensional facial data is projected onto a two-dimensional depth image to obtain target image data.

[0013] Optionally, determining the first glasses sample from a preset digital glasses library according to the facial feature vector includes:

[0014] Acquire glasses sample feature vectors of a plurality of glasses samples in a preset digital glasses library, and calculate feature similarity between the facial feature vector and the glasses sample feature vectors of the plurality of glasses samples;

[0015] A first glasses sample is determined from the multiple glasses samples according to the feature similarity.

[0016] Optionally, determining a first glasses sample from the multiple glasses samples according to the feature similarity includes:

[0017] sorting the plurality of glasses samples according to the feature similarities;

[0018] One or more glasses samples ranked first are determined as first glasses samples.

[0019] Optionally, before scanning the user's face by structured light to obtain the user's three-dimensional facial data, the method further includes:

[0020] By performing 3D scanning on the glasses, the feature vector of the glasses samples is obtained;

[0021] According to the glasses sample feature vector, glasses samples are created in a preset digital glasses library.

[0022] Optionally, the price budget includes a budget price range, and determining the second glasses sample from the first glasses sample according to the price budget of the user includes:

[0023] Obtaining price information of the first glasses sample;

[0024] From the first glasses samples, determine the first glasses sample whose price information is within the user's budget price range as the second glasses sample.

[0025] Optionally, the data model is a convolutional neural network model, and extracting a facial feature vector from the target image data using the pre-trained data model includes:

[0026] Inputting the target image data into a pre-trained convolutional neural network model, the convolutional neural network model extracting the facial feature vector from the target image data through multiple convolutional layers;

[0027] Receive the facial feature vector output by the convolutional neural network model.

[0028] Optionally, after recommending the second glasses sample to the user, the method further includes:

[0029] In response to the user's selection operation on the second glasses sample, determining a target glasses sample from the second glasses samples;

[0030] Glasses are manufactured based on the target glasses sample.

[0031] An automatic glasses fitting device, comprising:

[0032] A target image data generating module, configured to scan the face of a user by structured light to obtain three-dimensional facial data of the user, and generate target image data based on the three-dimensional facial data;

[0033] A first glasses sample determination module, configured to extract a facial feature vector from the target image data by using a pre-trained data model, and determine a first glasses sample from a preset digital glasses library according to the facial feature vector;

[0034] The second glasses sample determining module is used to determine a second glasses sample from the first glasses samples according to the price budget of the user, and recommend the second glasses sample to the user.

[0035] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method described above when executed by the processor.

[0036] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0037] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described above.

[0038] The embodiments of the present invention have the following advantages:

[0039] In an embodiment of the present invention, a user's face is scanned by structured light to obtain three-dimensional facial data of the user, and target image data is generated based on the three-dimensional facial data; a facial feature vector is extracted from the target image data by a pre-trained data model, and a first glasses sample is determined from a preset digital glasses library based on the facial feature vector; a second glasses sample is determined from the first glasses sample based on the user's price budget, and the second glasses sample is recommended to the user, thereby achieving screening of glasses samples that meet the user's facial features and price budget, thereby reducing the time the user spends selecting and trying on glasses. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0041] Figure 1 is a flowchart of the steps of an automatic glasses fitting method provided by one embodiment of the present invention;

[0042] Figure 2 is a flowchart of another method for automatic eyeglass fitting provided by an embodiment of the present invention;

[0043] Figure 3 is a flowchart of another method for automatic eyeglass fitting provided by an embodiment of the present invention;

[0044] Figure 4 is a flowchart of another method for automatic eyeglass fitting provided by an embodiment of the present invention;

[0045] Figure 5 It is a structural block diagram of an automatic glasses fitting device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Reference Figure 1 , shows a flowchart of a method for automatic eyeglass fitting provided by an embodiment of the present invention, which may specifically include the following steps:

[0048] Step 101 , scanning the face of a user by structured light to obtain three-dimensional facial data of the user, and generating target image data according to the three-dimensional facial data.

[0049] In practical applications, structured light is a technology that uses a projector to project specific light information onto the surface of an object, and uses a camera to collect changes in light signals to calculate information such as the position and depth of the object.

[0050] Specifically, structured light is a system structure composed of a projector and a camera. The projector can project specific light information onto the surface of an object and the background, which is then collected by the camera. The position, depth and other information of the object are calculated based on the changes in the light signal caused by the object, and the entire three-dimensional space is restored.

[0051] During the automatic glasses fitting process, the user's face can be scanned by structured light, and the user's facial three-dimensional data can be obtained based on the image data collected by the camera.

[0052] In some embodiments of the present invention, generating target image data according to the three-dimensional facial data includes:

[0053] Converting the facial three-dimensional data into a three-dimensional voxel image to obtain target image data;

[0054] Alternatively, the three-dimensional facial data is projected onto a two-dimensional depth image to obtain target image data.

[0055] As some examples, a three-dimensional voxel map may refer to a three-dimensional image composed of voxels.

[0056] Specifically, voxel is the abbreviation of volume pixel, which is the smallest unit for three-dimensional space segmentation, similar to pixels in two-dimensional space. Each voxel represents a spatial volume in a unified three-dimensional grid, usually a cube. Voxels are used to store volume data. Voxel maps are implemented by dividing three-dimensional space into voxels in the form of cubes. Each voxel has a position and an associated value, such as density or color, to form a three-dimensional voxel image.

[0057] In some examples, after obtaining the user's three-dimensional facial data, the three-dimensional facial data can be converted into a three-dimensional voxel map through image processing software, and the three-dimensional voxel map is used as target image data.

[0058] As some examples, since height or depth information can be stored in three-dimensional data, the three-dimensional facial data can be projected onto a two-dimensional depth image, and the projected two-dimensional depth image is used as the target image data.

[0059] In some embodiments of the present invention, before scanning the user's face by structured light to obtain the user's three-dimensional facial data, the method further includes:

[0060] By performing 3D scanning on the glasses, the feature vector of the glasses samples is obtained;

[0061] According to the glasses sample feature vector, glasses samples are created in a preset digital glasses library.

[0062] In some examples, the frame and lenses of the glasses may be 3D scanned to obtain the geometric shape and surface features of the frame and lenses of the glasses, and determine the glasses sample feature vector of the glasses; wherein the glasses sample feature vector includes the feature vectors of the frame and lenses, such as the formula:

[0063]

[0064] Among them, g i refers to the feature vector of the glasses sample; refers to the feature vector of the frame; refers to the eigenvector of the lens.

[0065] After the glasses sample feature vector is determined, a glasses sample of the glasses may be created in a preset digital glasses library according to the glasses sample feature vector.

[0066] In practical applications, different glasses can be collected, 3D scanned, and multiple glasses samples can be created in a preset digital glasses library.

[0067] As some examples, data annotation and classification can also be performed on each glasses sample, such as the price and size of the frames and lenses in the glasses sample, to ensure that the frame and lens information of each glasses sample can be extracted and used separately, and each glasses sample has its corresponding feature vector to represent it, which is convenient for matching with the customer's facial features.

[0068] Step 102: extracting a facial feature vector from the target image data through a pre-trained data model, and determining a first glasses sample from a preset digital glasses library according to the facial feature vector.

[0069] In some embodiments of the present invention, the data model is a convolutional neural network model.

[0070] In some examples, a convolutional neural network model may include a convolutional layer, a pooling layer, and a fully connected layer; wherein the convolutional layer is used to extract image features, the pooling layer reduces the spatial dimension of the feature map, reduces the model complexity and computational complexity; and the fully connected layer is used for final features and classification.

[0071] As some examples, let's take a simplified convolutional neural network structure:

[0072] The input data is:

[0073] X∈R H×W×D

[0074] Among them, X represents the input data; R represents three-dimensional data; H represents height, which refers to the vertical size of the image; W represents width, which refers to the horizontal size of the image; D represents depth, which refers to the size of the image in the direction perpendicular to the screen plane in three-dimensional space, and can also refer to the image data dimension or information depth; R H×W×D Indicates that the input data X is a tensor of dimension HxWxD.

[0075] The first convolutional layer input feature map:

[0076]

[0077] Among them, Z (1) It represents the output after the first layer of convolution; R represents the dimension and data type of this feature map; H 1 , W 1 , C 1 are the output height, width and number of channels of the first convolution layer respectively.

[0078] Output after the pooling layer:

[0079]

[0080] Among them, Z (2) Represents the output after the pooling layer; R represents the dimension and data type of this feature map; H 2 , W 2 , C 2 are the output height, width, and number of channels of the pooling layer, respectively.

[0081] Flatten the pooled feature map into a vector The flattened feature vector is then input into the fully connected layer to obtain the final feature representation:

[0082] f(X)∈R F

[0083] Among them, f(X) refers to the facial feature vector; R represents the dimension and data type of this feature map; F is the dimension of the feature vector.

[0084] In some embodiments of the present invention, extracting a facial feature vector from the target image data using a pre-trained data model includes:

[0085] Inputting the target image data into a pre-trained convolutional neural network model, the convolutional neural network model extracting the facial feature vector from the target image data through multiple convolutional layers;

[0086] Receive the facial feature vector output by the convolutional neural network model.

[0087] In some examples, after obtaining the target image data, the target image data can be input into a pre-trained convolutional neural network model for processing, and the convolutional neural network model can extract facial feature vectors from the target image data through multiple convolutional layers.

[0088] As some examples, the computational logic of a convolutional neural network model to extract facial feature vectors from target image data through multiple convolutional layers can be as follows:

[0089] The first convolutional layer operation:

[0090]

[0091] in, Refers to the value at the (i, j, k)th position in the output feature map Z; i, j, k represent the spatial position of the feature map; б is the activation function; is the value corresponding to position X i+m，j+n，l The first layer of convolution kernel; M / m, N / n, L / l correspond to the height, width and depth of the convolution kernel respectively.

[0092] Pooling layer operation:

[0093]

[0094] in, Refers to the value of the feature map at position (i, j, k) after the pooling operation; i, j, k represent the spatial position of the feature map; max m，n Refers to operating on the maximum value of all elements in the pooling window; is the feature map before pooling, s is the stride of the pooling operation, and m and n are the height and width of the convolution kernel, respectively.

[0095] Fully connected layer operation:

[0096] f(X)=σ(W (FC) Z (2) +b FC )

[0097] Where f(X) refers to the facial feature vector; б is the activation function; W (FC ) is the weight of the fully connected layer; Z (2) Represents the output after the pooling layer; b FC is the bias of the feature map.

[0098] In some examples, the automatic glasses fitting system may receive a facial feature vector output by a convolutional neural network model to determine a first glasses sample from a preset digital glasses library based on the facial feature vector.

[0099] In some embodiments of the present invention, determining the first glasses sample from a preset digital glasses library according to the facial feature vector includes:

[0100] Acquire glasses sample feature vectors of a plurality of glasses samples in a preset digital glasses library, and calculate feature similarity between the facial feature vector and the glasses sample feature vectors of the plurality of glasses samples;

[0101] A first glasses sample is determined from the multiple glasses samples according to the feature similarity.

[0102] In some examples, after determining the facial feature vector, glasses sample feature vectors corresponding to multiple glasses samples can be obtained from a preset digital glasses library, and feature similarities between the facial feature vector and the glasses sample feature vectors of the multiple glasses samples are calculated respectively. The specific formula is as follows:

[0103]

[0104] Among them, similarity refers to feature similarity; f(X) refers to the facial feature vector; g i Refers to the feature vector of the glasses sample.

[0105] In some embodiments of the present invention, determining the first glasses sample from the multiple glasses samples according to the feature similarity includes:

[0106] sorting the plurality of glasses samples according to the feature similarities;

[0107] One or more glasses samples ranked first are determined as first glasses samples.

[0108] In some examples, after respectively calculating the feature similarities between the facial feature vector and the glasses sample feature vectors of multiple glasses samples, the glasses samples can be sorted according to the calculation results of the feature similarities, and one or more glasses samples ranked first can be determined as the first glasses sample.

[0109] Step 103: Determine a second pair of glasses sample from the first pair of glasses sample according to the price budget of the user, and recommend the second pair of glasses sample to the user.

[0110] In some embodiments of the present invention, the price budget includes a budget price range, and determining the second glasses sample from the first glasses sample according to the price budget of the user includes:

[0111] Obtaining price information of the first glasses sample;

[0112] From the first glasses samples, determine the first glasses sample whose price information is within the user's budget price range as the second glasses sample.

[0113] In some examples, the price information of the glasses sample may be the sum of the frame price information and the lens price information of the glasses sample; for example, in glasses sample A, the frame price information is 200 yuan, and the lens price information is 300 yuan, then the price information of the glasses sample is 500 yuan.

[0114] In actual applications, the budget price range given by the user can be combined to screen out first glasses samples that meet the budget price range from one or more first glasses samples as second glasses samples; for example, the first glasses samples are A, B, and C, the price information of glasses sample A is 600 yuan, the price information of glasses sample B is 900 yuan, and the price information of glasses sample C is 1200 yuan, and the price budget (budget price range) given by the user is 500-1000, then the first glasses samples A and B are used as second glasses samples, and the second glasses samples are recommended to the user.

[0115] In some examples, based on the price budget given by the user, second glasses samples whose price information of the first glasses samples is within the budget price range are screened out, which can be calculated by the following formula:

[0116] G = {g i ∈EyeglassSamples|p i ∈[P min , P max ]}

[0117] Where G represents the second glasses sample; g i Refers to the feature vector of the glasses sample; EyeglassSamples represents the first glasses sample; p i represents the user's price budget; P min Indicates the minimum value of the budget price range given by the user; P max Indicates the maximum value of the budget price range given by the user; if there are multiple second glasses samples that meet the requirements, the user can choose them.

[0118] In some embodiments of the present invention, after recommending the second glasses sample to the user, the method further includes:

[0119] In response to the user's selection operation on the second glasses sample, determining a target glasses sample from the second glasses samples;

[0120] Glasses are manufactured based on the target glasses sample.

[0121] After the second glasses sample is recommended to the user, the user can perform a selection operation through the interactive interface to select a target glasses sample from the second glasses sample, and the optician makes the glasses based on the target glasses sample determined by the user.

[0122] In some examples, if the second glasses samples recommended to the user are A, B, and C, and the user selects glasses sample B as the target glasses sample through the interactive interface, the optician can find the frames and lenses from the inventory to make the glasses based on the frame and lens information in glasses sample B.

[0123] In an embodiment of the present invention, a user's face is scanned by structured light to obtain three-dimensional facial data of the user, and target image data is generated based on the three-dimensional facial data; a facial feature vector is extracted from the target image data by a pre-trained data model, and a first glasses sample is determined from a preset digital glasses library based on the facial feature vector; a second glasses sample is determined from the first glasses sample based on the user's price budget, and the second glasses sample is recommended to the user, thereby achieving screening of glasses samples that meet the user's facial features and price budget, thereby reducing the time the user spends selecting and trying on glasses.

[0124] The following combination Figure 2 The present invention is exemplified as follows:

[0125] S201, the customer selects a glasses fitting method (automatic glasses fitting or conventional glasses fitting) and proceeds to the next step S2 if automatic glasses fitting is selected.

[0126] S202, conduct vision test and cost estimation for the customer.

[0127] In actual application, the customer's vision needs to be tested during the glasses fitting process, such as standard eye chart screening, eye examination, etc. Because each person's myopia or astigmatism degree is different, it is necessary to make a cost estimate based on the vision test results for the customer's reference, and then determine the budget price range given by the customer.

[0128] S203, performing three-dimensional facial scanning through structured light.

[0129] In practical applications, structured light is a technology that uses a projector to project specific light information onto the surface of an object, and uses a camera to collect changes in light signals to calculate information such as the position and depth of the object.

[0130] Specifically, structured light is a system structure composed of a projector and a camera. The projector can project specific light information onto the surface of an object and the background, which is then collected by the camera. The position, depth and other information of the object are calculated based on the changes in the light signal caused by the object, and the entire three-dimensional space is restored.

[0131] During the automatic glasses fitting process, the user's face can be scanned by structured light, and the user's facial three-dimensional data can be obtained based on the image data collected by the camera.

[0132] In some examples, the three-dimensional facial data may be converted into a three-dimensional voxel map to obtain target image data; or the three-dimensional facial data may be projected onto a two-dimensional depth image to obtain target image data.

[0133] As some examples, a three-dimensional voxel map may refer to a three-dimensional image composed of voxels.

[0134] Specifically, voxel is the abbreviation of volume pixel, which is the smallest unit for three-dimensional space segmentation, similar to pixels in two-dimensional space. Each voxel represents a spatial volume in a unified three-dimensional grid, usually a cube. Voxels are used to store volume data. Voxel maps are implemented by dividing three-dimensional space into voxels in the form of cubes. Each voxel has a position and an associated value, such as density or color, to form a three-dimensional voxel image.

[0135] In some examples, after obtaining the user's three-dimensional facial data, the three-dimensional facial data can be converted into a three-dimensional voxel map through image processing software, and the three-dimensional voxel map is used as target image data.

[0136] As some examples, since height or depth information can be stored in three-dimensional data, the three-dimensional facial data can be projected onto a two-dimensional depth image, and the projected two-dimensional depth image is used as the target image data.

[0137] S204, facial feature extraction.

[0138] In some examples, after obtaining the target image data, the target image data can be input into a pre-trained convolutional neural network model for processing, and the convolutional neural network model can extract facial feature vectors from the target image data through multiple convolutional layers.

[0139] As some examples, the computational logic of a convolutional neural network model to extract facial feature vectors from target image data through multiple convolutional layers can be as follows:

[0140] The first convolutional layer operation:

[0141]

[0142] in, Refers to the value at the (i, j, k)th position in the output feature map Z; i, j, k represent the spatial position of the feature map; б is the activation function; is the value corresponding to position X i+m，j+n，lThe first layer of convolution kernel; M / m, N / n, L / l correspond to the height, width and depth of the convolution kernel respectively.

[0143] Pooling layer operation:

[0144]

[0145] in, Refers to the value of the feature map at position (i, j, k) after the pooling operation; i, j, k represent the spatial position of the feature map; max m，n Refers to operating on the maximum value of all elements in the pooling window; is the feature map before pooling, s is the stride of the pooling operation, and m and n are the height and width of the convolution kernel, respectively.

[0146] Fully connected layer operation:

[0147] f(X)=σ(W (FC) Z (2) +b FC )

[0148] Where f(X) refers to the facial feature vector; б is the activation function; W (FC) is the weight of the fully connected layer; Z (2) Represents the output after the pooling layer; b FC is the bias of the feature map.

[0149] S205, facial features (facial feature vectors) are matched with glasses (glasses samples) (similarity calculation).

[0150] In some examples, after determining the facial feature vector, glasses sample feature vectors corresponding to multiple glasses samples can be obtained from a preset digital glasses library, and feature similarities between the facial feature vector and the glasses sample feature vectors of the multiple glasses samples are calculated respectively. The specific formula is as follows:

[0151]

[0152] Among them, similarity refers to feature similarity; f(X) refers to the facial feature vector; g i Refers to the feature vector of the glasses sample.

[0153] In some examples, after respectively calculating the feature similarities between the facial feature vector and the glasses sample feature vectors of multiple glasses samples, the glasses samples can be sorted according to the calculation results of the feature similarities, and one or more glasses samples ranked first can be determined as the first glasses sample.

[0154] S206, screening glasses that meet the price range (second glasses sample).

[0155] In actual applications, the budget price range given by the user can be combined to screen out first glasses samples that meet the budget price range from one or more first glasses samples as second glasses samples; for example, the first glasses samples are A, B, and C, the price information of glasses sample A is 600 yuan, the price information of glasses sample B is 900 yuan, and the price information of glasses sample C is 1200 yuan, and the price budget (budget price range) given by the user is 500-1000, then the first glasses samples A and B are used as second glasses samples, and the second glasses samples are recommended to the user.

[0156] In some examples, based on the price budget given by the user, second glasses samples whose price information of the first glasses samples is within the budget price range are screened out, which can be calculated by the following formula:

[0157] G = {g i ∈EyeglassSamples|p i ∈[P min , P max ]}

[0158] Where G represents the second glasses sample; g i Refers to the feature vector of the glasses sample; EyeglassSamples represents the first glasses sample; p i represents the user's price budget; P min Indicates the minimum value of the budget price range given by the user; P max Indicates the maximum value of the budget price range given by the user; if there are multiple second glasses samples that meet the requirements, the user can choose them.

[0159] S207, glasses recommendation and confirmation of glasses production.

[0160] After the second glasses sample is recommended to the user, the user can perform a selection operation through the interactive interface to select a target glasses sample from the second glasses sample, and the optician makes the glasses based on the target glasses sample determined by the user.

[0161] Through some embodiments of the present invention, automatic glasses can be fitted efficiently and quickly, which improves the efficiency of glasses fitting and saves time costs.

[0162] Reference Figure 3 , shows a flowchart of another method for automatic eyeglass fitting provided by an embodiment of the present invention, which may specifically include the following steps:

[0163] Step 301, perform 3D scanning on the glasses to obtain a sample feature vector of the glasses.

[0164] In some examples, the frame and lenses of the glasses may be 3D scanned to obtain the geometric shape and surface features of the frame and lenses of the glasses, and determine the glasses sample feature vector of the glasses; wherein the glasses sample feature vector includes the feature vectors of the frame and lenses, such as the formula:

[0165]

[0166] Among them, g i refers to the feature vector of the glasses sample; refers to the feature vector of the frame; refers to the eigenvector of the lens.

[0167] Step 302: Create glasses samples in a preset digital glasses library according to the glasses sample feature vector.

[0168] After the glasses sample feature vector is determined, a glasses sample of the glasses may be created in a preset digital glasses library according to the glasses sample feature vector.

[0169] In practical applications, multiple pairs of glasses can be collected for 3D scanning, their corresponding glasses sample feature vectors can be determined, and multiple glasses samples can be created in a preset digital glasses library.

[0170] As some examples, data annotation and classification can also be performed on each glasses sample, such as the price and size of the frames and lenses in the glasses sample, to ensure that the frame and lens information of each glasses sample can be extracted and used separately, and each glasses sample has its corresponding feature vector to represent it, which is convenient for matching with the customer's facial features.

[0171] Step 303 , scanning the user's face by structured light to obtain the user's three-dimensional facial data, and generating target image data based on the three-dimensional facial data.

[0172] In practical applications, structured light is a technology that uses a projector to project specific light information onto the surface of an object, and uses a camera to collect changes in light signals to calculate information such as the position and depth of the object.

[0173] Specifically, structured light is a system structure composed of a projector and a camera. The projector can project specific light information onto the surface of an object and the background, which is then collected by the camera. The position, depth and other information of the object are calculated based on the changes in the light signal caused by the object, and the entire three-dimensional space is restored.

[0174] During the automatic glasses fitting process, the user's face can be scanned by structured light, and the user's facial three-dimensional data can be obtained based on the image data collected by the camera.

[0175] In some examples, the three-dimensional facial data may be converted into a three-dimensional voxel map to obtain target image data; or the three-dimensional facial data may be projected onto a two-dimensional depth image to obtain target image data.

[0176] As some examples, a three-dimensional voxel map may refer to a three-dimensional image composed of voxels.

[0177] Specifically, voxel is the abbreviation of volume pixel, which is the smallest unit for three-dimensional space segmentation, similar to pixels in two-dimensional space. Each voxel represents a spatial volume in a unified three-dimensional grid, usually a cube. Voxels are used to store volume data. Voxel maps are implemented by dividing three-dimensional space into voxels in the form of cubes. Each voxel has a position and an associated value, such as density or color, to form a three-dimensional voxel image.

[0178] In some examples, after obtaining the user's three-dimensional facial data, the three-dimensional facial data can be converted into a three-dimensional voxel map through image processing software, and the three-dimensional voxel map is used as target image data.

[0179] As some examples, since height or depth information can be stored in three-dimensional data, the three-dimensional facial data can be projected onto a two-dimensional depth image, and the projected two-dimensional depth image is used as the target image data.

[0180] Step 304 : extracting a facial feature vector from the target image data through a pre-trained data model, and determining a first glasses sample from a preset digital glasses library according to the facial feature vector.

[0181] In some examples, the data model can be a convolutional neural network model, wherein the convolutional neural network model can include a convolutional layer, a pooling layer, and a fully connected layer; wherein the convolutional layer is used to extract image features, the pooling layer reduces the spatial dimension of the feature map, reduces the model complexity and computational complexity; and the full connection is used for the final features and classification.

[0182] As some examples, let's take a simplified convolutional neural network structure:

[0183] The input data is:

[0184] X∈R H×W×D

[0185] Among them, X represents the input data; R represents three-dimensional data; H represents height, which refers to the vertical size of the image; W represents width, which refers to the horizontal size of the image; D represents depth, which refers to the size of the image in the direction perpendicular to the screen plane in three-dimensional space, and can also refer to the image data dimension or information depth; RH×V×D Indicates that the input data X is a tensor of dimension HxWxD.

[0186] The first convolutional layer input feature map:

[0187]

[0188] Among them, Z (1) It represents the output after the first layer of convolution; R represents the dimension and data type of this feature map; H 1 , W 1 , C 1 are the output height, width and number of channels of the first convolution layer respectively.

[0189] Output after the pooling layer:

[0190]

[0191] Among them, Z (2) Represents the output after the pooling layer; R represents the dimension and data type of this feature map; H 2 , W 2 , C 2 are the output height, width, and number of channels of the pooling layer, respectively.

[0192] Flatten the pooled feature map into a vector The flattened feature vector is then input into the fully connected layer to obtain the final feature representation:

[0193] f(X)∈R F

[0194] Among them, f(X) refers to the facial feature vector; R represents the dimension and data type of this feature map; F is the dimension of the feature vector.

[0195] In some examples, after obtaining the target image data, the target image data can be input into a pre-trained convolutional neural network model for processing, and the convolutional neural network model can extract facial feature vectors from the target image data through multiple convolutional layers.

[0196] As some examples, the computational logic of a convolutional neural network model to extract facial feature vectors from target image data through multiple convolutional layers can be as follows:

[0197] The first convolutional layer operation:

[0198]

[0199] in, Refers to the value at the (i, j, k)th position in the output feature map Z; i, j, k represent the spatial position of the feature map; б is the activation function; is the value corresponding to position X i+m，j+n，l The first layer of convolution kernel; M / m, N / n, L / l correspond to the height, width and depth of the convolution kernel respectively.

[0200] Pooling layer operation:

[0201]

[0202] in, Refers to the value of the feature map at position (i, j, k) after the pooling operation; i, j, k represent the spatial position of the feature map; max m，n Refers to operating on the maximum value of all elements in the pooling window; is the feature map before pooling, s is the stride of the pooling operation, and m and n are the height and width of the convolution kernel, respectively.

[0203] Fully connected layer operation:

[0204] f(X)=σ(W (FC) Z (2) +b FC )

[0205] Where f(X) refers to the facial feature vector; б is the activation function; W (FC) is the weight of the fully connected layer; Z (2 ) represents the output after the pooling layer; b FC is the bias of the feature map.

[0206] In some examples, after determining the facial feature vector, glasses sample feature vectors corresponding to multiple glasses samples can be obtained from a preset digital glasses library, and feature similarities between the facial feature vector and the glasses sample feature vectors of the multiple glasses samples are calculated respectively. The specific formula is as follows:

[0207]

[0208] Among them, similarity refers to feature similarity; f(X) refers to the facial feature vector; g i Refers to the feature vector of the glasses sample.

[0209] In some examples, after respectively calculating the feature similarities between the facial feature vector and the glasses sample feature vectors of multiple glasses samples, the glasses samples can be sorted according to the calculation results of the feature similarities, and one or more glasses samples ranked first can be determined as the first glasses sample.

[0210] Step 305: Determine a second pair of glasses sample from the first pair of glasses sample according to the price budget of the user, and recommend the second pair of glasses sample to the user.

[0211] In some examples, the price budget may be a budget price range.

[0212] In actual applications, the budget price range given by the user can be combined to screen out first glasses samples that meet the budget price range from one or more first glasses samples as second glasses samples; for example, the first glasses samples are A, B, and C, the price information of glasses sample A is 600 yuan, the price information of glasses sample B is 900 yuan, and the price information of glasses sample C is 1200 yuan, and the price budget (budget price range) given by the user is 500-1000, then the first glasses samples A and B are used as second glasses samples, and the second glasses samples are recommended to the user.

[0213] In some examples, based on the price budget given by the user, second glasses samples whose price information of the first glasses samples is within the budget price range are screened out, which can be calculated by the following formula:

[0214] G={gi∈EyeglassSamples|pi∈[Pm in , Pma x ]}

[0215] Where G represents the second glasses sample; g i Refers to the feature vector of the glasses sample; EyeglassSamples represents the first glasses sample; p i represents the user's price budget; P min Indicates the minimum value of the budget price range given by the user; P max Indicates the maximum value of the budget price range given by the user; if there are multiple second glasses samples that meet the requirements, the user can choose them.

[0216] After the second glasses sample is recommended to the user, the user can perform a selection operation through the interactive interface to select a target glasses sample from the second glasses sample, and the optician makes the glasses based on the target glasses sample determined by the user.

[0217] In some examples, if the second glasses samples recommended to the user are A, B, and C, and the user selects glasses sample B as the target glasses sample through the interactive interface, the optician can find the frames and lenses from the inventory to make the glasses based on the frame and lens information in glasses sample B.

[0218] In an embodiment of the present invention, a glasses sample feature vector is obtained by performing 3D scanning on the glasses; a glasses sample is created in a preset digital glasses library according to the glasses sample feature vector; a user's face is scanned by structured light to obtain the user's three-dimensional facial data, and target image data is generated according to the three-dimensional facial data; then, a facial feature vector is extracted from the target image data according to a pre-trained data model, and a first glasses sample is determined from the preset digital glasses library according to the facial feature vector; a second glasses sample is determined from the first glasses sample according to the user's price budget, and the second glasses sample is recommended to the user, thereby implementing screening of glasses samples that meet the user's facial features and price budget, so as to reduce the time the user spends selecting and trying on glasses.

[0219] Reference Figure 4 , shows a flowchart of another method for automatic eyeglass fitting provided by an embodiment of the present invention, which may specifically include the following steps:

[0220] Step 401, perform 3D scanning on the glasses to obtain a sample feature vector of the glasses.

[0221] Step 402: Create glasses samples in a preset digital glasses library according to the glasses sample feature vector.

[0222] Step 403: pre-train the data model.

[0223] In some embodiments of the present invention, the data model is a convolutional neural network model.

[0224] As some examples, 3D modeling software can be used to build different facial models, and the facial model three-dimensional data of the facial model can be obtained, and the facial model three-dimensional data can be converted into a three-dimensional voxel map to obtain image data; or, the facial model three-dimensional data can be projected onto a two-dimensional depth image to obtain image data, and then a convolutional neural network model can be trained based on the image data to optimize the reliability of the model in extracting facial feature vectors.

[0225] In some examples, a convolutional neural network model may include a convolutional layer, a pooling layer, and a fully connected layer; wherein the convolutional layer is used to extract image features, the pooling layer reduces the spatial dimension of the feature map, reduces the model complexity and computational complexity; and the fully connected layer is used for final features and classification.

[0226] As some examples, let's take a simplified convolutional neural network structure:

[0227] The input data is:

[0228] X∈R H×W×D

[0229] Among them, X represents the input data; R represents three-dimensional data; H represents height, which refers to the vertical size of the image; W represents width, which refers to the horizontal size of the image; D represents depth, which refers to the size of the image in the direction perpendicular to the screen plane in three-dimensional space, and can also refer to the image data dimension or information depth; R H×W×D Indicates that the input data X is a tensor of dimension HxWxD.

[0230] The first convolutional layer input feature map:

[0231]

[0232] Among them, Z (1) It represents the output after the first layer of convolution; R represents the dimension and data type of this feature map; H 1 , W 1 , C 1 are the output height, width and number of channels of the first convolution layer respectively.

[0233] Output after the pooling layer:

[0234]

[0235] Among them, Z (2) Represents the output after the pooling layer; R represents the dimension and data type of this feature map; H 2 , W 2 , C 2 are the output height, width, and number of channels of the pooling layer, respectively.

[0236] Flatten the pooled feature map into a vector The flattened feature vector is then input into the fully connected layer to obtain the final feature representation:

[0237] f(X)∈R F

[0238] Among them, f(X) refers to the facial feature vector; R represents the dimension and data type of this feature map; F is the dimension of the feature vector.

[0239] As some examples, the computational logic of a convolutional neural network model to extract facial feature vectors from image data through multiple convolutional layers can be as follows:

[0240] The first convolutional layer operation:

[0241]

[0242] in, Refers to the value at the (i, j, k)th position in the output feature map Z; i, j, k represent the spatial position of the feature map; б is the activation function; is the value corresponding to position X i+m,j+n,l The first layer of convolution kernel; M / m, N / n, L / l correspond to the height, width and depth of the convolution kernel respectively.

[0243] Pooling layer operation:

[0244]

[0245] in, Refers to the value of the feature map at position (i, j, k) after the pooling operation; i, j, k represent the spatial position of the feature map; max m,n Refers to operating on the maximum value of all elements in the pooling window; is the feature map before pooling, s is the stride of the pooling operation, and m and n are the height and width of the convolution kernel, respectively.

[0246] Fully connected layer operation:

[0247] f(X)=σ(W (FC) Z (2) +b FC )

[0248] Where f(X) refers to the facial feature vector; б is the activation function; W (FC) is the weight of the fully connected layer; Z (2) Represents the output after the pooling layer; b FC is the bias of the feature map.

[0249] Step 404 , scan the user's face by structured light to obtain the user's three-dimensional facial data, and generate target image data based on the three-dimensional facial data.

[0250] Step 405 , extracting a facial feature vector from the target image data through a pre-trained data model, and determining a first glasses sample from a preset digital glasses library according to the facial feature vector.

[0251] In some examples, after obtaining the target image data, the target image data can be input into a pre-trained convolutional neural network model for processing, and the convolutional neural network model can extract facial feature vectors from the target image data through multiple convolutional layers.

[0252] In some examples, after determining the facial feature vector, glasses sample feature vectors corresponding to multiple glasses samples can be obtained from a preset digital glasses library, and feature similarities between the facial feature vector and the glasses sample feature vectors of the multiple glasses samples are calculated respectively. The specific formula is as follows:

[0253]

[0254] Among them, similarity refers to feature similarity; f(X) refers to the facial feature vector; g i Refers to the feature vector of the glasses sample.

[0255] In some examples, after respectively calculating the feature similarities between the facial feature vector and the glasses sample feature vectors of multiple glasses samples, the glasses samples can be sorted according to the calculation results of the feature similarities, and one or more glasses samples ranked first can be determined as the first glasses sample.

[0256] Step 406: Determine a second pair of glasses sample from the first pair of glasses sample according to the price budget of the user, and recommend the second pair of glasses sample to the user.

[0257] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0258] Reference Figure 5 , shows a schematic structural diagram of an automatic glasses fitting device provided by an embodiment of the present invention, which may specifically include the following modules:

[0259] The target image data generating module 501 is used to scan the face of the user by structured light to obtain the three-dimensional facial data of the user, and generate the target image data according to the three-dimensional facial data;

[0260] A first glasses sample determining module 502 is used to extract a facial feature vector from the target image data by using a pre-trained data model, and determine a first glasses sample from a preset digital glasses library according to the facial feature vector;

[0261] The second glasses sample determining module 503 is used to determine a second glasses sample from the first glasses samples according to the price budget of the user, and recommend the second glasses sample to the user.

[0262] In some embodiments of the present invention, the target image data generating module 501 includes:

[0263] A facial three-dimensional data conversion submodule, used to convert the facial three-dimensional data into a three-dimensional voxel image to obtain target image data;

[0264] The facial three-dimensional data projection submodule is used to project the facial three-dimensional data onto a two-dimensional depth image to obtain target image data.

[0265] In some embodiments of the present invention, the first glasses sample determination module 502 includes:

[0266] A feature similarity calculation submodule, used for obtaining glasses sample feature vectors of a plurality of glasses samples in a preset digital glasses library, and calculating feature similarities between the facial feature vector and the glasses sample feature vectors of the plurality of glasses samples;

[0267] The first glasses sample determining submodule is used to determine a first glasses sample from the multiple glasses samples according to the feature similarity.

[0268] In some embodiments of the present invention, the first glasses sample determination submodule includes:

[0269] A sorting unit, configured to sort the plurality of glasses samples according to the feature similarity;

[0270] The first glasses sample determining unit is used to determine one or more glasses samples that are ranked first as first glasses samples.

[0271] In some embodiments of the present invention, the device is also used for:

[0272] By performing 3D scanning on the glasses, the feature vector of the glasses samples is obtained;

[0273] According to the glasses sample feature vector, glasses samples are created in a preset digital glasses library.

[0274] In some embodiments of the present invention, the second glasses sample determination module 503 includes:

[0275] A price information acquisition submodule, used to acquire price information of the first glasses sample;

[0276] The budget price range determination submodule is used to determine, from the first glasses samples, the first glasses sample whose price information is within the user's budget price range as the second glasses sample.

[0277] In some embodiments of the present invention, the data model is a convolutional neural network model.

[0278] In some embodiments of the present invention, the first glasses sample determination module 502 includes:

[0279] A target image data input submodule, used for inputting the target image data into a pre-trained convolutional neural network model, wherein the convolutional neural network model extracts the facial feature vector from the target image data through multiple convolutional layers;

[0280] The facial feature vector receiving submodule is used to receive the facial feature vector output by the convolutional neural network model.

[0281] In some embodiments of the present invention, the device further comprises:

[0282] a target glasses sample determining module, configured to determine a target glasses sample from the second glasses samples in response to the user's selection operation on the second glasses sample;

[0283] The glasses manufacturing module is used to manufacture glasses based on the target glasses sample.

[0284] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the above-mentioned automatic glasses fitting method is implemented.

[0285] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned method of automatic eyeglass fitting is implemented.

[0286] An embodiment of the present invention further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the method for automatically fitting glasses is implemented as described above.

[0287] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0288] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0289] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0290] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0291] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0292] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0293] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0294] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0295] The above is a detailed introduction to the provided method and device for automatic eyeglass fitting, electronic device, and storage medium. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for automatically fitting glasses, It is characterized in that The method comprises: Scanning the face of a user by structured light to obtain three-dimensional facial data of the user, and generating target image data based on the three-dimensional facial data; Extracting a facial feature vector from the target image data by using a pre-trained data model, and determining a first glasses sample from a preset digital glasses library according to the facial feature vector; According to the price budget of the user, a second glasses sample is determined from the first glasses sample, and the second glasses sample is recommended to the user.

2. The method according to claim 1, It is characterized in that The step of generating target image data according to the facial three-dimensional data comprises: Converting the facial three-dimensional data into a three-dimensional voxel image to obtain target image data; Alternatively, the three-dimensional facial data is projected onto a two-dimensional depth image to obtain target image data.

3. The method according to claim 1, It is characterized in that The step of determining a first glasses sample from a preset digital glasses library according to the facial feature vector comprises: Acquire glasses sample feature vectors of a plurality of glasses samples in a preset digital glasses library, and calculate feature similarity between the facial feature vector and the glasses sample feature vectors of the plurality of glasses samples; A first glasses sample is determined from the multiple glasses samples according to the feature similarity.

4. The method according to claim 3, It is characterized in that The step of determining a first glasses sample from the plurality of glasses samples according to the feature similarity comprises: sorting the plurality of glasses samples according to the feature similarities; One or more glasses samples ranked first are determined as first glasses samples.

5. The method according to claim 3, It is characterized in that Before scanning the user's face by structured light to obtain the user's three-dimensional facial data, the method further includes: By performing 3D scanning on the glasses, the feature vector of the glasses samples is obtained; According to the glasses sample feature vector, glasses samples are created in a preset digital glasses library.

6. The method according to any one of claims 1 to 5, It is characterized in that The price budget includes a budget price range, and determining the second glasses sample from the first glasses sample according to the price budget of the user includes: Obtaining price information of the first glasses sample; From the first glasses samples, determine the first glasses sample whose price information is within the user's budget price range as the second glasses sample.

7. The method according to any one of claims 1 to 5, It is characterized in that The data model is a convolutional neural network model, and the facial feature vector is extracted from the target image data through the pre-trained data model, including: Inputting the target image data into a pre-trained convolutional neural network model, the convolutional neural network model extracting the facial feature vector from the target image data through multiple convolutional layers; Receive the facial feature vector output by the convolutional neural network model.

8. The method according to any one of claims 1 to 5, It is characterized in that After recommending the second glasses sample to the user, the method further includes: In response to the user's selection operation on the second glasses sample, determining a target glasses sample from the second glasses samples; Glasses are manufactured based on the target glasses sample.

9. An automatic glasses fitting device, It is characterized in that The device comprises: A target image data generating module, configured to scan the face of a user by structured light to obtain three-dimensional facial data of the user, and generate target image data based on the three-dimensional facial data; A first glasses sample determination module, configured to extract a facial feature vector from the target image data by using a pre-trained data model, and determine a first glasses sample from a preset digital glasses library according to the facial feature vector; The second glasses sample determining module is used to determine a second glasses sample from the first glasses samples according to the price budget of the user, and recommend the second glasses sample to the user.

10. An electronic device, It is characterized in that The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 8 when executed by the processor.

11. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, It is characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images