A method and system for identifying and processing beauty contact lenses based on deep neural network

Abstract

The present invention discloses a method and system for identifying and processing beautiful contact lenses based on a deep neural network. The method comprises acquiring an eye region image and dividing it into an inner eye region image and an outer eye region image; calculating the grayscale mean of the inner eye region image; performing mask processing on the upper eyelid region and the lower eyelid region of the outer eye region image according to the grayscale mean; calculating the mean and standard deviation of the eye region image and inputting them into a constructed deep neural network model to normalize the pixel values ​​of the eye region image to obtain a data set; training the deep neural network model according to the data set; acquiring the eye region image for preprocessing to obtain the mean and standard deviation of the eye region image and transmitting them to the deep neural network model for processing to obtain classification result data; the system comprises an information acquisition module, an image data operation module, a neural network model construction module, a normalization operation module, a neural network training module, and a prediction module.

Description

Technical Field

[0001] The present invention relates to the field of methods and systems for identifying and processing colored contact lenses, and in particular to a method and system for identifying and processing colored contact lenses based on a deep neural network. Background Art

[0002] In iris recognition, the presence of people wearing colored contact lenses, or cosmetic contact lenses, will interfere with the results of iris recognition. Because the pattern of cosmetic contact lenses is superimposed on the texture of the iris itself, it will cause the user's iris texture features to change, resulting in the user's iris being unable to be correctly recognized. Therefore, when performing iris recognition, it is necessary to detect whether the user is wearing cosmetic contact lenses. Currently, there are the following patents on the market to meet the needs of cosmetic contact lens recognition in iris recognition:

[0003] Patent No. 2019110013640 for the method and device for detecting colored contact lenses for iris recognition;

[0004] Patent number is 2019109953829, a method and system for identifying beauty contact lenses based on deep learning;

[0005] Patent number is 2019109953890 A method and system for identifying colored contact lenses based on transfer learning;

[0006] Among them, the method and device for detecting beauty pupils used in iris recognition with patent number 2019110013640 need to first perform local binarization processing on the iris area image to obtain the LBP features of the iris area image, and then form a grayscale co-occurrence matrix based on the local binary image to obtain the GLCM features, and then combine the LBP features and GLCM features of the iris image and input them into a pre-trained beauty pupil detection model to output the beauty pupil detection results. This method is not only complicated and difficult to implement, but also has too many preprocessing links, which can easily lose information or introduce errors. In addition, in practical applications, due to various changes in lighting conditions and the existence of light spot interference, this method cannot meet the needs of accurate recognition of large-capacity samples in real scenes;

[0007] Patent No. 2019109953829 is a method and system for identifying beauty contact lenses based on deep learning. It uses deep learning technology to meet the needs of beauty contact lens identification. In the data preprocessing stage before network learning, this invention requires that the non-iris area in the sample be manually masked (the pupil part is filled with 0, and the iris and the area outside the pupil are filled with 255), and the iris area is locally binarized before network training. Due to the huge workload of manually marking masks, it becomes very difficult to train with a large number of learning samples. At the same time, the operation of local binarization of the iris area is prone to loss of information or interference from light spots, which affects the correctness of the results.

[0008] Patent No. 2019109953890, a method and system for identifying beauty contact lenses based on transfer learning, also uses deep learning technology to meet the needs of beauty contact lens identification. However, in order to achieve multiple iterative training of deep neural networks, this invention requires that the positive and negative sample sets be reclassified into multiple sample subsets through a clustering algorithm, and the iris area be expanded into a rectangular area and then normalized to obtain a feature template for training, learning, and reasoning comparison. This invention algorithm has too many operation links, which is not only too complicated and difficult to implement, but also difficult to ensure the absolute accuracy of the effect at each step. The more links in the algorithm, the easier it is to introduce errors or omit information;

[0009] In summary, the current methods for identifying cosmetic contact lenses have the following inconveniences in the data preprocessing stage before starting neural network training:

[0010] The pretreatment process is complicated, with many steps, and it is difficult to implement and obtain satisfactory results;

[0011] The mask needs to be filled manually, which is labor-intensive and cannot be generalized in the case of massive samples.

[0012] In the data preprocessing stage, the information irrelevant to the classification decision cannot be effectively filtered out, which increases the difficulty of network learning. Summary of the invention

[0013] The technical problem to be solved by the present invention is that the current method for identifying colored pupils has the following inconveniences in the data preprocessing stage before starting neural network training: the preprocessing process is complicated, with many links, which is difficult to implement and it is difficult to obtain satisfactory results; it is necessary to fill in the mask manually, which has a high labor cost and cannot be promoted in the case of massive samples; in the data preprocessing stage, it is not possible to effectively filter out information irrelevant to the classification decision, which increases the difficulty of network learning. The present invention provides a method for identifying and processing colored pupils based on a deep neural network, and also provides a system for identifying and processing colored pupils based on a deep neural network. The present invention also provides a system for identifying and processing colored pupils, which can automatically distinguish the inner eye area image and the outer eye area image domain in the eye area image through a semantic segmentation algorithm, and perform mask filling, thereby saving a lot of manpower for manual labeling, so that the network can be trained using massive samples; only the eyelid area that is easy to segment needs to be mask filled, and the pupil and sclera parts do not need to be filled, which reduces the workload of preprocessing and increases the robustness of the network, so as to solve the defects caused by the prior art.

[0014] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0015] In a first aspect, a method for identifying and processing colored contact lenses based on a deep neural network comprises the following steps:

[0016] Step A1: Acquire an eye region image;

[0017] Step A2: dividing the eye region image into an inner eye region image and an outer eye region image;

[0018] Step A3: calculating the grayscale mean of the image of the inner eye area, and performing mask processing on the upper eyelid area and the lower eyelid area of ​​the image of the outer eye area according to the grayscale mean;

[0019] Step A4: Calculating the mean and standard deviation of the eye region image;

[0020] Step A5: building a deep neural network model, inputting the mean and standard deviation into the deep neural network model to normalize the pixel values ​​of the eye area image to obtain a data set;

[0021] Step A6: training the deep neural network model according to the data set;

[0022] Step A7: Reacquire the eye area image and pre-process it through step A2-step A4 to obtain the mean and standard deviation of the eye area image and transmit it to the deep neural network model obtained in step A6 for processing to obtain classification result data.

[0023] The deep neural network model includes an input layer, a normalization layer, a Block layer, and an output layer. Considering the actual application environment of beauty contact lens recognition, the technical solution provides two processing systems, namely a CPU system and an ARM+DSP system considering embedded requirements. The beauty contact lens recognition processing method is divided into two processes, training and reasoning. The training process needs to be completed on the CPU platform, while the reasoning process can be completed on the CPU platform or on the ARM+DSP embedded platform. On the ARM+DSP embedded platform, the ARM device first performs part of the eye area image preprocessing process, including reading the eye area image, automatically segmenting the inner eye area and the outer eye area of ​​the image, calculating the average grayscale of the inner eye area and masking the outer eye area; the image obtained after masking will be saved as a two-dimensional unsigned array, and the ARM transmits the two-dimensional unsigned array to the DSP. After the DSP performs normalization processing on the image data, it is passed to the Block layer for reasoning to obtain the classification result of beauty contact lens or living body;

[0024] The input layer reads the masked image;

[0025] The normalization layer is used to complete the individual normalization of each eye image data. Considering that the beauty contact lens recognition often has the demand for application in embedded devices in practical applications, it is necessary to transplant the network model to an embedded device, such as a DSP. However, the DSP can usually only support fixed-point input and cannot accept floating-point input. After the input sample image is normalized, the data will be converted from fixed-point to floating-point, so that the input cannot be passed to the DSP without losing accuracy. Therefore, the present invention designs a separate normalization layer (SN layer) embedded in the deep neural network model, and the DSP implements the normalization operation to avoid the loss of calculation accuracy.

[0026] The Block layer includes five convolutional layers and three fully connected layers connected in sequence, wherein the convolution kernel size of the convolutional layer is 3*3, and each convolutional layer is connected to a mean pooling layer, the pooling kernel size is 3*3, and the step size is 2. The loss function of the deep neural network model is the cross entropy loss function.

[0027] The output layer is a Softmax classifier with a decision threshold of 0.5. If the threshold is higher than 0.5, it is judged as a cosmetic contact lens, and if the threshold is lower than 0.5, it is judged as a living object.

[0028] In the above-mentioned method for processing beauty contact lens recognition based on deep neural network, an intelligent positioning and segmentation algorithm is used in step A2 to segment the inner eye area image and the outer eye area image, and the inner eye area image (including the white of the eye, the iris, and the pupil) and the outer eye area image (including the upper eyelid and the lower eyelid) are automatically distinguished in the eye area image, and only the outer eye area is masked, and the inner eye area, i.e., the pupil, the iris, and the white of the eye area, is not masked, and the mask uses the grayscale average value of the inner eye area, rather than a fixed constant such as 0 or 255. Since the texture feature information that can be used to distinguish between beauty contact lenses and living bodies only occupies a part of the area of ​​the entire sample image, in many scenes of beauty contact lens recognition, the redundant information area will be masked, so that when the network learns, the focus of learning can be focused on the key area that can be used to distinguish between beauty contact lenses and living bodies. Compared with other technical solutions related to beauty contact lens recognition, although the present technical solution also needs to be masked, the specific implementation method of the mask is very different from the masking methods in other beauty contact lens recognition technical solutions;

[0029] Other cosmetic contact lens processing algorithms on the market, such as the cosmetic contact lens recognition method and system based on deep learning with patent number 2019109953829, require the pupil area to be fixedly set to 0 during sample image mask preprocessing, and the area other than the pupil iris (eyelid + white of the eye) to 255. However, this technical solution only needs to perform mask processing on the outer area of ​​the eye, namely the upper eyelid area and the lower eyelid area. The white of the eye and the pupil area do not need to be masked. In this technical solution, the average value of the inner area of ​​the eye is used as the filling value of the mask outside the eye. Although the white of the eye and the pupil do not carry feature information that can distinguish cosmetic contact lenses from living bodies, the deep learning network itself has a strong ability to process redundant information. At the same time, the preprocessing operation The more steps there are, the easier it is to introduce errors or increase the labor cost of annotation. Too many and too strict mask constraints will sometimes only reduce the network's anti-interference ability. Therefore, the requirements for the mask area of ​​this technical solution are not strict. It is only necessary to mask the area outside the eye that can be clearly segmented, and it is not required to mask the pupil and white of the eye area. At the same time, this technical solution does not use a fixed value, such as 255 or 0, as the mask fill value, but uses the average of the inner area of ​​the eye, that is, the grayscale mean of the white of the eye, iris, and pupil area as the fill value of the eyelid area mask. In the subsequent normalization process, the eyelid area is filled with the grayscale mean of the entire eye area image, so all of them will be normalized to 0, which means no information, and thus will not cause any interference to the network's learning process.

[0030] In the above-mentioned method for identifying and processing colored contact lens based on deep neural network, the intelligent positioning and segmentation algorithm is a semantic segmentation algorithm, which automatically distinguishes the inner eye area image and the outer eye area image domain in the eye area image through the semantic segmentation algorithm.

[0031] In the above-mentioned method for identifying and processing beautiful contact lens based on deep neural network, the image of the inner area of ​​the eye includes the white of the eye, the iris, and the pupil, and the image of the outer area of ​​the eye includes the upper eyelid and the lower eyelid.

[0032] In step A3, the mask filling is automatically performed on the area outside the eyes, thereby avoiding the huge workload of manual labeling and making it possible to use massive samples for network training.

[0033] Each of the eye region images is normalized using the mean and standard deviation of the image itself, which is different from other patented methods that uniformly use the mean and standard deviation of the entire sample set for normalization or use a fixed constant for normalization. This is because for the beauty contact lens recognition scene, only texture is the main basis for distinguishing beauty contact lenses from living bodies. The mean and standard deviation of the sample itself do not carry any classification information. Each sample image is normalized using the mean and standard deviation of the image itself, which can effectively filter out the differences in grayscale mean and standard deviation between different samples, thereby reducing the difficulty of network learning;

[0034] This preprocessing method is simple to implement and does not require manual labeling. In addition, since the noise irrelevant to the recognition of cosmetic contact lenses is filtered out in the preprocessing stage and the key texture information for distinguishing cosmetic contact lenses from living objects is completely retained, the difficulty of neural network learning is reduced, allowing the deep neural network to achieve better training results.

[0035] At the same time, this preprocessing process is relatively simple. Different from other patents, this preprocessing method does not require local binarization, nor does it require calculating the LBP and GLCM features of the sample image. It also does not require expanding the iris area into a rectangle and dividing the sample set into several sample subsets through a clustering algorithm. While being easy to implement, it also avoids information loss or errors introduced by too many processing steps.

[0036] Different from the conventional method of using a fixed constant for normalization or using a sample set mean for normalization, the present invention uses a method of normalizing each sample individually.

[0037] The method for beauty contact lens recognition based on deep neural network provided by this technical solution is specifically divided into four parts: beauty contact lens recognition data preprocessing, deep neural network construction, neural network training process, and neural network reasoning process;

[0038] Neural network training process (data set training process): The deep neural network model that has been built is trained by using the eye area images (sample data) collected in advance, the data set is input into the deep neural network model through the input layer, and the classification result is obtained at the output layer, the learning error is obtained by calculating the difference between the output result and the expected output result, and the connection weights in the deep neural network model are adjusted according to the size of the learning error. After repeated reasoning and feedback, the inference output obtained by the deep neural network for each training input can be consistent with the expected output.

[0039] Neural network reasoning process: After the deep neural network model has been trained with the data set and meets the standards, it can be used in the actual application of beauty contact lens recognition. When a new eye area image is input, the beauty contact lens recognition data is first pre-processed, and then the trained deep neural network model is input to obtain the correct classification result of whether the beauty contact lens is alive.

[0040] In the second aspect, a system for identifying and processing beauty contact lens based on a deep neural network includes an information acquisition module, an image data operation module, a neural network model building module, a normalization operation module, a neural network training module, and a prediction module;

[0041] The information acquisition module is used to obtain an eye region image of the eye and transmit it to the image data calculation module;

[0042] The image data operation module is used to divide the eye region image into an inner eye region image and an outer eye region image and calculate the grayscale mean of the inner eye region image, perform mask processing on the upper eyelid region and the lower eyelid region of the outer eye region image according to the grayscale mean, and calculate the mean and standard deviation of the eye region image;

[0043] The neural network model building module is used to build a deep neural network model;

[0044] The normalization operation module is used to input the mean and standard deviation into the deep neural network model to perform normalization processing on the pixel values ​​of the eye area image to obtain a data set;

[0045] The neural network training module is used to obtain the data set and train the deep neural network model according to the data set;

[0046] The prediction module is used to reacquire the eye area image and preprocess it through the image data operation module to obtain the mean and standard deviation of the eye area image and transmit it to the trained deep neural network model for processing to obtain classification result data.

[0047] In the above-mentioned system for identifying and processing beautiful pupils based on deep neural networks, the image data operation module uses an intelligent positioning and segmentation algorithm to segment the image of the inner area of ​​the eye and the image of the outer area of ​​the eye;

[0048] The above-mentioned system for identifying and processing colored contact lens based on deep neural network, wherein the intelligent positioning and segmentation algorithm is a semantic segmentation algorithm, which automatically distinguishes the inner eye area image and the outer eye area image domain in the eye area image through the semantic segmentation algorithm.

[0049] The above-mentioned system for identifying and processing beautiful contact lens based on deep neural network, wherein the image of the inner area of ​​the eye includes the white of the eye, the iris, and the pupil, and the image of the outer area of ​​the eye includes the upper eyelid and the lower eyelid.

[0050] Considering that deep learning technology often requires the network model to be transplanted to DSP in practical applications, and DSP usually only supports fixed-point input and cannot accept floating-point input. After the input sample image is normalized, the data will be converted from fixed-point to floating-point, making it impossible to pass the input to DSP without losing accuracy.

[0051] Therefore, the technical solution adds a separate normalization layer (SN layer). The position of the SN layer in the entire processing network is between the input layer and the Block layer. The significance of setting the SN layer is that on the embedded platform, the DSP side completes the normalization operation of the fixed-point image, so as not to lose any precision and improve the accuracy of reasoning;

[0052] The data processing process of a single normalization layer can be expressed as follows:

[0053]

[0054] Among them, X′ i is a single new sample data obtained after normalization, X i is the data of this picture in the original sample set, is the grayscale mean of the image after masking, s i is the standard deviation of the image after masking, α is the adjustment parameter of the standard deviation, and its value is between 1 and 3, so that the single new sample data X′ obtained after normalization i The mean value is controlled to be 0, and the range is basically within the range of -1 to 1;

[0055] The Block layer includes five convolutional layers and three fully connected layers connected in sequence. The convolution kernel size of the convolutional layer is 3*3. Each convolutional layer is connected to a mean pooling layer with a pooling kernel size of 3*3 and a step size of 2. The loss function of the deep learning network model is the cross entropy loss function.

[0056] The output layer is a Softmax classifier with a decision threshold of 0.5. If the threshold is higher than 0.5, it is judged as cosmetic contact lenses, and if the threshold is lower than 0.5, it is judged as alive.

[0057] In a third aspect, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0058] The technical solution provided by the method and system for color contact lens recognition based on deep neural network of the present invention has the following technical effects:

[0059] The semantic segmentation algorithm is used to automatically distinguish the inner eye area image and the outer eye area image domain in the eye area image, and perform mask filling, which saves a lot of manpower for manual labeling, so that the network can be trained using massive samples;

[0060] Only the easily segmented eyelid area needs to be masked and filled, and the pupil and sclera parts do not need to be filled, which reduces the workload of preprocessing and increases the robustness of the network. Instead of using fixed constants such as 0 or 255 as the mask filling value, the grayscale average value of the information area is used as the filling value of the mask area. After the mask area is normalized, the value is converted into a 0 value representing no information, so it will not interfere with the learning of the network.

[0061] Directly use the original image for training and reasoning, without the need to perform local binarization on the image, extract the LBP feature and GLCM feature of the image, or reclassify the training set of positive and negative samples into several training subsets, which reduces the complexity of preprocessing. At the same time, due to the reduction of processing links, it also reduces various errors or missing information that may be introduced in the processing process, and improves the recognition effect of the network in the case of large-capacity samples;

[0062] Each eye region image is normalized according to its own mean and standard deviation, rather than using a fixed value or the mean and variance of the entire sample set. This is because in the scene of cosmetic contact lens recognition, only the texture of the iris region carries the classification judgment information between cosmetic contact lenses and living objects, while the mean difference and variance difference between samples do not carry any information that can assist in the classification judgment between cosmetic contact lenses and living objects. Therefore, the operation of normalizing each eye region image according to its own mean and standard deviation can filter out the worthless information of the sample itself, so that the network learning can focus on the key texture information that can effectively distinguish cosmetic contact lenses from living objects;

[0063] In response to the hardware requirements of neural network models, a normalization layer (SN layer) design inside the network is proposed, so that the network can be hardware-based without losing accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a flow chart of a method for identifying and processing colored contact lenses based on a deep neural network according to the present invention;

[0065] Figure 2A schematic diagram of eyes with and without colored pupils using a second-order Bezier curve for segmenting an eye region image into an inner eye region image and an outer eye region image in a method for identifying and processing colored pupils based on a deep neural network of the present invention;

[0066] Figure 3 A schematic diagram of eyes with and without colored contact lenses in a method for identifying and processing colored contact lenses based on a deep neural network of the present invention;

[0067] Figure 4 A schematic diagram of eyes with and without colored pupils after the image of the eye region is normalized by the mean and standard deviation in a method for identifying and processing colored pupils based on a deep neural network of the present invention;

[0068] Figure 5 This is an overall flow chart of a method for identifying and processing colored contact lenses based on a deep neural network according to the present invention;

[0069] Figure 6 This is a structural schematic diagram of a cosmetic contact lens recognition and processing system based on a deep neural network according to the present invention.

[0070] The reference numerals are as follows:

[0071] Information collection module 101, image data operation module 102, neural network model building module 103, normalization operation module 104, neural network training module 105, prediction module 106. DETAILED DESCRIPTION

[0072] In order to make the technical means, creative features, objectives and effects of the invention easy to understand, the technical solutions in the embodiments of the present invention are clearly and completely described below in combination with specific diagrams. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them.

[0073] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0074] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportion or adjustment of size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical contents disclosed by the present invention.

[0075] At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" cited in this specification are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments to their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0076] The first embodiment of the present invention is to provide a method for identifying and processing beautiful contact lenses based on a deep neural network, the purpose of which is to automatically distinguish the inner eye area image and the outer eye area image domain in the eye area image through a semantic segmentation algorithm, and perform mask filling, thereby saving a lot of manpower for manual labeling, so that the network can be trained using massive samples; only the eyelid area that is easy to segment needs to be masked and filled, and there is no need to fill the pupil and sclera parts, which reduces the workload of preprocessing and increases the robustness of the network.

[0077] like Figure 1 As shown, in the first aspect, the first embodiment, a method for identifying and processing colored contact lenses based on a deep neural network, comprising the following steps:

[0078] Step A1: Acquire an eye region image;

[0079] Step A2: Segment the eye region image into an inner eye region image and an outer eye region image, and then fit the dividing point to a second-order Bezier curve, such as Figure 2 As shown;

[0080] Step A3: Calculate the grayscale average of the image in the eye area to obtain Figure 3 In the image shown, the upper eyelid region and the lower eyelid region of the outer eye region image are masked according to the grayscale mean;

[0081] Step A4: Calculate the mean and standard deviation of the eye area image;

[0082] Step A5: Build a deep neural network model, input the mean and standard deviation into the deep neural network model to normalize the pixel values ​​of the eye area image to obtain the data set as follows Figure 4 The image shown;

[0083] Step A6: training the deep neural network model based on the data set;

[0084] Step A7: Reacquire the eye area image and pre-process it through steps A2-A4 to obtain the mean and standard deviation of the eye area image and transmit it to the deep neural network model obtained in step A6 for processing to obtain classification result data.

[0085] The deep neural network model includes an input layer, a normalization layer, a Block layer, and an output layer. Considering the actual application environment of beauty contact lens recognition, this technical solution provides two processing systems, namely a CPU system and an ARM+DSP system considering embedded requirements. The beauty contact lens recognition processing method is divided into two processes: training and reasoning. The training process needs to be completed on the CPU platform, while the reasoning process can be completed through the CPU platform or through the ARM+DSP embedded platform. On the ARM+DSP embedded platform, the ARM device first performs part of the eye area image preprocessing process, including reading the eye area image, automatically segmenting the inner eye area and the outer eye area of ​​the image, calculating the average grayscale of the inner eye area and masking the outer eye area; the image obtained after masking will be saved as a two-dimensional unsigned array, and the ARM transmits this two-dimensional unsigned array to the DSP. After the DSP performs normalization processing on the image data, it is passed to the Block layer for reasoning to obtain the classification result of beauty contact lens or living body;

[0086] The input layer reads the masked image;

[0087] The normalization layer is used to complete the individual normalization of each eye image data. Considering that the beauty contact lens recognition often has the demand for application in embedded devices in practical applications, it is necessary to transplant the network model to an embedded device, such as a DSP. However, the DSP can usually only support fixed-point input and cannot accept floating-point input. After the input sample image is normalized, the data will be converted from fixed-point to floating-point, and thus the input cannot be passed to the DSP without losing accuracy. Therefore, the present invention designs a separate normalization layer (SN layer) embedded in the deep neural network model, and the DSP implements the normalization operation to avoid the loss of calculation accuracy.

[0088] The Block layer contains five convolutional layers and three fully connected layers connected in sequence. The convolution kernel size of the convolutional layer is 3*3. Each convolutional layer is connected to a mean pooling layer with a pooling kernel size of 3*3 and a step size of 2. The loss function of the deep neural network model is the cross entropy loss function.

[0089] The output layer is a Softmax classifier with a decision threshold of 0.5. If the threshold is higher than 0.5, it is judged as cosmetic contact lenses, and if the threshold is lower than 0.5, it is judged as alive.

[0090] The above-mentioned method for processing beauty contact lens recognition based on deep neural network, wherein in step A2, an intelligent positioning segmentation algorithm is used to segment the inner eye area image and the outer eye area image, and the inner eye area image (including the white of the eye, iris, pupil) and the outer eye area image (including the upper eyelid and the lower eyelid) are automatically distinguished in the eye area image, and only the outer eye area is masked, and the inner eye area, i.e., the pupil, iris, and the white of the eye area, is not masked, and the mask uses the grayscale average value of the inner eye area, rather than a fixed constant such as 0 or 255. Since the texture feature information that can be used to distinguish between beauty contact lenses and living objects only occupies a part of the area of ​​the entire sample image, in many scenes of beauty contact lens recognition, the redundant information area will be masked, so that when the network learns, the focus of learning can be focused on the key area that can be used to distinguish between beauty contact lenses and living objects. Compared with other technical solutions related to beauty contact lens recognition, although the present technical solution also needs to be masked, the specific implementation method of the mask is very different from the masking methods in other beauty contact lens recognition technical solutions;

[0091] Other cosmetic contact lens processing algorithms on the market, such as the cosmetic contact lens recognition method and system based on deep learning with patent number 2019109953829, require the pupil area to be fixedly set to 0 during sample image mask preprocessing, and the area other than the pupil iris (eyelid + white of the eye) to 255. However, this technical solution only needs to perform mask processing on the outer area of ​​the eye, namely the upper eyelid area and the lower eyelid area. The white of the eye and the pupil area do not need to be masked. In this technical solution, the average value of the inner area of ​​the eye is used as the filling value of the mask outside the eye. Although the white of the eye and the pupil do not carry feature information that can distinguish cosmetic contact lenses from living bodies, the deep learning network itself has a strong ability to process redundant information. At the same time, the preprocessing operation The more steps there are, the easier it is to introduce errors or increase the labor cost of annotation. Too many and too strict mask constraints will sometimes only reduce the network's anti-interference ability. Therefore, the requirements for the mask area of ​​this technical solution are not strict. It is only necessary to mask the area outside the eye that can be clearly segmented, and it is not required to mask the pupil and white of the eye area. At the same time, this technical solution does not use a fixed value, such as 255 or 0, as the mask fill value, but uses the average of the inner area of ​​the eye, that is, the grayscale mean of the white of the eye, iris, and pupil area as the fill value of the eyelid area mask. In the subsequent normalization process, the eyelid area is filled with the grayscale mean of the entire eye area image, so all of them will be normalized to 0, which means no information, and thus will not cause any interference to the network's learning process.

[0092] In the above-mentioned method for identifying and processing cosmetic contact lens based on deep neural network, the intelligent positioning and segmentation algorithm is a semantic segmentation algorithm, which automatically distinguishes the inner eye area image and the outer eye area image domain in the eye area image through the semantic segmentation algorithm.

[0093] In the above-mentioned method for identifying and processing cosmetic contact lens based on a deep neural network, the image of the inner area of ​​the eye includes the white of the eye, the iris, and the pupil, and the image of the outer area of ​​the eye includes the upper eyelid and the lower eyelid.

[0094] In step A3, the mask filling is automatically performed on the area outside the eyes, thereby avoiding the huge workload of manual labeling and making it possible to use massive samples for network training.

[0095] Each eye region image is normalized using the mean and standard deviation of the image itself, rather than using the mean and standard deviation of the entire sample set or a fixed constant. This is because for the beauty contact lens recognition scene, only texture is the main basis for distinguishing beauty contact lenses from living bodies. The mean and standard deviation of the sample itself do not carry any classification information. Each sample image is normalized using the mean and standard deviation of the image itself, which can effectively filter out the differences in grayscale mean and standard deviation between different samples, thereby reducing the difficulty of network learning;

[0096] This preprocessing method is simple to implement and does not require manual labeling. In addition, since the noise irrelevant to the recognition of cosmetic contact lenses is filtered out in the preprocessing stage and the key texture information for distinguishing cosmetic contact lenses from living objects is completely retained, the difficulty of neural network learning is reduced, allowing the deep neural network to achieve better training results.

[0097] Different from the conventional method of using a fixed constant for normalization or using a sample set mean for normalization, the present invention uses a method of normalizing each sample individually.

[0098] like Figure 5 As shown, the method for beauty contact lens recognition based on deep neural network provided by this technical solution is specifically divided into four parts: beauty contact lens recognition data preprocessing, deep neural network construction, neural network training process, and neural network reasoning process;

[0099] Neural network training process (dataset training process): The deep neural network model that has been built is trained through the eye area images (sample data) collected in advance, the data set is input into the deep neural network model through the input layer, and the classification result is obtained at the output layer. The learning error is obtained by calculating the difference between the output result and the expected output result, and the connection weights in the deep neural network model are adjusted according to the size of the learning error. After repeated reasoning and feedback, the inference output obtained by the deep neural network for each training input can be consistent with the expected output.

[0100] Neural network reasoning process: After the deep neural network model has passed the data set training and met the standards, it can be used in the actual application of beauty contact lens recognition. When a new eye area image is input, the beauty contact lens recognition data is first pre-processed, and then the trained deep neural network model is input to obtain the correct classification result of beauty contact lens or living body.

[0101] like Figure 6 As shown, in the second aspect, the second embodiment is a system for identifying and processing colored contact lenses based on a deep neural network, which includes an information acquisition module 101, an image data operation module 102, a neural network model building module 103, a normalization operation module 104, a neural network training module 105, and a prediction module 106;

[0102] The information acquisition module 101 is used to obtain an eye region image of the eye and transmit it to the image data calculation module 102;

[0103] The image data operation module 102 is used to divide the eye region image into an inner eye region image and an outer eye region image and calculate the grayscale mean of the inner eye region image, perform mask processing on the upper eyelid region and the lower eyelid region of the outer eye region image according to the grayscale mean, and calculate the mean and standard deviation of the eye region image;

[0104] The neural network model building module 103 is used to build a deep neural network model;

[0105] The normalization operation module 104 is used to input the mean and standard deviation into the deep neural network model to perform normalization processing on the pixel values ​​of the eye area image to obtain a data set;

[0106] The neural network training module 105 is used to obtain a data set and train a deep neural network model based on the data set;

[0107] The prediction module 106 is used to reacquire the eye area image and preprocess it through the image data operation module 102 to obtain the mean and standard deviation of the eye area image and transmit it to the trained deep neural network model for processing to obtain classification result data.

[0108] In the above-mentioned system for identifying and processing colored pupils based on a deep neural network, the image data operation module 102 uses an intelligent positioning and segmentation algorithm to segment the image of the inner area of ​​the eye and the image of the outer area of ​​the eye;

[0109] The above-mentioned system for identifying and processing colored contact lens based on deep neural network, wherein the intelligent positioning and segmentation algorithm is a semantic segmentation algorithm, which automatically distinguishes the inner eye area image and the outer eye area image domain in the eye area image through the semantic segmentation algorithm.

[0110] In the above-mentioned system for identifying and processing beauty contact lens images based on a deep neural network, the image of the inner area of ​​the eye includes the white of the eye, the iris, and the pupil, and the image of the outer area of ​​the eye includes the upper eyelid and the lower eyelid.

[0111] Considering that deep learning technology often requires the network model to be transplanted to DSP in practical applications, and DSP usually only supports fixed-point input and cannot accept floating-point input. After the input sample image is normalized, the data will be converted from fixed-point to floating-point, making it impossible to pass the input to DSP without losing accuracy.

[0112] Therefore, the technical solution adds a separate normalization layer (SN layer). The position of the SN layer in the entire processing network is between the input layer and the Block layer. The significance of setting the SN layer is that the DSP side completes the normalization operation of the fixed-point image, so as not to lose any precision and improve the accuracy of reasoning.

[0113] The data processing process of a single normalization layer can be expressed as follows:

[0114]

[0115] Among them, X′ i is a single new sample data obtained after normalization, X i is the data of this picture in the original sample set, is the grayscale mean of the image after masking, s i is the standard deviation of the image after masking, α is the adjustment parameter of the standard deviation, and its value is between 1 and 3, so that the single new sample data X′ obtained after normalization i The mean value is controlled to be 0, and the range is basically within the range of -1 to 1;

[0116] The Block layer includes five convolutional layers and three fully connected layers connected in sequence. The convolution kernel size of the convolutional layer is 3*3. Each convolutional layer is connected to a mean pooling layer with a pooling kernel size of 3*3 and a step size of 2. The loss function of the deep learning network model is the cross entropy loss function.

[0117] The output layer is a Softmax classifier with a decision threshold of 0.5. If the threshold is higher than 0.5, it is judged as cosmetic contact lenses, and if the threshold is lower than 0.5, it is judged as alive.

[0118] In a third aspect, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of any one of the methods in the first aspect are implemented.

[0119] For example, the memory may include random access memory, flash memory, read-only memory, programmable read-only memory, non-volatile memory or registers, etc.;

[0120] The processor may be a central processing unit (CPU) or a graphic processing unit (GPU). The memory may store executable instructions.

[0121] The processor may execute the execution instructions stored in the memory to implement the various processes described in this document.

[0122] It can be understood that the memory in this embodiment may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory;

[0123] Among them, the non-volatile memory can be ROM (Read-Only Memory), PROM (Programmable ROM), EPROM (Erasable PROM), EEPROM (Electrically EPROM) or flash memory.

[0124] The volatile memory may be a RAM (Random Access Memory), which is used as an external cache;

[0125] By way of example but not limitation, many forms of RAM are available, such as SRAM (Static RAM), DRAM (Dynamic RAM), SDRAM (Synchronous DRAM), DDR SDRAM (Double Data Rate SDRAM), ESDRAM (Enhanced SDRAM), SLDRAM (Synchlink DRAM), and DRRAM (Direct Rambus RAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0126] In some embodiments, the memory stores the following elements, upgrade packages, executable units or data structures, or subsets thereof, or extended sets thereof: operating systems and applications;

[0127] Among them, the operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., which are used to implement various basic services and process hardware-based tasks;

[0128] The application program includes various application programs and is used to implement various application services. The program for implementing the method of the embodiment of the present invention may be included in the application program.

[0129] It will be apparent to those skilled in the art that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of software and electronic hardware;

[0130] Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution;

[0131] Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0132] In the embodiments of the present application, the disclosed systems, devices and methods may be implemented in other ways;

[0133] For example, the division of units or modules is only a logical function division, and there may be other division methods in actual implementation;

[0134] For example, multiple units or modules or components may be combined or may be integrated into another system;

[0135] In addition, each functional unit or module in the embodiments of the present application may be integrated into one processing unit or module, or may be a separate physical entity, etc.

[0136] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0137] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a machine-readable storage medium;

[0138] Therefore, the technical solution of the present application may be embodied in the form of a software product, which may be stored in a machine-readable storage medium, and may include several instructions for enabling an electronic device to perform all or part of the process of the technical solution described in the embodiments of the present application;

[0139] The above-mentioned storage medium may include ROM, RAM, removable disk, hard disk, magnetic disk or optical disk, etc., which can store program codes.

[0140] In summary, the method and system for identifying and processing beautiful contact lens based on a deep neural network of the present invention can automatically distinguish the inner eye area image and the outer eye area image domain in the eye area image through a semantic segmentation algorithm, and perform mask filling, which saves a lot of manpower for manual labeling, so that the network can be trained using massive samples; only the eyelid area that is easy to segment needs to be masked and filled, and there is no need to fill the pupil and sclera parts, which reduces the workload of preprocessing and increases the robustness of the network.

[0141] The above describes the specific embodiments of the invention. It should be understood that the invention is not limited to the above specific embodiments, and the devices and structures not described in detail should be understood to be implemented in a common manner in the art; those skilled in the art can make various modifications or modifications within the scope of the claims, and make some simple deductions, deformations or substitutions, which does not affect the essence of the invention.

Claims

1. A method for identifying and processing colored contact lenses based on a deep neural network, characterized in that: The following steps are involved: Step A1: Acquire an eye region image; Step A2: dividing the eye region image into an inner eye region image and an outer eye region image; Step A3: calculating the grayscale mean of the image of the inner eye area, and performing mask processing on the upper eyelid area and the lower eyelid area of ​​the image of the outer eye area according to the grayscale mean; Step A4: Calculating the mean and standard deviation of the eye region image; Step A5: building a deep neural network model, inputting the mean and standard deviation into the deep neural network model to normalize the pixel values ​​of the eye area image to obtain a data set; Step A6: training the deep neural network model according to the data set; Step A7: Reacquire the eye area image and pre-process it through step A2-step A4 to obtain the mean and standard deviation of the eye area image and transmit it to the deep neural network model obtained in step A6 for processing to obtain classification result data.

2. A method for identifying and processing colored contact lenses based on a deep neural network as claimed in claim 1, characterized in that: In step A2, an intelligent positioning and segmentation algorithm is used to segment the image of the inner eye area and the image of the outer eye area.

3. A method for identifying and processing colored contact lenses based on a deep neural network as claimed in claim 2, characterized in that: The intelligent positioning and segmentation algorithm is a semantic segmentation algorithm, which is used to automatically distinguish the inner eye area image and the outer eye area image domain in the eye area image.

4. A method for identifying and processing colored contact lenses based on a deep neural network according to any one of claims 1 to 3, characterized in that: The image of the inner eye area includes the white of the eye, the iris, and the pupil, and the image of the outer eye area includes the upper eyelid and the lower eyelid.

5. A color contact lens recognition and processing system based on deep neural network, wherein: It includes information acquisition module, image data calculation module, neural network model building module, normalization calculation module, neural network training module and prediction module; The information acquisition module is used to obtain an eye region image of the eye and transmit it to the image data calculation module; The image data operation module is used to divide the eye region image into an inner eye region image and an outer eye region image and calculate the grayscale mean of the inner eye region image, perform mask processing on the upper eyelid region and the lower eyelid region of the outer eye region image according to the grayscale mean, and calculate the mean and standard deviation of the eye region image; The neural network model building module is used to build a deep neural network model; The normalization operation module is used to input the mean and standard deviation into the deep neural network model to perform normalization processing on the pixel values ​​of the eye area image to obtain a data set; The neural network training module is used to obtain the data set and train the deep neural network model according to the data set; The prediction module is used to reacquire the eye area image and preprocess it through the image data operation module to obtain the mean and standard deviation of the eye area image and transmit it to the trained deep neural network model for processing to obtain classification result data.

6. A system for identifying and processing colored contact lenses based on a deep neural network as claimed in claim 5, wherein: The image data operation module uses an intelligent positioning and segmentation algorithm to segment the image of the inner area of ​​the eye and the image of the outer area of ​​the eye.

7. A system for identifying and processing colored contact lenses based on a deep neural network as claimed in claim 6, wherein: The intelligent positioning and segmentation algorithm is a semantic segmentation algorithm, which is used to automatically distinguish the inner eye area image and the outer eye area image domain in the eye area image.

8. A system for identifying and processing colored contact lenses based on a deep neural network as claimed in any one of claims 5 to 7, wherein: The image of the inner eye area includes the white of the eye, the iris, and the pupil, and the image of the outer eye area includes the upper eyelid and the lower eyelid.

9. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images