Vision-based health and beauty detection and management method

Abstract

The invention relates to a makeup detection, physique analysis and management method suitable for family members and staff, in particular to a vision-based health and beauty detection and management system, and the detection system is composed of a sensing unit, an interaction unit, a data processing unit and a communication unit. Wherein the sensing unit is used for detecting human bodies, faces, cosmetics and health care products; the sensing unit adopts an image processing module with more than 5 million pixels to collect images, and adopts two image processing modules to respectively collect images outside or inside the system; the sensing unit image processing module sends the image signal to a data processing unit for processing through an MIPI interface; the display module interactively accesses the interface module through the MIPI, and the interface module accesses the data processing unit. And obtaining skin selection cosmetics, performing makeup pretreatment on eyes, noses, faces and lips, performing makeup setting and selecting suggestions on health care products.

Description

Technical Field

[0001] The invention belongs to the field of information technology. Background Art

[0002] CN202080087543.5 A makeup assisting device, a makeup assisting method, a makeup assisting program and a makeup assisting system guide the application of cosmetics. The makeup assisting device comprises: an area detection unit, which detects the area where the cosmetics are applied; and a display unit, which displays information associated with the area, wherein the information is information indicating the application status of the cosmetics in the area.

[0003] CN201680069623.1 A makeup simulation assisting device, a makeup simulation assisting method and a makeup simulation assisting program, wherein the makeup simulation assisting device comprises: a simulation image generating unit (210), which generates a simulation image by superimposing a makeup image on a facial image captured by a capturing unit that captures an image of a user's face as a facial image, wherein the makeup image represents a makeup state when makeup is applied to facial organs of the face; a simulation image output unit (220), which outputs the generated simulation image; an interest level detecting unit (230), which detects the user's interest level in the makeup image based on the facial image captured by the camera unit when the simulation image is output; and an interest level information output unit (240), which outputs interest level information representing the detected interest level.

[0004] CN202211566191.9 A simulated makeup method, simulated makeup system and intelligent makeup table The method, system and dressing table establish a physical makeup model composed of a human face model, and a number of touch sensors are densely distributed on the makeup model. The features of the photographed face image can be matched with the features of the makeup model image through an algorithm; in this way, when applying makeup on the makeup model, the makeup position detected by the touch sensor can be converted to a virtual face image, and the makeup effect can be displayed on the virtual image. For the user, although makeup is not applied on his own face, the real sensory effect of "looking in the mirror" can be achieved. Moreover, the makeup operation is performed on the physical model, and the simulated makeup tools can be used, so that the real touch makeup operation can be achieved, and the virtual simulated makeup effect can be obtained. The cosmetics can be tried on without loss, and the practice operation of makeup techniques can also be realized. Summary of the invention

[0005] The purpose of the present invention is to propose a comprehensive Kangmei (healthy beauty and makeup) detection and management method suitable for family members and employees. In response to the beauty issues that everyone is concerned about, multi-dimensional detection of human skin, facial skin, cosmetics, health care products, etc. is added to the robot display interface. It can analyze facial and skin characteristics, identify cosmetic types and makeup steps, and provide voice prompts; at the same time, a tongue coating detection function can be added to analyze a person's physique and recommend corresponding health care products by voice.

[0006] It can also measure the distance between people and output the unlocking signal of electronic locks. It can also detect the quantity and status of cosmetics and health products and give voice prompts. It can store and manage the cosmetics or health products used by different people. However, some existing beauty and health testing equipment has relatively single functions and cannot identify the makeup process and give voice prompts, so as to realize the whole process, all-round, multi-angle and multi-application testing, evaluation and monitoring.

[0007] The technical solution of the present invention is a makeup detection and management method suitable for family members and employees. Based on vision, the Kangmei detection and management system consists of a perception unit, an interaction unit, a data processing unit, and a communication unit;

[0008] The sensing unit is used to detect human bodies, faces, cosmetics, and health products. The sensing unit uses an image processing module with more than 5 million pixels to collect images. Two image processing modules are used to collect images from the outside or inside of the system.

[0009] The image processing module of the perception unit then sends the image signal to the data processing unit for processing through the MIPI interface;

[0010] The interactive unit includes a voice and display interactive module; the voice is collected and recognized and played; the display interactive module can also display the recognition result; the interactive unit transmits data to the data processing unit through the USB interface; the display module interacts with the interface module through the MIPI access interface module, and the interface module is connected to the data processing unit;

[0011] The data processing unit is installed with operating systems such as Android, Linux, HarmonyOS, and Windows, and runs software modules such as detection and management. It detects the human body, face, objects, usage process, and distance, and provides different cosmetics and makeup suggestions based on the user's different skin types.

[0012] The perception unit includes a detection module, a human recognition submodule, and a facial recognition submodule.

[0013] The detection module is used to sense the status of the human body, face, etc. It can also detect the status of items such as cosmetics and health products. The components of the detection module are:

[0014] The human body recognition submodule recognizes the human body and uses the YOLO method to identify people in the image. First, the image signal is passed through the Darknet-53 backbone network in Yolo3 to extract features, and then upsampling, feature fusion and regression analysis are performed to obtain the prediction box information, and the target feature modeling, matching and tracking are performed through the SORT algorithm;

[0015] The facial recognition submodule first detects the face, then extracts the facial features, and performs face shape, eye shape, nose shape, lip shape and skin texture recognition;

[0016] The management system includes a management module, which includes a detection and storage status management module, a human body recognition and distance management module, a skin type and physique recognition management module, a makeup process recognition management module, and other sub-modules. Fig.12 ):

[0017] In the storage status detection management submodule, the storage status of cosmetics, health products, etc. is detected, and the quantity is identified and counted; the above detection results or suggestions are displayed through voice prompts or on the screen; the image processing module in the perception unit collects the image inside the system and sends it to the data processing unit for identification and statistics;

[0018] In the human body recognition and distance management submodule, when the image processing module in the perception unit collects images outside the system and recognizes the face, it performs distance measurement;

[0019] Choose cosmetics according to skin type, perform pre-treatment and setting of eyes, nose, face and lips, and provide recommendations on the selection of health care products.

[0020] There are (i) face shape recognition, (ii) eye shape recognition, (iii) nose shape recognition, (iv) lip shape recognition, and (v) skin quality recognition modules; there is a tongue coating detection submodule to identify and detect the human tongue. The tongue image is first converted into a grayscale image using a weighted average method, which helps to simplify the subsequent processing flow and reduce the complexity of the processing. The pixel values ​​in the grayscale image represent the brightness information of each area of ​​the tongue. The implementation method will be described in detail.

[0021] It has a distance measurement submodule, and the distance recognition measures the distance between the camera and the face. It uses a monocular distance measurement method to locate the face first and output the distance of the face. It uses a deep network structure, which consists of Input, Backbone, Neck, and Prediction. The Input structure uses Mosaic data enhancement and adaptive image scaling. The Backbone structure is used for feature extraction. The Neck structure uses a high-to-low and low-to-high feature fusion method to enhance the model's feature extraction and feature fusion capabilities. The Predict structure outputs three scale features that are ultimately used for prediction.

[0022] There is an item detection submodule, in which items can be detected to achieve effective management of items; this module can identify cosmetics and health products, etc.

[0023] Object recognition can adopt a lightweight recognition method, the structure of which mainly consists of preprocessing, backbone network, Neck and Head detection end; preprocessing mainly focuses on Mosa-ic data enhancement module,

[0024] Random scaling, cropping, color gamut conversion and other operations are used to enhance the images and then stitch them together to make the data set richer. In the backbone network, the Focus module is first used to expand the input channel four times and complete the sampling without information loss. In the Neck layer, a convolutional attention module is introduced to improve the detection of small targets.

[0025] A gesture recognition submodule is provided. The gesture recognition module is used to recognize various actions in the makeup area during the makeup process, guide the makeup steps and make makeup suggestions. A neural network model based on multi-scale spatiotemporal feature fusion can be used. It is mainly composed of a spatiotemporal feature extraction network of gesture images, a multi-scale feature fusion network and a decoding module.

[0026] In the spatiotemporal feature extraction network of gesture images, f is the number of channels of each branch, t is the number of feedback cycles, and c is the number of convolution channels. The spatiotemporal feature extraction network is composed of a stack of 4 layers of spatiotemporal attention networks STA-Net. Attention networks with 16, 32, 64, and 128 channels are designed to extract spatiotemporal feature maps of different scales. FPN is designed to fuse spatiotemporal feature maps at multiple scales, and the channel attention network STA-Net is used to assign different weights to spatiotemporal features to suppress redundant information. Feedback loop branches are designed to iteratively update spatiotemporal features. Finally, the fused spatiotemporal features are passed through a decoding module to obtain the label of the gesture action.

[0027] There is a management module, which is mainly composed of sub-modules such as detection and storage status management, human body recognition and distance management, skin type and physique recognition management, and makeup process recognition management;

[0028] (1) In the storage status detection management submodule, the storage status of cosmetics, health products, etc. is detected, and the number is identified and counted. At this time, the image processing module in the perception unit collects the image inside the system and sends it to the data processing unit for identification and counting. First, the number of items is counted, and if the required cosmetics and health products are not available, a voice prompt is given; at the same time, the items can be identified and entered into the family member list, and the items and members are mapped to achieve classification management of items for different members;

[0029] (2) In the human body recognition and distance management sub-module, when the image processing module in the sensing unit captures an image outside the system and recognizes a human face, distance measurement is performed. At the same time, the recognized distance D_set is input into the data processing unit and compared with the face distance d. When d < D_set, the data processing unit outputs a high-level signal in the IO interface to control the opening of the selected electronic lock. When d > D_set, the data processing unit outputs a low-level signal in the IO interface to control the closing of the selected electronic lock;

[0030] (3) In the skin texture and physique recognition management sub-module, when the image processing module in the sensing unit captures an image outside the system, it is sent to the data processing unit for recognition and processing. For makeup, different skin textures are recognized, different cosmetics are selected, and makeup steps are recommended. For physique analysis, different tongue coatings are recognized, and different health products are selected. At the same time, the data processing unit sends the results to the interaction unit, and the interaction unit gives text display or voice prompts to identify the functions of the skin texture and physique recognition management sub-module. Recommendations for skin texture selection of cosmetics, pre-makeup treatment, setting makeup for eyes, nose, face, lips, and selection of health products are given.

[0031] (4) In the makeup process recognition management sub-module, the makeup process and steps are recognized:

[0032] During the facial contouring process, shadows, highlighters, blushes, and powder brushes are recognized in the image to identify the facial contouring process. Eyebrow shaping gestures such as eyebrow mascara and eyebrow pencils are recognized in the image to identify the makeup area of the eyebrow shaping process. When an error is recognized during the process, the data processing unit sends the result to the interaction unit, and the interaction unit gives a voice prompt.

[0033] The present invention can improve the intelligent level of home makeup and health care, and at the same time can monitor the whole process of the use and storage of cosmetics and health products to ensure the accuracy of use.

[0034] Beneficial effects, the present invention proposes a Kangmei AI detection and management method for ordinary family members and employees. This method can improve the convenience of home beauty and physique detection, improve the accuracy of the use of cosmetics and health products, and improve the intelligent and humanized level of home makeup and physique analysis. Brief Description of the Drawings

[0035] Figure 1 Schematic diagram of the system composition; Figure 2 Schematic diagram of the sensing unit interface;

[0036] Figure 3 Schematic diagram of the interaction unit processing; Figure 4 Schematic diagram of the communication unit signal transmission;

[0037] Figure 5 Schematic diagram of the functional composition of the detection module; Figure 6 Schematic diagram of human recognition processing;

[0038] Figure 7 Schematic diagram of facial recognition processing; Figure 8 Schematic diagram of face recognition architecture;

[0039] Fig. 9 Schematic diagram of face distance measurement; Fig.10 Schematic diagram of object recognition network processing;

[0040] Fig.11 Schematic diagram of the gesture recognition network model; Fig.12 Schematic diagram of the management module composition;

[0041] Fig.13 Detection storage status management submodule functional diagram;

[0042] Fig.14 Functional diagram of human body recognition and distance management submodules;

[0043] Fig.15 Schematic diagram of the functions of the skin type identification and physique management submodules; Fig.16 Schematic diagram of makeup process recognition and processing;

[0044] Fig.17 Schematic diagram of facial retouching process recognition; Fig.18 Schematic diagram of eye makeup process recognition and processing;

[0045] Fig.19 Schematic diagram of the nose contouring process recognition and processing; Fig. 20 Schematic diagram of lip makeup process recognition and processing;

[0046] Fig.21 Face recognition processing flow chart; Fig. 22 Eye shape recognition process flow chart;

[0047] Fig.23 Nose shape recognition process flow chart; Fig.24 Lip recognition process flow chart. DETAILED DESCRIPTION

[0048] 1 System composition

[0049] The present invention provides a vision-based Kangmei detection and management method. The system consists of a perception unit, an interaction unit, a data processing unit, a communication unit, etc. The system consists of Figure 1 As shown:

[0050] 2 Perception Unit

[0051] The perception unit is used to detect human bodies, faces, cosmetics, health products, etc. The unit uses an image processing module with more than 5 million pixels to capture images. Two image processing modules can be used to capture images outside or inside the system respectively. The image processing module is mainly composed of the IMX335 circuit. The IMX335 circuit has a built-in complete ISP function, supports backlight compensation, DOL mode wide dynamic, 2D / 3D noise reduction, defogging and other functions, and provides MIPI and other interfaces (mobile industry processor interface). The perception unit then sends the image signal to the data processing unit for processing through the MIPI interface. The interface of the perception unit is as follows Figure 2 As shown:

[0052] 3 interactive units, Figure 3

[0053] The system can perform voice interaction and provide a display screen for human-computer interaction. It can collect and recognize voice and play it. The display module can also display the recognition results. The interactive unit transmits data with the data processing unit through the USB interface.

[0054] Adopting multi-mic module LTM6216, dual mic or 4mic array for noise reduction, it can pick up 360 channels of sound, and 4mic supports 0-10m distance sound pickup. Adopting audio DAC module CJC4344, with built-in interpolation filter, multi bit digital-to-analog converter, output analog filter, etc., it is connected to TPA3116D2 amplifier module to output audio. The display module adopts OLED / LCD display screen of more than 7 inches, with a size ratio of 16:9 and supports touch function. Through MIPI and other access interface modules, the interface module uses SN2010 to convert into USB signal to access the data processing unit.

[0055] 4 Communication Unit

[0056] The communication unit mainly uses the Hi3861L WIFI chip. Hi3861L is a low-power wifi module with an integrated 2.4GHz wifi SoC chip, an integrated IEEE 802.11b / g / n baseband and RF circuit, including power amplifier PA, low noise amplifier LNA, antenna switch and other modules; supports 20MHz standard bandwidth and 5MHz / 10MHz narrow bandwidth, and provides a maximum physical layer rate of 72.2Mbit / s. The RF signal generated by Hi3861L is then filtered and amplified before being transmitted by the RF antenna. Communication unit signal transmission is as follows: Figure 4 As shown:

[0057] 5Data processing unit

[0058] The data processing unit mainly uses chips based on ARM Cortex-A7 cores with more than four cores, integrates NEON and FPU, supports two MIPI CSI interfaces and one DVP interface, and can also transmit data with other units through USB, RS232, IO and other interfaces.

[0059] The data processing unit can be installed with operating systems such as Android, Linux, HarmonyOS, and Windows, and run software modules such as detection and management. It can detect the human body, face, objects, usage process, and distance, and provide different cosmetics and makeup suggestions based on the different skin types of members, and different health care product suggestions based on different physiques. At the same time, it can detect the storage status of cosmetics and health care products. The above test results or suggestions can be displayed through voice prompts or on the screen.

[0060] 5.1 Detection module see Figure 5 ,

[0061] This unit is mainly used to sense the state of the human body, face, etc. It can also detect the state of cosmetics, health products, etc. The detection module functions are as follows: Figure 5 As shown:

[0062] The main components of YOLO include:

[0063] 1. Convolutional Neural Network (CNN) is used to extract features.

[0064] 2. Prediction results of the fully connected layer.

[0065] The following is the basic process of YOLO:

[0066] 1. Divide the image into grids.

[0067] 2. Predict B bounding boxes for each grid (B is usually set to 2, that is, 2 bounding boxes are predicted for each grid).

[0068] 3. Each bounding box includes the predicted coordinates, confidence score, and category prediction.

[0069] The Yolo algorithm, whose full name is You 0nly Look Once: Unified, Real-Time Object Detection, only requires one CNN operation.

[0070] 5.1.1 Human body recognition submodule, Figure 6

[0071] This module can identify human bodies. It can use the YOLO method to identify people in images. First, the image signal is passed through the Darknet-53 backbone network in Yolo3 to extract features, and then upsampling, feature fusion and regression analysis are performed to obtain prediction box information, and the target feature modeling, matching and tracking are performed through the SORT algorithm. The processing is shown in the figure below: (DarkNet-53 is composed of convolutional layers and residual layers. At the same time, it should be noted that the last three layers of Avgpool, Connected and Softmax layers play a major role).

[0072] 5.1.2 Facial Recognition Submodule

[0073] For facial recognition, you can first detect the face, then extract the facial features, and perform face shape and skin type recognition. Figure 7 As shown:

[0074] Face detection can be done using Figure 8 The architecture is used for recognition, which is mainly composed of Backbone, Neck and Head. CBSBlock mainly contains 1 convolution layer Conv, 1 batch normalization layer BN and 1 SiLU activation function. Conv is used to extract features, BN is used to regularize input data, and SiLU is used to activate output. The Stem module is mainly composed of 4 CBS, 1 maximum pooling layer Maxpool and 1 Concat. Maxpool is used for downsampling, Concat is used to splice image features from different paths, and the Stem module is mainly used to improve the generalization ability of the network and reduce the computational complexity. A 10*10*16 output block is added to improve the detection performance of large faces.

[0075] (a) Face recognition, Figure 8

[0076] It can identify the face shape of a person and classify the face shape into round face, square face, diamond face, triangular face, long face, oval face, etc. First, convert the face image into a grayscale image, use a Gaussian filter for smoothing, and then use the directional gradient histogram feature and cascade classifier to detect the facial feature points. The directional gradient histogram feature method can count the local morphological edge gradient information of the face:

[0077] Gradient component calculation:

[0078]

[0079] Gradient and direction calculation:

[0080]

[0081] The cascade classifier realizes face detection through the cascade method:

[0082] y=C1(x)*C2(x)*…C n (x)

[0083] Where x is the feature vector of the input sample, C1, C2, …, C n Represents the stages of the cascade classifier.

[0084] Line 1 connects the left and right edge coordinate points of the forehead, Line 2 connects the zygomatic line of the 2nd and 16th key points, Line 3 connects the jaw line of the 4th and 14th key points, Line 4 connects the forehead midpoint and the 9th key point, Line 5 connects the forehead length line, and Line 6 connects the facial width line. Use Euclidean distance to calculate the length of the line segment:

[0085]

[0086] Compute the standard deviation of the line:

[0087]

[0088] Calculate the standard deviation SD1 of Line1, Line2, and Line3, calculate the standard deviation SD2 of Line2 and Line4, calculate the angle between the two vectors from the 3rd key point to the 5th key point and from the 4th key point to the 6th key point and the x-axis respectively, take the difference between the two and convert it to the angle Angle.

[0089] By identifying SD1 and Angle, the face shape is divided into round face, square face, diamond face, triangle face, rectangular face, oval face, etc. (The processing flow is shown in the face shape recognition processing flow chart below)

[0090] (II) Eye shape recognition

[0091] For the eye feature points, you can use the left key point of the left eye (LL_x1, LL_y1), the right key point of the left eye (LR_x2, LR_y2), the left 1 key point above the left eye (LU_x3, LU_y3), the right 1 key point above the left eye (LU_x4, LU_y4), the left 1 key point below the left eye (LD_x5, LD_y5), the right 1 key point below the left eye (LD_x6, LD_y6), the left key point of the right eye (RL_x7, RL_y7), the right key point of the right eye (RR_x8, RR_y8), the left 1 key point above the right eye (RU_x9, RU_y9), the right 1 key point above the right eye (RU_x10, RU_y10), the left 1 key point below the right eye (RD_x11, RD_y11), and the right 1 key point below the right eye (RD_x12, RD_y12).

[0092] The left eye length DL is calculated by the Euclidean distance between (LL_x1, LL_y1) and (LR_x2, LR_y2).

[0093]

[0094] The right eye length DR is calculated by the Euclidean distance between (RL_x7, RL_y7) and (RR_x8, RR_y8).

[0095]

[0096] The interocular distance DLR is calculated by the Euclidean distance between (LR_x2, LR_y2) and (RL_x7, RL_y7).

[0097]

[0098] Calculate the Euclidean distances DW1, DW2, DW3, and DW4 between (LU_x3, LU_y3) and (LD_x5, LD_y5), (LU_x4, LU_y4) and (LD_x6, LD_y6), (RU_x9, RU_y9) and (RD_x11, RD_y11), and (RU_x10, RU_y10) and (RD_x12, RD_y12) respectively, and calculate the average width DW.

[0099]

[0100] By comparing DL, DR, DLR and DW, it can be identified as slender eyes, round eyes, small eyes, etc. (See the eye shape recognition process flow chart below for the process flow)

[0101] 3. Nose shape recognition

[0102] For the nose feature points, the nasal root position (A_x1, A_y1), the nose tip position (B_x2, B_y2), the left nose wing position (C_x3, C_y3), the right nose wing position (D_x4, D_y4), and the frontal length LM are used.

[0103] Calculate the nose length by the Euclidean distance between (A_x1, A_y1) and (B_x2, B_y2)

[0104]

[0105] Calculate the nose width by the Euclidean distance between (C_x3, C_y3) and (D_x4, D_y4)

[0106]

[0107] By comparing the values ​​of BL, BW, Line5 and Line6, wide nose, long nose, short nose, etc. can be identified. (For the processing flow, see the nose shape recognition processing flow chart below)

[0108] (IV) Lip shape recognition

[0109] First locate the lip area. According to the overall structure and proportion of the face, the position of the mouth can be obtained:

[0110]

[0111] Among them, H face and W face Indicates the height and width of the face area, H mouth and W mouth Indicates the height and width of the mouth area.

[0112] After further optimization and noise reduction, we get a lip image without teeth, beard, etc.:

[0113] (a) Calculate the grayscale of the mouth area. Assume that the mouth image is P, and its resolution is MxN. P(x,y) represents the grayscale value at (x,y). The grayscale range is [0,1,…,L-1]. The number of pixels corresponding to grayscale level l is m. l , the probability calculation formula for each gray level is The range of l is [0,1,…,L-1].

[0114] (b) The pixels in the mouth area are divided into the lip area and the non-lip area according to the grayscale level L using the threshold th. The grayscale range of the lip area is [0,…,th-1], and the grayscale range of the non-lip area is [th,…,L-1]. The probability of the lip area and the non-lip area appearing is:

[0115]

[0116] The average grayscale values ​​of the lip area and non-lip area are

[0117]

[0118] The total gray value of the mouth area μ = P T *μ T +P F *μ F

[0119]

[0120] The l value corresponding to l∈[0,1,…,L-1], max[σ(l)] can be calculated to obtain the threshold value T1, thereby distinguishing the lip area from the non-lip area.

[0121] (c) The inner lip edge contour is then constructed to obtain the binary image pixel value of the lip area.

[0122]

[0123] Select the first 15% of the binary image pixel values ​​and set them to 255, and the remaining pixel values ​​are 0 to fit the inner lip curve:

[0124]

[0125] Where h1 and h2 represent the height coefficients of the upper and lower parts of the inner lip, respectively, w represents the width coefficient, and x represents the inner lip edge motion variable.

[0126] (d) The outer lip edge contour is constructed again. The outer lip edge contour is reddish, and the outer lip edge skin is not red. The lip pixel set X1 and the surrounding skin pixel set X2 are obtained, and the mean vector of the lip and surrounding skin pixels is calculated.

[0127]

[0128] Where M and N represent the number of pixels in the set. Then calculate the intra-class discrete matrix S corresponding to the lips and surrounding skin b and S w , and the global divergence matrix S of the lip area image t

[0129] S b =(μ-μ1)(μ-μ1) -

[0130]

[0131] S t =S b +S w

[0132] By S t Find the intra-class projection weight k, calculate the mean y1 and y2 of the lip and skin pixel information projected into the one-dimensional linear space, and detect the lip area image:

[0133]

[0134] Where z represents the image, and k0 represents the standard mean value of the pixels in the lip area. When the above formula is greater than 0, the area is the outer lip contour, and when it is less than or equal to 0, the area is the skin around the outer lip contour, thus fitting the outer lip curve:

[0135]

[0136] Among them, h3 and h4 represent the height coefficients of the outer lip contour, q1 and q2 represent the coefficients of the upper and lower grooves of the outer lip, and w1 represents the outer lip width coefficient.

[0137] (e) Determine the lip shape standard: upper and lower lip width ratio coefficient hk = (h3-h1) / (h4-h1)

[0138] When hk>1 / 1.3, the lip shape is thick lips; when hk<1 / 1.5, the lip shape is thin lips;

[0139] When hk∈[1 / 1.3,1 / 1.5], the lip shape is standard lip.

[0140] (V) Skin type identification

[0141] Skin texture recognition identifies various skin characteristics, such as acne, wrinkles, spots, dark circles, etc., by analyzing specific areas in facial images, such as details on the skin surface.

[0142] 1. Image preprocessing

[0143] First, the skin image is preprocessed, including conversion to grayscale image and Gaussian filtering smoothing. This step helps to reduce image noise and unnecessary details, making subsequent feature extraction more accurate.

[0144] Grayscale: Convert color skin images to grayscale images.

[0145] Gray value (i, j) = 0.299*R(i, j) + 0.587*G(i, j) + 0.114*B(i, j)

[0146] Gaussian filtering: Use a Gaussian filter to smooth the grayscale image and remove noise.

[0147]

[0148] Where G(x,y) is the Gaussian kernel function, (x,y) is the pixel position in the image, and σ is the standard deviation of the Gaussian kernel.

[0149] 2. Skin quality characteristics detection

[0150] Next, based on the preprocessed images, skin characteristics such as acne, wrinkles, spots, dark circles, etc. are detected.

[0151] 1) Acne and acne scar detection

[0152] Detection of acne and acne marks can be achieved by analyzing areas of abnormal color on the skin surface.

[0153] Acne spot detection: Use color thresholding to segment the acne spot area.

[0154] Acne area = {(i,j)|pixel value (i,j)∈acne color range}

[0155] The color range of acne spots can be determined by color space conversion (such as RGB to HSV) and threshold setting.

[0156] Acne spots are classified as follows:

[0157] Mild acne spot: The area of ​​the acne spot is smaller than the threshold T1.

[0158] Severe acne spots: The area of ​​the acne spots is greater than the threshold T1 and smaller than T2.

[0159] Very serious: The area of ​​the acne spot is greater than the threshold T2.

[0160] No acne spots: No acne spots were detected.

[0161] 2) Wrinkle analysis

[0162] Wrinkles are one of the important signs of skin aging. In order to quantify the degree of wrinkles, we can use image texture analysis. First, the skin image is preprocessed, including conversion to grayscale image and Gaussian smoothing filtering to reduce noise and enhance image quality.

[0163] Texture feature extraction: Gray-level co-occurrence matrix (GLCM) is used to analyze the texture characteristics of the skin surface. GLCM can reflect the gray-level dependency between a pixel in an image and its neighboring pixels.

[0164] a. Contrast: A measure of the clarity or depth of image texture.

[0165]

[0166] Among them, L is the gray level, and P(i,j) is the element in the gray level co-occurrence matrix.

[0167] b. Energy: measures the consistency and stability of image texture.

[0168]

[0169] c. Entropy: reflects the complexity or amount of information of image texture.

[0170]

[0171] Wrinkle severity assessment: By comparing the texture features of different areas (such as forehead, corners of eyes, etc.), combined with the contrast, energy and entropy calculated by the above formula, the depth and distribution of wrinkles are comprehensively assessed.

[0172] 3) Color spot detection

[0173] The detection of color spots is also based on the identification of color abnormal areas.

[0174] Color spot detection: Use color thresholding to segment the color spot area.

[0175] Spot area = {(i,j)|pixel value (i,j)∈spot color range}

[0176] Pigmentation classification:

[0177] Color spots: The area of ​​the color spots is greater than the threshold T4.

[0178] Light stain: The stain area is smaller than threshold T4 but larger than T5.

[0179] Severe spots: The area of ​​the spots is greater than the threshold T4.

[0180] No stain: No stain areas were detected.

[0181] 4) Dark circles detection

[0182] The detection of dark circles focuses on the color abnormalities of the skin under the eyes.

[0183] Dark circle detection: Use color space conversion and color thresholding method to detect dark circle areas.

[0184] Dark circle area = {(i, j) | pixel value (i, j)∈dark circle color range and (i, j)∈under-eye area}

[0185] Dark circles classification:

[0186] Light dark circles: The dark circle area has low color saturation and its area is smaller than threshold T6.

[0187] Severe dark circles: The color saturation of the dark circles is high or the area is larger than the threshold T6.

[0188] 5.1.3 Tongue coating detection submodule

[0189] 1. The human tongue can be identified and detected. The tongue image is first converted into a grayscale image using the weighted average method, which helps to simplify the subsequent processing flow and reduce the complexity of the processing. The pixel values ​​in the grayscale image represent the brightness information of each area of ​​the tongue.

[0190] The RGB values ​​of each pixel in the color image are weighted averaged according to certain weights, and then the corresponding grayscale value is obtained. Since the human eye is most sensitive to green, followed by red, and least sensitive to blue, the usual weight setting is:

[0191] (a) Red channel weight: 0.299

[0192] (b) Green channel weight: 0.587

[0193] (c) Blue channel weight: 0.114

[0194] Then, for each pixel, its RGB value is multiplied by the weight of the corresponding channel respectively, and then the three products are added together, and the grayscale value = 0.299*R+0.587*G+0.114*B.

[0195] Then use Gaussian filter to smooth the tongue image to remove noise and slight irregular edges in the image:

[0196]

[0197] Where G(x,y) is the Gaussian kernel function, (x,y) is the pixel position in the image, and σ is the standard deviation of the Gaussian kernel. Smoothing makes subsequent feature extraction more accurate and stable.

[0198] 2. For tongue feature point detection, the directional gradient histogram (HOG) feature method is used to divide the image into several cell units. Adjacent cells do not overlap each other. In each cell, all gradient directions are divided into n bins (generally n=9) as the horizontal axis of the histogram, and the accumulated value of the gradient value corresponding to the angle range is used as the vertical axis of the histogram. The gradient direction histogram is obtained by statistics:

[0199] C=[x1,x2,…,x9]

[0200] Normalize the block and combine m cells into a large, spatially connected block (usually m=4). In this way, a block contains feature vectors of multiple cells. After normalization, the high-dimensional feature vector obtained by connecting them in series is the HOG feature of the block:

[0201] B=[C1,C2,C3,C4]

[0202] Combine the feature vectors of 1 Block, h depends on the width and height of the image, so that the final HOG feature is obtained:

[0203] HOG=[B1,B2,…,B l ]

[0204] These feature points include the key positions of the tongue root, tongue tip and tongue surface, and the local gradient information of the tongue surface is effectively captured through the HOG method.

[0205] 3. Based on the detected tongue feature points, use the straight-line distance calculation formula to perform the following measurements: Tongue length (CL): Measure the length from the tongue root to the tongue tip. Tongue width (CW): Measure the width of the tongue surface, usually the distance from the left side to the right side of the tongue surface.

[0206] A=Y2-Y1

[0207] B=X1-X2

[0208] C=X2*Y1-X1*Y2

[0209] Among them, A and B are the locations of image points.

[0210] For moss color analysis, common moss colors include light white, yellowish greasy, grayish black, etc. The characteristic information of moss color is extracted from the image through color space conversion and thresholding method, and the image color features are extracted using three dimensions: color, depth, and brightness:

[0211] V=max(R,G,B)

[0212]

[0213] Among them, H is color; S is depth, when S=0, there is only grayscale; V is brightness, which indicates the brightness of the color.

[0214] The moss quality is analyzed, such as greasy, thick, thin, moist, etc. The texture feature extraction method is used to characterize it. First, the texture primitive is determined, and then the arrangement rule is established. A texture primitive is a set of connected pixels characterized by a set of attributes. Suppose the texture primitive is h(x, y) and the arrangement rule is r(x, y), then the texture t(x, y) can be expressed as:

[0215]

[0216] Where r(x,y)=∑δ(xx m ,yy m ).

[0217] Tongue color analysis, such as light red, light white, purple red, etc., is achieved by extracting the same color features as the tongue coating.

[0218] The analysis results of tongue length (CL), tongue width (CW), coating color, coating quality and tongue color are combined for analysis, which can be used to identify different tongue quality types such as wet coating, yellow greasy coating, and red tongue. The combination of coating color, coating quality and tongue color can be used to obtain nine typical constitutions such as qi deficiency and yang deficiency.

[0219] 5.1.4 Ranging Submodule

[0220] Distance recognition can measure the distance between the camera and the face. It uses a monocular distance measurement method to locate the face first and output the distance of the face. A deep network structure can be used. The network structure consists of Input, Backbone, Neck, Prediction, etc. The Input structure uses Mosaic data enhancement and adaptive image scaling. The Backbone structure is used for feature extraction. The Neck structure uses a high-to-low and low-to-high feature fusion method to enhance the model's feature extraction and feature fusion capabilities. The Predict structure outputs three scale feature maps that are ultimately used for prediction. Fig. 9 .

[0221] Assume that the distance from the imaging plane to the focal plane of the camera is f, (u, v) is the coordinate of the imaging point on the imaging plane, (d x ,d y ) is the physical size of the pixel in the x and y directions of the coordinate plane, and (u0, v0) is the position of the origin of the imaging plane coordinate system in the pixel coordinate system.

[0222] Calculate the target pixel width:

[0223]

[0224] Then calculate the distance d between the camera and the face:

[0225]

[0226] Finally, the distance value d of the face is output to the management module.

[0227] 5.1.5 Object Detection Submodule ( Fig.10 )

[0228] In this module, objects can be detected to achieve effective management of objects. This module can identify cosmetics and health products.

[0229] The identification of objects can adopt a lightweight identification method, the structure mainly consists of preprocessing, backbone network, Neck and Head detection end. Preprocessing mainly focuses on the Mosaic data enhancement module.

[0230] Random scaling, cropping, color gamut conversion and other operations are used to enhance the image and then stitch it together to make the data set richer. In the backbone network, the Focus module is first used to expand the input channel four times and complete the sampling without information loss. In the neck layer, the convolutional attention module is introduced to improve the detection of small targets.

[0231] 5.1.6 Gesture Recognition Submodule

[0232] The gesture recognition module is mainly used to recognize various actions on the makeup parts during the makeup process, guide the makeup steps and give makeup suggestions. A neural network model based on the fusion of multi-scale spatio-temporal characteristics can be adopted, which mainly consists of a spatio-temporal feature extraction network for gesture images, a multi-scale feature fusion network, a decoding module, etc.

[0233] Fig.11 Among them, in the spatio-temporal feature extraction network, f is the number of channels of each branch, t is the number of feedback loops, and c is the number of convolutional channels. The spatio-temporal feature extraction network is stacked by 4 layers of spatio-temporal attention networks STA-Net, and attention networks with channel numbers of 16, 32, 64, and 128 are designed to extract spatio-temporal feature maps of different scales. The multi-scale feature fusion network consists of a feature pyramid network FPN, a channel attention mechanism, and a feedback loop branch. The FPN is designed to perform multi-scale fusion of spatio-temporal feature maps, and the channel attention network STA-Net is used to assign different weights to spatio-temporal features to suppress redundant information. At the same time, a feedback loop branch is designed to iteratively update spatio-temporal features. Finally, the fused spatio-temporal features are passed through the decoding module to obtain the labels of gesture actions.

[0234] 5.2 Management Module

[0235] The management module mainly consists of sub-modules such as detection and storage status management, human body and distance recognition management, skin type and physique recognition management, and makeup process recognition management. The composition is as Fig.12 shown:

[0236] In the detection and storage status management sub-module, the storage status of cosmetics, health products, etc. is detected, and the quantity is identified and counted. At this time, the image processing module in the perception unit collects the internal image of the system and sends it to the data processing unit for identification and statistics. First, the quantity of items is counted. If there are no required cosmetics and health products, a voice prompt is given; at the same time, the items can be identified and entered into the family member directory, and the item and member mapping can be performed to realize the classification management of items for different members. The functions of the sub-module are as Fig.13 shown:

[0237] (2) In the human body and distance recognition management sub-module, when the image processing module in the perception unit collects the external image of the system and recognizes the face, the distance is measured. At the same time, the recognition distance D_set is input into the data processing unit and compared with the face distance d. When d < D_set, the data processing unit outputs a high-level signal in the IO interface to control the opening of the selected electronic lock. When d > D_set, the data processing unit outputs a low-level signal in the IO interface to control the closing of the selected electronic lock. The functions of the human body and distance recognition management sub-module are as Fig.14 shown:

[0238] In the skin type and physique recognition management sub-module, when the image in the perception unit

[0239] The processing module collects images outside the system and sends them to the data processing unit for recognition and processing. Different skin types are identified for makeup, different cosmetics are selected, and makeup steps are recommended; different tongue coatings are identified for physical analysis, and different health products are selected. At the same time, the data processing unit sends the results to the interaction unit, which gives text display or voice prompts. The functions of the skin type recognition and physical condition management submodules are as follows: Fig.15 As shown:

[0240] The following table shows the cosmetic recommendations:

[0241] Table 1 Cosmetic Selection Recommendations

[0242]

[0243]

[0244] The following table shows the makeup steps suggested table:

[0245] Table 2 Makeup steps recommended

[0246]

[0247]

[0248] The following table shows the recommended selection of health care products:

[0249] Table 3 Recommendations for the selection of health products

[0250]

[0251]

[0252] (4) In the makeup process identification management submodule, Fig.16 , identify the makeup process, mainly identify the following makeup steps: Fig.17 In the process of facial contouring, shadows, highlights, blush and powder brushes are identified in the image to identify the facial contouring process. Eyebrow shaping gestures such as eyebrow dye and eyebrow pencil are identified in the image to identify the eyebrow shaping process. If an error is identified during the process, the data processing unit sends the result to the interaction unit, and the interaction unit gives a voice prompt.

[0253] (2) Fig.18 During the eye makeup process, eyeliner, eye shadow, etc. are identified in the image, and gesture recognition such as eye contouring is performed. If an error is found during the recognition process, the data processing unit sends the result to the interaction unit, and the interaction unit gives a voice prompt.

[0254] (3) Fig.19In the process of nose contouring, makeup brushes, shadows, highlights, etc. are identified in the image, and gestures such as nose contouring are recognized. If an error is found during the process, the data processing unit sends the result to the interaction unit, and the interaction unit gives a voice prompt.

[0255] (4) During lip makeup ( Fig. 20 ), identify lip liner, foundation, etc. in the image, and perform lip makeup and other gesture recognition. If there is an error in the recognition process, the data processing unit sends the result to the interaction unit, and the interaction unit gives a voice prompt.

[0256] The face recognition process flow chart is as follows: Fig.21 As shown: Eye shape recognition process flow chart is as follows Fig. 22 As shown:

[0257] Nose shape recognition process flow chart Fig.23 As shown; the lip shape recognition process is as follows Fig.24 shown.

Claims

1. A method of health testing, physical analysis and management applicable to family members and employees, Its characteristics are: Kangmei detection and management system based on vision. The detection system consists of a perception unit, an interaction unit, a data processing unit, and a communication unit. The sensing unit is used to detect human bodies, faces, cosmetics, and health products. The sensing unit uses an image processing module with more than 5 million pixels to collect images. Two image processing modules are used to collect images from the outside or inside of the system. The image processing module of the perception unit then sends the image signal to the data processing unit for processing through the MIPI interface; The interactive unit includes a voice and display interactive module; the voice is collected and recognized and played; the display interactive module can also display the recognition result; the interactive unit transmits data to the data processing unit through the USB interface; the display module interacts with the interface module through the MIPI access interface module, and the interface module is connected to the data processing unit; The data processing unit is installed with operating systems such as Android, Linux, HarmonyOS, and Windows, and runs software modules such as detection and management. It detects the human body, face, objects, usage process, and distance, and provides different cosmetics and makeup suggestions based on the user's different skin types. The perception unit includes a detection module, a human recognition submodule, and a facial recognition submodule. The detection module is used to sense the state of the human body and face, and can also detect the state of items such as cosmetics and health products; the components of the detection module are: The human body recognition submodule recognizes the human body and uses the YOLO method to identify people in the image. First, the image signal is passed through the Darknet-53 backbone network in Yolo3 to extract features, and then upsampling, feature fusion and regression analysis are performed to obtain the prediction box information, and the target feature modeling, matching and tracking are performed through the SORT algorithm; The facial recognition submodule first detects the face, then extracts the facial features, and performs face shape, eye shape, nose shape, lip shape and skin texture recognition; The management system includes a management module, which includes sub-modules such as detection and storage status management module, human body recognition and distance management, skin quality and physique recognition management, and makeup process recognition management: In the storage status detection management submodule, the storage status of cosmetics, health products, etc. is detected, and the quantity is identified and counted; the above detection results or suggestions are displayed through voice prompts or on the screen; the image processing module in the perception unit collects images inside the system and sends them to the data processing unit for identification and statistics; in the human body identification and distance management submodule, when the image processing module in the perception unit collects images outside the system, it measures the distance after identifying the face; select cosmetics based on skin type, perform pre-treatment and setting of eye, nose, face, and lip makeup, and make-up recommendations for health products.

2. The method for health testing, constitution analysis and management applicable to family members and employees according to claim 1, Its characteristics are: (I) Face shape recognition: recognize the face shape of a person and classify the face shapes into round face, square face, diamond face, triangle face, long face and oval face. First, convert the face image into a grayscale image and use a Gaussian filter for smoothing. Then, use the directional gradient histogram feature and cascade classifier to detect the facial feature points. The local morphological edge gradient information of the face is counted based on the directional gradient histogram feature method: Gradient component calculation: Gradient and direction calculation: The cascade classifier realizes face detection through the cascade method: y=C 1 (x)*C 2 (x)*…C n (x) Where x is the feature vector of the input sample, C 1 , C 2 ,…,C n Represents each stage of the cascade classifier; Line 1 connects the left and right edge coordinate points of the forehead, Line 2 connects the zygomatic line of the 2nd and 16th key points, Line 3 connects the jaw line of the 4th and 14th key points, Line 4 connects the forehead midpoint and the 9th key point, Line 5 connects the forehead length line, Line 6 connects the facial width line; use Euclidean distance to calculate the line segment length: Compute the standard deviation of the line: Calculate the standard deviation SD1 of Line1, Line2, and Line3, calculate the standard deviation SD2 of Line2 and Line4, calculate the angle between the two vectors from the third key point to the fifth key point and from the fourth key point to the sixth key point and the x-axis, take the difference between the two and convert it into the angle Angle; By identifying SD1 and Angle, the face shapes are divided into round face, square face, diamond face, triangle face, rectangular face and oval face; (ii) Eye shape recognition: for eye feature points, use the left key point of the left eye (LL_x1, LL_y1), the right key point of the left eye (LR_x2, LR_y2), the left key point above the left eye (LU_x3, LU_y3), the right key point above the left eye (LU_x4, LU_y4), the left key point below the left eye (LD_x5, LD_y5), the right key point below the left eye (LD_x6, LD_y6), the left key point of the right eye (RL_x7, RL_y7), the right key point of the right eye (RR_x8, RR_y8), the left key point above the right eye (RU_x9, RU_y9), the right key point above the right eye (RU_x10, RU_y10), left key point below the right eye (RD_x11, RD_y11), right key point below the right eye (RD_x12, RD_y12); The left eye length DL is calculated by the Euclidean distance between (LL_x1, LL_y1) and (LR_x2, LR_y2). The right eye length DR is calculated by the Euclidean distance between (RL_x7, RL_y7) and (RR_x8, RR_y8). The interocular distance DLR is calculated by the Euclidean distance between (LR_x2, LR_y2) and (RL_x7, RL_y7). Calculate the Euclidean distance DW between (LU_x3, LU_y3) and (LD_x5, LD_y5), (LU_x4, LU_y4) and (LD_x6, LD_y6), (RU_x9, RU_y9) and (RD_x11, RD_y11), (RU_x10, RU_y10) and (RD_x12, RD_y12) respectively. 1 , DW 2 , DW 3 , DW 4 Calculate the average width DW, By comparing DL, DR, DLR, and DW, slender eyes, round eyes, and small eyes were identified; 3. Nose shape recognition For the nose feature point, use the nasal root position (A_x 1 ,A_y 1 ), the position of the nose tip is (B_x 2 ,B_y 2 ), the position of the left nose wing is (C_x 3 ,C_y 3 ), the position of the right nose wing is (D_x 4 ,D_y 4 ), the frontal length is LM; By (A_x 1 ,A_y 1 ) and (B_x 2 ,B_y 2 ) to calculate the nose length By (C_x 3 ,C_y 3 ) and (D_x 4 ,D_y 4 ) to calculate the nose width By comparing the values ​​of BL, BW, Line5 and Line6, wide nose, long nose and short nose can be identified; (For the processing flow, see the nose shape recognition processing flow chart below) (IV) Lip shape recognition First locate the lip area. According to the overall structure and proportion of the face, the position of the mouth can be obtained: Among them, H face and W face Indicates the height and width of the face area, H mouth and W mouth Indicates the height and width of the mouth area; After further optimization and noise reduction, we get a lip image without teeth, beard, etc.: (a) Calculate the grayscale of the mouth area. Assume that the mouth image is P, and its resolution is MxN. P(x,y) represents the grayscale value at (x,y). The grayscale range is [0,1,…,L-1]. The number of pixels corresponding to grayscale level l is m. l , the probability calculation formula for each gray level is l ranges from [0,1,…,L-1]; (b) The pixels in the mouth area are divided into the lip area and the non-lip area according to the grayscale level L using the threshold th. The grayscale range of the lip area is [0,…,th-1], and the grayscale range of the non-lip area is [th,…,L-1]. The probability of the lip area and the non-lip area appearing is: The average grayscale values ​​of the lip area and non-lip area are The total gray value of the mouth area μ = P T *μ T +P F *μ F We can calculate l∈[0,1,…,L-1], the l value corresponding to max[σ(l)], and thus obtain the threshold value T 1 , thereby distinguishing the lip area from the non-lip area; (c) The inner lip edge contour is then constructed to obtain the binary image pixel value of the lip area. Select the first 15% of the binary image pixel values ​​and set them to 255, and the remaining pixel values ​​are 0 to fit the inner lip curve: where h 1 and h 2 represents the height coefficient of the upper and lower parts of the inner lip, respectively, w represents the width coefficient, and x represents the inner lip edge motion variable; (d) The outer lip edge contour is constructed again. The outer lip edge contour is reddish, and the outer lip edge skin is not red. The lip pixel set X1 and the surrounding skin pixel set X2 are obtained, and the mean vector of the lip and surrounding skin pixels is calculated. Where M and N represent the number of pixels in the set respectively; then calculate the intra-class discrete matrix S corresponding to the lips and surrounding skin b and S w , and the global divergence matrix S of the lip area image t By S t Find the intra-class projection weight k and calculate the mean y of the projection of lip and skin pixel information into one-dimensional linear space 1 and 2 , and detect the lip area image: Where z represents the image, k 0 Represents the standard mean value of the pixels in the lip area; when the above formula is greater than 0, the area is the outer lip contour, and when it is less than or equal to 0, the area is the skin around the outer lip contour, thereby fitting the outer lip curve: where h 3 and h 4 Represent the height coefficient of the outer lip contour, q 1 and q 2 Represent the coefficients of the upper and lower grooves of the outer lip, w 1 represents the outer lip width coefficient; (e) Determine the lip shape standard: the upper and lower lip width ratio coefficient hk = (h 3 -h 1 ) / (h 4 -h 1 ) When hk>1 / 1.3, the lip shape is thick lips; when hk<1 / 1.5, the lip shape is thin lips; When hk∈[1 / 1.3,1 / 1.5], the lip shape is the standard lip; (V) Skin type identification Skin recognition analyzes specific areas in facial images, including details on the skin surface, to identify various skin characteristics, such as acne, wrinkles, spots, and dark circles; (a) Image preprocessing First, the skin image is preprocessed, including conversion to grayscale image and Gaussian filtering smoothing; this step helps to reduce image noise and unnecessary details, making subsequent feature extraction more accurate; Grayscale: Convert color skin images to grayscale images; Gray value (i, j) = 0.299*R(i, j) + 0.587*G(i, j) + 0.114*B(i, j) Gaussian filtering: Use Gaussian filter to smooth grayscale images and remove noise; Where G(x,y) is the Gaussian kernel function, (x,y) is the pixel position in the image, and σ is the standard deviation of the Gaussian kernel; (b) Skin quality characteristics detection Next, based on the preprocessed images, skin characteristics such as acne, wrinkles, spots, dark circles, etc. are detected.

3. The method for health detection, constitution analysis and management applicable to family members and employees according to claim 1, Its characteristics are: 1) Acne and acne scar detection The detection of acne and acne marks is achieved by analyzing the abnormal color areas on the skin surface; Acne spot detection: Use color thresholding to segment the acne spot area; Acne area = {(i,j)|pixel value (i,j)∈acne color range} The color range of acne spots can be determined by color space conversion (such as RGB to HSV) and threshold setting; Acne spots are classified as follows: Mild acne spot: The area of ​​the acne spot is smaller than the threshold T1; Severe acne spots: The area of ​​acne spots is greater than threshold T1 and less than T2; Very serious: the area of ​​acne spots is larger than the threshold T2; No acne spots: No acne spots were detected; 2) Wrinkle analysis Wrinkles are one of the important signs of skin aging. To quantify the degree of wrinkles, we can use image texture analysis. First, the skin image is preprocessed, including conversion to grayscale image and Gaussian smoothing filtering to reduce noise and enhance image quality. Texture feature extraction: Use gray-level co-occurrence matrix (GLCM) to analyze the texture characteristics of the skin surface; GLCM can reflect the gray-level dependency between a pixel in an image and its neighboring pixels; a. Contrast: measures the clarity or depth of image texture; Where L is the gray level, and P(i,j) is the element in the gray level co-occurrence matrix; b. Energy: measures the consistency and stability of image texture; c. Entropy: reflects the complexity or amount of information of image texture; Wrinkle degree assessment: By comparing the texture features of different areas (such as forehead, corners of eyes, etc.), combined with the contrast, energy and entropy calculated by the above formula, the depth and distribution of wrinkles are comprehensively assessed; 3) Color spot detection The detection of color spots is also based on the identification of abnormal color areas; Color spot detection: Use color thresholding method to segment the color spot area; Spot area = {(i,j)|pixel value (i,j)∈spot color range} Pigmentation classification: Color spots: The area of ​​the color spots is greater than the threshold T4; Light stain: The stain area is smaller than threshold T4 but larger than T5; Severe spots: The area of ​​the spots is greater than the threshold T4; No stain: No stain area detected; 4) Dark circles detection The detection of dark circles focuses on the color abnormality of the skin under the eyes; Dark circle detection: Use color space conversion and color thresholding to detect dark circle areas; Dark circle area = {(i, j) | pixel value (i, j)∈dark circle color range and (i, j)∈under-eye area} Dark circles classification: Light dark circles: The color saturation of the dark circles is low and the area is smaller than the threshold T6; Severe dark circles: The color saturation of the dark circles is high or the area is larger than the threshold T6.

4. The method for health detection, constitution analysis and management applicable to family members and employees according to claim 1, Its characteristics are: A tongue coating detection submodule is provided to identify and detect the human tongue. The tongue image is first converted into a grayscale image using the weighted average method, which helps to simplify the subsequent processing flow and reduce the complexity of the processing. The pixel value in the grayscale image represents the brightness information of each area of ​​the tongue. The RGB values ​​of each pixel in the color image are weighted averaged according to certain weights, and then the corresponding grayscale value is obtained. Since the human eye is most sensitive to green, followed by red, and least sensitive to blue, the usual weight setting is: (a) Red channel weight: 0.299 (b) Green channel weight: 0.587 (c) Blue channel weight: 0.114 Then, for each pixel, multiply its RGB value by the weight of the corresponding channel respectively, and then add the three products together, grayscale value = 0.299*R+0.587*G+0.114*B; Then use Gaussian filter to smooth the tongue image to remove noise and slight irregular edges in the image: Where G(x,y) is the Gaussian kernel function, (x,y) is the pixel position in the image, and σ is the standard deviation of the Gaussian kernel; smoothing processing makes subsequent feature extraction more accurate and stable; For tongue feature point detection, the directional gradient histogram (HOG) feature method is used to divide the image into several cell units. Adjacent cells do not overlap each other. In each cell, all gradient directions are divided into n bins (generally n=9) as the horizontal axis of the histogram, and the accumulated value of the gradient value corresponding to the angle range is used as the vertical axis of the histogram. The gradient direction histogram is obtained by statistics: C=[x 1 ,x 2 ,…,x 9 ] Normalize the blocks and combine m cells into a large, spatially connected block (usually m=4). In this way, a block contains feature vectors of multiple cells. After normalization, the high-dimensional feature vector obtained by connecting them in series is the HOG feature of the block: B=[C 1 ,C 2 ,C 3 ,C 4 ] Combine the feature vectors of 1 Block, h depends on the width and height of the image, so that the final HOG feature is obtained: HOG=[B 1 ,B 2 ,…,B l ] These feature points include the key positions of the tongue root, tongue tip and tongue surface, and the HOG method is used to effectively capture the local gradient information of the tongue surface; Based on the detected tongue feature points, the following measurements are performed using the straight-line distance calculation formula: Tongue length (CL): measures the length from the tongue root to the tongue tip; Tongue width (CW): measures the width of the tongue surface, usually the distance from the left side to the right side of the tongue surface; A=Y 2 -AND 1 B=X 1 -X 2 C=X 2 *AND 1 -X 1 *AND 2 Among them, A and B are the locations of image points; For moss color analysis, common moss colors include light white, greasy yellow, gray and black, etc. The characteristic information of moss color is extracted from the image through color space conversion and thresholding method, and the image color features are extracted using three dimensions: color, depth, and brightness: V=max(R,G,B) Among them, H is color; S is depth, when S=0, there is only grayscale; V is brightness, indicating the brightness of the color; For the analysis of moss quality, greasy, thick, thin, and moist are all characterized by texture feature extraction methods. First, the texture primitives are determined, and then the arrangement rules are established. A texture primitive is a set of connected pixels characterized by a set of attributes. Suppose the texture primitive is h(x, y) and the arrangement rule is r(x, y), then the texture t(x, y) can be expressed as: where r(x, y) = ∑δ(x - x m , y - y m ); Tongue color analysis, such as light red, light white, purple red, etc., is achieved by extracting the same color features as the tongue coating; The analysis results of tongue length (CL), tongue width (CW), coating color, coating quality and tongue color are combined for analysis, which can be used to identify different tongue quality types such as wet coating, yellow greasy coating, and red tongue; nine typical constitutions such as Qi deficiency and Yang deficiency are obtained through the combination of coating color, coating quality and tongue color.

5. The method for health detection, constitution analysis and management applicable to family members and employees according to claim 1, Its characteristics are: There is a distance measurement submodule, which can identify and measure the distance between the camera and the face. The monocular distance measurement method is used to locate the face first. Output the distance of the face; adopt a deep network structure, which consists of Input, Backbone, Neck, and Prediction. The Input structure uses Mosaic data enhancement and adaptive image scaling, the Backbone structure is used for feature extraction, and the Neck structure uses a high-to-low and low-to-high feature fusion method to enhance the model's feature extraction and feature fusion capabilities; the Predict structure outputs three scale features that are ultimately used for prediction; Assume that the distance from the imaging plane to the focal plane of the camera is f, (u, v) is the coordinate of the imaging point on the imaging plane, (d x ,d y ) is the physical size of the pixel in the x and y directions of the coordinate plane, (u 0 ,v 0 ) is the position of the origin of the imaging plane coordinate system in the pixel coordinate system; Calculate the target pixel width: Then calculate the distance d between the camera and the face: Finally, the distance value d of the face is output to the management module.

6. The method for health detection, constitution analysis and management applicable to family members and employees according to claim 1, Its characteristics are: Equipped with object detection submodule In this module, objects can be detected to achieve effective management of objects; this module can identify cosmetics and health products; Object recognition can adopt a lightweight recognition method, the structure of which mainly consists of preprocessing, backbone network, Neck and Head detection end; preprocessing mainly focuses on Mosa-ic data enhancement module, Random scaling, cropping, color gamut conversion and other operations are used to enhance the image and then stitch it together to make the data set richer. In the backbone network, the Focus module is first used to expand the input channel four times and complete the sampling without information loss. At the neck layer, a convolutional attention module is introduced to improve the detection of small targets.

7. The method for health detection, constitution analysis and management applicable to family members and employees according to claim 1, Its characteristics are: A gesture recognition submodule is provided. The gesture recognition module is used to recognize various actions in the makeup area during the makeup process, guide the makeup steps and make makeup suggestions. A neural network model based on multi-scale spatiotemporal feature fusion can be used. It is mainly composed of a spatiotemporal feature extraction network of gesture images, a multi-scale feature fusion network and a decoding module. In the spatiotemporal feature extraction network of gesture images, f is the number of channels of each branch, t is the number of feedback cycles, and c is the number of convolution channels. The spatiotemporal feature extraction network is composed of a stack of 4 layers of spatiotemporal attention networks STA-Net. Attention networks with 16, 32, 64, and 128 channels are designed to extract spatiotemporal feature maps of different scales. FPN is designed to fuse spatiotemporal feature maps at multiple scales, and the channel attention network STA-Net is used to assign different weights to spatiotemporal features to suppress redundant information. Feedback loop branches are designed to iteratively update spatiotemporal features. Finally, the fused spatiotemporal features are passed through a decoding module to obtain the label of the gesture action.

8. The method for health detection, constitution analysis and management applicable to family members and employees according to claim 1, Its characteristics are: There is a management module, which is mainly composed of sub-modules such as detection and storage status management, human body recognition and distance management, skin type and physique recognition management, and makeup process recognition management; (1) In the detection storage status management sub-module, the storage status of cosmetics, health products, etc. is detected, and the quantity is identified and counted; at this time, the image processing module in the sensing unit collects the internal image of the system and sends it to the data processing unit for identification and statistics; first, the quantity of items is counted, and if there are no required cosmetics and health products, a voice prompt is given; at the same time, the items can be identified and entered into the family member directory, and the item and member mapping can be performed to achieve the classification management of items for different members. (2) In the human body and distance management sub-module, when the image processing module in the sensing unit collects the external image of the system and recognizes the human face, the distance is measured; at the same time, the recognized distance D_set is input into the data processing unit and compared with the face distance d. When d < D_set, the data processing unit outputs a high-level signal in the IO interface to control the selected electronic lock to open. When d > D_set, the data processing unit outputs a low-level signal in the IO interface to control the selected electronic lock to close. (3) In the skin type and physique management sub-module, when the image processing module in the sensing unit collects the external image of the system, it is sent to the data processing unit for recognition and processing; different skin types are recognized for makeup, different cosmetics are selected, and makeup steps are recommended; different tongue coatings are recognized for physique analysis, and different health products are selected; at the same time, the data processing unit sends the results to the interaction unit, and the interaction unit gives a text display or voice prompt to identify the functions of the skin type and physique management sub-module; suggestions for selecting cosmetics according to skin type, pre-makeup treatment, and setting makeup for eyes, nose, face, lips, and health products are given.

9. The Kangmei detection, physique analysis and management method applicable to family members and employees according to claim 8, characterized in that, (4) In the makeup process recognition management sub-module, the makeup process and steps are recognized: In the process of contouring the face, shadows, highlighters, blushes, and powder brushes are recognized in the image to identify the face contouring process; eyebrow shaping gestures such as eyebrow mascara and eyebrow pencils are recognized in the image to identify the makeup area of the eyebrow shaping process; when an error is recognized during the process, the data processing unit sends the result to the interaction unit, and the interaction unit gives a voice prompt.

Images