A control method of an intelligent bathroom mirror

Abstract

This invention discloses a control method for an intelligent bathroom mirror. The method includes the following steps: Step 1: An administrator collects user data and stores it in a user database. The data includes height, facial features, and facial contour data. The administrator sets the parameters for the lighting module. This invention integrates a controller, power supply module, 3D camera, touchscreen, lighting module, defogging module, and communication module into the bathroom mirror, achieving automated and intelligent detection of the mirror. It applies artificial intelligence technologies such as human key point and identity recognition algorithms, high-precision Anchor-Free target detection algorithms, and behavior analysis to the bathroom mirror, enabling the detection of human actions and behavior patterns in front of the mirror. Based on the detection results, it provides operation reminders, and the lighting module adjusts the light according to the behavior pattern, thus greatly improving the intelligence of the bathroom mirror.

Description

Technical Field

[0001] This invention belongs to the field of bathroom products and relates to a control method for an intelligent bathroom mirror. Background Technology

[0002] Bathroom mirrors are mirrors placed in the bathroom for grooming and are an indispensable part of the bathroom space. With the popularization and application of smart home technology, bathroom mirrors have also acquired intelligent application functions. Currently, smart bathroom mirrors on the market have LED lights and touch switches, which allow adjustment of the lighting color temperature via touch switches. Overall, their functions are relatively simple, and their intelligent applications are quite basic. Summary of the Invention

[0003] In order to overcome the shortcomings of the prior art, the present invention provides a control method for an intelligent bathroom mirror.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: a control method for an intelligent bathroom mirror, the method comprising the following steps:

[0005] Step 1: The administrator collects user data and stores it in the user database. The data includes height, facial features, and facial contour data; the administrator sets the lighting module parameters.

[0006] Step 2: Real-time monitoring of the 3D camera; the controller controls the 3D camera to acquire the color image Pm and depth image Dm in front of the bathroom mirror in real time.

[0007] Step 3: When someone enters the bathroom, determine whether the person is in front of the bathroom mirror. If yes, proceed to step 4; otherwise, proceed to step 2.

[0008] Step 4: Identify the entrant;

[0009] Step 5: Obtain the current time. The lighting module adjusts the brightness to Lh and the color temperature to Kh according to the time period Th to which the current time belongs.

[0010] Step 6: Determine whether the bathroom mirror and the 3D camera lens need defogging. If yes, execute the automatic defogging process and then proceed to Step 7. If no, proceed to Step 7.

[0011] Step 7: Process the color image Pm and obtain the face region F1;

[0012] Step 8: Process the color image Pm to obtain the left-hand region H1 and the right-hand region H2;

[0013] Step 9: Analyze the behavior to determine if it is a cleaning behavior. If yes, proceed to step 2 after the cleaning behavior is completed. If no, proceed to step 10.

[0014] Step 10: Determine if the hand is holding an object. If yes, proceed to step 12 to enter the object recognition process. If no, proceed to step 11.

[0015] Step 11: Determine if a certain area needs to be enlarged. If yes, execute the enlargement process; otherwise, execute step 13.

[0016] Step 12: The object recognition model identifies the object held by the hand. If the object recognition model outputs 1, the object is determined to be a toothbrush, and the brushing behavior mode is entered; if the object recognition model outputs 2, the object is determined to be a lipstick, and the lipstick behavior mode is entered; if the object recognition model outputs 3, the object is determined to be a razor, and the shaving behavior mode is entered.

[0017] Step 13: Determine if anyone is in the bathroom. If yes, proceed to step 2; otherwise, turn off the lighting module and end the process.

[0018] Furthermore, in step 1, the user-defined lighting module parameters are the brightness and color temperature of the light for different time periods and different behavior modes. The division of time periods is based on user-defined settings, and the brightness and color temperature for different time periods are set according to user-defined settings. The controller controls the lighting module to turn on and off, and turns on the brightness and color temperature of the corresponding time period according to the current time. Then, the brightness and color temperature are converted into the brightness and color temperature set for the corresponding behavior mode according to the current behavior mode.

[0019] Furthermore, the specific steps in step 3 for determining whether the entrant is active in front of the bathroom mirror are as follows:

[0020] Step 3.1: Calculate the distance d between the person entering and the bathroom mirror;

[0021] Step 3.2: Determine d and the threshold distance S between the entrant and the bathroom mirror. If d is not greater than S,

[0022] Execute step 3.3. If d is greater than S, determine that the person entering is not active in front of the bathroom mirror.

[0023] Step 3.3: Determine the activity time Δt of the entrant under the condition that d is not greater than S and the activity threshold time t. If Δt is not less than t, determine that the entrant is active in front of the bathroom mirror and proceed to step 4. If Δt is less than t, determine that the entrant is not active in front of the bathroom mirror and proceed to step 2.

[0024] Furthermore, in step 4, identifying the entrant's identity specifically involves: identifying facial information in the color image Pm, comparing the facial information with the user database to identify the user's identity; if the user's identity is identified, proceed to step 5; if the user's identity is not identified, the entrant is determined to be a stranger, and the touchscreen prompts "Does the administrator need to add data?"; if data needs to be added, the administrator collects the stranger's data; otherwise, proceed to step 5.

[0025] Furthermore, step 6 includes the following steps:

[0026] Step 6.1: The humidity sensor detects whether the humidity of the bathroom mirror is greater than the mirror humidity threshold. If yes, it is determined that the bathroom mirror needs to be defogged and proceeds to step 6.4. If no, proceed to step 6.2.

[0027] Step 6.2: The controller starts the defogging module, and the heating resistance wire defogs the bathroom mirror surface;

[0028] Step 6.3: The humidity sensor detects the humidity of the bathroom mirror in real time. When the humidity of the mirror is less than the humidity threshold, the controller controls the defogging module to stop and proceeds to step 6.4.

[0029] Step 6.4: Calculate the sharpness index C0(f) of the current 3D camera lens, and compare the sharpness index C0(f) with the sharpness threshold C. A If C0(f) > C A If the image is clear, the process ends; otherwise, if the image is blurry, proceed to step 6.5.

[0030]

[0031] In equation (4) above, x Pa y Pa For color image P m The horizontal and vertical coordinates, f(x) P ,y P ) is at coordinate point (x Pa y Pa The corresponding grayscale value;

[0032] Step 6.5: The controller activates the defogging module, heating the resistance wire to defog the lens;

[0033] Step 6.6: Calculate the lens's defogging sharpness index C t (f) Comparison of fog-clearance index C t (f) and the sharpness threshold C A If C t (f)>C A If the image is clear, proceed to step 6.8 and end the process. If not, compare with C. t (f) and C0(f), if |C t (f)-C0(f)| <C θ C θ If the sharpness difference threshold is set, it is assumed that the poor sharpness of the lens is caused by dirt that cannot pass through the defogging module, and the lens is determined to be dirty, proceeding to step 6.7; otherwise, continue to step 6.5.

[0034] Step 6.7: The touchscreen prompts the user to clean the lens and proceed to step 6.6;

[0035] Step 6.8: End of steps.

[0036] Furthermore, in step 7, the color image P is processed. m The mid-face area specifically includes the following steps:

[0037] Step 7.1: Extract the color image P m For key facial points 14-17, calculate the minimum bounding rectangle, expanding its length and width by n pixels to generate the facial region. Crop the facial region to obtain the facial image F1.

[0038] Step 7.2: Run the face_landmark_localization neural network model on the face image F1 to identify 70 key points of the face. Use two cascaded CNNs to detect the key points and identify the outer contour of the face, the eyebrow region, the nose region, the eye region, the lip region and the mouth region.

[0039] Step 7.3: Based on the regions identified in Step 7.2, obtain the skin color region and the lip region. Further, in Step 8, process the color image P. m The middle hand region, specifically: extracting the color image P m For key points 4 and 7 on the upper part, calculate the minimum bounding rectangle, expand the length and width by n pixels, and generate the left-hand region H1 and the right-hand region H2.

[0040] Furthermore, the formula for calculating the distance d in step 3.1 is as follows:

[0041] Color image P m Key points are marked as O Pa Depth image D m Key points are marked as O Da a = 1, 2...18, O Pa With O Da One-to-one correspondence;

[0042] Color image P m The pixel width is set to W P Pixel height set to H eP Depth image D m The pixel width is set to W D Pixel height set to H eD W P H eP W D H eD Given the parameters, we can obtain

[0043]

[0044] Take the key point O of P1 Pa O Pa The coordinates are (x Pa y Pa ), P m D of key point mapping m Key point O Da The coordinates are (x Da y Da ),(x Pa y Pa ) are known parameters that can be obtained. According to (1), we can obtain

[0045]

[0046] From equation (2), the coordinates (x, y) of the key point are obtained. Da y Da According to (x) Da y Da ), to obtain the depth image D m The distance value d of the corresponding target a d a The distance between the key points of the actual photographed object and the depth values ​​of the 3D camera is taken from key points 0-7 and 14-17.

[0047]

[0048] Furthermore, the specific steps for determining whether the behavior in step 9 is a cleaning behavior are as follows:

[0049] Step 9.1: Calculate the distances D1 and D2 between the left hand region H1 and the right hand region H2 and the lip region, respectively;

[0050] D1 is the actual distance between the center point of the left hand region H1 and the center point of the lip region, and D2 is the actual distance between the center point of the right hand region H2 and the center point of the lip region. (The last part, "P," appears to be an incomplete sentence or fragment.) m In the image, the center point of the lip region is Z. P1 The coordinates are set as (x P1 y P1 The center point of the left-hand region H1 is Z. P2 Z P2 The coordinates are set to (x) P2 y P2 The center point of the right-hand region H2 is Z. P3 Z P3 The coordinates are set to (x) P3 y P3 ),Z P1 Z P2 and Z P3The coordinates are known parameters, Z. P1 Z P2 and Z P3 In D m The mapping point on is Z D1 Z D2 and Z D3 Z D1 The coordinates are (x D1 y D1 ),Z D2 The coordinates are (x D2 y D2 ),Z D3 The coordinates are (x D3 y D3 According to equations (1) and (2) above, Z can be obtained. P1 Z P2 and Z P3 With Z D1 Z D2 and Z D3 The mapping relationship of coordinates;

[0051]

[0052]

[0053]

[0054] According to equations (5), (6), and (7) above, Z D1 Z D2 and Z D3 From the coordinates, the depth image D can be obtained. m China Z D1 Z D2 and Z D3 Distance from 3D camera depth value d D1 d D2 d D3 ;

[0055] Calculate depth image D m Compared to the magnification of the actual subject, D m Set the horizontal image magnification to M x D m Set the vertical image magnification to M y The actual physical field of view of the subject being photographed is set as W0×H0, where W0 is the horizontal width and H0 is the vertical height. Therefore, we can obtain...

[0056]

[0057]

[0058] In equation (9) above, α1 is the lateral field of view of the 3D camera, α2 is the longitudinal field of view of the 3D camera, and μ is the angle between the 3D camera and D. m The mean distance, α1, α2, and μ are known parameters. According to equations (8) and (9) above, we can obtain

[0059]

[0060]

[0061]

[0062] Establish a world coordinate system based on the actual photographed subject, Z D1 Z D2 and Z D3 Let the point corresponding to the world coordinate system be Z. S1 Z S2 and Z S3 According to equations (11), (12), and (13) above, Z D1 Z D2 and Z D3 The coordinates are transformed into Z. S1 Z S2 and Z S3 Coordinates in the world coordinate system, Z S1 The coordinates are (W x1 W y1 W z1 Z S2 The coordinates are (W x2 W y2 W z2 Z S3 The coordinates are (W x3 W y3 W z3 ); we can obtain

[0063]

[0064]

[0065] Step 9.2: Calculate the overlap area and intersection-union ratio between the left-hand region H1 and the right-hand region H2 and the lip region, respectively;

[0066] Let the area of ​​the lip region be SZ, the area of ​​the left-handed region H1 be SH1, the overlap area between the left-handed region H1 and the lip region be Si1, the intersection-union ratio of the left-handed region H1 be IoU1, the area of ​​the right-handed region H2 be SH2, the overlap area between the right-handed region H2 and the lip region be Si2, and the intersection-union ratio of the right-handed region H2 be IoU2. We can then obtain...

[0067]

[0068]

[0069] In equations (15) and (16) above, SZ, SH1, Si1, SH2, and Si2 are all known parameters.

[0070] According to the Pm image, SZ, SH1, Si1, and SH2 are all areas in the Pm image;

[0071] Step 9.3: Analyze the behavior:

[0072] If both D1 and D2 are greater than D0, and Si1 = 0 and Si2 = 0, then the user is determined to be in a cleaning behavior.

[0073] If D1 or / and D2 are greater than Ds but not greater than D0, and IoU1 and IoU2 are greater than 0, then the analysis continues.

[0074] If D1 or / and D2 are less than Ds, and IoU1 and IoU2 are greater than IoU0, the user is determined to be in a grooming behavior.

[0075] D0 is the maximum distance threshold between the hand and mouth areas. At this distance, it is considered that there is no interaction between the hand and mouth. D0 is a set value.

[0076] Ds is the minimum distance threshold between the hand and mouth areas. At this distance, interaction between the hand and mouth is considered to exist. Ds is a set value.

[0077] IoU0 is the threshold for the cross-union ratio.

[0078] Furthermore, the specific steps of the scale-up process in step 11 are as follows:

[0079] Step 11.1: The user draws a circle with their hand over the area of ​​the face that needs to be enlarged;

[0080] Step 11.2: Extract the trajectory of the hand making circles;

[0081] Acquire a real-time color image Pt of the hand circling motion from start to finish. Calculate the minimum bounding rectangle, expanding its length and width by n pixels to generate the face region. Crop the face region to obtain the face image Ft. Extract the hand key points 4 and 7 from the color image Pt. Calculate the minimum bounding rectangle, expanding its length and width by n pixels to generate the left hand region H1t and the right hand region H2t. Use a convolutional pose machine (CPM) to detect the left hand region H1t and the right hand region H2t, extract the finger key points, and obtain the points where the face image Ft overlaps with the finger key points. Several points form the hand circling trajectory.

[0082] Step 11.3: Fit the smallest horizontal outer rectangle to the trajectory of the hand drawing circles to form a trajectory image. Crop the trajectory image from the face image Ft and perform secondary interpolation on the trajectory image data to increase the resolution and generate a magnified image.

[0083] Step 11.4: The touchscreen displays a magnified image.

[0084] In summary, the advantages of this invention are:

[0085] 1) The bathroom mirror of the present invention has brushing standard detection, lipstick detection and shaving detection, and automatically adjusts the corresponding lights according to different behaviors, which enriches the functions of the bathroom mirror and improves the comfort and convenience of using the bathroom mirror.

[0086] 2) This invention integrates a controller, power supply module, 3D camera, touch screen, lighting module, defogging module, and communication module into a bathroom mirror, realizing the automation and intelligence of bathroom mirror detection. It applies artificial intelligence technologies such as human key point and identity recognition algorithms, high-precision Anchor-Free target detection algorithms, and behavior analysis to the bathroom mirror, enabling the detection of human operation and behavior patterns in front of the bathroom mirror, and providing operation reminders based on the detection results. The lighting module adjusts the light according to the behavior pattern, thereby greatly improving the intelligence of the bathroom mirror. Attached Figure Description

[0087] Figure 1 This is a schematic diagram of the structure of the smart bathroom mirror of the present invention.

[0088] Figure 2 This is a flowchart of the present invention.

[0089] Figure 3 This is a flowchart of the lighting module of the present invention.

[0090] Figure 4 This is a schematic diagram of key human body points according to the present invention.

[0091] Figure 5 This is a flowchart of the defogging module of the present invention.

[0092] Figure 6 This is a schematic diagram of key points in the facial region of the present invention.

[0093] Figure 7 a is a schematic diagram of key points in the left orbital region of the present invention.

[0094] Figure 7 b is a schematic diagram of key points in the right orbital region of the present invention.

[0095] Figure 7 c is a schematic diagram of key points in the lip region of this invention.

[0096] Figure 8 This is a schematic diagram of the facial region distribution according to the present invention.

[0097] Figure 9 This is a schematic diagram of the lip region and hand region of the present invention.

[0098] Figure 10 This is a schematic diagram of the lip region and hand region of the present invention. Detailed Implementation

[0099] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0100] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0101] In this embodiment of the invention, all directional indicators (such as up, down, left, right, front, back, lateral, longitudinal, etc.) are only used to explain the relative positional relationship and movement of each component in a specific posture. If the specific posture changes, the directional indicator will also change accordingly.

[0102] Due to installation errors and other reasons, the parallel relationship referred to in the embodiments of the present invention may actually be an approximate parallel relationship, and the perpendicular relationship may actually be an approximate perpendicular relationship.

[0103] Example 1:

[0104] like Figure 1-10 As shown, a control method for a smart bathroom mirror is disclosed. The method controls the smart bathroom mirror, which includes a controller, a power supply module, a 3D camera, a touchscreen, a lighting module, a defogging module, and a communication module. The power supply module supplies power to the controller. The controller is connected to the 3D camera, the touchscreen, the lighting module, and the defogging module. The controller controls the communication module to upload and download user data, including the following steps:

[0105] Step 1: The administrator collects user data and stores it in the user database. The data includes height, facial features, and facial contour data; the administrator sets the lighting module parameters.

[0106] Step 2: Real-time monitoring of the 3D camera; the controller controls the 3D camera to acquire the color image Pm and depth image Dm in front of the bathroom mirror in real time.

[0107] Step 3: When someone enters the bathroom, determine whether the person is in front of the bathroom mirror. If yes, proceed to step 4; otherwise, proceed to step 2.

[0108] Step 4: Identify the entrant;

[0109] Step 5: Obtain the current time. The lighting module adjusts the brightness to Lh and the color temperature to Kh according to the time period Th to which the current time belongs.

[0110] Step 6: Determine whether the bathroom mirror and the 3D camera lens need defogging. If yes, execute the automatic defogging process and then proceed to Step 7. If no, proceed to Step 7.

[0111] Step 7: Process the color image Pm and obtain the face region F1;

[0112] Step 8: Process the color image Pm to obtain the left-hand region H1 and the right-hand region H2;

[0113] Step 9: Analyze the behavior to determine if it is a cleaning behavior. If yes, proceed to step 2 after the cleaning behavior is completed. If no, proceed to step 10.

[0114] Step 10: Determine if the hand is holding an object. If yes, proceed to step 12 to enter the object recognition process. If no, proceed to step 11.

[0115] Step 11: Determine if a certain area needs to be enlarged. If yes, execute the enlargement process; otherwise, execute step 13.

[0116] Step 12: The object recognition model identifies the object held by the hand. If the object recognition model outputs 1, the object is determined to be a toothbrush, and the brushing behavior mode is entered; if the object recognition model outputs 2, the object is determined to be a lipstick, and the lipstick behavior mode is entered; if the object recognition model outputs 3, the object is determined to be a razor, and the shaving behavior mode is entered.

[0117] Step 13: Determine if anyone is in the bathroom. If yes, proceed to step 2; otherwise, turn off the lighting module and end the process.

[0118] In step 1, the user-defined lighting module parameters are the brightness and color temperature of the lights for different time periods and different behavior modes. The division of time periods is based on the user-defined settings, and the brightness and color temperature for different time periods are set according to the user-defined settings. The controller controls the lighting module to turn on and off, and turns on the brightness and color temperature of the corresponding time period according to the current time. Then, it converts the brightness and color temperature to the brightness and color temperature set for the corresponding behavior mode according to the current behavior mode.

[0119] In this embodiment, a 24-hour day is divided into H time periods, the current time period is Th, 1≤h≤H, h is the sequence number of the current time period, the brightness of the current time period is set to Lh, and the color temperature of the current time period is set to Kh.

[0120] This embodiment includes the following time periods: 1. Early morning (1:00-5:00); 2. Morning (5:00-8:00); 3. Morning (8:00-11:00); 4. Noon (11:00-13:00); 5. Afternoon (13:00-17:00); 6. Evening (17:00-19:00); 7. Midnight (19:00-20:00); and 8. Late night (20:00-24:00). The brightness and color temperature for the early morning (1:00-5:00) and late night (20:00-24:00) periods are set to 100-200 l. For the morning (5:00-8:00), late morning (8:00-11:00), evening (17:00-19:00), and midnight (19:00-20:00) periods, the brightness and color temperature should be set to 300-400 lux and 4000-5000 k respectively. These periods are generally for grooming and require higher brightness and color temperature. For the midday (11:00-13:00) and afternoon (13:00-17:00) periods, the brightness and color temperature should be set to 100-200 lux and 4000-5000 k respectively. These periods are not for grooming and require higher ambient light.

[0121] Behavioral patterns include lipstick behavior pattern, shaving behavior pattern, and brushing behavior pattern; the lipstick behavior pattern is labeled Q1, and the brightness setting under Q1 is... Color temperature set to The shaving behavior mode is marked with Q2, and the brightness setting under Q2 is... Color temperature set to Brushing behavior patterns are marked with Q3, and the brightness setting under Q3 is... Color temperature set to

[0122] If the user does not customize the lighting module parameters, the lighting module will turn on with the factory-set brightness and color temperature, and the touch screen will prompt whether to set them. If no setting is made after a certain period of time following the prompt, the touch screen will prompt to cancel.

[0123] In step 2, images Pm and Dm are real-time captured images, not a single image.

[0124] Step 3, "entry person's activity in front of the bathroom mirror" refers to the entry person being in a certain position and for a certain period of time using or waiting to use the bathroom mirror, excluding short-term activities such as entering or leaving the bathroom that are unrelated to the bathroom mirror.

[0125] The specific steps to determine whether an intruder is active in front of the bathroom mirror include:

[0126] Step 3.1: Calculate the distance d between the person entering and the bathroom mirror;

[0127] The openpose_hands_estimation model is used to analyze the color image Pm acquired in step 2 to determine if an entrant exists in Pm. If not, it is determined that the entrant is not active in front of the bathroom mirror. If so, the openpose_hands_estimation model obtains the entrant's key points. The number of key points is 18: key points 14-17 are facial key points, key points 0-1 are chest key points, key points 2-4 are left arm key points, key points 5-7 are right arm key points, key points 8-10 are left leg key points, and key points 11-13 are right leg key points. The color image P... m Key points are marked as O Pa Depth image D m Key points are marked as O Da a = 1, 2...18, O Pa With O Da One-to-one correspondence,

[0128] Color image P m The pixel width is set to W P Pixel height set to H eP Depth image D m The pixel width is set to W D Pixel height set to H eD W P H eP W D H eD Given parameters, known

[0129]

[0130] Establish P m Establish coordinate system A1, and set up Dm The coordinate system A2, coordinate system A1, and coordinate system A2 have corresponding origins and the same coordinate axis directions, which facilitates coordinate transformation. The key point O of P1 is taken. Pa O Pa The coordinates are (x Pa y Pa ), P m D of key point mapping m Key point O Da The coordinates are (x Da y Da ),(x Pa y Pa ) are known parameters that can be obtained. According to (1), we can obtain

[0131]

[0132] From equation (2), the coordinates (x, y) of the key point are obtained. Da y Da According to (x) Da y Da ), to obtain the depth image D m The distance value d of the corresponding target a d a Let d be the distance between the key points of the actual subject and the 3D camera depth value, taking key points 0-7 and 14-17 as values.

[0133]

[0134] Step 3.2: Determine d and the threshold distance S between the entrant and the bathroom mirror. If d is not greater than S, proceed to step 3.3. If d is greater than S, determine that the entrant is not active in front of the bathroom mirror.

[0135] Step 3.3: Determine the activity time Δt of the entrant under the condition that d is not greater than S and the activity threshold time t. If Δt is not less than t, determine that the entrant is active in front of the bathroom mirror and proceed to step 4. If Δt is less than t, determine that the entrant is not active in front of the bathroom mirror and proceed to step 2.

[0136] Mark the time when d is first detected to be no greater than S, and set it as t1. Detect P in subsequent real-time shooting. m With D m In the case of d, when d is not greater than S, the marking time is t2. If P is subsequently captured... m With D m If the detected value d is not greater than S, the current time is assigned to t2, and Δt = t2 - t1. If Δt is not less than t, it is determined that the person entering is active in front of the bathroom mirror, and step 4 is executed. If Δt is less than t, it is determined that the person entering is not active in front of the bathroom mirror, and step 2 is executed.

[0137] Step 4: Identify the entrant;

[0138] Identify color image P m The system uses facial information to compare with the user database to identify the user. If the user is identified, proceed to step 5. If the user is not identified, the person entering is determined to be a stranger, and the touchscreen prompts "Does the administrator need to add data?" If data needs to be added, the administrator collects the stranger's data. If not, proceed to step 5.

[0139] This embodiment configures corresponding behavior patterns according to different identities. For example, if the identified user is a man, the configured behavior patterns are shaving and brushing teeth; if the identified user is a woman, the configured behavior patterns are lipstick and brushing teeth; and if the identified user is a child, the configured behavior pattern is brushing teeth, in order to reduce computation and increase computation speed.

[0140] Step 6 includes the following steps:

[0141] Step 6.1: The humidity sensor detects whether the humidity of the bathroom mirror is greater than the mirror humidity threshold. If yes, it is determined that the bathroom mirror needs to be defogged and proceeds to step 6.4. If no, proceed to step 6.2.

[0142] Step 6.2: The controller starts the defogging module, and the heating resistance wire defogs the bathroom mirror surface;

[0143] Step 6.3: The humidity sensor detects the humidity of the bathroom mirror in real time. When the humidity of the mirror is less than the humidity threshold, the controller controls the defogging module to stop and proceeds to step 6.4.

[0144] Step 6.4: Calculate the sharpness index C0(f) of the current 3D camera lens, and compare the sharpness index C0(f) with the sharpness threshold C. A If C0(f) > C A If the image is clear, the process ends; otherwise, if the image is blurry, proceed to step 6.5.

[0145]

[0146] In equation (4) above, x Pa y Pa For color image P m The horizontal and vertical coordinates, f(x) P ,y P ) is at coordinate point (x Pa y Pa The corresponding grayscale value;

[0147] Under normal circumstances, when the defogging module uses a heating wire to defog the mirror surface of the bathroom mirror, the water mist on the surface of the lens will be evaporated synchronously. However, due to the requirements for the clarity of the images captured by the 3D camera, the requirements for the clarity and cleanliness of the position where the lens is located are higher, so further defogging is required.

[0148] Step 6.5: The controller controls the defogging module to start, and the heating resistance wire defogs the lens.

[0149] Step 6.6: Calculate the defogging clarity index C t (f), compare the defogging clarity index Ct(f) with the clarity threshold CA. If Ct(f)>CA, it is determined that the lens is clear, and step 6.8 is executed to end the step. If not, compare Ct(f) with C0(f). If |Ct(f)-C0(f)<Cθ, where Cθ is the clarity difference threshold, if so, it is considered that the poor clarity of the lens is caused by dirt that cannot pass through the defogging module, and it is determined that the lens is dirty, and step 6.7 is executed. If not, step 6.5 is continued to be executed.

[0150] Step 6.7: The touch screen gives a prompt, prompting the user that the lens is dirty and to deal with it, and then execute step 6.6.

[0151] Step 6.8: End the step.

[0152] The specific steps for processing the face area in the color image Pm in step 7 are as follows:

[0153] Step 7.1: Extract the face key points 14-17 on the color image Pm, calculate the minimum bounding rectangle, expand each of the length and width by n pixel points to generate the face area, and crop the face area to obtain the face image F1.

[0154] Step 7.2: Run the face_landmark_localization neural network model on the face image F1 to identify 70 key points of the face, and use two cascaded CNNs to detect the key points to identify the outer contour, eyebrow area, nose area, eye area, lip area and oral area of the face.

[0155] The 70 facial keypoints include 17 external keypoints and 53 internal keypoints. The internal keypoints are detected using a four-level cascaded network: Level-1 obtains the bounding boxes of facial organs; Level-2 outputs the predicted keypoint positions, achieving coarse keypoint localization and initializing Level-3; Level-3 performs coarse-to-fine localization based on different organs; Level-4 takes the output of Level-3 as input, rotates it, and finally outputs the positions of the 53 keypoints. The external keypoints are detected using a two-level cascaded network: Level-1 obtains the bounding boxes of the outer contour of the face; Level-2 directly predicts the positions of 17 keypoints. The 70 facial keypoints are obtained by superimposing the outputs of the two cascaded CNNs.

[0156] The outer contour region is the area formed by calculating the minimum bounding rectangle for keypoints 0-16. The eyebrow region is the left and right eyebrow regions formed by calculating the minimum bounding rectangle for keypoints 17-21 and 22-26. The nose region is the area formed by calculating the minimum bounding rectangle for keypoints 27-35. The eye region is the left and right eye socket regions formed by calculating the minimum bounding rectangle for keypoints 36-41, 68, 42-47, 69. The lip region is the area formed by calculating the minimum bounding rectangle for keypoints 48-59. The oral cavity region is the area formed by calculating the minimum bounding rectangle for keypoints 60-67.

[0157] Step 7.3: Based on the areas identified in Step 7.2, obtain the skin color area and the lip area;

[0158] The skin color area consists of the forehead area, cheek area, chin area, and upper chin area. The forehead area is the facial area above the eyebrows. The cheek area is the total area formed by the first area calculated by the minimum inscribed rectangle of key points 0-4, 36-39, 28-30, 32-31, and 49-48, and the second area calculated by the minimum inscribed rectangle of key points 16-12, 45-42, 28-30, 34-35, and 53-54. The chin area is the area formed by the minimum inscribed rectangle calculated by key points 4-12 and 48-54. The upper chin area is the area formed by the minimum inscribed rectangle calculated between key points 32-35 and 48-54.

[0159] The lip region is the area formed by subtracting the oral cavity region from the lip region.

[0160] Step 8 involves processing the color image P. m The middle hand region, specifically: extracting the color image P m For key points 4 and 7 on the upper part, calculate the minimum bounding rectangle, expand the length and width by n pixels, and generate the left hand region H1 and the right hand region H2.

[0161] The cleaning behavior in step 9 refers to actions such as washing hands and clothes, not the act of washing face and mouth.

[0162] The specific steps for determining whether a behavior is a cleaning behavior in step 9 are as follows:

[0163] Step 9.1: Calculate the distances D1 and D2 between the left hand region H1 and the right hand region H2 and the lip region, respectively;

[0164] D1 is the actual distance between the center point of the left hand region H1 and the center point of the lip region, and D2 is the actual distance between the center point of the right hand region H2 and the center point of the lip region. (The last part, "P," appears to be an incomplete sentence or fragment.) m In the image, the center point of the lip region is Z. P1 The coordinates are set as (x P1 y P1 The center point of the left-hand region H1 is Z. P2 Z P2 The coordinates are set to (x) P2 y P2 The center point of the right-hand region H2 is Z. P3 Z P3 The coordinates are set to (x) P3 y P3 ),Z P1 Z P2 and Z P3 The coordinates are known parameters, Z. P1 Z P2 and Z P3 In D m The mapping point on is Z D1 Z D2 and Z D3 Z D1 The coordinates are (x D1 y D1 ),Z D2 The coordinates are (x D2 y D2 ),Z D3 The coordinates are (x D3 y D3 According to equations (1) and (2) above, Z can be obtained. P1 Z P2 and Z P3 With Z D1 Z D2 and Z D3 Coordinate mapping relationship

[0165]

[0166]

[0167]

[0168] According to equations (5), (6), and (7) above, Z D1 Z D2 and Z D3 From the coordinates, the depth image D can be obtained. m China Z D1 Z D2 and Z D3 Distance from 3D camera depth value d D1 d D2 d D3 ;

[0169] Calculate depth image D m Compared to the magnification of the actual subject, D m Set the horizontal image magnification to M x D m Set the vertical image magnification to M y The actual physical field of view of the subject being photographed is set as W0×H0, where W0 is the horizontal width and H0 is the vertical height. Therefore, we can obtain...

[0170]

[0171]

[0172] In equation (9) above, α1 is the lateral field of view of the 3D camera, α2 is the longitudinal field of view of the 3D camera, and μ is the angle between the 3D camera and D. m The mean distance, α1, α2, and μ are known parameters. According to equations (8) and (9) above, we can obtain

[0173]

[0174]

[0175]

[0176] Establish a world coordinate system based on the actual photographed subject, Z D1 Z D2 and Z D3 Let the point corresponding to the world coordinate system be Z. S1 Z S2 and Z S3 According to equations (11), (12), and (13) above, Z D1 Z D2 and Z D3 The coordinates are transformed into Z. S1 Z S2 and Z S3 Coordinates in the world coordinate system, Z S1 The coordinates are (W x1 Wy1 W z1 Z S2 The coordinates are (W x2 W y2 W z2 Z S3 The coordinates are (W x3 W y3 W z3 ); we can obtain

[0177]

[0178]

[0179] Step 9.2: Calculate the overlap area and intersection-union ratio between the left-hand region H1 and the right-hand region H2 and the lip region, respectively;

[0180] The area of ​​the lip region is set to S. Z Let the area of ​​the left-hand region H1 be S. H1 The overlap area between the left hand region H1 and the lip region is set as S. i1 Let the intersection-union ratio of the left-hand region H1 be IoU1, and let the area of ​​the right-hand region H2 be S. H2 Let the overlap area between the right-hand region H2 and the lip region be S. i2 Let the intersection-union ratio of the right-hand region H2 be IoU2, then we can obtain

[0181]

[0182]

[0183] In equations (15) and (16) above, SZ, SH1, Si1, SH2 and Si2 are all known parameters. According to the Pm image, SZ, SH1, Si1 and SH2 are all areas in the Pm image.

[0184] Step 9.3: Analyze the behavior:

[0185] If both D1 and D2 are greater than D0, and Si1 = 0 and Si2 = 0, then the user is determined to be in a cleaning behavior.

[0186] If D1 or / and D2 are greater than Ds but not greater than D0, and IoU1 and IoU2 are greater than 0, then the analysis continues.

[0187] If D1 or / and D2 are less than Ds, and IoU1 and IoU2 are greater than IoU0, the user is determined to be in a grooming behavior.

[0188] D0 is the maximum distance threshold between the hand and mouth areas. At this distance, it is considered that there is no interaction between the hand and mouth. D0 is a set value.

[0189] Ds is the minimum distance threshold between the hand and mouth areas. At this distance, interaction between the hand and mouth is considered to exist. Ds is a set value.

[0190] IoU0 is the threshold for the cross-union ratio.

[0191] The specific steps of the amplification process in step 11 are as follows:

[0192] Step 11.1: The user draws a circle with their hand over the area of ​​the face that needs to be enlarged;

[0193] Step 11.2: Extract the trajectory of the hand making circles;

[0194] Acquire a real-time color image Pt of the hand drawing circles from start to finish. Calculate the minimum bounding rectangle, expanding its length and width by n pixels to generate the face region. Crop the face region to obtain the face image Ft.

[0195] Extract key points 4 and 7 of the hand from the color image Pt, calculate the minimum bounding rectangle, expand the length and width by n pixels, and generate the left hand region H1t and the right hand region H2t. Use a convolutional pose machine (CPM) to detect the left hand region H1t and the right hand region H2t, extract the key points of the fingers, obtain the points where the key points of the face image Ft overlap with the key points of the fingers, and form the hand circling trajectory from several points.

[0196] Step 11.3: Fit the smallest horizontal outer rectangle to the trajectory of the hand drawing circles to form a trajectory image. Crop the trajectory image from the face image Ft and perform secondary interpolation on the trajectory image data to increase the resolution and generate a magnified image.

[0197] Step 11.4: The touchscreen displays a magnified image.

[0198] In step 12, the object recognition model adopts a high-precision Anchor-Free object detection algorithm with ESNet (Enhanced ShuffleNet) as the backbone network and PAN (Path Aggregation Structure) structure. This is used to obtain multi-level feature maps, and CSP cross-stage network is used to connect and fuse features between adjacent feature maps.

[0199] Anchor-Free object detection algorithm performs deep learning on toothbrushes, lipsticks, and razors, setting the label for toothbrush as 1, lipstick as 2, and razor as 3;

[0200] Lightweight pico-det is run in the left-hand region H1 and the right-hand region H2. pico-det adds a feature map scale on top of the CSP-PAN to detect more targets.

[0201] In step 12, the behavioral pattern associated with the identified identity is first obtained based on the identity identified in step 4, and then the item is detected.

[0202] The brushing behavior mode in step 12 is as follows: the lighting module adjusts the light brightness to... Color temperature adjusted to Deep learning is used to identify the brushing area and determine whether the brushing posture is correct based on the toothbrush angle, brushing time, and brushing effect after brushing, such as cleanliness.

[0203] Applying lipstick is the final step in makeup. Lipstick color testing is performed in lipstick behavior mode to check the saturation of the lipstick after application and to check the evenness of the makeup.

[0204] Step 12, executing the lipstick behavior pattern, includes the following steps:

[0205] Step S1.1: The lighting module adjusts the light brightness to... Color temperature adjusted to Determine if the makeup is even in the skin tone area. If yes, proceed to step S1.2; if no, the touchscreen will prompt the user that the makeup is uneven and display the uneven area on the touchscreen. If the user decides to redo the makeup, they can click "Yes" on the touchscreen to re-evaluate the evenness of the adjusted makeup. If the user decides that redoing the makeup is not necessary, they can click "No" on the touchscreen and proceed to step S1.2.

[0206] Step S1.2: Determine whether to perform lipstick swatching. If yes, proceed to step S1.3; otherwise, proceed to step S1.5.

[0207] Step S1.3: Extract the lip region from the face image F2, record the lip color without lipstick, establish an HSV model, and set the values ​​of the three components of the HSV model: hue H′, saturation S′, and lightness V′; the user adjusts the values ​​of the components by touching the touch screen; the matching color number of the three components is displayed in the lip region; if the user confirms the selection of the color number, the color number is recorded as the target color number, and the three matching HSV components at this moment are stored;

[0208] Step S1.4: After applying the lipstick, check the evenness of the lipstick on the lip area after testing the color.

[0209] Convert the three HSV components (hue H′, saturation S′, and lightness V′) stored in step S1.3 into RGB.

[0210] (R,G,B)=((R'+q)×255+(G'+q)×255+(B'+q)×255); (17)

[0211]

[0212] In equation (18) above, M, N, q, and (R', G', B') are all variables, and M = S' × V'. q = V′ - M;

[0213] Converting RGB to XYZ tristimulus values ​​yields equation (19):

[0214]

[0215] Converting the XYZ tristimulus values ​​to LAB values ​​yields equation (20):

[0216]

[0217] In equation (20) above, XYZ are the tristimulus values ​​of the transformation, X0Y0Z0 are the tristimulus values ​​of the light source, and X0Y0Z0 are known parameters; (L z A z B z The LAB value is the result of converting the three HSV components (hue H', saturation S', and lightness V') stored in step S1.3, which is the target LAB value.

[0218] Obtain the face image after color testing (F3) and extract the lip area image;

[0219] Transform the lip region image to the LAB color space; a set of (L, A, B) in the LAB color space forms a one-to-one correspondence with a color, where L is the brightness, and A and B are the green-red and blue-yellow hues of the color, respectively.

[0220] Take the pixel G on the lip region image c G c Represented as (L) c A c B c ), (L c A c B c ) represents a known parameter, c is the index of any pixel in the image, and the value of c is no greater than the total number of pixels in the image. Pixel G c The color difference from the target LAB value is:

[0221] ΔE c =[(L c -L z ) 2 +(A c -A z ) 2 +(B c -B z )2] 1 / 2 ; (twenty one)

[0222] ΔEc Color difference threshold E (standard uniformity) M In comparison, if ΔE c Not less than E M If the uniformity is poor, the touchscreen will display which area of ​​the lip area is not evenly applied. If ΔE c Less than E M To determine if the lipstick is applied evenly.

[0223] Step S1.5: After applying the lipstick, check the evenness of the lipstick on the lip area;

[0224] The detection steps in this step are the same as those in step S1.1 for determining the evenness of makeup application in skin-tone areas, and will not be repeated here.

[0225] Step S1.6: End of step.

[0226] The step in S1.1 to determine the evenness of makeup application in skin-tone areas is as follows:

[0227] Step S1.1.1: Obtain the color image P at the current time. m Extract color image P m For key facial points 14-17, calculate the minimum bounding rectangle, expand the length and width by n pixels to generate the facial region. Crop the facial region to obtain facial image F2. Image F2 is the user's face image with makeup on but without lipstick.

[0228] Step S1.1.2: Establish the digital image processing model LAB;

[0229] Step S1.1.3: Calculate the non-uniformity rate E c ;

[0230] Transform the face image F2 to the LAB color space; a set of (L, A, B) in the LAB color space forms a one-to-one correspondence with a color, where L is the brightness, and A and B are the green-red and blue-yellow hues of the color, respectively.

[0231] Take pixel G in the skin color region of image F2. l and G j G l Represented as (L) l A l B l ), G j Represented as (L) j A j B j ), (L l A l B l ) and (L j A j Bj Let f(x) be a known parameter, and l and j be the numbers of any pixel in image F2. l and j are less than the total number of pixels g in image F2, and l ≠ j. The color difference between two pixels is:

[0232] ΔE lj =[(ΔL) 2 +(ΔA) 2 +(ΔB) 2 ] 1 / 2 ; (twenty two)

[0233] In equation (22) above, ΔE lj For pixel G l and G j The color difference, ΔL is the pixel G l and G j The brightness difference, ΔA and ΔB are the pixel points G. l and G j The color coordinate differences, ΔL, ΔA, and ΔB, can be obtained according to equation (23);

[0234]

[0235] ΔE is the set of color differences between any two pixels in image F2, ΔE = {ΔE} 12 ...ΔE lj ...ΔE (g-1)g}, count the number of color differences in ΔE, denoted as E1; count the number of color differences in ΔE greater than the uniformity color difference threshold ΔE0, denoted as E2; calculate the non-uniformity rate E. c ,

[0236] Step S1.1.4: Compare E c With standard non-uniformity E v If E c Greater than E v If the skin tone is uneven, determine if the skin tone is even; otherwise, determine if the skin tone is even. v This is the set value.

[0237] The specific steps of the shaving behavior pattern in step 12 are as follows:

[0238] Step S2.1: The lighting module adjusts the light brightness to... Color temperature adjusted to

[0239] Step S2.2: Obtain the color image P at the current time. m Extract color image P mFor key points 14-17 on the upper face, calculate the minimum bounding rectangle, expand the length and width by n pixels to generate the face region. Crop the face region to obtain face image F3. Obtain the chin region image between the upper lip and nose and the chin region image between the lower lip and lower contour of F3.

[0240] Step S2.3: Convert the upper chin region image and the lower chin region image into grayscale image g1. Use a 5x5 Gaussian filter to process the upper chin region image and the lower chin region image to obtain image m1. Subtract image m1 from grayscale image g1 to obtain image c1.

[0241] Step S2.4: Set the threshold t θ Using threshold t θ Binarize image c1 and extract the gray values ​​in image c1 that are less than the threshold t. θ The extracted regions are merged into one region, and the position, size, and number of each black dot (beard is a black dot) are calculated.

[0242] Step S2.5: After shaving with the razor, obtain the current color image Pt. Using steps S2.2-S2.4, obtain the position and size of each black dot in the current image, as well as the number of black dots.

[0243] Step S2.6: Compare the position, size, and number of black dots in Step S2.4 with the position, size, and number of black dots in Step S2.5. Count the number of black dots that disappear at the same position. If the black dot area shrinkage rate is greater than 90%, it is determined that the shave is clean. If the black dot area shrinkage rate is not greater than 90%, it is determined that the shave is not clean. The touch screen will display the location of the shave that is not clean. If the user decides to shave again, click "Yes" on the touch screen to execute Step S2.5 and re-evaluate. If the user decides that shaving is not necessary, click "No" on the touch screen.

[0244] The steps of the lipstick behavior pattern and shaving behavior pattern involve a magnification process. The magnification process involves cropping the merged area of ​​the face and hands, fitting the minimum horizontal outer rectangle of the area to form a region image, performing secondary interpolation on the data of the region image to increase the resolution, and generating a magnified image of the area.

[0245] The items specified in this embodiment are toothbrushes, lipsticks, and razors, but are not limited to these. The types of items can be increased or decreased according to actual needs.

[0246] Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.

Claims

1. A control method for an intelligent bathroom mirror, characterized in that: This method is used to control a smart bathroom mirror and includes the following steps: Step 1: The administrator collects user data and stores it in the user database. The data includes height, facial features, and facial contour data; the administrator sets the lighting module parameters. Step 2: Real-time monitoring with a 3D camera; the controller controls the 3D camera to acquire color images in front of the bathroom mirror in real time. With depth images ; Step 3: When someone enters the bathroom, determine whether the person is in front of the bathroom mirror. If yes, proceed to step 4; otherwise, proceed to step 2. Step 4: Identify the entrant; Step 5: Obtain the current time. The lighting module determines the time period based on the current time. Adjust the brightness to Color temperature adjusted to ; Step 6: Determine whether the bathroom mirror and the 3D camera lens need defogging. If yes, execute the automatic defogging process and then proceed to Step 7. If no, proceed to Step 7. Step 6, the automatic defogging process, includes the following steps: Step 6.1: The humidity sensor detects whether the humidity of the bathroom mirror is greater than the mirror humidity threshold. If yes, it is determined that the bathroom mirror needs to be defogged and proceeds to step 6.

4. If no, proceed to step 6.

2. Step 6.2: The controller starts the defogging module, and the heating resistance wire defogs the bathroom mirror surface; Step 6.3: The humidity sensor detects the humidity of the bathroom mirror in real time. When the humidity of the mirror is less than the humidity threshold, the controller controls the defogging module to stop and proceeds to step 6.

4. Step 6.4: Calculate the sharpness index of the current 3D camera lens. Comparison of sharpness indicators With sharpness threshold ,like > If the image is clear, the process ends; otherwise, if the image is blurry, proceed to step 6.

5. In equation (4) above, For color images The horizontal and vertical coordinates, For at coordinate point The corresponding grayscale value; Step 6.5: The controller activates the defogging module, heating the resistance wire to defog the lens; Step 6.6: Calculate the lens's defogging sharpness index Comparison of defogging clarity index With sharpness threshold ,like > If the image is clear, proceed to step 6.8 and end the process. If not, compare... and ,like < , If the sharpness difference threshold is set, it is assumed that the poor sharpness of the lens is caused by dirt that cannot pass through the defogging module, and the lens is determined to be dirty, proceeding to step 6.7; otherwise, continue to step 6.

5. Step 6.7: The touchscreen prompts the user to clean the lens and proceed to step 6.6; Step 6.8: End of steps; Step 7: Process the color image Get facial region ; Step 8: Process the color image Get the left-hand region and right-hand area ; Step 9: Analyze the behavior to determine if it is a cleaning behavior. If yes, proceed to step 2 after the cleaning behavior is completed. If no, proceed to step 10. The specific steps for determining whether a behavior is a cleaning behavior in step 9 are as follows: Step 9.1: Calculate the left-hand region and right-hand area Distance from the lip area and ; The left hand area of ​​the human body The actual distance between the center point and the center point of the lip area. Right hand region of the human body The actual distance between the center point and the center point of the lip area is... In the image, the center point of the lip area is The coordinates are set to Left-hand area The center point is , Coordinates are set to Right-hand area The center point is , Coordinates are set to , , as well as The coordinates are known parameters. , as well as exist The mapping point on is , as well as , The coordinates are , The coordinates are , The coordinates are According to equations (1) and (2) above, we can obtain , as well as and , as well as The mapping relationship of coordinates; ； ； ； According to equations (5), (6), and (7) above... , as well as From the coordinates, the depth image can be obtained. middle , as well as Distance to 3D camera depth value , , ; Calculate depth image Compared to the magnification of the actual subject, Horizontal image magnification set to , Vertical image magnification set to The physical field of view of the actual subject being photographed is set to , Horizontal width Vertical height, can be obtained ； ； In the above formula (9), The lateral field of view of the 3D camera. The longitudinal field of view of the 3D camera. For 3D cameras and The mean distance, Given the parameters, according to equations (8) and (9) above, we can obtain ； ； ； Establish a world coordinate system based on the actual subjects being photographed. , as well as Set the point corresponding to the world coordinate system as , as well as According to equations (11), (12), and (13) above, , as well as The coordinates are transformed , as well as Coordinates in the world coordinate system The coordinates are , The coordinates are , The coordinates are ; can be obtained ； ； Step 9.2: Calculate the left-hand region and right-hand area The area of ​​overlap with the lip region and the intersection-union ratio, respectively; The area of ​​the lip region is set to Left hand area The area is set as Left-hand area The overlap area with the lip area is set to Left-hand area The intersection-union ratio is set as Right-hand area The area is set as Right-hand area The overlap area with the lip area is set to right-hand area The intersection-union ratio is set as , can be obtained ； ； In equations (15) and (16) above, , , , as well as All parameters are known, according to The image shows that , , , All Area in the image; Step 9.3: Analyze the behavior: like , All greater than , , The system determines that the user is engaged in cleaning activities. like or / and Greater than and not greater than , and If the value is greater than 0, further analysis is required. like or / and Less than , and Greater than The system determines that the user is performing grooming or washing activities. This is the maximum distance threshold between the hand and mouth areas; at this distance, no interaction between the hand and mouth is considered. Set value; This is the minimum distance threshold between the hand and mouth areas; at this distance, interaction between the hand and mouth is considered to exist. Set value; The threshold for the crossover-union ratio; Step 10: Determine if the hand is holding an object. If yes, proceed to Step 12 to enter the object recognition process; otherwise, proceed to Step 11. Step 11: Determine if a certain area needs to be enlarged. If yes, execute the enlargement process; otherwise, execute step 13. Step 12: The object recognition model identifies the object held by the hand. If the object recognition model outputs 1, the object is determined to be a toothbrush, and the brushing behavior mode is entered; if the object recognition model outputs 2, the object is determined to be a lipstick, and the lipstick behavior mode is entered; if the object recognition model outputs 3, the object is determined to be a razor, and the shaving behavior mode is entered. Step 13: Determine if anyone is in the bathroom. If yes, proceed to step 2; otherwise, turn off the lighting module and end the process.

2. The control method for an intelligent bathroom mirror according to claim 1, characterized in that: In step 1, the user-defined lighting module parameters are the brightness and color temperature of the lights for different time periods and different behavior modes; the controller controls the lighting module to turn on and off, and turns on the brightness and color temperature of the corresponding time period according to the current time, and then converts the brightness and color temperature into the brightness and color temperature set for the corresponding behavior mode according to the current behavior mode.

3. The control method for an intelligent bathroom mirror according to claim 1, characterized in that: The specific steps for determining whether the person entering is active in front of the bathroom mirror in step 3 are as follows: Step 3.1: Calculate the distance between the person entering and the bathroom mirror. ; Step 3.2: Judgment Threshold distance between the entrant and the bathroom mirror ,like Not greater than Execute step 3.3, if Greater than It was determined that the person entering was not moving in front of the bathroom mirror; Step 3.3: Determine if the entrant meets the following conditions. Not greater than Activity time under the conditions With activity threshold time ,like Not less than If it is determined that the person entering is active in front of the bathroom mirror, proceed to step 4. Less than If it is determined that the person entering is not moving in front of the bathroom mirror, proceed to step 2.

4. The control method for an intelligent bathroom mirror according to claim 1, characterized in that: Step 4, identifying the entrant's identity, specifically involves: identifying a color image. The system uses facial information to compare with the user database to identify the user. If the user is identified, proceed to step 5. If the user is not identified, the person entering is determined to be a stranger, and the touchscreen prompts "Does the administrator need to add data?" If data needs to be added, the administrator collects the stranger's data. If not, proceed to step 5.

5. The control method for an intelligent bathroom mirror according to claim 1, characterized in that: Step 7 involves processing the color image. The mid-face area specifically includes the following steps: Step 7.1: Extract the color image For key facial points 14-17, calculate the minimum bounding rectangle, expanding its length and width by n pixels to generate the facial region. Then, crop the facial region to obtain the facial image. : Step 7.2: Process the face image The face_landmark_localization neural network model was run to identify 70 key points on a face. Two cascaded CNNs were used to detect the key points, identifying the outer contour of the face, the eyebrow region, the nose region, the eye region, the lip region, and the mouth region. Step 7.3: Based on the area identified in Step 7.2, obtain the skin color area and the lip area.

6. The control method for an intelligent bathroom mirror according to claim 1, characterized in that: Step 8 involves processing the color image. The middle hand region specifically involves: extracting the color image. For key points 4 and 7 on the upper hand, calculate the minimum bounding rectangle, expand its length and width by n pixels, and generate the left hand region. and right-hand area .

7. The control method for an intelligent bathroom mirror according to claim 3, characterized in that: In step 3.1, the distance The calculation formula is: Color images Key points are marked as Depth image Key points are marked as , , and One-to-one correspondence; Color images The pixel width is set to Pixel height set to Depth image The pixel width is set to Pixel height set to , , , , Given the parameters, we get ； Take P1 key point , The coordinates are , Key point mapping Key points The coordinates are , Given the available known parameters, according to (1), we get ； The coordinates of the key points are obtained from equation (2). ,according to To obtain a depth image Distance value of the corresponding target , The distance between the key points of the actual photographed object and the depth values ​​of the 3D camera is taken from key points 0-7 and 14-17. 。 8. The control method for an intelligent bathroom mirror according to claim 1, characterized in that: The specific steps of the amplification process in step 11 are as follows: Step 11.1: The user draws a circle with their hand over the area of ​​the face that needs to be enlarged; Step 11.2: Extract the trajectory of the hand making circles; Acquire real-time color images of the hand making a circular motion from start to finish. Calculate the minimum bounding rectangle, expand its length and width by n pixels to generate the face region, and then crop the face region to obtain the face image. Extract color images For key points 4 and 7 on the upper hand, calculate the minimum bounding rectangle, expand its length and width by n pixels, and generate the left hand region. and right-hand area The left-hand region was detected using a convolutional attitude machine (CPM). and right-hand area Extract key points of the fingers to obtain facial images. Several points that overlap with the key points of the fingers form the circular trajectory of the hand; Step 11.3: Fit the smallest horizontal outer rectangle to the extracted hand-circling trajectory to form a trajectory image. Then, extract the trajectory image from the face image. The data of the trajectory image is cropped and then subjected to secondary interpolation to increase the resolution and generate a magnified image. Step 11.4: The touchscreen displays a magnified image.

Images