A tooth brushing detection method, detection device and bathroom mirror

Abstract

The application discloses a tooth brushing detection method, a detection device and a bathroom mirror, and comprises the following steps: 1, an administrator collects toothbrush parameters and user data, and stores the data in a user database; 2, a three-dimensional camera is used for real-time monitoring, and a controller controls the three-dimensional camera to collect color images Pm and depth images Dm of a user in real time; 3, it is judged whether the images have a face, if yes, step 3 is executed, and if no, step 2 is executed, 1) the application monitors the tooth brushing behavior of a user, detects the position, duration and frequency of tooth brushing of the user, realizes standard detection of tooth brushing, and is beneficial to the user to form a standard tooth brushing habit and ensure tooth health; the application applies artificial intelligence technologies such as human body key point and identity recognition algorithm, high-precision Anchor-Free target detection algorithm and behavior analysis to standard tooth brushing detection, and realizes automation and intelligentization of detection.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of detection, and relates to a tooth brushing detection method, a detection device and a bathroom mirror. BACKGROUND

[0002] Tooth brushing is an important measure for protecting teeth, and an incorrect tooth brushing method not only cannot clean teeth, but also may damage teeth,

[0003] A correct tooth brushing method is to control a tooth brushing posture and a tooth brushing time length of different parts, and in daily tooth brushing, a self tooth brushing standardization detection cannot be realized due to a lack of a detection means, especially for children, due to a lack of an effective detection means for standardizing a tooth brushing method, not only teeth are damaged, but also a correct tooth brushing habit is not formed. SUMMARY

[0004] In order to overcome the defects of the prior art, the present application provides a tooth brushing detection method, a detection device and a bathroom mirror.

[0005] In order to achieve the above-mentioned purpose, the present application adopts the following technical scheme: a tooth brushing detection method, comprising the following steps:

[0006] Step 1: an administrator collects toothbrush parameters and user data, and stores the data in a user database;

[0007] Step 2: a three-dimensional camera is monitored in real time, and a controller controls the three-dimensional camera to collect color images of a user in real time and depth images ;

[0008] Step 3: whether the images have a face is judged, if yes, step 3 is executed, and if no, step 2 is executed;

[0009] Step 4: an identity of the user and a toothbrush are recognized;

[0010] Step 5: the controller processes the color images , obtains a lip region, and obtains an area of the lip region;

[0011] Step 6: the controller processes the color images , obtains a toothbrush region and a hand region;

[0012] Step 7: whether tooth brushing is started is judged, if yes, step 8 is executed, and if no, step 2 is executed;

[0013] Step 8: whether the user brushes teeth with the mouth open is judged, if yes, a mouth open tooth brushing process is executed, and if no, a mouth closed tooth brushing process is executed;

[0014] Step 9: the mouth open tooth brushing process is executed;

[0015] Step 10: closed mouth brushing procedure, proceed to step 11;

[0016] Step 11: detect whiteness of teeth after brushing;

[0017] Step 12: initial screening of dental disease;

[0018] Step 13: end procedure.

[0019] Further, the step of collecting toothbrush parameters in step 1 is:

[0020] Step 1.1: three-dimensional camera collects front color image, front depth image, back color image and back depth image of toothbrush;

[0021] Step 1.2: distortion correction and denoising enhancement processing are performed on the front color image to obtain a first image, and distortion correction and denoising enhancement processing are performed on the back color image of the toothbrush to obtain a second image;

[0022] Step 1.3: set threshold , use threshold to binarize and extract the first image and the second image, extract the area with a gray value less than the threshold in the first image and the second image, and form a first binarized image and a second binarized image;

[0023] Step 1.4: calculate total length of toothbrush , store to database:

[0024] Obtain the coordinates of the toothbrush head and handle of the first binarized image or the second binarized image, denoted as and , the coordinates are known parameters,

[0025]

[0026] In formula (1) is the magnification of the color image collected by the three-dimensional camera in step 1;

[0027] Step 1.5: set the midpoint of the toothbrush head as the center point of the toothbrush , calculate and the length of the toothbrush head , and the length of the toothbrush handle , store , , to database;

[0028] Step 1.6: Obtain the depth value of each pixel point according to the front depth image and the back depth image, obtain the toothbrush front feature parameters and the toothbrush back feature parameters; store the toothbrush front feature parameters and the toothbrush back feature parameters to the database;

[0029] The front feature parameters and the toothbrush back feature parameters are the brush head front parameters, the brush head back parameters, the brush handle front curved surface parameters, and the brush handle back curved surface parameters. The front of the brush head has bristles. The brush head front parameters or the brush head back parameters recognized by matching are matched with the front feature parameters and the toothbrush back feature parameters in the database. By matching with the toothbrush front feature parameters and the toothbrush back feature parameters in the database, it is determined that the current toothbrush is the front or the back.

[0030] Further, the face in step 3 is determined as:

[0031] The controller uses the openpose_hands_estimation model to process the color image to obtain the key points of the user's body, and determines whether the obtained key points of the body include face key points 14-17, left hand key points 4 and right hand key points 7. If yes, it is determined that there is a face in the image, and step 4 is executed; if not, it is determined that there is no face in the image.

[0032] Further, the color image in step 5 is processed to obtain the lip region, specifically:

[0033] Step 5.1: Extract the face key points 14-17, calculate the minimum bounding rectangle, and dilate the length and width by a certain number of pixel points to generate a face region, and crop the face region to obtain a face image ;

[0034] Step 5.2: Run the face_landmark_localization neural network model on the face image to identify the face key points, and use two cascaded CNNs to detect the key points to identify the lip region and the oral cavity region of the face;

[0035] The lip region is the region formed by calculating the minimum bounding rectangle of the key points 48-59; the oral cavity region is the region formed by calculating the minimum bounding rectangle of the key points 60-67;

[0036] Step 5.3: Obtain the lip region according to the lip region and the oral cavity region identified in step 5.2;

[0037] The lip region is the region formed by subtracting the oral cavity region from the lip region;

[0038] Step 5.4: Obtain the area of the lip region ;

[0039] The specific steps of step 6 are as follows:

[0040] Step 6.1: Extract hand key points 4 and 7, calculate the minimum circumscribed rectangle, and expand the length and width by several pixels to generate the right hand region ;

[0041] Step 6.2: Perform image processing on the right hand region to obtain the toothbrush region and the hand region, and obtain the area sum of the toothbrush region and the hand region, and the area sum is denoted as .

[0042] Further, the specific steps of step 7 are as follows:

[0043] Step 7.1: Extract the color image at the current time , and perform processing on the color image to obtain the contour line of the lip region, the toothbrush region and the hand region, calculate the minimum circumscribed rectangle, and generate the action region;

[0044] Step 7.2: Obtain the area of the action region ;

[0045] Step 7.3: Calculate the overlapping area ;

[0046] ;

[0047] Step 7.4: Calculate the distance between the center point of the lip region and the center point of the toothbrush ;

[0048]

[0049] In the above formula (3), is the center point coordinate of the lip region in the color image, is the center point coordinate of the toothbrush in the color image;

[0050] Step 7.3: Determine whether is greater than the overlapping area threshold, is less than the center distance threshold, if so, it is determined that tooth brushing has started, if is not greater than the overlapping area threshold or / and is not less than the center distance threshold, it is determined that tooth brushing has not started.

[0051] Further, the specific steps of step 8 for determining whether the user is brushing teeth with mouth open are as follows:

[0052] Step 8.1: Obtain and process the color image at the current time point , the minimum circumscribed rectangle is calculated based on the key points 48-59 and 60-67, forming a lip region;

[0053] Step 8.2: Calculate the distance between the key points 62 and 66 ;

[0054]

[0055] In the above formula (4), is a color image The distance between the upper lip region key points 62 and 66 is calculated.

[0056] Step 8.3: Determine the key point distance threshold , if is greater than , it is determined that the user is brushing teeth with mouth open, if is not greater than , it is determined that the user is brushing teeth with mouth closed.

[0057] Further, the step 9 mouth open brushing process includes the following steps:

[0058] Step 9.1: Obtain a color image including a brush head during brushing and a depth image , obtain images of the oral cavity region, lip region and toothbrush region through image processing ; process the oral cavity region image using a triangular mesh model, segment the teeth, gums and tongue, and generate the tooth region , gum region and tongue region , establish a coordinate system with the center of the lip region as the coordinate origin;

[0059] Extract the coordinates of the center point of the lip region , the coordinates of the toothbrush center point of the brush head position ,

[0060] Step 9.2: Calculate the tilt angle , preliminarily confirm the brushing position according to ;

[0061]

[0062] The brushing position is set to , is the number of brushing positions, the brushing positions include the inner and outer walls of the left upper molar, the inner and outer walls of the left lower molar, the inner and outer walls of the upper and lower incisors, the inner and outer walls of the right lower molar, the inner and outer walls of the right upper molar and the tongue, a total of 13 brushing positions, , set the brushing position set , ,

[0063] In the process of brushing teeth, the tilt angle is detected in real time, not the set value,

[0064] Step 9.3: accurately confirm the position of brushing teeth;

[0065] Step 9.4: calculate the real-time tilt angle If , it is determined that the current brushing position is unchanged, otherwise, it is determined that the brushing position has changed.

[0066] Set the set of brushed teeth positions , store the brushed teeth positions to the set ;

[0067] Step 9.5: record the brushing time of the user in and the total brushing time ;

[0068]

[0069] Step 9.6: extract the coordinates of the contact point of the toothbrush and the lip area in the image , set the contact point of the toothbrush when brushing into the lip area as , , set the coordinates of , set the coordinates of the end point of the toothbrush handle as , , set the coordinates of , the mapping point of the contact point in the depth image is , , set the coordinates of , the mapping point of the end point of the toothbrush handle is , , set the coordinates of , and are known parameters,

[0070]

[0071]

[0072] In the above formulas (7) and (8), the pixel width of the color image is set as , the pixel height is set as , and the depth image Set the pixel width to Pixel height set to , , , , The parameters are known.

[0073] Depth image according to The coordinates are set to as well as The coordinates are set to ,get depth value , depth value ,

[0074] Depth image The horizontal image magnification relative to the actual subject is set to , Vertical image magnification set to The physical field of view of the actual subject being photographed is set to , Horizontal width Vertical height Set the pixel width to Pixel height set to ,have to

[0075]

[0076]

[0077] In the above formula (10), The lateral field of view of the 3D camera. The longitudinal field of view of the 3D camera. For 3D cameras and The mean distance, Given the parameters, according to equations (9) and (10) above, we can obtain

[0078]

[0079]

[0080] Establish a world coordinate system based on the actual subjects being photographed. , Set the point corresponding to the world coordinate system as , , The coordinates are , The coordinates are ,have to

[0081]

[0082] In the above formula (13), L is the shortest length of the outer part of the actual toothbrush when brushing the lip area, and similarly, L is the longest length of the outer part of the actual toothbrush when brushing the lip area, and the recognition is once , is the longest length of the outer part of the actual toothbrush when brushing the lip area, and the recognition is once and once is recorded as a tooth brushing time. In a tooth brushing time , the number of tooth brushing times is recorded as ;

[0083] that is, when the tooth brushing position is , the number of tooth brushing times is in the tooth brushing time ;

[0084] Step 9.7: Compare the set with the set . If the set is missing elements, the touch screen prompts the user to brush the position; if the elements are the same, compare the recorded , with the set standard value. If , reaches the standard value, it is determined that the user brushes the teeth regularly, otherwise, the touch screen displays a prompt that the user does not brush the teeth regularly.

[0085] A tooth brushing detection device for implementing the tooth brushing detection method, comprising a controller and a three-dimensional camera, a touch screen, a communication module, a data acquisition module, a face recognition module, a data recognition module, an image processing module, a tooth brushing behavior module, a tooth brushing module, a detection module, a disease detection module connected with the controller. The controller controls the three-dimensional camera to synchronously shoot color images and depth images. The controller receives and processes the color images and the depth images, and transmits the color images and the depth images to the touch screen for display. The touch screen receives user operation input for command response. The controller controls the communication module to upload and download user data.

[0086] Further, the data acquisition module is used to acquire toothbrush parameters and user data, and store the data in a user database.

[0087] The face recognition module is used to identify whether there is a face in the photographed image.

[0088] The data recognition module is used to identify the identity of the user and the toothbrush.

[0089] The image processing module is used to process the color images and the depth images, and obtain corresponding regions.​

[0090] Brushing behavior module: for judging whether the user starts brushing and the mode of brushing;

[0091] Brushing module: for monitoring the position, duration and frequency of the user's brushing;

[0092] Detection module: for detecting the whiteness of the teeth after brushing;

[0093] Disease detection module: for preliminary screening of dental diseases.

[0094] A bathroom mirror comprising the tooth brushing detection device.

[0095] In summary, the present application has the following advantages:

[0096] The present application monitors the user's brushing behavior, detects the position, duration and frequency of the user's brushing, and realizes the standardization detection of brushing, which is beneficial to the user to develop the habit of standard brushing and ensure the health of the teeth.

[0097] The present application applies human key point and identity recognition algorithm, high-precision Anchor-Free target detection algorithm, behavior analysis and other artificial intelligence technologies to the standardization detection of brushing, realizes the automation and intelligentization of detection.

[0098] The present application applies the whiteness detection of brushing and the preliminary screening of dental diseases to the standardization detection of brushing, which enriches the function. BRIEF DESCRIPTION OF DRAWINGS

[0099] Figure 1 The flowchart of the tooth brushing detection method of the present application.

[0100] Figure 2 The tooth brushing flowchart of the present application.

[0101] Figure 3 The schematic diagram of the toothbrush image collected by the three-dimensional camera of the present application.

[0102] Figure 4 The schematic diagram of the toothbrush image collected by the three-dimensional camera of the present application. DETAILED DESCRIPTION

[0103] The embodiments of the present application will be described below through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the present application from the disclosure of the present specification. The present application can also be implemented or applied through other different specific embodiments, and each detail in the present specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following examples and features in the examples can be combined with each other without conflict.

[0104] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0105] In this embodiment of the invention, all directional indicators (such as up, down, left, right, front, back, lateral, longitudinal, etc.) are only used to explain the relative positional relationship and movement of each component in a specific posture. If the specific posture changes, the directional indicator will also change accordingly.

[0106] Due to installation errors and other reasons, the parallel relationship referred to in the embodiments of the present invention may actually be an approximate parallel relationship, and the perpendicular relationship may actually be an approximate perpendicular relationship.

[0107] Example 1:

[0108] like Figures 1-4 As shown, a tooth brushing test method includes the following steps:

[0109] Step 1: The administrator collects toothbrush parameters and user data, and stores the data in the user database. The data includes facial features and facial contour data.

[0110] Step 2: Real-time monitoring of the 3D camera; the controller controls the 3D camera to acquire the user's color images in real time. With depth images ;

[0111] Color images With depth images The images are detected in real time, not a single image;

[0112] Step 3: Determine if there is a human face in the image. If yes, proceed to step 3; otherwise, proceed to step 2.

[0113] Step 4: Identify the user and toothbrush;

[0114] Step 5: The controller processes the color image. Obtain the lip region, and the area of ​​the lip region is denoted as . ;

[0115] Step 6: The controller processes the color image. Get the toothbrush area and hand area;

[0116] Step 7: Determine if you want to start brushing your teeth. If yes, proceed to step 8; otherwise, proceed to step 2.

[0117] Step 8: judging whether the user is brushing teeth with mouth open, if yes, executing step 9 of mouth open brushing teeth process; if no, executing step 10 of mouth closed brushing teeth process;

[0118] Step 9: mouth open brushing teeth process;

[0119] Step 10: mouth closed brushing teeth process;

[0120] Step 11: detecting the whiteness of teeth after brushing;

[0121] Step 12: preliminary screening of dental diseases;

[0122] Step 13: ending step.

[0123] The step of collecting toothbrush parameters in step 1 is:

[0124] Step 1.1: collecting front color image, front depth image, back color image and back depth image of toothbrush by three-dimensional camera;

[0125] The front color image and the front depth image are images taken by three-dimensional camera with bristles facing the camera as shown in Figure 3 The back color image and the back depth image are images taken by three-dimensional camera with bristles facing away from the camera as shown in Figure 4

[0126] When collecting front and back images of toothbrush, the toothbrush is located in the same horizontal plane to eliminate data errors caused by placement errors.

[0127] Step 1.2: performing distortion correction and denoising enhancement processing on the front color image to obtain a first image, and performing distortion correction and denoising enhancement processing on the back color image of the toothbrush to obtain a second image;

[0128] Step 1.3: setting threshold value , using threshold value to perform binaryzation extraction on the first image and the second image, extracting the area with gray value less than threshold value in the first image and the second image to form a first binary image and a second binary image;

[0129] Step 1.4: calculating total length of toothbrush , storing to database:

[0130] Obtaining coordinates of toothbrush head and handle of the first binary image or the second binary image, denoted as and , the coordinates are known parameters,

[0131]

[0132] In formula (1)​ magnification of the color image collected by the three-dimensional camera in step 1;

[0133] Step 1.5: Set the midpoint of the brush head as the center point of the toothbrush , calculate the length of the toothbrush head , the length of the toothbrush handle , and store , , to the database;

[0134] With the center point set, , known parameters; thus, with the coordinates of the toothbrush head and the handle known, the coordinates of the center point can be obtained.

[0135] Step 1.6: Obtain the depth value of each pixel point according to the front and back depth images, and obtain the front and back feature parameters of the toothbrush; store the front and back feature parameters of the toothbrush to the database.

[0136] The front and back feature parameters of the toothbrush are the front and back head parameters, the front and back handle curve parameters. The front of the head has bristles, so the front and back head parameters can be matched by matching the recognized front and back head parameters with the front and back feature parameters in the database, so that the current toothbrush is determined to be front or back by matching with the front and back feature parameters in the database.

[0137] The method for face judgment in step 3 is:

[0138] The controller uses the openpose_hands_estimation model to process the color image to obtain the key points of the user's body, and judges whether the obtained key points of the body include face key points 14-17, left hand key points 4 and right hand key points 7. If yes, it is determined that there is a face in the image, and step 4 is executed; if not, it is determined that there is no face in the image.

[0139] The specific steps of step 4 to identify the user's identity and the user's toothbrush are:

[0140] Step 4.1: According to the color image The middle face key points 14-17 identify the face information, compare the face information with the database to identify the user identity, if the user identity is identified, select the toothbrush, brushing standard posture and brushing standard time length matched with the identity, if the user identity is not identified, determine that the entrant is a stranger, the touch screen prompts "whether the administrator needs to add data", if yes, the administrator collects the stranger's data, if no, execute step 5;

[0141] Step 4.2; according to the color image The middle left hand key point 4 and right hand key point 7, calculate the minimum circumscribed rectangle, and the length and width are each inflated by a plurality of pixel points, generate the left hand region And the right hand region Identify the left hand region Or the right hand region The parameter of the toothbrush; compare the parameter with the database to identify the toothbrush, if the toothbrush is identified, execute step 5, if the toothbrush is not identified, determine that the toothbrush is a new toothbrush, the touch screen prompts "whether the administrator needs to add toothbrush data", if yes, the administrator collects the data of the new toothbrush, if no, execute step 5

[0142] Step 5, process the color image Obtain the lip region step, specifically:

[0143] Step 5.1: extract the face key points 14-17, calculate the minimum circumscribed rectangle, and the length and width are each inflated by a plurality of pixel points, generate the face region, and cut the face region to obtain the face image ;

[0144] Step 5.2: run the face_landmark_localization neural network model on the face image , identify the face key points, adopt two cascaded CNNs to detect the key points, identify the lip region of the face to the oral cavity region;

[0145] The face key points include 17 external key points and 53 internal key points, the internal key points are detected by a four-level cascaded network level; Level-1 obtains the boundary box of the face organ; Level-2 outputs the key point prediction position, realizes the rough positioning of the key points, and initializes Level-3; Level-3 performs rough-to-precise positioning according to different organs; The input of Level-4 is to rotate the output of Level-3 to a certain extent, and finally output the positions of the 53 key points; The external key points are detected by a two-level cascaded network, Level-1 obtains the boundary box of the face contour; Level-2 directly predicts the positions of the 17 key points, and the 70 key points of the face are obtained by superimposing the outputs of the two cascaded CNNs.

[0146] The mouth region is the region formed by the minimum circumscribed rectangle of the key points 48-59; the oral cavity region is the region formed by the minimum circumscribed rectangle of the key points 60-67;

[0147] Step 5.3: Obtain the lip region according to the mouth region and the oral cavity region identified in step 5.2;

[0148] The lip region is the region formed by subtracting the oral cavity region from the mouth region;

[0149] Step 5.4: Obtain the area of the lip region ;

[0150] The area of the upper lip region is the area of the upper lip region of the face image, not the actual area of the user's upper lip;

[0151] The specific steps of step 6 are as follows:

[0152] Step 6.1: Extract the hand key points 4 and 7, calculate the minimum circumscribed rectangle, and dilate the length and width by a certain number of pixels to generate the right hand region ;

[0153] The left hand or the right hand is used during regular brushing, so only the left hand or the right hand needs to be identified. This embodiment takes the right hand holding the toothbrush as an example for description.

[0154] Step 6.2: Perform image processing on the right hand region to obtain the toothbrush region and the hand region, and obtain the area sum of the toothbrush region and the hand region, denoted as .

[0155] The area sum of the toothbrush region and the hand region is obtained in the same way as the area of the lip region in step 5, and thus is not described here.

[0156] The specific steps of step 7 are as follows:

[0157] Step 7.1: Extract the current time color image , perform processing on the color image , obtain the contour lines of the lip region, the toothbrush region, and the hand region, calculate the minimum circumscribed rectangle, and generate the action region;

[0158] Step 7.2: Obtain the area of the action region ;

[0159] The area of the action region is the area of the action region of the color image , not the actual action region area;​

[0160] Step 7.3: Calculate the overlapping area ;

[0161] ;

[0162] Step 7.4: Calculate the distance between the center point of the lip region and the center point of the toothbrush ;

[0163]

[0164] In the above formula (3), is the center point coordinate of the lip region in the color image, is the center point coordinate of the toothbrush in the color image;

[0165] In different color images, due to the change of the position of the lip region and the toothbrush region, and synchronously change, that is, and are variable values, not constant values, and in this embodiment, refers to the center point coordinate of the lip region in the color image, refers to the center point coordinate of the toothbrush in the color image;

[0166] Step 7.3: Determine whether is greater than the overlapping area threshold value, is less than the center distance threshold value, if so, it is determined that the toothbrushing has started, if is not greater than the overlapping area threshold value or / and is not less than the center distance threshold value, it is determined that the toothbrushing has not started.

[0167] The difference between open-mouth toothbrushing and closed-mouth toothbrushing lies in that when the three-dimensional camera collects images, the image of open-mouth toothbrushing includes the oral cavity region and the brush head of the toothbrush, and whether the bristles are recognized can be used to judge the inside and outside of the toothbrushing position, while the image of closed-mouth toothbrushing does not include the oral cavity region and the brush head of the toothbrush, and it is impossible to judge the inside and outside of the toothbrushing position by whether the bristles are recognized;

[0168] The specific steps of step 8 to determine whether the user is open-mouth toothbrushing are as follows:

[0169] Step 8.1: Obtain and process the color image at the current time point , extract the key points 48-59 and 60-67 to calculate the minimum circumscribed rectangle to form the lip region;

[0170] Step 8.2: Calculate the distance between the key points 62 and 66 ;

[0171]

[0172] In the above formula (4), is a color image The distance between the upper lip region key points 62 and 66;

[0173] Step 8.3: judging The key point distance threshold If is greater than , it is determined that the user is brushing teeth with mouth open, and if is not greater than , it is determined that the user is brushing teeth with mouth closed.

[0174] The step 9 mouth open brushing process includes the following steps:

[0175] Step 9.1: acquiring a color image including a brush head when brushing and a depth image , obtaining images of the oral cavity region, the lip region, and the toothbrush region through image processing ; using a triangular mesh model to process the oral cavity region image, segmenting the teeth, gums, and tongue, and generating a tooth region , a gum region , and a tongue region , establishing a coordinate system with the center of the lip region as the coordinate origin;

[0176] Extracting the coordinates of the center point of the lip region , the coordinates of the toothbrush center point of the brush head position ,

[0177] Step 9.2: calculating the tilt angle , according to to preliminarily determine the brushing position;

[0178]

[0179] The brushing position is set to , is the number of brushing positions, and the brushing positions in this embodiment include the inner and outer walls of the left upper molar, the inner and outer walls of the left lower molar, the inner and outer walls of the upper and lower incisors, the inner and outer walls of the right lower molar, the inner and outer walls of the right upper molar, and the tongue, a total of 13 brushing positions, , set the set of brushing positions , ,

[0180] During the brushing process, the tilt angle is detected in real time, and is not a set value,

[0181] Step 9.3: Precisely identify the brushing area;

[0182] according to Preliminary confirmation ,

[0183] when At that time, recognize the image The position of the brush head is determined, and the first distance between the center point of the brush head and the key point 62 of the lip area is calculated. If the first distance is less than the first set value, the brushing position is determined to be the upper front teeth. The second distance between the center point of the brush head and the key point 66 of the lip area is calculated. If the second distance is less than the second set value, the brushing position is determined to be the lower front teeth. Otherwise, the brushing position is determined to be the tongue.

[0184] If the distance is not less than the set value, the brushing position is determined to be the lower front teeth;

[0185] The steps for identifying the inner and outer surfaces of the brushing area are as follows:

[0186] Image recognition The brush head is processed, and its parameter information is extracted and entered into the database for matching. If the matching result is the front of the toothbrush, that is, the bristles are detected, and the current brushing position is determined to be the inner wall of the toothbrush. If the matching result is the back of the toothbrush, that is, the bristles are not detected, and the current brushing position is determined to be the outer wall of the toothbrush.

[0187] This embodiment sets The inner wall of the upper left molar, The outer wall of the upper left molar, The inner wall of the lower left molar, The inner and outer walls of the left lower molar The inner wall of the upper incisors, The outer wall of the upper incisors, The inner wall of the lower incisors, The outer wall of the lower incisors The inner wall of the lower right molar The outer wall of the lower right molar The inner wall of the upper right molar It is the outer wall of the upper right molar. For the tongue;

[0188] Step 9.4: Calculate the real-time tilt angle ,like If the current brushing position remains unchanged, it is determined that the brushing position has changed; otherwise, it is determined that the brushing position has changed.

[0189] In this embodiment, Set to 5°;

[0190] During brushing, the tilt angle is monitored in real time, and once the real-time tilt angle is detected... If the angle is not in the set angle range, it is considered that the brushing position has changed;

[0191] Set the set of brushed teeth positions Store the brushed teeth positions to the set ;

[0192] Step 9.5: Record the brushing time of the user in and the total brushing time ; ;

[0193]

[0194] For example, the brushing position is in at , the brushing time is ;

[0195] Step 9.6: Extract the coordinates of the contact point between the toothbrush and the lip area in the image , set the contact point when the toothbrush brushes into the lip area as , at this time, the coordinates of are set as , the endpoint of the toothbrush handle is set as , , the coordinates of are set as , at this time, the external distance of the toothbrush is the shortest, the mapping point of the contact point in the depth image , , the coordinates of are set as , the mapping point of the endpoint of the toothbrush handle is set as , , and are known parameters,

[0196]

[0197]

[0198] In the above formulas (7) and (8), the pixel width of the color image is set as , the pixel height is set as , the pixel width of the depth image is set as , the pixel height is set as , , , , are known parameters;

[0199] The depth image ​According to the coordinates of and are set as , the depth value of is obtained , the depth value of ,

[0200] the depth image is set as , the longitudinal image magnification is set as , the physical field of view range of the actual shooting object is set as , the lateral width is the longitudinal height is , the pixel width of is set as , the pixel height is set as , and

[0201]

[0202]

[0203] In the above formula (10), is the lateral field of view angle of the three-dimensional camera, is the longitudinal field of view angle of the three-dimensional camera, is the average distance between the three-dimensional camera and , is a known parameter, according to the above formula (9) and (10), the following formula (11) can be obtained

[0204]

[0205]

[0206] A world coordinate system is established with the actual shooting object, , the corresponding point of the world coordinate system is set as , , the coordinates of are , , and

[0207]

[0208] In the above formula (13), is the shortest external length of the actual toothbrush when brushing the lip area, and , the longest length of the outer part of the actual toothbrush when brushing into the lip area, to identify the first time and the first time is recorded as the first time of brushing. In a brushing time , the number of times of brushing is recorded as ;

[0209] that is, when the brushing position is in , in the brushing time , the number of times of brushing is ;

[0210] Step 9.7: compare the set with the set . If the set is missing elements, the touch screen prompts the user to brush the position; if the elements are the same, compare the recorded , with the set standard value, if , reaches the standard value, it is determined that the user brushes regularly, otherwise, the touch screen displays a prompt that the user does not brush regularly.

[0211] Step 10: the closed tooth brushing process includes the following steps:

[0212] Step 10.1: obtain color images and depth images when brushing, and obtain images of the lip area and toothbrush area through image processing ; establish a coordinate system with the center of the lip area as the coordinate origin;

[0213] Step 10.2: calculate the tilt angle , and preliminarily confirm the brushing position according to ;

[0214] Let the center point of the lip area be , the endpoint of the toothbrush handle be , the contact point of the toothbrush and the lip be , the center point of the brush head be , the coordinates , , , , , , , , , , , , and in the color image are known parameters,

[0215] color image the tilt angle of the upper toothbrush is set to then

[0216]

[0217] According to the above formula (7), the depth image is obtained the mapping point and the coordinates of and , the coordinates , the coordinates ;

[0218]

[0219]

[0220] known and the length of the toothbrush handle ,

[0221]

[0222] According to formulas (14) and (17), the coordinates of ,

[0223]

[0224] According to the above formula, we can obtain:

[0225]

[0226] The tooth brushing positions are set to , is the number of tooth brushing positions, and the tooth brushing positions in this embodiment include the inner wall and outer wall of the upper left molar, the inner wall and outer wall of the lower left molar, the inner wall and outer wall of the upper and lower incisors, the inner wall and outer wall of the lower right molar, the inner wall and outer wall of the upper right molar, and the tongue, a total of 13 tooth brushing positions, , the set of tooth brushing positions is set to , ;

[0227] During tooth brushing, the tilt angle is detected in real time, not the set value,

[0228] Step 10.3: accurately confirm the tooth brushing position;

[0229] according to​ Preliminary confirmation ,

[0230] when At that time, calculate the image The position of the center point of the brush head is calculated, and the first distance between the center point of the brush head and the key point 62 of the lip area is calculated. If the first distance is less than the first set value, the brushing position is determined to be the upper front teeth. The second distance between the brush head and the key point 66 of the lip area is calculated. If the second distance is less than the second set value, the brushing position is determined to be the lower front teeth. Otherwise, the brushing position is determined to be the tongue.

[0231] If the distance is not less than the set value, the brushing position is determined to be the lower front teeth;

[0232] The steps for identifying the inner and outer surfaces of the brushing area are as follows:

[0233] Image recognition The brush handle is extracted, and its surface feature parameters are input into the database for matching. If the matching result is the front of the toothbrush, the current brushing position is determined to be the inner wall; if the matching result is the back of the toothbrush, the current brushing position is determined to be the outer wall.

[0234] Step 10.4: Calculate the real-time tilt angle ,like If the current brushing position remains unchanged, it is determined that the brushing position has changed; otherwise, it is determined that the brushing position has changed.

[0235] In this embodiment, Set to 5°;

[0236] During brushing, the tilt angle is monitored in real time, and once the real-time tilt angle is detected... If the angle is outside the set range, it is assumed that the brushing position has changed.

[0237] Set the set of brushed positions Store brushed areas to a collection ;

[0238] Step 10.5: Record user activity Brushing time and total brushing time ;

[0239] by For example, the brushing position is... At that time, the brushing time is ;

[0240] Step 10.6: Record the brushing position. At the time of brushing teeth The number of brushing times within the period is ,

[0241] The determination and record of the number of tooth brushing is the same as step 9.6, which is not repeated here;

[0242] Step 10.7: Set the set and the set In contrast, if the set is missing, the touch screen prompts the user to the position of not brushing teeth; if the elements are the same, the recorded , is compared with the set standard value, if , reach the standard value, it is determined that the user brushes teeth regularly, otherwise, the touch screen displays a prompt that the user does not brush teeth regularly.

[0243] The steps for detecting the whiteness of the teeth after brushing in step 11 are as follows:

[0244] Step 11.1: Extract the color image before brushing teeth , get the tooth area image; extract the color image before brushing teeth , get the tooth area image;

[0245] Step 11.2: Establish a digital image model ;

[0246] Step 11.3: Calculate the whiteness ;

[0247] Transform the tooth area image to color difference space; In the color difference space, a set of is in one-to-one correspondence with a color, is the brightness, is the green-red degree and blue-yellow degree of the color;

[0248] Extract the pixel points of the tooth area image, , , set the target whiteness, convert the target whiteness to value, denoted as , , , and are known parameters, is the number of image pixel points, the total number of image pixel points , the color difference between two pixel points is:

[0249]

[0250] In the above formula (20), For pixels The color difference from the target whiteness For image The set of color differences between all pixels and the target whiteness. A certain pixel is smaller than the color difference threshold. The whiteness of the dot was considered to have reached the target level. (Statistics) Smaller than the color difference threshold The number of them is denoted as Calculate whiteness. ,contrast Compared with standard whiteness ,like Greater than The system determines that teeth are white after brushing; otherwise, it determines that teeth are not white after brushing and displays the affected area on the touchscreen. Images and tooth areas Images and locations where teeth are not white.

[0251] Step 12, the initial screening for dental diseases, involves the following steps:

[0252] Step 12.1: Acquire a color image Image of the oral cavity region was obtained through image processing. ;

[0253] Step 12.2: Transfer the image The image is segmented into R, G, and B channels, with each channel having a grayscale value of 0-255. The segmentation is then refined based on three regions: gums, teeth, and tongue, to generate the tooth region. gingival area and the tongue area ;

[0254] When R>240, G>240, and B>240, the selected area is a white area, which represents the tooth region. ;

[0255] When R>240, 80>G>60, and 120>B>80, the selection area is a dark red area located in the middle of the teeth area, which is the tongue area. ;

[0256] When R>240, 190>G>170, and 240>B>220, the selection area is a light red area, located above the upper row of teeth or below the lower row of teeth, i.e., the gingival region. ;

[0257] Step 12.3: Tooth Area After the connected domain processing and the close operation, the region contour is obtained, the edge processing is performed on the contour, the straight line fitting is performed on the edge points, the contour straight line is obtained, the maximum distance of the gap between the adjacent teeth is calculated, and if the distance is greater than a set value 2MM, the user is prompted that the teeth are not neat;

[0258] Step 12.4: Identify the gum region Between the tooth region Whether a black region appears between them, the black region is R<20; G<20; B<20, and it is determined that the tooth whiteness is abnormal, and the user is prompted that the tooth has a tooth knot;

[0259] Step 12.5: Identify the tooth region Whether a light red shape appears, which has a color difference much larger than the gum color, if so, the user is prompted that the gum is bleeding, and the user is prompted to use the brushing force and clean as soon as possible.

[0260] A tooth brushing detection device for implementing tooth brushing detection, comprising a controller and a three-dimensional camera, a touch screen, a communication module, a data acquisition module, a face recognition module, a data recognition module, an image processing module, a tooth brushing behavior module, a tooth brushing module, a detection module, and a disease detection module connected with the controller, the controller controls the three-dimensional camera to synchronously shoot color images and depth images, the controller receives and processes the color images and the depth images, and transmits the color images and the depth images to the touch screen for display, the touch screen receives user operation input for command response; the controller controls the communication module to upload and download user data.

[0261] Data acquisition module: used for acquiring toothbrush parameters and user data, and storing the data in a user database;

[0262] Face recognition module: used for recognizing whether there is a face in the shot image;

[0263] Data recognition module: used for recognizing the identity of the user and the toothbrush;

[0264] Image processing module: used for processing the color images and the depth images, and acquiring corresponding regions;

[0265] Tooth brushing behavior module: used for judging whether the user starts to brush teeth and the mode of tooth brushing;

[0266] Tooth brushing module: used for monitoring the position, duration and frequency of tooth brushing of the user;

[0267] Detection module: used for detecting the whiteness of teeth after tooth brushing;

[0268] Disease detection module: used for preliminary screening of tooth diseases.

[0269] A bathroom mirror comprises a mounting module and the tooth brushing detection device as described above, the mounting module is used for mounting the tooth brushing detection device.

[0270] Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present application.

Claims

1. A tooth brushing detection method, characterized in that: The method comprises the following steps: Step 1: the administrator collects toothbrush parameters and user data and stores the data in a user database; Step 2: Real-time monitoring by the three-dimensional camera, and the controller controls the three-dimensional camera to collect color images of the user in real time with the depth image ; Step 3: determining whether the image contains a face, if yes, executing step 3, if no, executing step 2; Step 4: identifying the identity of the user and the toothbrush; Step 5: Controller processes color image Obtain lip region, lip region area; Step 6: Controller processes the color image obtains the toothbrush region and the hand region; Step 7: determining whether to start brushing, if yes, executing step 8, if no, executing step 2; Step 8: determining whether the user is brushing with the mouth open, if yes, executing step 9 for the mouth-open brushing process, if no, executing step 10 for the mouth-closed brushing process; The step 9 mouth-open brushing process comprises the following steps: Step 9.3: accurately confirming the position of brushing; Step 9.1: Obtain a color image including a brush head when brushing and a depth image , obtain images of an oral cavity region, a lip region, and a toothbrush region through image processing ; process the image of the oral cavity region using a triangular mesh model to segment teeth, gums, and a tongue, thereby generating a tooth region , a gum region , and a tongue region , and establish a coordinate system with the center of the lip region as the coordinate origin; extracting the lip region center point coordinates , toothbrush center point coordinates of the brush head position , Step 9.2: Calculate the tilt angle , according to Preliminary confirmation of the position of the brushing ； The brushing position is set to , The number of brushing positions is set to 13, and the brushing positions include the inner wall and outer wall of the left upper molar, the inner wall and outer wall of the left lower molar, the inner wall and outer wall of the upper and lower incisors, the inner wall and outer wall of the right lower molar, the inner wall and outer wall of the right upper molar, and the tongue, The set of brushing positions is set , , During the tooth brushing, the tilt angle is detected in real time, not a set value, Step 10: the mouth-closed brushing process, executing step 11; Step 9.4: Calculate real-time tilt angle , if , determine that the current brushing position is unchanged, otherwise, determine that the brushing position has changed; Setting a set of brushed teeth locations Storing the brushed teeth locations to the set ; Step 9.5: Record the user's brushing time at and the total brushing time ;​ ； Step 9.6: Extracting the image The coordinates of the contact point of the toothbrush with the lip region are set to be , The coordinates of the contact point of the toothbrush with the lip region are set to be The coordinates of the end point of the toothbrush handle are set to be , The coordinates of the end point of the toothbrush handle are set to be The coordinates of the contact point of the toothbrush with the lip region are set to be The mapping point of the contact point is , The coordinates of the contact point of the toothbrush with the lip region are set to be The mapping point of the end point of the toothbrush handle is , The coordinates of the contact point of the toothbrush with the lip region are set to be , and are known parameters, ； ； In the above equations (7) and (8), the pixel width of the color image is set to , the pixel height is set to , the pixel width of the depth image is set to , and the pixel height is set to , , , , are known parameters. Depth image According to The coordinates of And The coordinates of , we get The depth value of , The depth value of , Depth image The lateral image magnification is set to , The longitudinal image magnification is set to The physical field of view range of the actual photographic subject is set to , The pixel width is set to The longitudinal height, The pixel height is set to , and the pixel height is set to , ； ； In the above formula (10), is the lateral field of view angle of the three-dimensional camera, is the longitudinal field of view angle of the three-dimensional camera, is the distance mean value of the three-dimensional camera and is the distance mean value of the three-dimensional camera and is a known parameter, according to the above formula (9) and (10), ； ； A world coordinate system is established with an actual shooting object, 、 A point corresponding to the world coordinate system is set as 、 , The coordinate of the point is , The coordinate of the point is , ； In the above formula (13), is the shortest length of the outer part of the actual toothbrush when brushing the lip area, and similarly , is the longest length of the outer part of the actual toothbrush when brushing the lip area, and is identified to and once is recorded as a tooth brushing time. In a tooth brushing time , the number of tooth brushing times is ; The brushing position is in the brushing time is ; Step 9.7: compare the set with the set If the sets are different, the touch screen reminds the user of the position where the teeth have not been brushed; if the sets are the same, the recorded , , are compared with the set standard value, if , the standard value is reached, the user's brushing is determined to be standard, otherwise, the touch screen displays a prompt that the user's brushing is not standard. Step 11: detecting the whiteness of the teeth after brushing; Step 12: preliminary screening of dental diseases; Step 13: ending the steps. The step of collecting toothbrush parameters in step 1 is:

2. A tooth brushing detection method according to claim 1, characterized in that: Step 1.1: a three-dimensional camera collects a front color image, a front depth image, a back color image and a back depth image of the toothbrush; Step 1.2: performing distortion correction and denoising enhancement processing on the front color image to obtain a first image, and performing distortion correction and denoising enhancement processing on the back color image of the toothbrush to obtain a second image; Step 1.6: obtaining the depth value of each pixel point according to the front depth image and the back depth image, obtaining the front feature parameters of the toothbrush and the back feature parameters of the toothbrush; storing the front feature parameters of the toothbrush and the back feature parameters of the toothbrush to the database; Step 1.3: Setting threshold , using threshold Binarizing the first image and the second image, extracting regions in the first image and the second image with a gray value less than the threshold , to form a first binarized image and a second binarized image; Step 1.4: Calculate total length of toothbrush , the store to database: Obtaining the coordinates of the first or second binary image of the brush head and the handle, denoted as and , the coordinates are known parameters, ； In formula (1) is the magnification of the color image captured by the three-dimensional camera in step 1; Step 1.5: Set midpoint of brush head as center point of toothbrush , calculate Length of toothbrush head , Length of toothbrush handle , , , Store to database; The front feature parameters and the back feature parameters of the toothbrush are the front head parameters, the back head parameters, the front handle curved surface parameters and the back handle curved surface parameters of the toothbrush. The front of the tooth head has bristles. The front head parameters or the back head parameters identified by matching are matched with the front feature parameters and the back feature parameters in the database. By matching with the front feature parameters and the back feature parameters of the toothbrush in the database, it is determined that the current toothbrush is the front or the back. The face determination in step 3 is:

3. The tooth brushing detection method of claim 1, wherein: The mouth region is a region formed by calculating the minimum circumscribed rectangle of the key points 48-59; and the oral cavity region is a region formed by calculating the minimum circumscribed rectangle of the key points 60-67; The controller uses an openpose_hands_estimation model to process the color image to obtain the key points of the user's body, and determines whether the obtained key points of the body include face key points 14-17, left hand key points 4, and right hand key points 7. If yes, it is determined that there is a face in the image, and step 4 is executed. If no, it is determined that there is no face in the image.

4. The tooth brushing detection method of claim 1, wherein: The step 5 of processing the color image The step of obtaining the lip region, specifically: Step 5.1: Extract face key points 14-17, calculate the minimum circumscribed rectangle, and expand the length and width by several pixels to generate a face region, and crop the face region to obtain a face image ; Step 5.2: On face image Running the face landmark localization neural network model to identify the facial landmarks, using two cascaded CNNs to detect the landmarks, and identify the lip region of the face to the oral cavity region; Step 5.3: obtaining a lip region according to the mouth region and the oral cavity region identified in step 5.2; The lip region is a region formed by subtracting the oral cavity region from the mouth region; The specific steps of step 6 are: Step 5.4: Obtain the area of the lip region ; The specific steps of step 7 are as follows: Step 6.1: Extract hand key points 4 and 7, calculate the minimum circumscribed rectangle, and expand the length and width by several pixels to generate the right hand region ; Step 6.2: Right hand region Image processing is performed to obtain the toothbrush region and the hand region. The area sum of the toothbrush region and the hand region is obtained, and the area sum is denoted as .

5. The tooth brushing detection method of claim 1, wherein: The specific steps of step 8 for determining whether the user is brushing with the mouth open are as follows: Step 7.1: Extracting the current time color image The color image is processed to obtain the contour line of the lip region, toothbrush region and hand region, calculate the minimum circumscribed rectangle, and generate the action region. The color image is processed to obtain the contour line of the lip region, toothbrush region and hand region, calculate the minimum circumscribed rectangle, and generate the action region. Step 7.2: Obtain the area of the action region ; Step 7.3: Calculate the overlapping area ; ； Step 7.4: Calculate the distance between the center point of the lip region and the center point of the toothbrush ; ； In the above formula (3), is the center point coordinate of the lip region in the color image, is the center point coordinate of the toothbrush in the color image; Step 7.3: judging whether or not the overlap area is greater than an overlap area threshold, whether or not the center distance is less than a center distance threshold, if so, determining that tooth brushing has started, if not whether or not the overlap area is not greater than the overlap area threshold or / and whether or not the center distance is not less than the center distance threshold, determining that tooth brushing has not started.

6. The method of claim 1, wherein: ​ Step 8.1 : Acquire and process the color image at the current time point , extract the key points 48-59 and 60-67 and calculate the minimum circumscribed rectangle to form the lip region; Step 8.2: Calculate the distance of the key points 62 and 66 ; ； In the above formula (4), for a color image the distance between the upper lip region key points 62 and 66; Step 8.3: judging distance threshold , if is greater than , it is determined that the user is brushing teeth with mouth open, if is not greater than , it is determined that the user is brushing teeth with mouth closed.

7. A tooth brushing detection apparatus, characterized by: A tooth brushing detection method according to any one of claims 1-6, comprising a controller and a three-dimensional camera, a touch screen, a communication module, a data acquisition module, a face recognition module, a data recognition module, an image processing module, a tooth brushing behavior module, a tooth brushing module, a detection module, a disease detection module connected with the controller, the controller controls the three-dimensional camera to synchronously shoot color images and depth images, the controller receives and processes the color images and the depth images, and transmits the color images and the depth images to the touch screen for display, the touch screen receives user operation input for command response; the controller controls the communication module to upload and download user data.

8. A tooth brushing detection apparatus according to claim 7, wherein: The data acquisition module is used for acquiring toothbrush parameters and user data, and storing the data in a user database; The face recognition module is used for recognizing whether there is a face in the shot image; The data recognition module is used for recognizing the identity of the user and the toothbrush; The image processing module is used for processing the color images and the depth images, and obtaining corresponding regions; The tooth brushing behavior module is used for judging whether the user starts to brush teeth and the mode of tooth brushing; The tooth brushing module is used for monitoring the position, duration and frequency of tooth brushing of the user; The detection module is used for detecting the whiteness of teeth after tooth brushing; The disease detection module is used for preliminary screening of tooth diseases.

9. A bathroom mirror, characterised in that: The installation module and the tooth brushing detection device according to any one of claims 7-8 are included, and the installation module is used for installation of the tooth brushing detection device.

Images