Infant monitoring method and device

Abstract

The invention relates to a monitoring technology, and discloses an infant monitoring method and device, and the method comprises the steps: collecting an infant image, and carrying out the real-time collection of the image of an infant; face and key point features of the baby are extracted, and the face and key point features of the baby are extracted from the collected image; and obtaining the head orientation and the face shielding state, wherein the head orientation and the face shielding state of the baby are obtained according to the extracted face and key point features of the baby. According to the invention, each state of the infant can be well monitored, when the infant sleeps, if the face is covered or the head is shielded, the face can be detected in real time, and the user experience is further improved.

Description

Technical Field

[0001] The present invention relates to monitoring technology, and in particular to a baby monitoring method and device. Background Art

[0002] Smart infant care is highly sought after by young parents, and providing accurate smart care is a key factor in the market share of AI-powered smart care devices. Existing smart care solutions for infant sleep scenarios can only detect a human head through a head detection algorithm. Once a head is detected, the algorithm then distinguishes whether it is an adult head or an infant head. Sleep activity is determined by determining whether the infant's head remains motionless for an extended period. This simple solution only considers head detection. Because the head detection algorithm focuses solely on the presence of a head, it cannot detect if the infant's face or head is covered while they sleep. Furthermore, there are times when an infant's head does not move, yet the infant remains awake. In these cases, this solution cannot accurately determine whether the infant is asleep.

[0003] For example, prior art 1: CN116386278B is an intelligent recognition reminder method, device and equipment based on the sleeping posture of infants; the breathing status of the infant is detected based on the picture, and then the infant's breathing is judged to be dangerous based on the breathing status; this method has the following problems: a. The judgment of the pixel points of the weakly moving target is relatively difficult for general camera equipment, b. Even if the pixel points in a can be distinguished, the requirements for the equipment are particularly high, and the cost of the overall solution will become extremely high. c. For the algorithm that distinguishes such subtle differences, its prediction accuracy is also relatively low, and it is not suitable for detecting the vital status of the infant. d. The infant's breathing detection is only detected when the physical characteristics show abnormal performance, which is a post-alarm.

[0004] Prior Art 2: CN113963424B, a method for early warning of infant suffocation or sudden death based on a single-stage face positioning algorithm, detects the position of the face, determines the presence of key facial points, and ultimately determines whether the infant is obscured based on the occlusion information of the key points. This method suffers from the following issues: a. Because the head detection algorithm only focuses on the presence of the head, it may detect sleeping infants whose faces are covered or whose heads are obscured. b. False alarms may occur when the infant's head is lying on its side, impacting the user experience. c. It cannot account for infants lying on their stomachs and is easily overlooked.

[0005] Prior Art 3: CN115880852A - A method and device for monitoring an infant. This method detects key points on the infant's head, body, and limbs. The angle between each key point and the center of the infant's body is calculated to comprehensively determine the infant's sleeping position: supine, prone, upright, or unknown. This method suffers from the following issues: a. When covered with a blanket, key points on the body and limbs cannot be accurately detected due to obstruction. Summary of the Invention

[0006] The present invention addresses the problem that the monitoring system in the prior art cannot accurately determine whether an infant is in a sleeping state, and provides an infant monitoring method and device.

[0007] In order to solve the above technical problems, the present invention solves them through the following technical solutions: a method for monitoring infants, which includes: collecting infant images, collecting infant images in real time; extracting the infant's facial and key point features, extracting the infant's facial and key point features from the collected images; obtaining the status of the infant's head orientation and face occlusion, obtaining the status of the infant's head orientation and face occlusion based on the extracted infant's facial and key point features.

[0008] Preferably, after obtaining the head orientation and face occlusion status, the method also includes obtaining alarm information of the baby status.

[0009] Preferably, extracting facial and key point features of the baby from the collected image includes: obtaining a first feature map Faet1, a second feature map Faet2 and a third feature map Faet3 from the input baby image after passing it through Backbone and Neck; For the obtained feature map, the center point mapping range is extracted, and the center point mapping ranges are pos1, pos2, and pos3 respectively; the feature module area corresponding to pos1 is x1, the feature module area corresponding to pos2 is x2, and the feature module area corresponding to pos3 is x3, where gt is the baby target box, pos1 is the gt center point mapping range, pos2 is the second center point mapping range, and pos3 is the third center point mapping range; For feature module regional feature compression and cosine similarity comparison: ; ; in, is a compressed network, a1 is the cosine similarity between the feature module area x1 and the feature module area x2, and a2 is the cosine similarity between the feature module area x1 and the feature module area x2; The positive sample of the center point mapping range is determined based on the obtained cosine similarity a1. When a1 is greater than the threshold β, the second center point mapping range pos2 is a positive sample, and when a1 is greater than the threshold β, the third center point mapping range pos3 is a positive sample.

[0010] As a preference, the acquisition of the state of the head orientation and the face occlusion includes judging the head orientation and judging the state of the face; for judging the head orientation, the head regression model M of resnet18 is used. reg , the face status is judged by the face occlusion judgment model M of resnet18 cls accomplish.

[0011] As a preference, the face status is judged by the face occlusion judgment model M of resnet18 cls Implementation, based on resnet18 face occlusion judgment model M cls ,The training data is a corrected human face picture, and the output results are 2 categories: mouth and nose occluded, not occluded, ; Among them, y f is the detected face area image, S cls Classification results for face occlusion.

[0012] As a preference, the head regression model M of resnet18 is used reg To judge the direction of the head, the head regression model M based on resnet18 reg , the training data is human head data, and the output result is 6 key points in the face; ; Among them, yh is the output head area image, S reg The key points of the human head are (c0, c1, c2, c3, c4, c5), where c0 is the key point of the top of the head, c1 is the key point of the first ear, c2 is the key point of the second ear, c5 is the key point of the chin, c3 is the key point of the first eye, and c4 is the key point of the second eye; Two ears c0 (x0, y0, 1|0), c2 (x2, y2, 1|0); two eyes c3 (x3, y3, 1|0), c4 (x4, y4, 1|0); top of the head c1 (x1, y1, 1|0); chin vertex c5 (x5, y5, 1|0); where 1|0 represents whether the position exists or not, 0 represents being blocked, and 1 represents being normal; ; ; Among them, x5 is the X-axis position of the chin vertex, y5 is the Y-axis position of the chin vertex, x1 is the X-axis position of the top of the head key point, y1 is the Y-axis position of the top of the head key point, x2 is the X-axis position of the second ear key point, y2 is the y-axis position of the second ear key point, xh2 is the x-axis position of the second ear key point corresponding to the vertical foot, yh2 is the y-axis position of the second ear key point corresponding to the vertical foot, xh4 is the x-axis position of the second eye key point corresponding to the vertical foot, yh4 is the x-axis position of the second eye key point corresponding to the vertical foot, The y-axis position of the foot of the key point corresponding to the vertical point is inferred from the above formula. The corresponding vertical point ch2 (xh2, yh2) corresponding to the ear key point c2 and the corresponding vertical point ch4 (xh4, yh4) corresponding to the eye key point c4 are inferred. ; Among them, d2 represents the difference from the ear to the middle position, and d4 represents the difference from the eye to the middle position.

[0013] Preferably, after obtaining the head direction and face occlusion status, the alarm information of the baby status is also included, including no baby information, head occlusion information, baby front or side occlusion information, baby lying information, baby face occlusion information and baby no occlusion information.

[0014] In order to solve the above technical problems, the present application also provides an infant monitoring device, which includes: an infant image acquisition module, which acquires infant images in real time; an infant facial and key point feature extraction module, which extracts the infant's facial and key point features from the acquired images; and a head orientation and face occlusion status acquisition module, which acquires the infant's head orientation and face occlusion status based on the extracted infant's facial and key point features.

[0015] Due to the adoption of the above technical solutions, the present invention has significant technical effects: the present invention can monitor infants in various states very well, and can detect in real time if the face is covered or the head is blocked when the infant is sleeping, thereby further improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the present invention.

[0017] Figure 2 Flowchart of infant status determination according to the present invention.

[0018] Figure 3 It is a distribution diagram of the mapping range of the positive sample features of the present invention in the original image.

[0019] Figure 4 This is a flow chart of the positive and negative samples based on feature comparison of the present invention.

[0020] Figure 5-1 is the detection image 1 of the present invention; Figure 5-2 This is the detection image 2 of the present invention.

[0021] Figure 5-3 is the detection image 3 of the present invention; Figure 5-4 This is the detection image 4 of the present invention. DETAILED DESCRIPTION

[0022] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0023] Example 1

[0024] A baby monitoring method includes: collecting baby images, collecting baby images in real time; extracting baby facial and key point features, extracting baby facial and key point features from the collected images; The state of the head orientation and face occlusion is obtained based on the extracted facial and key point features of the baby.

[0025] Extracting facial and key point features of the baby from the captured image includes: obtaining the first feature map Faet1, the second feature map Faet2, and the third feature map Faet3 after the input baby image passes through Backbone and Neck; wherein Backbone represents the backbone network, which is used to extract image features; Neck represents the neck network, which is used to combine image features; extracting the center point mapping range from the acquired feature map, the center point mapping ranges are pos1, pos2, and pos3 respectively; the feature module area corresponding to pos1 is x1, the feature module area corresponding to pos2 is x2, and the feature module area corresponding to pos3 is x3, wherein gt is the baby target frame, pos1 is the gt center point mapping range, pos2 is the second center point mapping range, and pos3 is the third center point mapping range; for the feature module area, feature compression is performed and cosine similarity comparison is performed: ; ; in, is a compressed network, a1 is the cosine similarity between the feature module area x1 and the feature module area x2, and a2 is the cosine similarity between the feature module area x1 and the feature module area x2; The positive sample of the center point mapping range is determined based on the obtained cosine similarity a1. When a1 is greater than the threshold β, the second center point mapping range pos2 is a positive sample, and when a1 is greater than the threshold β, the third center point mapping range pos3 is a positive sample.

[0026] The acquisition of the state of head orientation and face occlusion includes judging the direction of the head and judging the state of the face; the judgment of the direction of the head is performed through the head regression model M of resnet18 reg , the face status is judged by the face occlusion judgment model M of resnet18 cls accomplish.

[0027] The face status is judged by the face occlusion judgment model M of resnet18 cls Implementation, based on resnet18 face occlusion judgment model M cls ,The training data is a corrected human face picture, and the output results are 2 categories: mouth and nose occluded, not occluded, ; Among them, y f is the detected face area image, S cls Classification results for face occlusion.

[0028] Through the resnet18 head regression model M reg To judge the direction of the head, the head regression model M based on resnet18 reg , the training data is human head data, and the output result is 6 key points in the face; ; Among them, yh is the output head area image, S reg The key points of the human head are (c0, c1, c2, c3, c4, c5), where c0 is the key point of the top of the head, c1 is the key point of the first ear, c2 is the key point of the second ear, c5 is the key point of the chin, c3 is the key point of the first eye, and c4 is the key point of the second eye; Two ears c0 (x0, y0, 1|0), c2 (x2, y2, 1|0); two eyes c3 (x3, y3, 1|0), c4 (x4, y4, 1|0); top of the head c1 (x1, y1, 1|0); chin vertex c5 (x5, y5, 1|0); where 1|0 represents whether the position exists or not, 0 represents being blocked, and 1 represents being normal; ; ; Among them, x5 is the X-axis position of the chin vertex, y5 is the Y-axis position of the chin vertex, x1 is the X-axis position of the top of the head key point, y1 is the Y-axis position of the top of the head key point, x2 is the X-axis position of the second ear key point, y2 is the y-axis position of the second ear key point, xh2 is the x-axis position of the second ear key point corresponding to the vertical foot, yh2 is the y-axis position of the second ear key point corresponding to the vertical foot, xh4 is the x-axis position of the second eye key point corresponding to the vertical foot, yh4 is the x-axis position of the second eye key point corresponding to the vertical foot, The y-axis position of the foot of the key point corresponding to the vertical point is inferred from the above formula. The corresponding vertical point ch2 (xh2, yh2) corresponding to the ear key point c2 and the corresponding vertical point ch4 (xh4, yh4) corresponding to the eye key point c4 are inferred. ; Among them, d2 represents the difference from the ear to the middle position, and d4 represents the difference from the eye to the middle position.

[0029] Determine whether conf_c3 and conf_c4 are greater than the threshold conf_thred case(conf_c3,conf_c4 are both less than conf_thred) Sleeping on one's stomach, such as Figure 5-1 case(conf_c3,conf_c4 are both greater than conf_thred) sleep on the front, such as Figure 5-2 case (conf_c3, conf_c4 has one greater than conf_thred, here it is assumed to be visible, invisible) { Determine the signs of d2 and d4 Case (d2 and d4 have different symbols) Sleeping on the side or front, as shown in Figure 5-3 Case (d2 and d4 have the same sign) Sleeping on your stomach, as shown in Figure 5-4 } Among them, conf_c3 is the confidence of the first eye key point; conf_c4 is the confidence of the second eye key point; conf_thred is the confidence threshold.

[0030] Example 2

[0031] Based on the embodiment 1, this embodiment further includes obtaining the alarm information of the baby status after obtaining the head direction and the face occlusion status.

[0032] Figure 2In the process, after obtaining the head direction and face occlusion status, the alarm information of the baby status is also included, including no baby information, head occlusion information, baby front or side occlusion information, baby lying information, baby face occlusion information and baby no occlusion information.

[0033] For the output structure, it is determined whether there is a baby's head. If there is no baby's head, the baby's body is determined. If there is no baby's body, the baby is not detected, and the output alarm information is that there is no baby information; When a baby's body is detected, when a baby's head is present, the information of the baby's head occlusion is determined; when a baby's head is detected, the baby's face is detected. When no baby's face information is detected, the baby's head direction is determined. When the baby's head direction is detected to be front or side, the information of the baby's occlusion is output; If the baby's head is detected to be facing backward, the system outputs information indicating the baby is lying prone. If the face is detected to be facing forward or sideways, the system further determines whether the baby's face is blocked. If the face is blocked, the system outputs information indicating the baby is blocked. Otherwise, the system outputs information indicating the baby is in a normal state.

[0034] Is there a head? case (with head) { Whether a face is detected { case (face) { Is the face obscured? { case (occlusion): occlusion warning case (unobstructed): no warning } } case (no face) { Determine the direction of the head { Case (back): Lying down warning case (front or side): face occlusion warning } } } } case (no head) { Is there a body { case (with body): head occlusion warning case(no body): no baby } }.

[0035] Example 3

[0036] Based on the above embodiment, this embodiment is a baby monitoring device, which includes: a baby image acquisition module, which acquires baby images in real time; a baby facial and key point feature extraction module, which extracts the baby's facial and key point features from the acquired images; and a head orientation and face occlusion status acquisition module, which acquires the baby's head orientation and face occlusion status based on the extracted baby's facial and key point features.

[0037] Example 4

[0038] Based on the third embodiment, this embodiment further includes a module for obtaining alarm information of the baby status after obtaining the head orientation and the face occlusion status. Figure 2 In the process, after obtaining the head direction and face occlusion status, the alarm information of the baby status is also included, including no baby information, head occlusion information, baby front or side occlusion information, baby lying information, baby face occlusion information and baby no occlusion information.

Claims

1. A method for monitoring an infant, comprising: Collection of infant images: real-time collection of infant images; Extraction of facial and key point features of infants: Extraction of facial and key point features of infants from the collected images; The state of head orientation and face occlusion is obtained based on the extracted facial and key point features of the baby.

2. A baby monitoring method according to claim 1, characterized in that: After obtaining the head orientation and face occlusion status, the alarm information of the baby status is also obtained.

3. A baby monitoring method according to claim 1, characterized in that: Extraction of facial and key point features of infants from the collected images includes: For the input baby image, after Backbone and Neck, the first feature map Faet1, the second feature map Faet2 and the third feature map Faet3 are obtained; For the obtained feature map, the center point mapping range is extracted, and the center point mapping ranges are pos1, pos2, and pos3 respectively; the feature module area corresponding to pos1 is x1, the feature module area corresponding to pos2 is x2, and the feature module area corresponding to pos3 is x3, where gt is the baby target frame, pos1 is the gt center point mapping range, pos2 is the second center point mapping range, and pos3 is the third center point mapping range; For feature module regional feature compression and cosine similarity comparison: ； ； in, is a compression network, a1 is the cosine similarity between feature module region x1 and feature module region x2, a2 ​​is the cosine similarity between feature module region x1 and feature module region x2; The positive sample of the center point mapping range is determined based on the obtained cosine similarity a1. When a1 is greater than the threshold β, the second center point mapping range pos2 is a positive sample, and when a1 is greater than the threshold β, the third center point mapping range pos3 is a positive sample.

4. A method for monitoring infants according to claim 1, characterized in that: The acquisition of the head orientation and face occlusion status includes judging the head orientation and the face status; the head orientation is judged by the resnet18 head regression model M reg , the face status is judged through the face occlusion judgment model M of resnet18 cls accomplish.

5. A method for monitoring infants according to claim 1, characterized in that: The face status is judged through the face occlusion judgment model M of resnet18 cls Implementation, face occlusion judgment model M based on resnet18 cls ,The training data is a corrected human face picture, and the output results are 2 categories: mouth and nose occluded, not occluded, ； Among them, y f is the detected face area image, S cls Classification results for face occlusion.

6. A method for monitoring infants according to claim 1, characterized in that: Through the head regression model M of resnet18 reg Determine the head orientation based on the resnet18 head regression model M reg , the training data is human head data, and the output result is 6 key points in the face; ； Among them, yh is the output head area image, S reg The key points of the head are (c0, c1, c2, c3, c4, c5), where c0 is the key point of the top of the head, c1 is the key point of the first ear, c2 is the key point of the second ear, c5 is the key point of the chin, c3 is the key point of the first eye, and c4 is the key point of the second eye; Two ears c0 (x0, y0, 1|0), c2 (x2, y2, 1|0); two eyes c3 (x3, y3, 1|0), c4 (x4, y4, 1|0); top of the head c1 (x1, y1, 1|0); chin vertex c5 (x5, y5, 1|0); where 1|0 means whether the position exists or not, 0 means being blocked, and 1 means normal; ； ； Among them, x5 is the X-axis position of the chin vertex, y5 is the Y-axis position of the chin vertex, x1 is the X-axis position of the top of the head key point, y1 is the Y-axis position of the top of the head key point, x2 is the X-axis position of the second ear key point, y2 is the y-axis position of the second ear key point, xh2 is the x-axis position of the second ear key point corresponding to the vertical foot, yh2 is the y-axis position of the second ear key point corresponding to the vertical foot, xh4 is the x-axis position of the second eye key point corresponding to the vertical foot, yh4 is the second eye key point corresponding to the vertical foot. The y-axis position of the foot of the key point corresponding to the vertical point is inferred through the above formula to be the corresponding vertical point ch2 (xh2, yh2) corresponding to the ear key point c2 and the corresponding vertical point ch4 (xh4, yh4) corresponding to the eye key point c4; ； Among them, d2 represents the difference from the eyes to the middle position, and d4 represents the difference from the ears to the middle position.

7. A baby monitoring method according to claim 1, characterized in that: After obtaining the head direction and face occlusion status, the alarm information of the baby status is also obtained, including no baby information, head occlusion information, baby front or side occlusion information, baby lying information, baby face occlusion information and baby no occlusion information.

8. A baby monitoring device, characterized in that: include: The baby image acquisition module acquires the baby's image in real time; A module for extracting facial and key point features of infants, which extracts facial and key point features of infants from the collected images; The head orientation and face occlusion state acquisition module acquires the head orientation and face occlusion state of the baby based on the extracted facial and key point features of the baby.

Images