Sleep detection method and device, electronic equipment and storage medium

Abstract

The invention provides a sleep detection method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: inputting an image frame in an obtained video stream into a target detection model, so as to output a human body frame and a face frame of a target object in each frame of image; extracting a human body image from the human body frame, inputting the human body image into a human body skeleton point detection model to obtain a skeleton point information sequence, and inputting the skeleton point information sequence into a human body resting state detection model to obtain human body state parameters; extracting a face image from the face frame, and respectively inputting the face image into a face key point detection model and a health monitoring model to obtain corresponding eye activity parameters and physiological activity parameters; and determining whether the target object is in a sleep state or not according to at least one of the human body state parameters, the eye activity parameters and the physiological activity parameters. According to the method, the problems of robustness and accuracy in user sleep detection are effectively solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a sleep detection method, device, electronic device and storage medium. Background Art

[0002] With the continuous advancement of smart car technology, traditional cockpits are gradually being replaced by highly intelligent digital cockpits, which significantly improve the driving experience and passenger comfort. More and more passengers choose to rest in the car, which has prompted an urgent need for passenger sleep detection functions. Existing technologies mainly use cameras to detect whether passengers close their eyes, or use sleep detection devices and wearable devices to obtain EEG signals and ultrasonic detection results. However, the former has extremely high requirements for algorithm robustness and is limited by the camera image, while the latter faces the problems of high cost and cumbersome operation. Summary of the invention

[0003] The present application provides a sleep detection method, device, electronic device and storage medium, aiming to solve the problems of robustness and accuracy in user sleep detection.

[0004] In a first aspect, the present application provides a sleep detection method, the method comprising:

[0005] The image frames in the acquired video stream are input into the target detection model to output the human body frame and face frame of the target object in each frame of the image;

[0006] Extracting a human body image from the human body frame, and inputting it into a human skeleton point detection model to obtain a skeleton point information sequence, and inputting it into a human resting state detection model to obtain human body state parameters; extracting a face image from the face frame, and inputting it into a face key point detection model and a health monitoring model respectively to obtain corresponding eye activity parameters and physiological activity parameters;

[0007] Whether the target object is in a sleeping state is determined according to at least one of the human body state parameter, the eye activity parameter and the physiological activity parameter.

[0008] In a second aspect, the present application further provides a sleep detection device, the device comprising:

[0009] The target detection module is used to input the image frames in the acquired video stream into the target detection model to output the human body frame and face frame of the target object in each frame of the image;

[0010] An image processing module is used to extract a human body image from the human body frame, and input it into a human skeleton point detection model to obtain a skeleton point information sequence, and input it into a human resting state detection model to obtain human body state parameters; extract a face image from the face frame, and input it into a face key point detection model and a health monitoring model respectively to obtain corresponding eye activity parameters and physiological activity parameters;

[0011] The sleep determination module is used to determine whether the target object is in a sleep state according to at least one of the human body state parameter, the eye activity parameter and the physiological activity parameter.

[0012] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the steps of the sleep detection method as described in any one of the first aspects are implemented.

[0013] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the sleep detection method as described in any one of the first aspects are implemented.

[0014] The present application provides a sleep detection method, device, electronic device and storage medium. The method inputs the image frame in the video stream into the target detection model, extracts the human body frame and the face frame, and then processes the human body and face images respectively, obtains the human body skeleton point information sequence, and then obtains the human body state parameters, as well as the eye activity parameters and the physiological activity parameters, and finally comprehensively determines the sleep state of the target object. Therefore, the method realizes multimodal data fusion by utilizing the human body state parameters, eye activity parameters and physiological activity parameters, and improves the robustness and accuracy of the detection results. The combination of the human body skeleton point detection model and the face key point detection model is adopted to ensure the accuracy of the skeleton point information and the position of the face key points. And through multi-level judgment conditions, that is, combining the human body state parameters, eye activity parameters and physiological activity parameters, the accurate judgment of the sleep state of the target object is ensured. And the image frames in the video stream are processed in real time to ensure the continuous monitoring of the sleep state of the target object and avoid the judgment errors caused by the single frame detection error.

[0015] Therefore, this application effectively solves the problems of robustness and accuracy in user sleep detection through multimodal data fusion, combination of models, multi-level judgment and real-time monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 is a flow chart of the sleep detection method provided by the present application;

[0018] Figure 2 is a schematic diagram of each model provided in this application;

[0019] Figure 3 is a schematic diagram of a human body frame and a human face frame provided in an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of the positions of key points of a face provided in an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of a sleep detection method provided in an embodiment of the present application;

[0022] Figure 6 is a schematic diagram of calculating eye activity parameters provided by an embodiment of the present application;

[0023] Figure 7 is a structural schematic diagram of the sleep detection device provided by the present application;

[0024] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0027] In order to solve the problem of robustness and accuracy in sleep detection of users, this application proposes a sleep detection method, device, electronic device and storage medium. The method inputs the image frames in the video stream into the target detection model, extracts the human body frame and the face frame, and then processes them separately to obtain human body state parameters, eye activity parameters and physiological activity parameters. Through multimodal data fusion, multi-level judgment conditions and real-time monitoring, the high robustness and accuracy of sleep state detection are ensured.

[0028] Combine the following Figure 1-Figure 8 The present application describes a sleep detection method, device, electronic device and storage medium.

[0029] Please refer to Figure 1 , Figure 2 , Figure 1 is a flow chart of the sleep detection method provided by this application, Figure 2 is a schematic diagram of each model provided in this application. A sleep detection method includes:

[0030] S110, inputting the image frames in the acquired video stream into the target detection model to output the human body frame and the face frame of the target object in each frame of the image.

[0031] Specifically, this step is to input each frame of the video stream into the target detection model. The role of the target detection model is to identify and locate the target object in the image. In the car cabin scene, the target object is the passenger. Through this step, the target detection model will output the location of the person in each frame of the image, which is specifically expressed as a human body frame (i.e., a rectangular frame surrounding the human body) and a face frame (i.e., a rectangular frame surrounding the face), such as Figure 3 By obtaining the human body frame and face frame, it can help the subsequent steps to accurately locate and process the human body image and face image, so as to further extract relevant information, such as human body bone points, face key points, etc., so as to analyze and determine the sleep state of the target object.

[0032] S120, extracting a human body image from the human body frame, and inputting it into a human body skeleton point detection model to obtain a skeleton point information sequence, and inputting it into a human body resting state detection model to obtain human body state parameters.

[0033] Specifically, this step is to extract a human body image from a human body frame, and then input the extracted human body image into a human body skeleton point detection model. The task of the human body skeleton point detection model is to identify and locate key skeleton points in the human body image, such as joints and limb connection points. Through this step, the human body skeleton point detection model will output a skeleton point information sequence, and the skeleton point information sequence contains the position information of various parts of the human body, for example, it contains a preset number (such as at least 17) of skeleton point positions and their visible and invisible attributes. Then, the skeleton point information sequence is input into the human body resting state detection model, and the role of the human body resting state detection model is to analyze and calculate the human body state parameters according to the movement and position changes of the skeleton points. The human body state parameters can reflect the human body state, determine whether the target object is in a resting state or a moving state, and attach corresponding confidence. In short, this step detects the human body skeleton point information, and then uses the human body skeleton point information to evaluate the human body's motion state, providing data support for subsequent sleep state determination.

[0034] S130, extracting a face image from the face frame, and inputting it into a face key point detection model and a health monitoring model respectively, to obtain corresponding eye activity parameters and physiological activity parameters.

[0035] Specifically, this step is to extract the face image from the face frame, and then input the extracted face image into two different models, namely the face key point detection model and the health monitoring model. Among them, the task of the face key point detection model is to identify and locate the key points in the face image, such as the positions of the eyes, mouth, nose and other parts. Figure 4 As shown, Figure 4 At least 68 key point positions of the face are shown. Through this step, the face key point detection model will output the face key point positions, and then the eye activity parameters can be calculated based on the face key point positions. The eye activity parameters can reflect the open state, closed state and closed duration of the eyes. At the same time, the health monitoring model will analyze the physiological characteristics in the face image, such as skin color, blood flow changes, etc., to obtain physiological activity parameters, which can reflect physiological states such as heart rate and breathing rate. In short, this step obtains eye activity parameters and physiological activity parameters respectively by detecting facial key points and analyzing physiological characteristics, providing multi-dimensional data support for subsequent sleep state determination.

[0036] S140, determining whether the target object is in a sleeping state according to at least one of the human body state parameter, the eye activity parameter, and the physiological activity parameter.

[0037] Specifically, this step is to comprehensively analyze these data to determine whether the target object is in a sleeping state based on at least one of the human body state parameters, eye activity parameters and physiological activity parameters obtained in the previous step. Among them, the human body state parameters can reflect whether it is in a resting state or a moving state, and the corresponding confidence score; the eye activity parameters can reflect the eye's closed state, open state and closed time; and the physiological activity parameters can reflect the heart rate, breathing rate and other physiological states. By comprehensively analyzing these parameters, the system can identify whether the target object is in a sleeping state, such as whether there is regular breathing, whether the eyes are closed and how long they are closed, whether the body is still and other characteristics. In short, this step finally determines the sleep state of the target object through the fusion analysis of multimodal data.

[0038] It can be seen that the sleep detection method described in this application analyzes the human skeleton points and facial key points in the video stream, without the need for additional equipment, low cost and natural experience. Even if the face is blocked or turned, sleep detection can be performed based on the human body resting state detection model of the skeleton points. At the same time, the integration of the duration of eye closure and the changing trends of heart rate and breathing frequency can significantly reduce the false alarm rate and improve the detection accuracy.

[0039] In some embodiments, the method further comprises training a human skeleton point detection model, wherein the training step comprises:

[0040] S1011, collecting video data containing different sleeping postures, and marking the skeleton point positions and visible and invisible human skeleton points of the target object in the video data to obtain marked human skeleton point data.

[0041] For example, human skeleton point estimation data of different sleeping postures can be produced based on the collected video data. That is, the positions of at least 17 human skeleton points are marked for passengers with different sleeping postures. The visible attributes are marked for the skeleton points of visible body parts, and the invisible attributes are marked for the occluded (i.e. invisible) body parts.

[0042] S1012, fusing the labeled human skeleton point data with the preset human skeleton point data to obtain a training data set, and using the training data set to perform model training according to the human body frame obtained by the target detection model.

[0043] For example, the labeled real-car passenger human skeleton point data and the existing publicly available human skeleton point data are used to train a top-down human skeleton point prediction model, that is, the human body frame of the passenger distribution is first obtained, and then the human skeleton points are predicted from top to bottom based on the human body frame.

[0044] S1013, after training, a human skeleton point detection model capable of predicting a skeleton point information sequence is obtained, wherein the skeleton point information sequence includes a preset number of skeleton point positions and their visible and invisible attributes.

[0045] For example, a human skeleton point detection model can output a sequence of skeleton point information for each passenger in a car cabin.

[0046] In some embodiments, the method further includes training the human body resting state detection model, and the training step includes:

[0047] S1021, recording the skeleton point information sequence output by the human skeleton point detection model within a preset time length.

[0048] Specifically, the skeleton point information of passengers at various positions in the car cabin is recorded over a period of time, for example, a sequence of skeleton point information over a period of time is recorded in the form of a sliding window, including the skeleton point position and its visible and invisible attributes, wherein the skeleton point position can be represented by coordinates.

[0049] S1022, dividing the recorded skeleton point information sequence into two categories: one is a first key point sequence representing a resting state, and the other is a second key point sequence representing a moving state;

[0050] S1023, using the first key point sequence and the second key point sequence as training data to train a human body resting state detection model.

[0051] Specifically, the input of the human resting state detection model can be the bone point position coordinates and visible and invisible attribute sequences of the same target object within a period of time, and the output is a binary classification, which indicates whether movement occurs during this period, that is, resting state or moving state.

[0052] S1024, after training, a human skeleton point detection model capable of predicting human state parameters is obtained, wherein the human state parameters include a resting state and a moving state, and are accompanied by corresponding confidence scores.

[0053] The above steps S110 to S140 are described in detail below through an embodiment.

[0054] Please refer to Figure 5 , Figure 5 Schematic diagram of a sleep detection method provided in an embodiment of the present application. A sleep detection method comprises the following steps:

[0055] S501, receiving a video stream of a sleep detection area.

[0056] For example, the sleep detection area can be the area inside the car cabin. If an OMS (Occupant Monitoring System) camera is already installed in the car, the existing OMS camera in the car can be used to obtain the video stream without the need to install additional special sleep detection devices or wearable devices. This method is contactless, which means that it does not interfere with passengers or require them to wear any equipment. Therefore, the increased cost and cumbersome operation caused by adding additional equipment are avoided, making sleep detection more convenient and economical.

[0057] S502: Input the acquired image frames in the video stream into the target detection model to output the human body frame and face frame of the target object in each frame of the image.

[0058] The human body frame represents the overall body position of the target object, and the face frame represents the facial position of the target object. A real-time video stream can be obtained from the camera in the car cabin, and each frame image in the video stream is input into the target detection model, which outputs the human body frame and face frame of the target object detected in each frame.

[0059] For example, suppose a frame image in the video stream shows passenger A sitting on the left side of the car and passenger B sitting on the right side, both facing forward. This frame image is input to the target detection model, and the output of the target detection model is:

[0060] Passenger A:

[0061] Human body frame: (x1, y1, x2, y2) = (100, 200, 300, 500);

[0062] Face frame: (x1, y1, x2, y2) = (150, 250, 250, 350).

[0063] Passenger B:

[0064] Human body frame: (x1, y1, x2, y2) = (400, 200, 600, 500);

[0065] Face frame: (x1, y1, x2, y2) = (450, 250, 550, 350).

[0066] S503, determining whether the output of the target detection model contains a human frame.

[0067] If the output of the target detection model includes a human body frame, execute step S504;

[0068] If the output of the target detection model does not contain a human frame, the operation ends.

[0069] S504, extracting a human body image from the human body frame, and inputting it into a human skeleton point detection model to obtain a skeleton point information sequence.

[0070] Specifically, the human frame coordinates output by the target detection model can be used to crop the human image from the original image frame, and the cropped human image can be input into the pre-trained human skeleton point detection model. The human skeleton point detection model will output a sequence of skeleton point information detected in the human image, including skeleton point position coordinates and visible and invisible attribute sequences. Among them, the skeleton key points can use 17 skeleton points of the human body. For example, assuming that the human frame coordinates of passenger A output by the target detection model are (100, 200, 300, 500), the human image of this area is cropped from the original image frame. The human image of passenger A is input into the human skeleton point detection model, and the output of the human skeleton point detection model is:

[0071] Passenger A's skeleton point information sequence:

[0072] Skeleton point 1: (x1, y1) = (150, 250), visible;

[0073] Skeleton point 2: (x1, y1) = (200, 300), visible;

[0074] Skeleton point 3: (x1, y1) = (250, 350), invisible (such as blocked); ...

[0076] Skeleton point 17: (x1, y1) = (280, 450), visible.

[0077] S505, inputting the skeleton point information sequence into a human body resting state detection model to obtain human body state parameters.

[0078] Specifically, the skeleton point information sequence is input into the human body resting state detection model, and the human body resting state detection model will output human body state parameters, including resting state and motion state, with corresponding confidence scores.

[0079] For example, the skeleton point information sequence of passenger A in each frame image is obtained from the human skeleton point detection model, and the skeleton point information sequence of passenger A is input into the human resting state detection model. The prediction results output by the human resting state detection model are as follows:

[0080] Passenger A: is in a resting state within the preset time, and his confidence score is 90.

[0081] S506, determining whether the output of the target detection model contains a face frame.

[0082] If the output of the target detection model contains a face frame, indicating that the face of the target object is visible, then step S507 is executed;

[0083] If there is no face frame in the output of the target detection model, indicating that the face of the target object is invisible, step S513 is executed.

[0084] S507, extracting the face image from the face frame and inputting it into the face key point detection model to obtain the face key point positions, confidence scores and head posture angles.

[0085] Specifically, the face frame coordinates output by the target detection model are used to crop the face image from the original image frame. The cropped face image is input into the face key point detection model. The face key point detection model outputs the positions of the face key points detected in the face image and their confidence scores, as well as the head pose angles. The face key points can be expressed as follows: Figure 4 The 68 key points shown. Head pose angles refer to the three angles that describe the posture of the head in three-dimensional space, such as pitch angle, yaw angle, and roll angle. The pitch angle describes the angle of the head turning up and down. The yaw angle describes the angle of the head turning left and right. The roll angle describes the angle of the head rotating around its own vertical axis. For example, the prediction result of the facial key point detection model for passenger A is:

[0086] Passenger A’s facial key points:

[0087] [(100, 200), (105, 210), ..., (300, 400)];

[0088] Corresponding confidence scores: [0.95, 0.96, ..., 0.94];

[0089] Head attitude angle (pitch angle, yaw angle, roll angle): [5.0, -2.0, 1.0].

[0090] Among them, the facial key point position of passenger A is a list of 68 points, each of which represents a key point on the face. The confidence score corresponding to each key point indicates the model's confidence in the key point position. For example, a pitch angle of 5.0 degrees indicates that the head is slightly tilted upward; a yaw angle of -2.0 degrees indicates that the head is slightly turned to the left. A roll angle of 1.0 degrees indicates that the head is slightly tilted to the left.

[0091] S508, calculating the closure degree of the left eye and the closure degree of the right eye respectively according to the positions of the key points of the face to obtain eye activity parameters.

[0092] Please refer to Figure 6 , Figure 6is a schematic diagram of calculating eye activity parameters provided by an embodiment of the present application. The steps of calculating the closure degree of the left eye and the closure degree of the right eye include:

[0093] S5081, determining whether the degree of closure of the left eye is less than a preset eye closure threshold.

[0094] If yes, execute step S5082; otherwise, execute step S5084.

[0095] Among them, the degree of closure of the left eye = (the absolute value of the distance from the left end point of the upper eyelid of the left eye to the left end point of the lower eyelid + the absolute value of the distance from the right end point of the upper eyelid of the left eye to the right end point of the lower eyelid) / (2×the absolute value of the distance from the left end point to the right end point of the left eye).

[0096] For example, see Figure 4 , the calculation formula of left eye closure Left Eye Mar is:

[0097]

[0098] Among them, P 37 Indicates the position of the left end point of the left upper eyelid, P 41 Indicates the position of the left end point of the lower eyelid of the left eye, P 38 Indicates the position of the right end point of the left upper eyelid, P 40 Indicates the position of the right end point of the left eyelid lower eyelid, P 36 The position of the left endpoint of the left eye, P 39 Indicates the position of the right endpoint of the left eye.

[0099] S5082, determining whether the degree of closure of the right eye is less than a preset eye closure threshold.

[0100] If yes, execute step S5083; otherwise, execute step S5084.

[0101] Among them, the degree of right eye closure = (the absolute value of the distance from the left end point of the upper eyelid to the left end point of the lower eyelid of the right eye + the absolute value of the distance from the right end point of the upper eyelid to the right end point of the lower eyelid of the right eye) / (2×the absolute value of the distance from the left end point to the right end point of the right eye).

[0102] For example, see Figure 4 , the calculation formula of right eye closure degree Right Eye Mar is:

[0103]

[0104] Among them, P 43 Indicates the position of the left end point of the upper eyelid of the right eye, P 47 Indicates the position of the left end point of the lower eyelid of the right eye, P 44 Indicates the position of the right end point of the upper eyelid of the right eye, P 46The position of the right end point of the lower eyelid of the right eye, P 42 Indicates the position of the left endpoint of the right eye, P 45 Indicates the position of the right endpoint of the right eye.

[0105] In the above steps S5081 and S5082, assuming that the values ​​of the left eye closure degree Left Eye Mar and the right eye closure degree Right Eye Mar are both 0.1, the preset eye closure threshold is set to 0.2, and 0.1 is less than 0.2, so it is determined to be a closed eye state, and the continuous closed eye duration needs to be counted.

[0106] S5083, in the eyes-closed state, counting the duration of eyes-closed continuation.

[0107] S5084, clear the counted eye closing time.

[0108] S5085, using the counted continuous eye closure duration as an eye activity parameter.

[0109] For example, if the counted continuous eye closure duration is 5 minutes, it means that the total length of time the eyes are closed is 5 minutes, and these 5 minutes are used as eye activity parameters.

[0110] S509, determining whether the head posture of the target object is correct.

[0111] If the head posture of the target object is within the preset range, it means that the head posture of the target object is correct, and step S510 is executed.

[0112] If the head posture of the target object is not within the preset range, it means that the head posture of the target object is not correct, and step S511 is executed.

[0113] For example, for the head posture angle (pitch angle, yaw angle, roll angle) of passenger A in step S507: [5.0, -2.0, 1.0], it can be determined whether the head posture of passenger A is correct based on the set threshold range (for example, the pitch angle, yaw angle, and roll angle are all within ±10 degrees).

[0114] Pitch angle: 5.0 degrees, within ±10 degrees, in line with the correct standard.

[0115] Yaw angle: -2.0 degrees, within ±10 degrees, meets the correct standard.

[0116] Roll angle: 1.0 degrees, within ±10 degrees, in line with the standard.

[0117] Therefore, it can be judged that passenger A's head posture is correct.

[0118] S510, extracting a face image from the face frame and inputting it into a health monitoring model to obtain physiological activity parameters.

[0119] Specifically, step S510 includes the following steps:

[0120] S5101, the health monitoring model outputs physiological activity parameters by analyzing pixel changes of the face image, and the physiological activity parameters include heart rate and respiratory rate.

[0121] S5102, based on the heart rate and respiratory rate, statistics the changing trends of the heart rate and respiratory rate within a preset time period: if the fluctuation range of the heart rate within the preset time period is less than the preset heart rate threshold, and the change range of the respiratory rate within the preset time period is less than the preset respiratory threshold, it is determined to be a sleeping state; otherwise, it is determined to be an awake state.

[0122] For example, assuming the preset duration is 10 minutes, the preset heart rate threshold is ±5 bpm (heart beats per minute), and the preset breathing threshold is ±2 breaths / min (breaths per minute). Then within 10 minutes, record the heart rate and breathing rate every 1 minute, and get the following data:

[0123] Time Heart rate (bpm) Respiratory rate (breaths / min) 0:00 60 12 1:00 61 12 2:00 60 11 3:00 62 12 4:00 59 11 5:00 60 12 6:00 61 12 7:00 60 11 8:00 62 12 9:00 60 12

[0124] (1) Calculate the fluctuation range of heart rate:

[0125] Maximum heart rate: 62bpm

[0126] Minimum heart rate: 59bpm;

[0127] Heart rate fluctuation range: 62-59=3bpm.

[0128] (2) Calculate the change in respiratory rate:

[0129] Maximum respiratory rate: 12breaths / min;

[0130] Minimum respiratory rate: 11breaths / min;

[0131] Respiratory rate change range: 12-11=1breaths / min.

[0132] (3) Determine the status:

[0133] Heart rate fluctuation range: 3bpm, less than the preset heart rate threshold (±5bpm).

[0134] Respiratory rate variation: 1 breaths / min, less than the preset respiratory threshold (±2 breaths / min).

[0135] According to the above analysis, the changing trends of heart rate and respiratory rate are within the preset threshold range, so it is determined to be a sleeping state.

[0136] S511, determining whether the human body state parameters and eye activity parameters meet a first preset condition.

[0137] The first preset condition is: the human body state parameter is displayed as a resting state, and the accompanying confidence score is greater than the preset first threshold (for example, the first threshold is 85 points, and the full score is 100), and at the same time, the continuous eye closure time displayed by the eye activity parameter (for example, the continuous eye closure time is 5 minutes) is greater than the preset time (for example, the preset time is 3 minutes).

[0138] If the first preset condition is met, it is determined that the target object is in a sleeping state.

[0139] If the first preset condition is not met, it is determined that the target object is in a sober state.

[0140] S512, determining whether the human body state parameters, eye activity parameters, and physiological activity parameters meet a second preset condition.

[0141] The second preset condition is: the human body state parameter is displayed as a resting state, and the accompanying confidence score is greater than a preset second threshold (for example, the second threshold is 75 points, and the full score is 100 points), and at the same time, the eye activity parameter shows a continuous eye closure time (for example, the continuous eye closure time is 5 minutes) greater than the preset time (for example, the preset time is 3 minutes), and the physiological activity parameter is displayed as a sleeping state, where the second threshold is less than the first threshold.

[0142] If the second preset condition is met, it is determined that the target object is in a sleeping state.

[0143] If the second preset condition is not met, it is determined that the target object is in a sober state.

[0144] S513, determining whether the human body state parameter satisfies a third preset condition.

[0145] The third preset condition is: the human body state parameter is displayed as a resting state, and the accompanying confidence score is greater than a preset third threshold (for example, the third threshold is 95 points, and the full score is 100 points), where the third threshold is greater than the first threshold.

[0146] If the third preset condition is met, it is determined that the target object is in a sleeping state.

[0147] If the third preset condition is not met, it is determined that the target object is in a sober state.

[0148] In summary, the sleep detection method described in this application realizes the passenger sleep detection function in the car cabin by detecting the resting state based on the human skeleton points and integrating the passenger sleep detection scheme of the eye closing duration, heart rate and breathing rate change trend. This function can intelligently adjust the cabin volume, air conditioning temperature and air volume, seat angle and window opening and closing status, so as to provide passengers with a quiet and comfortable sleeping environment.

[0149] The sleep detection device provided in the present application is described below. The sleep detection device described below and the sleep detection method described above can refer to each other.

[0150] Please refer to Figure 7 , Figure 7 Schematic diagram of the structure of the sleep detection device provided by the present application. A sleep detection device 700 includes a target detection module 710 , an image processing module 720 and a sleep determination module 730 .

[0151] Exemplarily, the target detection module 710 is used to input the image frames in the acquired video stream into the target detection model to output the human body frame and the face frame of the target object in each frame of the image.

[0152] Exemplarily, the image processing module 720 is used to extract a human body image from the human body frame, and input it into a human skeleton point detection model to obtain a skeleton point information sequence, and input it into a human resting state detection model to obtain human body state parameters; extract a face image from the face frame, and input it into a face key point detection model and a health monitoring model respectively to obtain corresponding eye activity parameters and physiological activity parameters.

[0153] Exemplarily, the sleep determination module 730 is used to determine whether the target object is in a sleep state according to at least one of the human body state parameter, the eye activity parameter, and the physiological activity parameter.

[0154] Exemplarily, the sleep detection device 700 is further used for:

[0155] Collect video data containing different sleeping postures, and mark the skeleton point positions and visible and invisible human skeleton points of the target object in the video data to obtain marked human skeleton point data;

[0156] The labeled human skeleton point data is merged with the preset human skeleton point data to obtain a training data set, and the training data set is used to perform model training according to the human body frame obtained by the target detection model;

[0157] After training, a human skeleton point detection model capable of predicting the skeleton point information sequence is obtained. The skeleton point information sequence includes a preset number of skeleton point positions and their visible and invisible attributes.

[0158] Exemplarily, the sleep detection device 700 is further used for:

[0159] Record the skeleton point information sequence output by the human skeleton point detection model within a preset time length;

[0160] The skeleton point information sequence is divided into two categories: one is a first key point sequence representing a resting state, and the other is a second key point sequence representing a moving state;

[0161] Using the first key point sequence and the second key point sequence as training data to train the human body resting state detection model;

[0162] After training, a human body resting state detection model is obtained that can predict the human body state parameters, wherein the human body state parameters include a resting state and a motion state, and are accompanied by corresponding confidence scores.

[0163] Exemplarily, the image processing module 720 is further used for:

[0164] Calculating the closure degree of the left eye and the closure degree of the right eye respectively according to the positions of the facial key points extracted from the facial image by the facial key point detection model;

[0165] If the closure degree of the left eye and the closure degree of the right eye are both less than a preset eye-closing threshold, it is determined to be an eye-closing state; if the closure degree of the left eye or the closure degree of the right eye is greater than the eye-closing threshold, it is determined to be an eye-opening state;

[0166] In the eyes-closed state, the continuous closing time is counted; when entering the eyes-open state, the counted eyes-closed time is cleared, and the counted continuous closing time is used as the eye activity parameter.

[0167] Exemplarily, the formula for calculating the closure degree of the left eye and the closure degree of the right eye is:

[0168] Left eye closure degree = (the absolute value of the distance from the left end point of the upper eyelid to the left end point of the lower eyelid + the absolute value of the distance from the right end point of the upper eyelid to the right end point of the lower eyelid) / (2 × the absolute value of the distance from the left end point to the right end point of the left eye);

[0169] Right eye closure degree = (absolute value of the distance from the left end point of the right upper eyelid to the left end point of the lower eyelid + absolute value of the distance from the right end point of the right upper eyelid to the right end point of the lower eyelid) / (2×absolute value of the distance from the left end point to the right end point of the right eye).

[0170] Exemplarily, the image processing module 720 is further used for:

[0171] The health monitoring model outputs physiological activity parameters by analyzing pixel changes of the face image, and the physiological activity parameters include heart rate and respiratory rate;

[0172] Based on the heart rate and the respiratory rate, the changing trends of the heart rate and the respiratory rate within a preset time period are counted: if the fluctuation range of the heart rate within the preset time period is smaller than the preset heart rate threshold, and the change amplitude of the respiratory rate within the preset time period is smaller than the preset respiratory threshold, it is determined to be a sleeping state; otherwise, it is determined to be an awake state.

[0173] Exemplarily, the sleep determination module 730 is further configured to:

[0174] If the human body state parameter indicates a resting state, and the confidence score thereof is greater than a preset first threshold, and the continuous eye closure time indicated by the eye activity parameter is greater than a preset time, it is determined that the target object is in a sleeping state;

[0175] If the human body state parameter is displayed as a resting state, and the confidence score thereof is greater than a preset second threshold, and the continuous eye closure time displayed by the eye activity parameter is greater than a preset time, and the physiological activity parameter is displayed as a sleeping state, it is determined that the target object is in a sleeping state, wherein the second threshold is less than the first threshold;

[0176] If the human body state parameter is displayed as a resting state and its accompanying confidence score is greater than a preset third threshold, it is determined that the target object is in a sleeping state, wherein the third threshold is greater than the first threshold.

[0177] Exemplarily, the skeleton point information sequence output by the human skeleton point detection model includes at least 17 skeleton point positions of the human body and their visibility and invisibility attributes; the key point position information output by the facial key point detection model includes at least 68 key point positions of the face.

[0178] It should be noted here that the above-mentioned sleep detection device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0179] Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 8As shown, the electronic device may include: a processor (Processor) 810, a communication interface (Communications Interface) 820, a memory (Memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the sleep detection method.

[0180] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to ensure that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0181] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the sleep detection method provided by the above methods.

[0182] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the sleep detection method provided above.

[0183] The embodiments of the present application provide an electronic device, a computer program product, and a processor-readable storage medium, on which the computer program stored enables the processor to implement all the method steps implemented in the above-mentioned method embodiments and to achieve the same technical effects. The parts and beneficial effects of this embodiment that are the same as those in the method embodiments will not be described in detail herein.

[0184] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0185] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions to ensure that a computer device (which can be a personal computer, a server, or a network device, etc.) executes the methods described in each embodiment or some parts of the embodiments.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A sleep detection method, characterized in that: The method comprises: The image frames in the acquired video stream are input into the target detection model to output the human body frame and face frame of the target object in each frame of the image; Extracting a human body image from the human body frame, and inputting it into a human skeleton point detection model to obtain a skeleton point information sequence, and inputting it into a human resting state detection model to obtain human body state parameters; extracting a face image from the face frame, and inputting it into a face key point detection model and a health monitoring model respectively to obtain corresponding eye activity parameters and physiological activity parameters; Whether the target object is in a sleeping state is determined according to at least one of the human body state parameter, the eye activity parameter and the physiological activity parameter.

2. The sleep detection method according to claim 1, characterized in that: The method further comprises training the human skeleton point detection model, wherein the training step comprises: Collect video data containing different sleeping postures, and mark the skeleton point positions and visible and invisible human skeleton points of the target object in the video data to obtain marked human skeleton point data; The labeled human skeleton point data is merged with the preset human skeleton point data to obtain a training data set, and the training data set is used to perform model training according to the human body frame obtained by the target detection model; After training, a human skeleton point detection model capable of predicting the skeleton point information sequence is obtained. The skeleton point information sequence includes a preset number of skeleton point positions and their visible and invisible attributes.

3. The sleep detection method according to claim 1, characterized in that: The method further includes training the human body resting state detection model, wherein the training step includes: Record the skeleton point information sequence output by the human skeleton point detection model within a preset time length; The skeleton point information sequence is divided into two categories: one is a first key point sequence representing a resting state, and the other is a second key point sequence representing a moving state; Using the first key point sequence and the second key point sequence as training data to train the human body resting state detection model; After training, a human body resting state detection model is obtained that can predict the human body state parameters, wherein the human body state parameters include a resting state and a motion state, and are accompanied by corresponding confidence scores.

4. The sleep detection method according to claim 1, characterized in that: The steps of inputting the face image into the face key point detection model and obtaining the eye activity parameters include: Calculating the closure degree of the left eye and the closure degree of the right eye respectively according to the positions of the facial key points extracted from the facial image by the facial key point detection model; If the closure degree of the left eye and the closure degree of the right eye are both less than a preset eye-closing threshold, it is determined to be an eye-closing state; if the closure degree of the left eye or the closure degree of the right eye is greater than the eye-closing threshold, it is determined to be an eye-opening state; In the eyes-closed state, the continuous closing time is counted; when entering the eyes-open state, the counted eyes-closed time is cleared, and the counted continuous closing time is used as the eye activity parameter.

5. The sleep detection method according to claim 4, characterized in that: The formula for calculating the closure degree of the left eye and the closure degree of the right eye is: Left eye closure degree = (the absolute value of the distance from the left end point of the upper eyelid to the left end point of the lower eyelid + the absolute value of the distance from the right end point of the upper eyelid to the right end point of the lower eyelid) / (2 × the absolute value of the distance from the left end point to the right end point of the left eye); Right eye closure degree = (absolute value of the distance from the left end point of the right upper eyelid to the left end point of the lower eyelid + absolute value of the distance from the right end point of the right upper eyelid to the right end point of the lower eyelid) / (2×absolute value of the distance from the left end point to the right end point of the right eye).

6. The sleep detection method according to claim 1, characterized in that: The steps of inputting the face image into the health monitoring model to obtain the physiological activity parameters include: The health monitoring model outputs physiological activity parameters by analyzing pixel changes of the face image, and the physiological activity parameters include heart rate and respiratory rate; Based on the heart rate and the respiratory rate, the changing trends of the heart rate and the respiratory rate within a preset time period are counted: if the fluctuation range of the heart rate within the preset time period is smaller than the preset heart rate threshold, and the change amplitude of the respiratory rate within the preset time period is smaller than the preset respiratory threshold, it is determined to be a sleeping state; otherwise, it is determined to be an awake state.

7. The sleep detection method according to claim 1, characterized in that: The step of determining whether the target object is in a sleeping state according to at least one of the human body state parameter, the eye activity parameter and the physiological activity parameter comprises: If the human body state parameter indicates a resting state, and the confidence score thereof is greater than a preset first threshold, and the continuous eye closure time indicated by the eye activity parameter is greater than a preset time, it is determined that the target object is in a sleeping state; If the human body state parameter is displayed as a resting state, and the confidence score thereof is greater than a preset second threshold, and the continuous eye closure time displayed by the eye activity parameter is greater than a preset time, and the physiological activity parameter is displayed as a sleeping state, it is determined that the target object is in a sleeping state, wherein the second threshold is less than the first threshold; If the human body state parameter is displayed as a resting state and its accompanying confidence score is greater than a preset third threshold, it is determined that the target object is in a sleeping state, wherein the third threshold is greater than the first threshold.

8. The sleep detection method according to claim 1, characterized in that: The skeleton point information sequence output by the human skeleton point detection model includes at least 17 skeleton point positions of the human body and their visibility and invisibility attributes; the key point position information output by the face key point detection model includes at least 68 key point positions of the face.

9. A sleep detection device, characterized in that: The device comprises: The target detection module is used to input the image frames in the acquired video stream into the target detection model to output the human body frame and face frame of the target object in each frame of the image; An image processing module is used to extract a human body image from the human body frame, and input it into a human skeleton point detection model to obtain a skeleton point information sequence, and input it into a human resting state detection model to obtain human body state parameters; extract a face image from the face frame, and input it into a face key point detection model and a health monitoring model respectively to obtain corresponding eye activity parameters and physiological activity parameters; The sleep determination module is used to determine whether the target object is in a sleep state according to at least one of the human body state parameter, the eye activity parameter and the physiological activity parameter.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the sleep detection method according to any one of claims 1 to 8 are implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the sleep detection method according to any one of claims 1 to 8 are implemented.

Images