Method and apparatus for recognizing sleep in vehicle cabin, electronic device, and medium

Abstract

The present disclosure provides a method and apparatus for recognizing sleep in a vehicle cabin, an electronic device, and a medium. The method comprises: acquiring target images of a vehicle cabin, each target image comprising passengers; on the basis of an eye confidence level determination result of each passenger in the target image, using a sleep state recognition method matching the eye confidence level determination result to determine whether the passenger is in a sleep state, the eye confidence level determination result representing whether the eyes are reliable for sleep recognition; if yes, on the basis of a detection result for a target behavior of the passenger in the target image, determining a verification result for the sleep state of the passenger; and on the basis of the verification result for the sleep state, determining a sleep state recognition result for the passenger in the vehicle cabin, thereby increasing the determination range for sleep recognition, improving the detection accuracy of sleep recognition, and suppressing false detection of the sleep state, thus further improving the detection accuracy of sleep recognition.

Description

A method, device, electronic device, and medium for recognizing sleep in a car cabin.

[0001] Cross-reference to related applications

[0002] This disclosure claims priority to Chinese Patent Application No. 202410931779.2, filed on July 12, 2024, entitled "A method, device, electronic device and medium for recognizing sleep in a car cabin", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of vehicle services, and more specifically, to a method, device, electronic device, and medium for recognizing sleep in a car cabin. Background Technology

[0004] With the continuous development of the automotive industry, the application scenarios of vehicles are becoming more and more diverse, and people's requirements for vehicle services are also increasing. Passengers may sleep inside the vehicle during driving or in leisure settings, requiring the vehicle to be able to recognize the passenger's sleep state and adjust the interior environment accordingly to promote rest and sleep quality.

[0005] Existing sleep recognition technologies determine whether a passenger is asleep by analyzing their movements and behavioral patterns. Typically, sleep behavior is judged based on head posture. However, the results of sleep recognition are affected by the Euler angles of the head posture, as well as the passenger's height and position, which can lead to misjudgments. Furthermore, there is no good sleep recognition solution for situations where the passenger is lying down or reclining and cannot see their eyes.

[0006] Summary of the Invention

[0007] In view of this, the purpose of this disclosure is to provide a method, device, electronic device and medium for recognizing sleep in a car cabin, which can improve the accuracy of sleep recognition for sleeping passengers in a car cabin.

[0008] This disclosure provides a method for recognizing sleep in a car cabin, the method comprising:

[0009] Acquire a target image of the vehicle cabin; the target image includes passengers;

[0010] Based on the confidence score of the passenger's eyes in the target image, a sleep state recognition method that matches the confidence score is used to determine whether the passenger is asleep; the confidence score of the eyes indicates whether the eyes are reliable for sleep recognition.

[0011] If so, the verification result for the passenger's sleep state is determined based on the detection result of the target behavior of the passenger in the target image;

[0012] Based on the verification results for sleep status, the sleep status identification results of passengers in the car cabin are determined.

[0013] In some embodiments, the method for recognizing sleep in a car cabin, wherein determining whether a passenger is asleep using a sleep state recognition method that matches eye confidence judgment results includes:

[0014] When the confidence score of the passenger's eyes in the target image is qualified, it is determined whether the passenger is asleep based on the state of the passenger's eyes in the target image.

[0015] If the confidence score of the passenger's eyes in the target image is unqualified, the passenger's body posture in the target image is used to determine whether the passenger is asleep.

[0016] In some embodiments, the method for recognizing sleep in a car cabin, based on the eye state of a passenger in the target image, determines whether the passenger is asleep, including:

[0017] Based on the eye state of the passenger in the target image, determine the cumulative duration of closed-eye images within a preset first time period;

[0018] Determine whether the cumulative duration of the closed-eye images meets the preset closed-eye duration requirement;

[0019] If so, then the passenger is confirmed to be asleep.

[0020] In some embodiments, the method for recognizing sleep in a car cabin, based on the passenger's body posture in the target image, determines whether the passenger is asleep, including:

[0021] Identify the target human skeleton points of the passenger in the target image;

[0022] Based on the target human skeleton points of the passenger, the head-to-body length ratio and the tilt of the human trunk of the passenger are determined; the head-to-body length ratio represents the length ratio of the passenger's head and torso in the target image; the tilt of the human trunk represents the tilt of the passenger's human trunk in the target image along the horizontal direction of the image.

[0023] Based on the head-to-body length ratio and the tilt of the human body trunk, it is determined whether the passenger is asleep.

[0024] In some embodiments, the method for recognizing sleep in a car cabin, based on the head-to-body length ratio and the tilt of the human trunk, determines whether the passenger is asleep, including:

[0025] Based on the target human skeleton points of the passenger, the passenger's seating area in the cockpit is determined; different seating areas correspond to different head-to-body length ratio conditions and human trunk tilt conditions.

[0026] Determine whether the passenger's head-to-body length ratio meets the head-to-body length ratio conditions corresponding to the seating area, and whether the passenger's trunk inclination meets the trunk inclination conditions corresponding to the seating area.

[0027] If both are true, then the passenger is determined to be asleep.

[0028] In some embodiments, the different head-to-body length ratio conditions and human trunk tilt conditions corresponding to different seat areas are determined based on images of passengers lying down or tilting their trunks in different vehicle models.

[0029] In some embodiments, the in-car sleep recognition method includes different seat areas corresponding to different human trunk tilt conditions, including:

[0030] The condition for the trunk tilt of the passenger on the right side of the first row is: the trunk tilts to the left.

[0031] The condition for the trunk tilt of the passenger on the left side of the first row is: the trunk tilts to the right.

[0032] The condition for the trunk inclination of the second row of passengers is that the trunk inclination meets the preset threshold range.

[0033] In some embodiments, the car cabin sleep recognition method is described in such a way that the condition for the human trunk tilt of the passenger on the right side of the first row is that the tangent of the human trunk tilt is less than 0.

[0034] The condition for the human trunk inclination of the passenger on the left side of the first row is: the tangent of the human trunk inclination is greater than 0.

[0035] The condition for the trunk inclination of the second row of passengers is: the tangent of the trunk inclination is greater than the preset first threshold and less than the preset second threshold.

[0036] In some embodiments, before determining whether a passenger is asleep based on the eye confidence judgment result of the passenger in the target image and using a sleep state recognition method that matches the eye confidence judgment result to determine whether the passenger is asleep, the method further includes:

[0037] The confidence level of the eye assessment is determined based on the angle between the face orientation and the camera imaging direction. The smaller the angle, the higher the confidence level of the eye for sleep recognition; the larger the angle, the lower the confidence level of the eye for sleep recognition.

[0038] In some embodiments, the in-vehicle sleep recognition method determines the eye confidence judgment result based on the angle between the face orientation and the camera imaging direction; including:

[0039] Determine the angle parameter corresponding to the angle between the face orientation and the camera imaging direction;

[0040] When the included angle parameter meets the preset confidence conditions, the eye confidence judgment result is determined to be qualified;

[0041] Conversely, the eye confidence assessment result is determined to be unqualified.

[0042] In some embodiments, the method for recognizing sleep in a car cabin includes determining an angle parameter corresponding to the angle between the face orientation and the camera orientation, including:

[0043] Define the coordinates of the camera coordinate system origin in the human head coordinate system;

[0044] Calculate the first angle between the line connecting the origin of the camera coordinate system and the origin of the head coordinate system and the z-axis of the head coordinate system, and determine the angle parameter corresponding to the first angle; the angle parameter corresponding to the first angle is used for eye confidence judgment.

[0045] Simultaneously calculate the projection of the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system onto the xOy plane of the human head coordinate system and the second included angle of the x-axis of the human head coordinate system, and determine the included angle parameter corresponding to the second included angle.

[0046] In some embodiments, the in-vehicle sleep recognition method determines a verification result for the sleep state based on the detection result of the target behavior of the passenger in the target image, including:

[0047] Determine the motion detection results of the target part and / or the detection results of the use of the target item by the passenger in the target image within a preset second time period;

[0048] Determine whether the motion detection results of the target body part and / or the detection results of the use of the target item meet the corresponding sleep inhibition conditions;

[0049] If any condition is met, the verification result for the sleep state is determined to be sleep suppression.

[0050] In some embodiments, the method for recognizing sleep in a car cabin includes determining the motion detection results of a target part of a passenger in the target image within a preset second time period, including:

[0051] Determine the rotation angle parameter of the passenger's face in the target image within a preset second time period; the rotation angle parameter represents the rotation angle of the face;

[0052] Determine the distance the passenger's arm moves within a preset second time period in the target image.

[0053] In some embodiments, in the in-vehicle sleep recognition method, determining whether the motion detection result of the target part meets the corresponding sleep inhibition condition includes:

[0054] Determine whether the rotation angle parameter of the face is greater than a preset third threshold;

[0055] Determine whether the distance the passenger's arm moves is greater than a preset fourth threshold; wherein, the preset fourth threshold is different for different vehicle models and different seating areas.

[0056] In some embodiments, the method for recognizing sleep in a car cabin includes determining whether the detection result of using a target item meets the corresponding sleep inhibition conditions; including:

[0057] Detect whether there is a target object in the image;

[0058] If so, then based on the contact relationship between the target item and the hand, and the contact relationship between the target item and the preset reference object, it is determined whether the target item is in a handheld state;

[0059] If so, then the test result is determined to be sleep suppression.

[0060] In some embodiments, in the vehicle cabin sleep recognition method, determining the sleep state recognition result of passengers in the vehicle cabin based on the verification result of the sleep state includes:

[0061] When the verification result is "sleep suppression", the passenger's sleep state identification result is updated to "non-sleep state".

[0062] When the verification result is non-suppressed sleep, the passenger's sleep state recognition result remains as sleep state.

[0063] In some embodiments, the sleep recognition method for a car cabin uses the sleep state recognition result of the passenger in the car cabin to adjust the state of at least one device in the car cabin according to a preset sleep scenario.

[0064] In some embodiments, an in-vehicle sleep detection device is also provided, the device comprising:

[0065] The acquisition module is used to acquire a target image of the car cabin; the target image includes passengers;

[0066] The judgment module is used to determine whether the passenger is asleep based on the eye confidence judgment result of the passenger in the target image and a sleep state recognition method that matches the eye confidence judgment result; the eye confidence judgment result indicates whether the eye is reliable for sleep recognition.

[0067] The first determining module is used to determine the verification result of the passenger's sleep state based on the state of the target object in the target image when it is determined that the passenger is in a sleep state.

[0068] The second determining module is used to determine the sleep state identification result of passengers in the car cabin based on the verification result of the sleep state.

[0069] In some embodiments, an electronic device is also provided, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the in-vehicle sleep recognition method are performed.

[0070] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, performs the steps of the in-vehicle sleep recognition method.

[0071] This disclosure provides a method, device, electronic device, and medium for sleep recognition in a car cabin. The method first acquires a target image of the car cabin. Then, based on the reliability of sleep recognition using the passenger's eyes in the target image, different sleep state recognition methods are used to determine whether the passenger is asleep. Furthermore, sleep state detection results are suppressed based on the detection results of the passenger's target behavior in the image. This increases the scope of sleep recognition by using different detection methods, improving the detection accuracy of sleep recognition. Additionally, the detection results based on the passenger's target behavior prevent false detections of sleep states, further improving the detection accuracy of sleep recognition. Attached Figure Description

[0072] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this disclosure and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 shows a flowchart of the in-vehicle sleep recognition method according to an embodiment of the present disclosure;

[0074] Figure 2 shows a schematic diagram of the head coordinate system and camera coordinate system according to an embodiment of the present disclosure;

[0075] Figure 3 shows a flowchart of the method for determining whether a passenger is asleep according to an embodiment of the present disclosure;

[0076] Figure 4 shows a flowchart of the method for determining whether a passenger is asleep based on the passenger's body posture in the target image according to an embodiment of the present disclosure;

[0077] Figure 5 shows the recognition results of human skeleton points in the target image according to an embodiment of this disclosure;

[0078] Figure 6 shows the recognition result of human skeleton points in another target image according to an embodiment of this disclosure;

[0079] Figure 7 shows the recognition result of human skeleton points in another target image according to an embodiment of this disclosure;

[0080] Figure 8 shows the recognition result of human skeleton points in another target image according to an embodiment of this disclosure;

[0081] Figure 9 shows a schematic diagram of the structure of the in-vehicle sleep recognition device according to an embodiment of the present disclosure;

[0082] Figure 10 shows a schematic diagram of the structure of the electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this disclosure are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this disclosure. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this disclosure illustrate operations implemented according to some embodiments of this disclosure. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this disclosure, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0084] Furthermore, the described embodiments are merely some, not all, of the embodiments of this disclosure. The components of the embodiments of this disclosure typically described and illustrated in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the drawings is not intended to limit the scope of the claimed disclosure, but merely to illustrate selected embodiments of the disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0085] It should be noted that the term "comprising" will be used in the embodiments of this disclosure to indicate the presence of the features subsequently declared, but does not exclude the addition of other features.

[0086] Current sleep detection technologies determine whether a passenger is asleep by analyzing their movements and behavioral patterns, typically based on head posture. This approach has low accuracy and lacks a suitable solution for situations where passengers are lying down or reclining and their eyes are not visible. Specifically, existing technologies using head posture to determine sleep behavior usually employ Euler angles. This involves judging whether the passenger's head is resting on their shoulders by checking if the head roll and yaw angles are within a certain range, and whether the head is tilted down by checking if the head pitch angle is within a certain range. However, using only Euler angles can result in multiple solutions for a specific spatial direction, affecting the accuracy of sleep detection. Furthermore, the inconsistent passenger positions within the cabin make it impossible to adapt a single configuration to passengers of different heights and distances, leading to errors in sleep detection. It is particularly important to note that for situations where passengers are lying down or reclining and their eyes are not visible, the inability to accurately identify the open or closed state of the eyes prevents effective sleep detection.

[0087] In addition to sleep recognition based on behavioral identification, sleep can also be recognized based on physiological signal detection. Physiological signal detection is a method of identifying a passenger's sleep state by monitoring physiological parameters, including heart rate, respiration, muscle activity, and sound. This typically requires integration with sensors and equipment within the vehicle. These devices may need to be installed and fixed within the vehicle, increasing cost and complexity. Furthermore, physiological signals are usually weak and easily interfered with by various noises, affecting the accuracy of sleep recognition. Different individuals may exhibit significant differences in physiological signals, such as heart rate and respiratory rate. These individual differences can also affect the accuracy of physiological signal monitoring.

[0088] Based on this, this disclosure provides a method, device, electronic device, and medium for sleep recognition in a car cabin. The method first acquires a target image of the car cabin, then determines whether the passenger is asleep based on the reliability of sleep recognition using the passenger's eyes in the target image, employing different sleep state recognition methods. Next, it suppresses sleep state detection results based on the detection results of the passenger's target behavior in the image, thereby increasing the scope of sleep recognition by using different detection methods, improving the detection accuracy of sleep recognition, and further preventing false detections of sleep state by using the detection results based on the passenger's target behavior, further improving the detection accuracy of sleep recognition.

[0089] Please refer to Figure 1, which shows a flowchart of the vehicle cabin sleep recognition method according to an embodiment of the present disclosure; as shown in Figure 1, the vehicle cabin sleep recognition method includes the following steps S101-S104:

[0090] S101. Obtain a target image of the car cabin; the target image includes passengers;

[0091] S102. Based on the confidence judgment result of the passenger's eyes in the target image, a sleep state recognition method matching the confidence judgment result is used to determine whether the passenger is in a sleep state; the confidence judgment result of the eyes represents whether the eyes are reliable for sleep recognition.

[0092] S103. If so, then based on the detection results of the target behavior of the passenger in the target image, determine the verification result of the passenger's sleep state.

[0093] S104. Based on the verification results for the sleep state, determine the sleep state identification result of the passenger in the car cabin.

[0094] In step S101, a target image of the car cabin is acquired; the target image includes passengers.

[0095] The target image of the vehicle cabin refers to the image captured by the in-vehicle camera inside the cabin. Different vehicle models may have the in-vehicle camera mounted in different locations, but typically, it is located in the center of the cabin (above the center of the windshield). Mounting the camera above the center of the windshield ensures that it has a wide field of view, capturing more areas inside the vehicle and providing a more comprehensive image of the interior, while also ensuring that as many passengers' faces as possible are captured.

[0096] The target image includes passengers. It should be noted that the passengers here include passengers in each seat, that is, both drivers and non-drivers.

[0097] When a car is in motion, the driver is unlikely to be asleep, especially not lying flat or reclining. However, the driver may be in a state of fatigue and doze off. Therefore, the car cabin sleep recognition method described in this embodiment is still necessary to apply to drivers.

[0098] When a car is not in a driving state, that is, a leisure state, passengers in every seat (including the driver's seat) may be in a lying or reclining sleeping state.

[0099] Since vehicle-mounted cameras typically capture in-vehicle video, the target image, i.e., the image frame in the video captured by the vehicle-mounted camera inside the car cabin, can also be referred to as an in-vehicle video image.

[0100] The target image can be a portion of the image frames within the vehicle-mounted video.

[0101] For example, in some embodiments, the target image is an image frame in the vehicle video that meets preset image quality conditions; that is, only those frames that meet specific image quality standards will be selected as the target image.

[0102] The preset image quality conditions can be such as sharpness greater than a preset sharpness threshold, image brightness greater than a preset brightness threshold, noise lower than a preset noise threshold, etc.

[0103] For example, in some embodiments, the target image is an image frame extracted from the vehicle video according to a preset frame interval; for example, the frame rate of the vehicle camera may be 30fps or 60fps, and the computing resources required to process dozens of images per second are large, which reduces the real-time performance of sleep recognition. Therefore, a portion of the image frames can be used for sleep recognition at intervals. For example, if the preset frame interval is 5 frames, then one image will be extracted as the target image from every 5 image frames, and 6 frames will be extracted as the target image per second.

[0104] In some embodiments, the target image may be a combination of an image frame in the vehicle video that meets a preset image quality condition and an image frame extracted from the vehicle video at preset intervals.

[0105] Specifically, firstly, each image frame in the in-vehicle video is analyzed in real time to determine whether it meets preset image quality conditions. For image frames that meet the preset image quality conditions, they are then extracted according to a preset frame interval. The final target image not only meets the image quality requirements but also follows the preset frame interval. This combined strategy ensures that high-quality and representative in-vehicle video image frames are obtained with limited resources, providing strong support for subsequent applications.

[0106] In some embodiments, when the vehicle control system is configured to an automatic sleep recognition mode, sleep recognition is automatically performed when a target image of the vehicle cabin is acquired.

[0107] In other words, the automatic sleep recognition mode of the vehicle control system can be turned on or off.

[0108] In step S102, based on the confidence judgment result of the passenger's eyes in the target image, a sleep state recognition method matching the confidence judgment result is used to determine whether the passenger is in a sleep state; the confidence judgment result of the eyes characterizes whether the eyes are reliable for sleep recognition.

[0109] Determine whether the confidence level of the passenger's eyes in the target image meets the preset confidence level conditions. If it does, the confidence level judgment result is determined to be qualified; otherwise, the confidence level judgment result is determined to be unqualified.

[0110] The confidence level of the eyes is determined based on the angle between the face orientation and the camera imaging direction; the smaller the angle between the face orientation and the camera imaging direction, the higher the confidence level of the eyes for sleep recognition; the larger the angle between the face orientation and the camera imaging direction, the lower the confidence level of the eyes for sleep recognition.

[0111] The angle between the face orientation and the camera imaging direction refers to the angle in three-dimensional space. For example, if the camera takes a picture from the side of the passenger's face or from above the passenger's head, the passenger's eyes are basically invisible in the image. In this case, using the eyes for sleep recognition is unreliable.

[0112] When there are multiple passengers in the target image, it is necessary to determine whether the passenger is asleep by using a sleep state recognition method that matches the eye confidence judgment result for each passenger.

[0113] In other words, when there are multiple passengers in the target image, the same sleep state recognition method is not used for all of them; the eye confidence judgment results of different passengers are different, and the corresponding sleep state recognition methods are different.

[0114] For example, if the target image contains passenger A and passenger B, with passenger A showing a frontal view and passenger B showing a side view, then different sleep state recognition methods can be used to determine whether passenger A and passenger B are in a sleep state.

[0115] Based on this, in the car cabin sleep recognition method described in this embodiment, before determining whether the passenger is asleep by using a sleep state recognition method that matches the eye confidence judgment result based on the passenger's eye confidence judgment result in the target image, the method further includes:

[0116] The confidence level of the eye assessment is determined based on the angle between the face orientation and the camera imaging direction. The smaller the angle, the higher the confidence level of the eye for sleep recognition; the larger the angle, the lower the confidence level of the eye for sleep recognition.

[0117] Specifically, the confidence level of the eye assessment is determined based on the angle between the face orientation and the camera imaging direction; including:

[0118] Determine the angle parameter corresponding to the angle between the face orientation and the camera imaging direction;

[0119] When the included angle parameter meets the preset confidence conditions, the eye confidence judgment result is determined to be qualified;

[0120] Conversely, the eye confidence assessment result is determined to be unqualified.

[0121] The face orientation refers to the direction of the face, which is determined based on the head posture; and the angle between the face orientation and the camera imaging direction is also related to the position of the passenger's head.

[0122] When detecting the head position and posture of a passenger, the head position can be represented as the coordinates of the head relative to the camera, and the head posture can be represented as the rotation relationship of the head coordinate system relative to the camera coordinate system. Please refer to Figure 2, which shows a schematic diagram of the head coordinate system and camera coordinate system according to an embodiment of this disclosure.

[0123] In this embodiment of the disclosure, when defining the passenger's head posture, two angle parameters are used to define the angle between the face orientation and the camera imaging direction.

[0124] Specifically, in the car cabin sleep recognition method described in this embodiment, determining the angle parameter corresponding to the angle between the face orientation and the camera orientation includes:

[0125] Define the coordinates of the camera coordinate system origin in the human head coordinate system;

[0126] Calculate the first angle between the line connecting the origin of the camera coordinate system and the origin of the head coordinate system and the z-axis of the head coordinate system, and determine the angle parameter corresponding to the first angle; the angle parameter corresponding to the first angle is used for eye confidence judgment.

[0127] Simultaneously calculate the projection of the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system onto the xOy plane of the human head coordinate system and the second included angle of the x-axis of the human head coordinate system, and determine the included angle parameter corresponding to the second included angle.

[0128] Specifically, by defining the coordinates of the origin of the camera coordinate system in the head coordinate system, the relative positional relationship between the passenger's head and the camera is represented.

[0129] Please refer to Figure 2, where the origin of the camera coordinate system is the optical center of the camera, the x-axis and y-axis are parallel to the X and Y axes of the image, and the z-axis points towards the optical axis of the camera and is perpendicular to the image plane. Specifically, since the image coordinate system has the x-axis to the right and the y-axis downwards, the camera coordinate system has its origin at the center of the lens's principal optical axis, with the positive x-direction to the right, the positive y-direction downwards, and the positive z-direction forwards. In this way, the x and y directions coincide with the directions of the image coordinate system, the z-direction represents the depth of field, and it also conforms to the definition of a right-handed coordinate system, facilitating vector calculations in the algorithm.

[0130] In summary, the z-axis of the camera coordinate system is aligned with the optical axis of the camera, both pointing in front of the camera.

[0131] The head coordinate system is defined as follows: the origin of the head coordinate system is located at the center of the head; the X-axis is parallel to the line connecting the left and right eyes, the Y-axis is parallel to the line connecting the end of the nose and the end of the chin, and the Z-axis is perpendicular to the X-axis and Y-axis, representing the direction of the face.

[0132] Based on this, the angle between the line connecting the origin of the camera coordinate system and the origin of the head coordinate system and the first angle of the z-axis of the head coordinate system can characterize whether the eye is reliable for sleep recognition.

[0133] The included angle parameter is used to characterize the size, direction, and other attributes of the included angle. It can be an angle or the trigonometric function value corresponding to the angle.

[0134] The following details the process of calculating the first angle between the line connecting the origin of the camera coordinate system and the origin of the head coordinate system and the z-axis of the head coordinate system, and the second angle between the projection of the line connecting the origin of the camera coordinate system and the origin of the head coordinate system onto the xOy plane of the head coordinate system and the x-axis of the head coordinate system. For ease of explanation, in this embodiment, the first angle between the line connecting the origin of the camera coordinate system and the origin of the head coordinate system and the z-axis of the head coordinate system is referred to as latitude; the second angle between the projection of the line connecting the origin of the camera coordinate system and the origin of the head coordinate system onto the xOy plane of the head coordinate system and the x-axis of the head coordinate system is referred to as longitude.

[0135] The second angle between the projection of the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system onto the xOy plane of the human head coordinate system and the x-axis of the human head coordinate system, i.e., the second angle between the projection of the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system onto the xOy plane of the human head coordinate system and the x-axis of the human head coordinate system.

[0136] Specifically, the coordinates of the origin of the camera coordinate system in the human head coordinate system are defined as (x, y, z). Then, (x, y, z) are normalized to obtain a normalized vector.

[0137] Specifically,

[0138] By normalizing vectors This allows us to obtain the representations of longitude and latitude; specifically, longitude = latitude

[0139] As can be seen from Figure 2, the The projection of the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system onto the xOy plane of the human head coordinate system and the second angle between the line and the x-axis of the human head coordinate system; The line connecting the origin of the camera coordinate system and the origin of the human head coordinate system, and the second included angle of the z-axis of the human head coordinate system.

[0140] According to the definitions of latitude and longitude, when a person's face is facing the camera, the camera is located on the z-axis of the head coordinate system, and the normalized vector is represented as... At this point, latitude = 0 and longitude = 0. Similarly, when the camera is located on the positive x-axis of the head coordinate system, latitude = 90 and longitude = 0; when the camera is located on the positive y-axis of the head coordinate system, latitude = 90 and longitude = 90. Based on the calculation method of latitude and longitude, this is used as the basis for judging eye quality. The smaller the latitude, the smaller the angle between the face orientation and the camera imaging direction, and the higher the confidence level of using the eye for quality judgment. Conversely, the larger the latitude, the more the eye is affected by the side of the face, making the eye invisible and thus lowering the confidence level.

[0141] However, using only Euler angles for judgment can lead to multiple different Euler angle solutions for a specific spatial direction. Furthermore, since passengers are in different positions within the cabin, a single configuration cannot be used to accommodate passengers of varying heights and distances, requiring different Euler angle configurations for each position. Correspondingly, this embodiment proposes using latitude and longitude to analyze eye visibility. The same configuration constraint can be used for latitude and longitude at different positions within the cabin, improving computational efficiency.

[0142] Please refer to Figure 3, which shows a flowchart of the method for determining whether a passenger is asleep according to an embodiment of this disclosure; as shown in Figure 3, in this embodiment of the disclosure, the method for determining whether a passenger is asleep using a sleep state recognition method that matches the eye confidence judgment result includes the following steps S301-S302:

[0143] S301. When the confidence judgment result of the passenger's eyes in the target image is qualified, determine whether the passenger is in a sleeping state based on the state of the passenger's eyes in the target image.

[0144] S302. When the confidence judgment result of the passenger's eyes in the target image is unqualified, determine whether the passenger is asleep based on the passenger's body posture in the target image.

[0145] In other words, the sleep state recognition method includes two types: a sleep recognition method based on eye state and a sleep recognition method based on body posture.

[0146] The eye states include both open and closed.

[0147] Specifically, determining whether a passenger is asleep based on the state of their eyes in the target image includes:

[0148] Based on the eye state of the passenger in the target image, determine the cumulative duration of closed-eye images within a preset first time period;

[0149] Determine whether the cumulative duration of the closed-eye images meets the preset closed-eye duration requirement;

[0150] If so, then the passenger is confirmed to be asleep.

[0151] The requirement of meeting the preset eye-closed duration is that the cumulative duration of the eye-closed image is greater than the preset eye-closed duration threshold.

[0152] In some embodiments, the cumulative duration of closed-eye images within a preset first time period is determined based on the number of closed-eye images and the sampling duration.

[0153] Based on this, the number of closed-eye images with the eyes closed within the preset first time period can also be determined.

[0154] Determine whether the number of the closed-eye images meets the preset quantity requirement;

[0155] If so, then the passenger is confirmed to be asleep.

[0156] The number that meets the preset requirement can be greater than the first preset number threshold.

[0157] Specifically, after identifying the eye state of the passenger in the target image, the eye state identification result of the current frame target image is updated to the sleep window; the window duration of the sleep window is a preset first time period.

[0158] When the number of target images with closed eyes in the sleep window meets a preset requirement, the passenger is determined to be in a sleep state.

[0159] Based on the sleep window, it is possible to determine more promptly whether a passenger is asleep.

[0160] Identify the eye state of passengers in the target image; specifically, use an eye-open / closed classification model to identify the eye state of passengers in the target image.

[0161] In other words, the target image is input into the eye-opening / closing classification model, and the eye-opening / closing classification model outputs the passenger's eye-opening / closing information.

[0162] Please refer to Figure 4, which shows a flowchart of the method for determining whether a passenger is asleep based on the passenger's body posture in the target image according to an embodiment of the present disclosure; as shown in Figure 4, determining whether a passenger is asleep based on the passenger's body posture in the target image includes the following steps S401-S403:

[0163] S401. Identify the target human skeleton points of the passenger in the target image;

[0164] S402. Based on the target human skeleton points of the passenger, determine the head-to-body length ratio and the tilt of the human body trunk of the passenger; the head-to-body length ratio represents the length ratio of the passenger's head and torso in the target image; the tilt of the human body trunk represents the tilt of the passenger's human body trunk in the target image along the horizontal direction of the image.

[0165] S403. Based on the head-to-body length ratio and the tilt of the human body trunk, determine whether the passenger is asleep.

[0166] Human skeletal points, also known as skeletal keypoints, generally refer to specific locations used in human pose recognition to describe human posture. These points are typically located at joints, feature areas, etc., and are used to construct the skeletal model of the human body. Taking a common pose recognition system as an example, skeletal keypoints include: top of the head (the highest point of the head); neck (the top of the cervical vertebrae); shoulders (including the left and right shoulders, located at the left and right shoulder joints respectively); elbows (including the left and right elbows, located at the left and right elbow joints respectively); wrists (including the left and right wrists, located at the left and right wrist joints respectively); and hips (including the left and right hips, located at the left and right hip joints respectively).

[0167] It should be noted that, in this embodiment of the disclosure, due to the limitations of space in the cockpit and camera position, it is usually only necessary to identify the human skeletal points of the upper body.

[0168] The main body of the human body includes the head and torso.

[0169] The horizontal direction of the target image is from left to right, and the origin of the coordinate system is the upper left corner of the target image.

[0170] Please refer to Figures 5, 6, 7, and 8. Figure 5 shows the recognition result of human skeletal points in the target image according to an embodiment of the present disclosure; Figure 6 shows the recognition result of human skeletal points in another target image according to an embodiment of the present disclosure; Figure 7 shows the recognition result of human skeletal points in another target image according to an embodiment of the present disclosure; and Figure 8 shows the recognition result of human skeletal points in another target image according to an embodiment of the present disclosure.

[0171] As shown in Figure 5 above, the head length of the person sitting on the left side of the front row is L1, and the torso length is L2, with a length ratio of approximately 1:1. The head length of the person lying on the right side of the front row is L3, and the torso length is L4, with a significantly smaller length ratio between L3 and L4. The ratio of head length to torso length in the target image is called the head-to-body ratio (head_body_ratio). This ratio is used to determine whether the person in the front row is sitting or lying down, thus indicating whether the passenger is sleeping. A head-to-body ratio threshold, th_head_body_ratio, is set; a head-to-body ratio less than th_head_body_ratio is considered to meet criterion 1.

[0172] Taking Figure 6 above as an example, for the person on the right side of the front row, when lying down, the main body (marked by the line segment from the top of the head to the middle of the left and right hip joints) tilts to the left. The angle between the white line and the X-axis of the target image (horizontally to the right at the bottom) is called body_ang, which is the tilt of the main body. When the main body tilts to the left, tan(body_ang) < 0; when sitting, the main body tilts to the right, tan(body_ang) > 0. For the person on the left side of the front row, it is exactly the opposite: when lying down, the main body tilts to the right (tan(body_ang) > 0); when sitting, the main body tilts to the left (tan(body_ang) < 0). Therefore, tan(body_ang) < 0 on the right side of the front row and tan(body_ang) > 0 on the left side of the front row is called satisfying index 2. Index 2 can also determine whether the person in the front row is sitting or lying down.

[0173] To reduce misjudgment, a person is considered to be asleep if they meet both Indicator 1 and Indicator 2.

[0174] For the people in the back row, please refer to FIGS. 7 and 8. It is possible to determine whether a person is sleeping in a reclined position by the body inclination angle body_ang. Preset the first threshold th_back_left and the second threshold th_back_right to define the preset threshold range of the body inclination angle. When th_back_left < tan(body_ang) < th_back_right, it is considered that this person is in a sleeping state.

[0175] At the same time, as can be seen from FIGS. 7 and 8, for the people in the back row, the head-trunk length ratio when sitting is greater than the head-trunk length ratio when reclined.

[0176] Based on the exemplary analysis of FIGS. 5-8 above, when determining whether the passenger is in a sleeping state based on the head-trunk length ratio and the body inclination angle, the head-trunk length ratio conditions and the body inclination angle conditions corresponding to different seat areas are different.

[0177] The seat areas include the left side of the first row, the right side of the first row, and the second row.

[0178] Based on this, in the embodiments of the present disclosure, when determining whether the passenger is in a sleeping state based on the head-trunk length ratio and the body inclination angle, it includes:

[0179] Based on the target human body bone points of the passenger, determine the seat area of the passenger in the cockpit;

[0180] Judge whether the head-trunk length ratio of the passenger meets the head-trunk length ratio condition corresponding to the seat area, and whether the body inclination angle of the passenger meets the body inclination angle condition corresponding to the seat area;

[0181] If both are yes, it is determined that the passenger is in a sleeping state.

[0182] Since the positions of the seats in the car cabin in the target image are basically unchanged, the positions of the left and right hip joints of the passenger in the target image are within a certain range.

[0183] Even if the positions of the seats in the car cabin can be adjusted forward and backward, the positions of the left and right hip joints of the passenger in the target image are still within a certain range.

[0184] Therefore, when determining the seat area of the passenger in the cockpit based on the target human body bone points of the passenger, specifically, according to the positions of the target human body bone points of the passenger in the target image and the position ranges of different seat areas in the target image, determine the seat area of the passenger in the cockpit.

[0185] The target human body bone points are the bone points of the left and right hip joints of the passenger.

[0186] Different seating areas correspond to different human trunk tilt conditions, including:

[0187] The condition for the trunk tilt of the passenger on the right side of the first row is: the trunk tilts to the left.

[0188] The condition for the trunk tilt of the passenger on the left side of the first row is: the trunk tilts to the right.

[0189] The condition for the trunk inclination of the second row of passengers is that the trunk inclination meets the preset threshold range.

[0190] The human trunk tilt condition is quantified for easier calculation. Specifically, the human trunk tilt condition for the passenger on the right side of the first row is: the tangent of the human trunk tilt is less than 0; that is, tan(body_ang) < 0.

[0191] The condition for the body trunk inclination of the passenger on the left side of the first row is: the tangent of the body trunk inclination is greater than 0; that is, tan(body_ang)>0.

[0192] The condition for the trunk inclination of passengers in the second row is: the tangent of the trunk inclination is greater than a preset first threshold and less than a preset second threshold; that is, th_back_left <tan(body_ang)<th_back_right。

[0193] The term body_ang refers to the tilt angle of the human body's main trunk.

[0194] Specifically, the different head-to-body length ratio conditions and human trunk tilt conditions corresponding to different seating areas are determined based on images of passengers lying down or with their trunks tilted in different vehicle models.

[0195] Parameters such as the interior space of the car cabin, the position of the camera, the internal parameters of the camera, and the position of the seat in different car models will affect the head-to-body length ratio and the tilt of the human body in the target image captured by the camera; therefore, the head-to-body length ratio conditions and the human body tilt conditions are related to the corresponding car model.

[0196] Specifically, for this type of car, multiple images of passengers of different heights, weights, and body shapes lying down or with their torsos tilted can be used to determine the different head-to-body length ratios and torso tilt conditions corresponding to different seating areas.

[0197] In step S103, when it is determined whether the passenger is asleep, a verification result for the passenger's sleep state is determined based on the detection result of the target behavior of the passenger in the target image.

[0198] If passengers can still move, such as moving their arms, turning their heads, or playing on their phones, reading, or eating snacks, it means that the passengers are not asleep.

[0199] Therefore, in this embodiment of the disclosure, some target behaviors are set. When a passenger is detected to have performed the target behaviors, it is determined that the passenger is in a non-sleep state, that is, the verification result is inconsistent with sleep.

[0200] The detection results of the passenger's target behavior include two categories: target behavior detected and target behavior not detected.

[0201] In this embodiment of the disclosure, the target behavior is divided into two types, and two types of detection strategies are formulated; the target behavior includes the movement of the target part of the passenger and the use of the target item by the passenger.

[0202] Specifically, determining the verification result for the sleep state based on the detection results of the target behavior of the passenger in the target image includes:

[0203] Determine the motion detection results of the target part and / or the detection results of the use of the target item by the passenger in the target image within a preset second time period;

[0204] Determine whether the motion detection results of the target body part and / or the detection results of the use of the target item meet the corresponding sleep inhibition conditions;

[0205] If any condition is met, the verification result for the sleep state is determined to be sleep suppression.

[0206] The target body part refers to a location within the cockpit where significant movements can be performed and which is easily detected, such as head rotation or arm movement. Conversely, leg movements cannot be detected, and the torso cannot move significantly on its own (torso movement would cause arm movement).

[0207] Based on this, in this embodiment of the disclosure, the target parts are specifically the hands and the head.

[0208] Specifically, determining the motion detection results of the target part of the passenger in the target image within a preset second time period includes:

[0209] Determine the rotation angle parameter of the passenger's face in the target image within a preset second time period; the rotation angle parameter represents the rotation angle of the face;

[0210] Determine the distance the passenger's arm moved in the target image.

[0211] In this embodiment of the disclosure, the rotation angle parameter of the face specifically represents the change in the angle between the face's orientation and the camera's orientation. Specifically, it is determined based on longitude and latitude.

[0212] Specifically, when calculating longitude and latitude based on eye confidence, these values ​​are added to the head posture time window to determine whether the head posture has changed significantly within that time window. A set of preset third thresholds, th_rotate_lat and th_rotate_long, are set; that is, if latitude > th_rotate_lat or longitude > th_rotate_long, it indicates that the head posture has changed significantly.

[0213] The distance the passenger's arm moves in the target image is determined, and the arm movement information is used to suppress sleep. The arm position is added to the arm movement time window to determine whether the arm moves significantly within a preset second time period.

[0214] Specifically, the preset fourth threshold for the front arm is set to th_front_hand_distance, and the preset fourth threshold for the back arm is set to th_back_hand_distance; that is, if the distance the front arm moves is greater than th_front_hand_distance or the distance the back arm moves is greater than th_back_hand_distance, it indicates that the arm has moved significantly within the preset second time period.

[0215] The front and rear arms correspond to different preset fourth thresholds because the distances from the front and rear arms to the camera are different, and their sizes in the image are different. Therefore, the same distance moved in real space corresponds to different distances changed in the image.

[0216] The determination of whether the motion detection results of the target area meet the corresponding sleep inhibition conditions includes:

[0217] Determine whether the rotation angle parameter of the face is greater than a preset third threshold;

[0218] Determine whether the distance the passenger's arm moves is greater than a preset fourth threshold; wherein, the preset fourth threshold is different for different vehicle models and different seating areas.

[0219] It should be noted that the position of the passenger's arm in the target image is determined based on the corresponding human skeletal points, such as the elbow and wrist bones. In other words, the identified human skeletal points are used not only for sleep state recognition but also for detecting the passenger's target behavior in the target image, achieving multiple functions with a single detection.

[0220] The determination of whether the motion detection results of the target area meet the corresponding sleep inhibition conditions includes:

[0221] Determine whether the rotation angle parameter of the face is greater than a preset third threshold;

[0222] Determine whether the distance the passenger's arm moves is greater than a preset fourth threshold; wherein, the preset fourth threshold is different for different vehicle models and different seating areas.

[0223] Correspondingly, if any judgment result is yes, then the verification result for the sleep state is determined to be sleep suppression; if all judgment results are no, then the verification result for the sleep state is determined to be non-sleep suppression.

[0224] In this embodiment of the disclosure, sleep is also suppressed by a person using the target item.

[0225] Specifically, in the aforementioned in-car sleep recognition method, determining whether the detection result of using the target item meets the corresponding sleep inhibition conditions includes:

[0226] Detect whether there is a target object in the image;

[0227] If so, then based on the contact relationship between the target item and the hand, and the contact relationship between the target item and the preset reference object, it is determined whether the target item is in a handheld state;

[0228] If so, then the test result is determined to be sleep suppression.

[0229] The target item can be a mobile phone, food, books, tablets, etc.

[0230] The preset reference object can be the seat surface.

[0231] Based on the contact relationship between the target item and the hand, it is determined whether the target item is held in the hand; based on the contact relationship between the target item and the preset reference object, it is distinguished whether the target item is placed on the preset reference object and simultaneously comes into contact with the hand, or whether the item is used by hand.

[0232] Therefore, based on the contact relationship between the target item and the hand, and the contact relationship between the target item and the preset reference object, it is possible to more accurately determine whether the target item is being held in hand, that is, to more accurately determine whether the passenger is using the target item.

[0233] The contact relationship between the target item and the hand is determined based on the position of the hand skeletal points and the position of the target item; if the positional difference is less than a preset fifth threshold, then the target item and the hand are in contact.

[0234] The target item and the preset reference object are determined based on the positions of the preset reference object and the target item. If the positional difference is less than the preset sixth threshold, it is determined that the target item and the preset reference object are in contact.

[0235] For example, when using information about a person using their mobile phone to suppress sleep, the system first detects whether a mobile phone is being held in the target image (the hand is in contact with the phone). If so, it determines whether the phone is placed on a chair; otherwise, sleep is suppressed.

[0236] In step S104, the sleep state identification result of the passenger in the car cabin is determined based on the verification result of the sleep state.

[0237] Specifically, determining the sleep state identification result of passengers in the car cabin based on the verification result of the sleep state includes:

[0238] When the verification result is "sleep suppression", the passenger's sleep state identification result is updated to "non-sleep state".

[0239] When the verification result is non-suppressed sleep, the passenger's sleep state recognition result remains as sleep state.

[0240] Based on the verification of the sleep state, some false sleep detections were suppressed, and the accuracy of sleep state recognition was improved.

[0241] In the car cabin sleep recognition method described in this embodiment, the sleep state recognition result of the passenger in the car cabin is used to adjust the state of at least one device in the car cabin according to a preset sleep scenario.

[0242] The equipment includes: seat angle, vehicle volume, interior lighting, temperature control devices (such as vehicle air conditioning), etc.

[0243] Specifically, when the sleep state recognition result indicates that the passenger is in a sleep state, a sleep signal is generated and sent to the device control module. After receiving the sleep signal, the device control module adjusts the device state according to the preset sleep scenario, such as adjusting the seat angle, vehicle volume, interior lighting, and temperature of the temperature control device.

[0244] Based on the same inventive concept, this disclosure also provides a device for recognizing sleep in the car cabin, which is similar to the car cabin sleep recognition method described above. Since the principle of the device in this disclosure is similar to that of the car cabin sleep recognition method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0245] Please refer to Figure 9, which shows a structural schematic diagram of the in-vehicle sleep recognition device according to an embodiment of the present disclosure; the device includes:

[0246] Acquisition module 901 is used to acquire a target image of a car cabin; the target image includes passengers;

[0247] The judgment module 902 is used to determine whether the passenger is in a sleep state based on the eye confidence judgment result of the passenger in the target image and using a sleep state recognition method that matches the eye confidence judgment result; the eye confidence judgment result indicates whether the eye is reliable for sleep recognition.

[0248] The first determining module 903 is used to determine the verification result of the passenger's sleep state based on the state of the target object in the target image when it is determined that the passenger is in a sleep state.

[0249] The second determining module 904 is used to determine the sleep state identification result of the passenger in the car cabin based on the verification result of the sleep state.

[0250] In some embodiments, in the in-vehicle cabin sleep recognition method, when the judgment module determines whether the passenger is asleep using a sleep state recognition method that matches the eye confidence judgment result, it is specifically used for:

[0251] When the confidence score of the passenger's eyes in the target image is qualified, it is determined whether the passenger is asleep based on the state of the passenger's eyes in the target image.

[0252] If the confidence score of the passenger's eyes in the target image is unqualified, the passenger's body posture in the target image is used to determine whether the passenger is asleep.

[0253] In some embodiments, in the in-vehicle sleep recognition device, when the determining module determines whether a passenger is asleep based on the passenger's eye state in the target image, it is specifically used for:

[0254] Based on the eye state of the passenger in the target image, determine the cumulative duration of closed-eye images within a preset first time period;

[0255] Determine whether the cumulative duration of the closed-eye images meets the preset closed-eye duration requirement;

[0256] If so, then the passenger is confirmed to be asleep.

[0257] In some embodiments, in the in-vehicle sleep recognition device, when the determination module determines whether the passenger is asleep based on the passenger's body posture in the target image, it is specifically used for:

[0258] Identify the target human skeleton points of the passenger in the target image;

[0259] Based on the target human skeleton points of the passenger, the head-to-body length ratio and the tilt of the human trunk of the passenger are determined; the head-to-body length ratio represents the length ratio of the passenger's head and torso in the target image; the tilt of the human trunk represents the tilt of the passenger's human trunk in the target image along the horizontal direction of the image.

[0260] Based on the head-to-body length ratio and the tilt of the human body trunk, it is determined whether the passenger is asleep.

[0261] In some embodiments, in the in-vehicle sleep recognition device, the determination module, when determining whether the passenger is asleep based on the head-to-body length ratio and the tilt of the human trunk, is specifically used for:

[0262] Based on the target human skeleton points of the passenger, the passenger's seating area in the cockpit is determined; different seating areas correspond to different head-to-body length ratio conditions and human trunk tilt conditions.

[0263] Determine whether the passenger's head-to-body length ratio meets the head-to-body length ratio conditions corresponding to the seating area, and whether the passenger's trunk inclination meets the trunk inclination conditions corresponding to the seating area.

[0264] If both are true, then the passenger is determined to be asleep.

[0265] In some embodiments, the different head-to-body length ratio conditions and human trunk tilt conditions corresponding to different seat areas in the car cabin sleep recognition device are determined based on images of passengers lying down or tilting their trunks in different car models.

[0266] In some embodiments, in the car cabin sleep recognition device, the different seating areas correspond to different human trunk tilt conditions, including:

[0267] The condition for the trunk tilt of the passenger on the right side of the first row is: the trunk tilts to the left.

[0268] The condition for the trunk tilt of the passenger on the left side of the first row is: the trunk tilts to the right.

[0269] The condition for the trunk inclination of the second row of passengers is that the trunk inclination meets the preset threshold range.

[0270] In some embodiments, in the car cabin sleep recognition device, the condition for the trunk tilt of the human body corresponding to the passenger on the right side of the first row is: the tangent of the trunk tilt is less than 0.

[0271] The condition for the human trunk inclination of the passenger on the left side of the first row is: the tangent of the human trunk inclination is greater than 0.

[0272] The condition for the trunk inclination of the second row of passengers is: the tangent of the trunk inclination is greater than the preset first threshold and less than the preset second threshold.

[0273] In some embodiments, in the in-vehicle sleep recognition device, the judgment module is further configured to determine the eye confidence judgment result based on the angle between the face orientation and the camera imaging direction before determining whether the passenger is in a sleep state based on the eye confidence judgment result of the passenger in the target image and using a sleep state recognition method that matches the eye confidence judgment result; the smaller the angle between the face orientation and the camera imaging direction, the higher the confidence of the eyes for sleep recognition; the larger the angle between the face orientation and the camera imaging direction, the lower the confidence of the eyes for sleep recognition.

[0274] In some embodiments, in the in-car sleep recognition device, the judgment module, when determining the eye confidence judgment result based on the angle between the face orientation and the camera imaging direction, is specifically used for:

[0275] Determine the angle parameter corresponding to the angle between the face orientation and the camera imaging direction;

[0276] When the included angle parameter meets the preset confidence conditions, the eye confidence judgment result is determined to be qualified;

[0277] Conversely, the eye confidence assessment result is determined to be unqualified.

[0278] In some embodiments, in the in-car sleep recognition device, the determining module, when determining the angle parameter corresponding to the angle between the face orientation and the camera orientation, is specifically used for:

[0279] Define the coordinates of the camera coordinate system origin in the human head coordinate system;

[0280] Calculate the first angle between the line connecting the origin of the camera coordinate system and the origin of the head coordinate system and the z-axis of the head coordinate system, and determine the angle parameter corresponding to the first angle; the angle parameter corresponding to the first angle is used for eye confidence judgment.

[0281] Simultaneously calculate the projection of the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system onto the xOy plane of the human head coordinate system and the second included angle of the x-axis of the human head coordinate system, and determine the included angle parameter corresponding to the second included angle.

[0282] In some embodiments, in the in-vehicle sleep recognition device, the first determining module, when determining the verification result for the sleep state based on the detection result of the target behavior of the passenger in the target image, is specifically used for:

[0283] Determine the motion detection results of the target part and / or the detection results of the use of the target item by the passenger in the target image within a preset second time period;

[0284] Determine whether the motion detection results of the target body part and / or the detection results of the use of the target item meet the corresponding sleep inhibition conditions;

[0285] If any condition is met, the verification result for the sleep state is determined to be sleep suppression.

[0286] In some embodiments, in the car cabin sleep recognition device, the first determining module, when determining the motion detection result of the passenger's target part in the target image within a preset second time period, is specifically used for:

[0287] Determine the rotation angle parameter of the passenger's face in the target image within a preset second time period; the rotation angle parameter represents the rotation angle of the face;

[0288] Determine the distance the passenger's arm moves within a preset second time period in the target image.

[0289] In some embodiments, in the in-vehicle sleep recognition device, the first determining module, when determining whether the motion detection result of the target part meets the corresponding sleep inhibition condition, is specifically used for:

[0290] Determine whether the rotation angle parameter of the face is greater than a preset third threshold;

[0291] Determine whether the distance the passenger's arm moves is greater than a preset fourth threshold; wherein, the preset fourth threshold is different for different vehicle models and different seating areas.

[0292] In some embodiments, in the in-car sleep recognition device, the first determining module, when determining whether the detection result of using the target item meets the corresponding sleep inhibition condition, is specifically used for:

[0293] Detect whether there is a target object in the image;

[0294] If so, then based on the contact relationship between the target item and the hand, and the contact relationship between the target item and the preset reference object, it is determined whether the target item is in a handheld state;

[0295] If so, then the test result is determined to be sleep suppression.

[0296] In some embodiments, in the in-vehicle cabin sleep recognition device, the second determining module, when determining the sleep state recognition result of a passenger in the vehicle cabin based on the verification result for the sleep state, is specifically used for:

[0297] When the verification result is "sleep suppression", the passenger's sleep state identification result is updated to "non-sleep state".

[0298] When the verification result is non-suppressed sleep, the passenger's sleep state recognition result remains as sleep state.

[0299] In some embodiments, the sleep state recognition result of the passenger in the car cabin is used to adjust the state of at least one device in the car cabin according to a preset sleep scenario in the car cabin.

[0300] Based on the same inventive concept, this disclosure also provides an electronic device corresponding to the car cabin sleep recognition method. Since the principle of the electronic device in this disclosure is similar to the car cabin sleep recognition method described above, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be described again.

[0301] Please refer to Figure 10, which shows a schematic diagram of the structure of the electronic device according to an embodiment of the present disclosure. The electronic device 1000 includes a processor 1002, a memory 1001, and a bus. The memory 1001 stores machine-readable instructions executable by the processor 1002. When the electronic device 1000 is running, the processor 1002 communicates with the memory 1001 through the bus. When the machine-readable instructions are executed by the processor 1002, the steps of the in-vehicle sleep recognition method are performed.

[0302] Based on the same inventive concept, this disclosure also provides a computer-readable storage medium corresponding to the car cabin sleep recognition method. Since the principle of the computer-readable storage medium in this disclosure is similar to the car cabin sleep recognition method described above, the implementation of the computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be described again.

[0303] A computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the aforementioned in-vehicle sleep recognition method.

[0304] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or modules may be electrical, mechanical, or other forms.

[0305] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0306] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0307] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, platform server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0308] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims. Industrial applicability

[0309] This disclosure provides a method, device, electronic device, and medium for sleep recognition in a car cabin. The method first acquires a target image of the car cabin. Then, based on the reliability of sleep recognition using the passenger's eyes in the target image, different sleep state recognition methods are used to determine whether the passenger is asleep. Furthermore, sleep state detection results are suppressed based on the detection results of the passenger's target behavior in the image. This increases the scope of sleep recognition by using different detection methods, improving the detection accuracy of sleep recognition. Additionally, the detection results based on the passenger's target behavior prevent false detections of sleep states, further improving the detection accuracy of sleep recognition.

[0310] Furthermore, it is understood that the in-vehicle sleep recognition method, apparatus, electronic device, and medium provided in this disclosure are reproducible and can be used in various industrial applications. For example, the in-vehicle sleep recognition method, apparatus, electronic device, and medium provided in this disclosure can be used in the field of vehicle services.

Claims

1. A method for recognizing sleep patterns in a car cabin, characterized in that, The method includes: Acquire a target image of the vehicle cabin; the target image includes passengers; Based on the confidence score of the passenger's eyes in the target image, a sleep state recognition method that matches the confidence score is used to determine whether the passenger is asleep; the confidence score of the eyes indicates whether the eyes are reliable for sleep recognition. If so, the verification result for the passenger's sleep state is determined based on the detection result of the target behavior of the passenger in the target image; Based on the verification results for sleep status, the sleep status identification results of passengers in the car cabin are determined.

2. The method for recognizing sleep in a car cabin according to claim 1, characterized in that, The sleep state recognition method, which uses matching with eye confidence judgment results to determine whether the passenger is asleep, includes: When the confidence score of the passenger's eyes in the target image is qualified, it is determined whether the passenger is asleep based on the state of the passenger's eyes in the target image. If the confidence score of the passenger's eyes in the target image is unqualified, the passenger's body posture in the target image is used to determine whether the passenger is asleep.

3. The method for recognizing sleep in a car cabin according to claim 2, characterized in that: Based on the passenger's eye state in the target image, determining whether the passenger is asleep includes: Based on the eye state of the passenger in the target image, determine the cumulative duration of closed-eye images within a preset first time period; Determine whether the cumulative duration of the closed-eye images meets the preset closed-eye duration requirement; If so, then the passenger is confirmed to be asleep.

4. The method for recognizing sleep in a car cabin according to claim 2, characterized in that, Based on the passenger's body posture in the target image, determining whether the passenger is asleep includes: Identify the target human skeleton points of the passenger in the target image; Based on the target human skeleton points of the passenger, the head-to-body length ratio and the tilt of the human trunk of the passenger are determined; the head-to-body length ratio represents the length ratio of the passenger's head and torso in the target image; the tilt of the human trunk represents the tilt of the passenger's human trunk in the target image along the horizontal direction of the image. Based on the head-to-body length ratio and the tilt of the human body trunk, it is determined whether the passenger is asleep.

5. The method for recognizing sleep in a car cabin according to claim 4, characterized in that, Based on the head-to-body length ratio and the tilt of the human trunk, determining whether the passenger is asleep includes: Based on the target human skeleton points of the passenger, the passenger's seating area in the cockpit is determined; different seating areas correspond to different head-to-body length ratio conditions and human trunk tilt conditions. Determine whether the passenger's head-to-body length ratio meets the head-to-body length ratio conditions corresponding to the seating area, and whether the passenger's trunk inclination meets the trunk inclination conditions corresponding to the seating area. If both are true, then the passenger is determined to be asleep.

6. The method for recognizing sleep in a car cabin according to claim 5, characterized in that, Different seating areas correspond to different head-to-body length ratios and trunk tilt conditions, which are determined based on images of passengers lying down or with their trunks tilted in different vehicle models.

7. The method for recognizing sleep in a car cabin according to claim 5, characterized in that, Different seating areas correspond to different human trunk tilt conditions, including: The condition for the trunk tilt of the passenger on the right side of the first row is: the trunk tilts to the left. The condition for the trunk tilt of the passenger on the left side of the first row is: the trunk tilts to the right. The condition for the trunk inclination of the second row of passengers is that the trunk inclination meets the preset threshold range.

8. The method for recognizing sleep in a car cabin according to claim 5 or 7, characterized in that, The condition for the human trunk inclination of the passenger on the right side of the first row is: the tangent of the human trunk inclination is less than 0. The condition for the human trunk inclination of the passenger on the left side of the first row is: the tangent of the human trunk inclination is greater than 0. The condition for the trunk inclination of the second row of passengers is: the tangent of the trunk inclination is greater than the preset first threshold and less than the preset second threshold.

9. The method for recognizing sleep in a car cabin according to claim 1, characterized in that, Before determining whether the passenger is asleep based on the eye confidence score of the passenger in the target image and using a sleep state recognition method that matches the eye confidence score, the method further includes: The confidence level of the eye assessment is determined based on the angle between the face orientation and the camera imaging direction. The smaller the angle, the higher the confidence level of the eye for sleep recognition; the larger the angle, the lower the confidence level of the eye for sleep recognition.

10. The method for recognizing sleep in a car cabin according to claim 9, characterized in that... The confidence level for eye assessment is determined based on the angle between the face orientation and the camera's imaging direction; this includes: Determine the angle parameter corresponding to the angle between the face orientation and the camera imaging direction; When the included angle parameter meets the preset confidence conditions, the eye confidence judgment result is determined to be qualified; Conversely, the eye confidence assessment result is determined to be unqualified.

11. The method for recognizing sleep in a car cabin according to claim 10, characterized in that... Determine the angle parameter corresponding to the angle between the face orientation and the camera orientation, including: Define the coordinates of the camera coordinate system origin in the human head coordinate system; Calculate the first angle between the line connecting the origin of the camera coordinate system and the origin of the head coordinate system and the z-axis of the head coordinate system, and determine the angle parameter corresponding to the first angle; the angle parameter corresponding to the first angle is used for eye confidence judgment. Simultaneously calculate the projection of the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system onto the xOy plane of the human head coordinate system and the second included angle of the x-axis of the human head coordinate system, and determine the included angle parameter corresponding to the second included angle.

12. The method for recognizing sleep in a car cabin according to claim 1, characterized in that, Based on the detection results of the target behavior of the passenger in the target image, the verification result for the sleep state is determined, including: Determine the motion detection results of the target part and / or the detection results of the use of the target item by the passenger in the target image within a preset second time period; Determine whether the motion detection results of the target body part and / or the detection results of the use of the target item meet the corresponding sleep inhibition conditions; If any condition is met, the verification result for the sleep state is determined to be sleep suppression.

13. The method for recognizing sleep in a car cabin according to claim 12, characterized in that, Determining the motion detection results of the target part of the passenger in the target image within a preset second time period includes: Determine the rotation angle parameter of the passenger's face in the target image within a preset second time period; the rotation angle parameter represents the rotation angle of the face; Determine the distance the passenger's arm moves within a preset second time period in the target image.

14. The method for recognizing sleep in a car cabin according to claim 13, characterized in that, The determination of whether the motion detection results of the target area meet the corresponding sleep inhibition conditions includes: Determine whether the rotation angle parameter of the face is greater than a preset third threshold; Determine whether the distance the passenger's arm moves is greater than a preset fourth threshold; wherein, the preset fourth threshold is different for different vehicle models and different seating areas.

15. The method for recognizing sleep in a car cabin according to claim 13, characterized in that, Determine whether the test results using the target item meet the corresponding sleep inhibition criteria; including: Detect whether there is a target object in the image; If so, then based on the contact relationship between the target item and the hand, and the contact relationship between the target item and the preset reference object, it is determined whether the target item is in a handheld state; If so, then the test result is determined to be sleep suppression.

16. The method for recognizing sleep in a car cabin according to claim 1, characterized in that, The determination of the sleep state identification result of passengers in the car cabin based on the verification result of the sleep state includes: When the verification result is "sleep suppression", the passenger's sleep state identification result is updated to "non-sleep state". When the verification result is non-suppressed sleep, the passenger's sleep state recognition result remains as sleep state.

17. The method for recognizing sleep in a car cabin according to claim 1, characterized in that, The sleep state recognition results of passengers in the car cabin are used to adjust the state of at least one device in the car cabin according to a preset sleep scenario.

18. A car cabin sleep recognition device, characterized in that, The device includes: The acquisition module is used to acquire a target image of the car cabin; the target image includes passengers; The judgment module is used to determine the confidence level of the passenger's eyes in the target image and to apply the confidence level judgment result to the judgment. The matching sleep state recognition method determines whether the passenger is asleep; the eye confidence judgment result characterizes whether the eye is reliable for sleep recognition. The first determining module is used to determine the verification result of the passenger's sleep state based on the state of the target object in the target image when it is determined that the passenger is in a sleep state. The second determining module is used to determine the sleep state identification result of passengers in the car cabin based on the verification result of the sleep state.

19. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the in-vehicle sleep recognition method as described in any one of claims 1 to 17.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the in-vehicle sleep recognition method as described in any one of claims 1 to 17.

Images