Automotive interior sleep recognition method, apparatus, electronic device, and medium

Abstract

This disclosure provides a method, apparatus, electronic device, and medium for recognizing sleep inside an automobile cabin. The method includes the steps of: acquiring a target image of the automobile cabin, including an occupant; determining whether the occupant is in a sleep state using a sleep state recognition method that conforms to the eye reliability determination result, which represents the reliability when using the eyes for sleep recognition, based on the eye reliability determination result of the occupant in the target image; if it is determined that the occupant is in a sleep state, determining the verification result for the occupant's sleep state based on the detection result for the occupant's target behavior in the target image; and determining the recognition result for the occupant's sleep state inside the automobile cabin based on the verification result for the sleep state. This expands the determination range of sleep recognition, improves the detection accuracy of sleep recognition, suppresses false detection of sleep states, and further improves the detection accuracy of sleep recognition.

Description

Technical Field

[0001] The present disclosure belongs to the field of vehicle services, and specifically relates to a method, apparatus, electronic device, and medium for recognizing sleep in a vehicle interior.

[0002] Cross-reference to related applications The present disclosure claims priority based on a Chinese application with an application number of 202410931779.2 and a title of "Method, Apparatus, Electronic Device, and Medium for Recognizing Sleep in a Vehicle Interior" filed with the Chinese Patent Office on July 12, 2024, and all of its contents are incorporated herein by reference.

Background Art

[0003] With the development of the automotive industry, the application scenarios of vehicles are becoming increasingly numerous, and the requirements that people place on vehicle services are also increasing. During the driving or non-driving rest scenarios of a vehicle, passengers may sometimes sleep inside the vehicle. Therefore, it is required that the vehicle recognize the sleep state of the passengers and adjust the vehicle interior environment accordingly to promote the quality of the passengers' rest and sleep.

[0004] The recognition of sleep in the prior art analyzes the actions and behavior patterns of passengers to determine whether they are in a sleep state, and usually determines the sleep behavior of passengers based on the head posture. The result of that sleep recognition is easily misjudged due to the calculation result of the Euler angles of the head posture and the influence of the height and position of the passengers. Also, there has been no more appropriate sleep determination method for the case when the eyes cannot be confirmed when lying horizontally / leaning against the seat back.

Summary of the Invention

[0005] In view of the above, an object of the present disclosure is to provide a method, apparatus, electronic device, and medium for recognizing sleep in a vehicle interior, which can improve the accuracy of sleep recognition for passengers sleeping in the vehicle interior.

[0006] Embodiments of the present disclosure provide a method for recognizing sleep inside an automobile cabin, the method comprising: acquiring a target image of the automobile cabin, including an occupant; determining whether the occupant is in a sleep state using a sleep state recognition method that conforms to the eye reliability determination result, which represents the reliability when the eye area is used for sleep recognition, based on the eye reliability determination result of the occupant in the target image; if it is determined that the occupant is in a sleep state, determining a verification result for the occupant's sleep state based on the detection result for the occupant's target behavior in the target image; and determining a recognition result for the sleep state of the occupant inside the automobile cabin based on the verification result for the sleep state.

[0007] In some embodiments, the method for recognizing sleep inside an automobile, in which a step is taken to determine whether or not an occupant is in a sleeping state using a sleep state recognition method that conforms to the determination result of the eye reliability, includes the steps of: determining whether or not an occupant is in a sleeping state based on the state of the occupant's eyes in the target image if the determination result of the eye reliability of the occupant in the target image is satisfactory; and determining whether or not an occupant is in a sleeping state based on the body posture of the occupant in the target image if the determination result of the eye reliability of the occupant in the target image is unsatisfactory.

[0008] In some embodiments, the method for recognizing sleep inside an automobile includes the steps of determining whether an occupant is in a sleep state based on the state of the occupant's eyes in the target image, determining the cumulative duration of closed-eye images in a state where the eyes are closed during a predetermined first period based on the state of the occupant's eyes in the target image, determining whether the cumulative duration of closed-eye images satisfies a predetermined closed-eye duration requirement, and, if so, determining that the occupant is in a sleep state.

[0009] In some embodiments, the method for recognizing sleep inside an automobile interior includes the steps of determining whether an occupant is in a sleeping state based on the occupant's body posture in the target image, the steps of: recognizing the occupant's target skeletal point in the target image; determining the occupant's head-to-torso length ratio and trunk inclination based on the occupant's target skeletal point; and determining whether the occupant is in a sleeping state based on the head-to-torso length ratio and trunk inclination, wherein the head-to-torso length ratio represents the ratio between the length of the occupant's head and the length of their torso in the target image, and the trunk inclination represents the inclination of the occupant's trunk in the target image with respect to the horizontal direction of the image.

[0010] In some embodiments, the method for recognizing sleep inside an automobile includes the steps of determining whether an occupant is in a sleeping state based on the head-to-torso length ratio and trunk inclination, determining the occupant's seat area in the vehicle based on the occupant's target skeletal point, determining whether the occupant's head-to-torso length ratio satisfies the head-to-torso length ratio requirement corresponding to the seat area, and determining whether the occupant's trunk inclination satisfies the trunk inclination requirement corresponding to the seat area, and determining that the occupant is in a sleeping state if both requirements are met, wherein different seat areas correspond to different head-to-torso length ratio requirements and trunk inclination requirements.

[0011] In some embodiments, in the car interior sleep recognition method, different head-torso length proportionality requirements and trunk inclination degree requirements corresponding to different seating areas are determined based on images of the occupant lying down or with their torso tilted in different vehicle types.

[0012] In some embodiments, the car interior sleep recognition method provides different trunk inclination requirements corresponding to different seating areas: the trunk inclination requirement for the right-hand occupant in the first row is that the torso is inclined to the left; the trunk inclination requirement for the left-hand occupant in the first row is that the torso is inclined to the right; and the trunk inclination requirement for the second-row occupant is that the trunk inclination is within a predetermined threshold range.

[0013] In some embodiments, the car interior sleep recognition method is such that the trunk inclination requirement for the right-hand occupant in the first row is that the tangent value of the trunk inclination is less than 0, the trunk inclination requirement for the left-hand occupant in the first row is that the tangent value of the trunk inclination is greater than 0, and the trunk inclination requirement for the second-row occupant is that the tangent value of the trunk inclination is greater than a predetermined first threshold and less than a predetermined second threshold.

[0014] In some embodiments, the car interior sleep recognition method further includes a step of determining the eye reliability result based on the angle formed by the orientation of the face and the imaging direction of the camera, prior to the step of determining whether or not the occupant is in a sleep state using a sleep state recognition method that is suitable for the eye reliability determination result based on the eye reliability determination result of the target image, wherein the smaller the angle formed by the orientation of the face and the imaging direction of the camera, the higher the reliability when using the eyes for sleep recognition, and the larger the angle formed by the orientation of the face and the imaging direction of the camera, the lower the reliability when using the eyes for sleep recognition.

[0015] In some embodiments, the method for recognizing sleep inside an automobile interior includes the steps of determining the eye reliability determination result based on the angle formed by the orientation of the face and the imaging direction of the camera, determining an angle parameter corresponding to the angle formed by the orientation of the face and the imaging direction of the camera, determining the eye reliability determination result as a pass if the angle parameter satisfies predetermined reliability conditions, and determining the eye reliability determination result as a fail if the angle parameter does not satisfy predetermined reliability conditions.

[0016] In some embodiments, the method for recognizing sleep inside an automobile interior includes the steps of: defining the coordinates of the origin of the camera coordinate system in the human head coordinate system; calculating the first angle formed by the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system and the z-axis of the human head coordinate system, and determining the angle parameter corresponding to the first angle; and calculating the second angle formed by the projection of the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system in the xOy plane of the human head coordinate system and the x-axis of the human head coordinate system, and determining the angle parameter corresponding to the second angle, wherein the angle parameter corresponding to the first angle is used to determine the eye reliability.

[0017] In some embodiments, the method for recognizing sleep inside an automobile includes the steps of determining the verification result for the sleep state based on the detection result for the occupant's target behavior in the target image, determining the detection result for the movement of the occupant's target part in the target image and / or the use of the target item within a predetermined second period, determining whether the detection result for the movement of the target part and / or the use of the target item satisfies the corresponding sleep false detection suppression conditions, and determining that if the detection result for the movement of the target part and / or the use of the target item satisfies the corresponding sleep false detection suppression conditions, the verification result for the sleep state is sleep false detection suppression.

[0018] In some embodiments, the car interior sleep recognition method includes the step of determining the detection result of the movement of the occupant's target part in the target image within a predetermined second period, which includes the step of determining a face rotation angle parameter representing the rotation angle of the occupant's face in the target image within a predetermined second period, and the step of determining the distance traveled by the occupant's arm in the target image within a predetermined second period.

[0019] In some embodiments, the method for recognizing sleep inside an automobile interior includes the step of determining whether the detection result of movement of a target area satisfies the corresponding sleep false detection suppression conditions, the step of determining whether the rotation angle parameter of the face is greater than a predetermined third threshold, and the step of determining whether the distance traveled by the occupant's arm is greater than a predetermined fourth threshold. Different predetermined fourth thresholds are corresponding to different vehicle types and different seating areas.

[0020] In some embodiments, the step of determining whether the detection result of using a target item satisfies the corresponding sleep false detection suppression condition in the automobile interior sleep recognition method includes: detecting whether a target item exists in the image; if present, determining whether the target item is being held in the hand based on the contact relationship between the target item and the hand, and the contact relationship between the target item and a predetermined reference object; and if it is determined that the target item is being held in the hand, confirming that the determination result is sleep false detection suppression.

[0021] In some embodiments, the method for recognizing sleep inside an automobile, based on the verification results for the sleep state, includes the step of determining the recognition result of the sleep state of an occupant inside an automobile based on the verification results for the sleep state, the step of updating the recognition result of the occupant's sleep state to a non-sleep state if the verification result is suppression of false sleep detection, and the step of maintaining the recognition result of the occupant's sleep state to a sleep state if the verification result is not suppression of false sleep detection.

[0022] In some embodiments, the car interior sleep recognition method uses the recognition result of the sleep state of the occupant in the car interior to adjust the state of at least one device in the car interior according to a predetermined sleep scene.

[0023] In some embodiments, an in-car sleep recognition device is further provided, the device including the following modules.

[0024] The acquisition module acquires target images of the car's interior, including the occupants.

[0025] Based on the determination result of the eye part reliability of the occupant in the target image, the determination module determines whether the occupant is in a sleeping state by using a sleep state recognition method that conforms to the determination result of the eye part reliability representing the reliability when using the eye part for sleep recognition.

[0026] When the first determination module determines that the occupant is in a sleeping state, it determines the verification result for the sleeping state of the occupant based on the state of the target object in the target image.

[0027] Based on the verification result for the sleeping state, the second determination module determines the recognition result of the sleeping state of the occupant in the vehicle interior.

[0028] In some embodiments, an electronic device is further provided. The electronic device includes a processor, a memory storing machine-readable commands executable by the processor, and a bus. When the electronic device operates, the processor and the memory communicate via the bus, and when the machine-readable commands are executed by the processor, the steps of the vehicle interior sleep recognition method are executed.

[0029] In some embodiments, a computer-readable storage medium storing a computer program is further provided. When the computer program is executed by a processor, the steps of the vehicle interior sleep recognition method are executed.

[0030] In an embodiment of the present disclosure, a method, apparatus, electronic device, and medium for recognizing sleep in a vehicle interior are provided. The method for recognizing sleep in the vehicle interior first obtains a target image of the vehicle compartment, and determines whether the occupant is in a sleep state using different sleep state recognition methods based on the reliability of using the eyes of the occupant in the target image for sleep recognition, and suppresses false detection of the sleep state detection result according to the detection result of the target action of the occupant in the image. Thereby, different detection methods expand the determination range of sleep recognition, improve the detection accuracy of sleep recognition, and further prevent false detection of the sleep state based on the detection result of the target action of the occupant, and further improve the detection accuracy of sleep recognition.

Brief Description of the Drawings

[0031] To more clearly illustrate the technical solutions of the embodiments in the present disclosure, the drawings necessary for the description of the embodiments are briefly described below. The drawings to be described only show some embodiments of the present disclosure and do not limit the scope. Those skilled in the art can obtain other related drawings based on these drawings without using inventive capabilities.

[0032] [Figure 1] A flowchart of a method for recognizing sleep in a vehicle interior according to an embodiment of the present disclosure is shown. [Figure 2] A schematic diagram of a human head coordinate system and a camera coordinate system according to an embodiment of the present disclosure is shown. [Figure 3] A flowchart of a method for determining whether the occupant is in a sleep state according to an embodiment of the present disclosure is shown. [Figure 4] A flowchart of a method for determining whether the occupant is in a sleep state based on the body posture of the occupant in the target image according to an embodiment of the present disclosure is shown. [Figure 5] The recognition result of the skeleton points of the human body in the target image according to an embodiment of the present disclosure is shown. [Figure 6] The recognition result of the skeleton points of the human body in another target image according to an embodiment of the present disclosure is shown. [Figure 7] The recognition result of the skeleton points of the human body in another target image according to an embodiment of the present disclosure is shown. [Figure 8] The results of recognizing skeletal points of the human body in other target images according to the embodiments of this disclosure are shown. [Figure 9] A schematic diagram of the automotive in-car sleep recognition device according to the embodiment of this disclosure is shown. [Figure 10] A schematic diagram of an electronic device according to an embodiment of this disclosure is shown. [Modes for carrying out the invention]

[0033] To clarify the purpose, technical concepts, and advantages of the embodiments of this disclosure, the technical concepts in the embodiments of this disclosure will be clearly and completely described below with reference to the drawings used in the embodiments of this disclosure. The drawings in this disclosure are for illustrative purposes only and do not limit the scope of protection of this disclosure. Furthermore, schematic drawings are not drawn to scale with actual objects. The flowcharts used in this disclosure illustrate operations that are realized by some embodiments of this disclosure. The operations in the flowcharts can be realized in any order. Steps that do not have a logical order may be reversed or performed simultaneously. A person skilled in the art may add one or more other operations to the flowcharts or remove one or more operations from the flowcharts by teaching the contents of this disclosure.

[0034] Furthermore, the embodiments described herein represent only a portion of the embodiments of this disclosure, not all embodiments. The components in the embodiments of this disclosure shown herein with reference to the drawings can be arranged and designed in various ways. For this reason, the following detailed description of the embodiments of this disclosure shown in the drawings is merely an example of selected embodiments of this disclosure and does not limit the scope of the disclosure that is intended to be protected. All other embodiments that a person skilled in the art could obtain without using their inventive ability based on the embodiments of this disclosure are all within the scope of protection of this disclosure.

[0035] In the embodiments of this disclosure, the term “includes” is used to indicate the presence of the feature in question, but does not preclude the addition of other features.

[0036] Conventional sleep recognition methods analyze the occupant's movements and behavioral patterns to determine whether or not they are asleep. Typically, sleep behavior is determined based on head posture, resulting in low accuracy. Furthermore, there was no more appropriate method for determining sleep when the eyes cannot be seen while the occupant is lying down or leaning against the seatback. Specifically, conventional methods for determining occupant sleep behavior using head posture typically utilize Euler angles. First, the roll and yaw angles of the head are determined to determine if the occupant's head is leaning against the shoulders, and the pitch angle is determined to determine if the occupant is looking downwards. However, because multiple different Euler angle solutions can exist for a given spatial direction, using only Euler angles affects the accuracy of sleep recognition. Additionally, occupants in a vehicle are not always in the same position, and a single setting cannot accommodate occupants of different heights and distances, leading to errors in sleep recognition. In particular, when the eyes cannot be seen while lying down or leaning back in a seat, it is impossible to accurately determine whether the eyes are open or closed, so there was no more appropriate method for determining sleep.

[0037] In addition to the sleep recognition based on behavioral recognition described above, sleep recognition based on physiological signal detection is also available. Physiological signal detection is a method of recognizing the sleep state of occupants by monitoring human physiological parameters such as heart rate, respiration, muscle activity, and voice, and usually requires the use of sensors and devices within the vehicle. Because these devices need to be installed and fixed within the vehicle, costs and complexity increase. Furthermore, physiological signals are usually weak and susceptible to interference from various types of noise, which affects the accuracy of sleep recognition. Physiological signals such as heart rate and respiratory rate can vary greatly from person to person. This individual difference may affect the accuracy of monitoring physiological signals.

[0038] Based on the above, embodiments of this disclosure provide a method, apparatus, electronic device, and medium for recognizing sleep inside an automobile cabin. The automobile cabin sleep recognition method first acquires a target image of the automobile cabin, and then, based on the reliability of using the occupant's eyes in the target image for sleep recognition, determines whether or not the occupant is in a sleep state using a different sleep state recognition method, and then suppresses false detection of the sleep state according to the detection result of the occupant's target behavior in the image. By using different detection methods, the determination range of sleep recognition is expanded, the detection accuracy of sleep recognition is improved, and furthermore, false detection of the sleep state is prevented based on the detection result of the occupant's target behavior, thereby further improving the detection accuracy of sleep recognition.

[0039] Figure 1 shows a flowchart of a car interior sleep recognition method according to an embodiment of the present disclosure. As shown in Figure 1, the car interior sleep recognition method includes the following steps S101 to S104.

[0040] Step S101 acquires a target image of the car's interior, including the occupants.

[0041] Step S102 determines whether or not the occupant is in a sleep state using a sleep state recognition method that is suitable for the eye reliability determination result, which represents the reliability when the eye area is used for sleep recognition, based on the determination result of the eye reliability determination result of the occupant's eye area in the target image.

[0042] Step S103 determines, if it is determined that the occupant is in a sleep state, confirms the verification result for the occupant's sleep state based on the detection result for the occupant's target behavior in the target image.

[0043] Step S104 determines the recognition result of the sleep state of the occupants inside the automobile based on the verification results regarding the sleep state.

[0044] In step S101, a target image of the car's interior, including the occupants, is acquired.

[0045] The aforementioned target image of the vehicle's interior is an image captured by an onboard camera located inside the vehicle. While the installation location of the onboard camera varies depending on the vehicle model, it is generally installed in the center of the front of the vehicle's interior (above the center of the windshield). By mounting the camera at the top center of the windshield, a wide field of view can be secured, allowing for a larger area of ​​the interior to be captured, providing a more complete interior image, and ensuring that images of all occupants facing forward are taken as much as possible.

[0046] The aforementioned target image includes occupants. Here, "occupants" includes all occupants in all seats, i.e., the driver and non-drivers.

[0047] While it is generally unlikely that a driver would be asleep, especially lying down or leaning back in a seat, when a vehicle is in operation, there is a possibility of falling asleep or experiencing fatigue while driving. Therefore, the vehicle interior sleep recognition method according to the embodiments of this disclosure needs to be applicable to the driver as well.

[0048] When a vehicle is not in operation, i.e., stationary, all occupants in each seat (including the driver's seat) may be asleep, either lying down or leaning back against the seatback.

[0049] Since in-vehicle cameras typically record in-vehicle videos, the target image is, in other words, an image frame from a video of the interior of a car captured by the in-vehicle camera, and can also be called an in-vehicle video image.

[0050] The target image may be a part of the image frame within the in-vehicle video.

[0051] For example, in some embodiments, the target image is an image frame in an in-vehicle video that satisfies predetermined image quality conditions; that is, only frames that meet specific image quality criteria are selected as the target image.

[0052] The aforementioned predetermined image quality conditions include, for example, that the sharpness exceeds a predetermined sharpness threshold, the image brightness exceeds a predetermined brightness threshold, and the noise falls below a predetermined noise threshold.

[0053] For example, in some embodiments, the target image is an image frame extracted from in-vehicle video at a predetermined frame interval. For instance, if the frame rate of the in-vehicle camera is 30fps or 60fps, processing dozens of images per second requires significant computational resources, reducing the real-time capability of sleep recognition. Therefore, it is sufficient to use only a portion of the image frames acquired at predetermined intervals for sleep recognition. For example, if the predetermined frame interval is 5 frames, one frame is extracted from every 5 image frames to be used as the target image, resulting in 6 frames being extracted as the target image per second.

[0054] In some embodiments, the target image may be a combination of an image frame from an in-vehicle video that satisfies predetermined image quality conditions and an image frame extracted from the in-vehicle video at predetermined frame intervals.

[0055] Specifically, the system first analyzes each image frame in the in-vehicle video in real time to determine whether it meets predetermined image quality criteria. For image frames that meet these criteria, they are extracted based on predetermined frame intervals. The resulting target image meets the image quality requirements and is extracted at predetermined frame intervals. This combination method allows for the acquisition of high-quality and representative in-vehicle video image frames with limited resources, providing excellent support for subsequent use.

[0056] In some embodiments, when the in-vehicle control system is set to automatic sleep recognition mode, it automatically performs sleep recognition upon acquiring a target image of the vehicle's interior.

[0057] In other words, the automatic sleep recognition mode of the in-vehicle control system can be set to on or off.

[0058] In step S102, based on the determination result of the eye reliability of the occupant in the target image, it is determined whether or not the occupant is in a sleep state using a sleep state recognition method that is suitable for the determination result of the eye reliability, which represents the reliability when the eye is used for sleep recognition.

[0059] The system determines whether the reliability of the occupant's eyes in the target image meets the predetermined reliability conditions. If it does, the result of the eye reliability determination is confirmed as a pass; otherwise, the result of the eye reliability determination is confirmed as a fail.

[0060] The reliability of the eye area is determined based on the angle formed by the orientation of the face and the imaging direction of the camera. The smaller the angle formed by the orientation of the face and the imaging direction of the camera, the higher the reliability of using the eye area for sleep recognition. Conversely, the larger the angle formed by the orientation of the face and the imaging direction of the camera, the lower the reliability of using the eye area for sleep recognition.

[0061] The angle between the orientation of the face and the camera's imaging direction refers to the angle in three-dimensional space. For example, when the camera captures an image from the side of the occupant's face or from above the occupant's head, the occupant's eyes are barely visible in the image, and in this case, using the eyes for sleep recognition is unreliable.

[0062] If the target image contains multiple occupants, it is necessary to determine whether or not each occupant is in a sleep state using a sleep state recognition method that is appropriate to the eye reliability assessment result for each occupant.

[0063] In other words, when multiple occupants are present in the target image, the same sleep state recognition method is not used for all of them. Because the eye confidence assessment results differ for different occupants, the corresponding sleep state recognition methods also differ.

[0064] For example, if the target image contains crew members A and B, with crew member A showing a frontal view and crew member B showing a profile view, then different sleep state recognition methods are used to determine whether or not each crew member A and crew member B are in a sleep state.

[0065] As a result, in the automobile interior sleep recognition method according to the embodiment of the present disclosure, before the step of determining whether or not the occupant is in a sleep state using a sleep state recognition method that matches the eye reliability determination result based on the eye reliability determination result of the occupant's eyes in the target image, the method further includes a step of determining the eye reliability determination result based on the angle formed by the orientation of the face and the imaging direction of the camera, wherein the smaller the angle formed by the orientation of the face and the imaging direction of the camera, the higher the reliability when using the eyes for sleep recognition, and the larger the angle formed by the orientation of the face and the imaging direction of the camera, the lower the reliability when using the eyes for sleep recognition.

[0066] Specifically, the step of determining the eye reliability determination result based on the angle formed by the orientation of the face and the imaging direction of the camera includes the steps of determining an angle parameter corresponding to the angle formed by the orientation of the face and the imaging direction of the camera, determining the eye reliability determination result as pass if the angle parameter satisfies predetermined reliability conditions, and determining the eye reliability determination result as fail if the angle parameter does not satisfy predetermined reliability conditions.

[0067] The orientation of the face is determined based on the head posture. On the other hand, the angle formed by the orientation of the face and the camera's imaging direction is also related to the position of the occupant's head.

[0068] When detecting the position and orientation of the occupant's head, the head position is represented by the coordinates of the head relative to the camera, and the head orientation is represented by the rotational relationship of the human head coordinate system relative to the camera coordinates. Figure 2 shows a schematic diagram of the human head coordinate system and camera coordinate system according to an embodiment of this disclosure.

[0069] In the embodiments of this disclosure, the occupant's head posture is defined using two included angle parameters: the angle formed by the orientation of the face and the imaging direction of the camera.

[0070] Specifically, in the car interior sleep recognition method according to the embodiment of the present disclosure, the step of determining the included angle parameter corresponding to the included angle between the orientation of the face and the orientation of the camera includes the steps of defining the coordinates of the origin of the camera coordinate system in the human head coordinate system, calculating the first included angle between the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system and the z-axis of the human head coordinate system, and determining the included angle parameter corresponding to the first included angle, and calculating the second included angle between the projection of the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system in the xOy plane of the human head coordinate system and the x-axis of the human head coordinate system, and determining the included angle parameter corresponding to the second included angle, wherein the included angle parameter corresponding to the first included angle is used to determine the eye reliability.

[0071] By defining the coordinates of the origin of the camera coordinate system in the human head coordinate system, the relative positional relationship between the crew member's head and the camera is represented.

[0072] As shown in Figure 2, the origin of the camera coordinate system is the optical center of the camera, the x and y axes are parallel to the X and Y axes of the image, and the direction of the z axis is the same as the optical axis of the camera and perpendicular to the image plane. Specifically, in the image coordinate system, the x axis is to the right and the y axis is downward, whereas in the camera coordinate system, the origin is the center of the principal optical axis of the lens, the positive x direction is to the right, the positive y direction is downward, and the positive z direction is forward. In this way, the x and y directions coincide with the directions of the image coordinate system, the z direction becomes the depth of field, which conforms to the definition of a right-handed coordinate system and contributes to the vector calculation of the algorithm.

[0073] In other words, the z-axis of the camera coordinate system and the optical axis of the camera coincide, and both are in the direction forward of the camera.

[0074] The definition of the aforementioned human head coordinate system is as follows: The origin of the human head coordinate system is located at the center of the head, the X-axis is parallel to the line connecting the left and right eyes, the Y-axis is parallel to the line connecting the tip of the nose and the tip of the chin, and the Z-axis is perpendicular to the X and Y axes and represents the direction of the face's orientation.

[0075] As a result, the first angle formed by the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system, and the z-axis of the human head coordinate system, can represent the reliability when using the eye region for sleep recognition.

[0076] The aforementioned included angle parameter represents attributes such as the magnitude and direction of the included angle, and may be an angle or a trigonometric function value corresponding to the angle.

[0077] The following describes in detail the calculation process for the first included angle formed by the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system, and the z-axis of the human head coordinate system, and the calculation process for the second included angle formed by the projection of the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system in the xOy plane of the human head coordinate system, and the x-axis of the human head coordinate system. For simplicity of expression, in the embodiment of this disclosure, the first included angle formed by the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system, and the z-axis of the human head coordinate system, is called latitude, and the second included angle formed by the projection of the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system, in the xOy plane of the human head coordinate system, and the x-axis of the human head coordinate system, is called longitude.

[0078] The second included angle formed by the projection of the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system onto the xOy plane of the human head coordinate system, and the x-axis of the human head coordinate system, is that is, the second included angle formed by the projection of the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system onto the xOy plane of the human head coordinate system, and the x-axis of the human head coordinate system.

[0079] Specifically, the coordinates of the origin of the camera coordinate system in the human head coordinate system are defined as (x, y, z), and (x, y, z) is vector normalized to obtain the normalized vector

number

[0080] specifically,

number

[0081] Normalized vector

number

number

[0082] As can be seen in Figure 2,

number

number

[0083] As can be seen from the definition of latitude and longitude, when the face is facing the camera, the camera is located on the z-axis of the human head coordinate system, and the normalized vector is

number

[0084] When making a determination using only Euler angles, multiple different Euler angle solutions may exist for a given spatial direction. Furthermore, occupants in a vehicle cabin are not always in the same position, and a single setting cannot accommodate occupants of different heights and distances. However, different settings for Euler angles are required depending on the location. In contrast, the embodiments of this disclosure propose a method that analyzes whether or not the eyes can be seen using latitude and longitude, allowing the same design to be applied to the latitude and longitude of different locations in the vehicle cabin, thereby improving computational efficiency.

[0085] Figure 3 shows a flowchart of a method for determining whether or not a occupant is in a sleep state according to an embodiment of the present disclosure. As shown in Figure 3, in an embodiment of the present disclosure, the step of determining whether or not a occupant is in a sleep state using a sleep state recognition method that conforms to the eye reliability determination result includes the following steps S301 to S302.

[0086] Step S301 determines, if the occupant's eye reliability in the target image is satisfactory, whether or not the occupant is asleep based on the state of the occupant's eyes in the target image.

[0087] Step S302 determines, if the occupant's eye reliability in the target image is unsatisfactory, whether or not the occupant is asleep based on the occupant's body posture in the target image.

[0088] In other words, the aforementioned sleep state recognition method includes two types: a sleep recognition method based on the state of the eyes and a sleep recognition method based on body posture.

[0089] The aforementioned eye states include two types: open and closed.

[0090] Specifically, the step of determining whether a occupant is in a sleeping state based on the state of the occupant's eyes in the target image includes: determining the cumulative duration of closed-eye images in a state where the eyes are closed during a predetermined first period based on the state of the occupant's eyes in the target image; determining whether the cumulative duration of closed-eye images satisfies a predetermined closed-eye duration requirement; and, if the cumulative duration of closed-eye images satisfies the predetermined closed-eye duration requirement, determining that the occupant is in a sleeping state.

[0091] Meeting the predetermined eye-closed time requirement means that the cumulative time length of the closed-eye images exceeds the predetermined eye-closed time threshold.

[0092] In some embodiments, the cumulative time length of closed-eye images, in which the eyes are closed during a predetermined first period, is determined based on the number of closed-eye images and the sampling time.

[0093] Based on this, the number of closed-eye images in a predetermined first period may be determined, and it may be determined whether the number of closed-eye images meets a predetermined number. If the number of closed-eye images meets the predetermined number, it may be determined that the occupant is in a sleep state.

[0094] To satisfy a predetermined number means exceeding the first predetermined number threshold.

[0095] Specifically, after recognizing the state of the occupant's eyes in the target image, the recognition result of the eye state in the target image of the current frame is updated in the sleep window. The window duration of the sleep window is a predetermined first period.

[0096] If the number of target images showing the eyes closed within the aforementioned sleep window reaches a predetermined number, it is determined that the occupant is in a sleep state.

[0097] By using the aforementioned sleep window, it is possible to determine the sleep state of the crew members in a more timely manner.

[0098] Specifically, the recognition of the occupant's eye state in the target image is performed using an eye open / closed classification model.

[0099] In other words, the target image is input into the eye open / closed classification model, and the eye open / closed classification model outputs information about the occupants' eye open / closed state.

[0100] Figure 4 shows a flowchart of a method for determining whether an occupant is asleep based on the occupant's body posture in the target image according to an embodiment of the present disclosure. As shown in Figure 4, the steps for determining whether an occupant is asleep based on the occupant's body posture in the target image include the following steps S401 to S403.

[0101] Step S401 involves recognizing the target human skeletal point of the occupant in the target image.

[0102] Step S402 determines the head-to-torso length ratio and trunk inclination of the occupant based on the target human skeleton point of the occupant. The head-to-torso length ratio represents the ratio between the length of the occupant's head and the length of their torso in the target image, and the trunk inclination represents the degree of inclination of the occupant's trunk in the target image relative to the horizontal direction of the image.

[0103] Step S403 determines whether the occupant is in a sleeping state based on the head-to-torso length ratio and the trunk inclination.

[0104] Skeletal points in the human body, also known as skeletal keypoints, typically refer to specific locations that represent human posture in the recognition of human posture. These points are usually located at joints and characteristic parts of the human body and are points that make up the skeletal model of the human body. In a typical posture recognition system, skeletal keypoints include the top of the head (the highest point of the head), the neck (the top of the cervical spine), the shoulder region (including the left and right shoulders at the locations of the left and right shoulder joints), the elbow region (including the left and right elbows at the locations of the left and right elbow joints), the arm region (including the left and right arms at the locations of the left and right wrists), and the hip region (including the left and right hips at the locations of the left and right hip joints).

[0105] Also, in the embodiments of the present disclosure, due to the space inside the vehicle cabin and the limitation of the camera position, usually only the upper body part of the skeletal points of the human body needs to be recognized.

[0106] The torso includes a head and a torso.

[0107] The horizontal direction of the target image is from left to right, and the origin of the coordinate system is the upper left corner of the target image.

[0108] Referring to FIGS. 5 to 8, FIG. 5 shows the recognition result of the skeletal points of the human body in the target image according to the embodiment of the present disclosure. FIG. 6 shows the recognition result of the skeletal points of the human body in another target image according to the embodiment of the present disclosure. FIG. 7 shows the recognition result of the skeletal points of the human body in another target image according to the embodiment of the present disclosure. FIG. 8 shows the recognition result of the skeletal points of the human body in another target image according to the embodiment of the present disclosure.

[0109] Taking FIG. 5 as an example, the person sitting on the left side of the front row has a head length of L1, a torso length of L2, and the length ratio of L1 to L2 is approximately 1:1. The person lying horizontally on the right side of the front row has a head length of L3 and a torso length of L4, and obviously the length ratio of L3 to L4 is relatively small. The ratio of the head length to the torso length in the target image is called the head-body length ratio head_body_ratio. By determining whether the person in the front row is sitting or lying horizontally based on this head-body length ratio, it is determined whether the occupant is in a sleeping state. A threshold value of the head-body length ratio, th_head_body_ratio, is set, and when head_body_ratio < th_head_body_ratio, it is said that the index 1 is satisfied.

[0110] Taking the above Figure 6 as an example, when the person on the right side of the front row lies horizontally, the trunk (the line segment connecting the centers of the left and right hip joints above the head) is inclined to the left. The included angle formed by the white line and the X-axis (the horizontal right direction at the bottom) of the target image is called body_ang, that is, the trunk inclination. When the trunk is inclined to the left, tan(body_ang) < 0, and when sitting, the trunk is inclined to the right (tan(body_ang) > 0). For the person on the left side of the front row, it is exactly the opposite. When lying horizontally, the trunk is inclined to the right (tan(body_ang) > 0), and when sitting, the trunk is inclined to the left (tan(body_ang) < 0). Therefore, when tan(body_ang) < 0 for the right side of the front row and tan(body_ang) > 0 for the left side of the front row, it meets Index 2. Index 2 can also be used to determine whether the person in the front row is sitting or lying horizontally.

[0111] To reduce misjudgment, when both Index 1 and Index 2 are met, this person is considered to be in a sleeping state.

[0112] As shown in Figures 7 and 8, for the person in the rear row, it can be determined whether the person is leaning against the seat back according to the trunk inclination body_ang. Set a predetermined first threshold th_back_left and a predetermined second threshold th_back_right to limit the predetermined threshold range of the trunk inclination. When th_back_left < tan(body_ang) < th_back_right, this person is considered to be in a sleeping state. <00>

[0111]

[0113] Also, from Figures 7 and 8, it can be seen that for the person in the rear row, the head-to-trunk length ratio when sitting is larger than the head-to-trunk length ratio when leaning against the seat back.

[0114] According to the exemplary analysis of Figures 5 to 8 above, when determining whether the corresponding occupant is in a sleeping state based on the head-to-trunk length ratio and the trunk inclination, the head-to-trunk length ratio requirements and trunk inclination requirements corresponding to different seat areas are different. ​The aforementioned seating area includes the left side of the first row, the right side of the first row, and the second row.

[0116] In this configuration, in the embodiment of the present disclosure, the step of determining whether the occupant is in a sleeping state based on the head-to-torso length ratio and trunk inclination includes the steps of determining the occupant's seat area in the vehicle based on the occupant's target skeletal point, determining whether the occupant's head-to-torso length ratio satisfies the head-to-torso length ratio requirement corresponding to the seat area, and whether the occupant's trunk inclination satisfies the trunk inclination requirement corresponding to the seat area, and determining that the occupant is in a sleeping state if both requirements are met.

[0117] Since the seat positions inside the car interior in the target image remain essentially unchanged, the positions of the occupants' left and right hip joints in the target image will fall within a certain range.

[0118] Even if the seat position inside the car is adjustable fore and aft, the positions of the occupant's left and right hip joints in the target image will remain within a certain range.

[0119] Therefore, determining the seating area of ​​an occupant within the vehicle based on the occupant's target skeletal points specifically involves determining the occupant's seating area within the vehicle based on the position of the occupant's target skeletal points in the target image and the positional range of different seating areas in the target image.

[0120] The aforementioned target human skeletal points are the skeletal points of the left and right hip joints of the crew member.

[0121] The trunk inclination requirements differ for different seating areas. For the occupant on the right side of the first row, the trunk inclination requirement is that the torso is tilted to the left. For the occupant on the left side of the first row, the trunk inclination requirement is that the torso is tilted to the right. For the occupant in the second row, the trunk inclination requirement is that the trunk inclination is within a predetermined threshold range.

[0122] Quantifying the aforementioned trunk inclination requirement contributes to the calculation. Specifically, the trunk inclination requirement corresponding to the occupant on the right side of the first row is that the tangent value of the trunk inclination is less than 0, i.e., tan(body_ang) < 0.

[0123] The trunk inclination requirement for the left-hand occupant in the first row is that the tangent value of the trunk inclination is greater than 0, i.e., tan(body_ang) > 0.

[0124] The trunk inclination requirement for the second-row occupants is that the tangent value of the trunk inclination is greater than a predetermined first threshold and less than a predetermined second threshold, i.e., th_back_left <tan(body_ang)<th_back_rightとなる。

[0125] The aforementioned body_ang represents the trunk inclination.

[0126] Specifically, different head-to-torso length ratio requirements and trunk tilt requirements corresponding to different seating areas are determined based on images of occupants lying down or with their torsos tilted in different vehicle types.

[0127] Spatial parameters within the vehicle interior, camera position, camera internal parameters, and seat position in different vehicle models affect the head-to-torso length ratio and trunk inclination in the target image captured by the camera. Therefore, the requirements for head-to-torso length ratio and trunk inclination differ depending on the vehicle model.

[0128] Specifically, using multiple images of occupants of the relevant vehicle type lying down or with their torsos tilted, different head-to-torso length ratio requirements and trunk inclination requirements corresponding to different seating areas are measured.

[0129] In step S103, when determining whether or not the occupant is in a sleep state, the verification result regarding the occupant's sleep state is determined based on the detection result for the occupant's target behavior in the target image.

[0130] If the crew is still active, for example, moving their arms, turning their head, using a cell phone, reading a book, or eating a snack, then the crew is not asleep.

[0131] Therefore, in the embodiments of this disclosure, several target behaviors are set, and when it is detected that an occupant has performed one of the target behaviors, it is determined that the occupant is not in a sleep state. In other words, the verification result is that the occupant is not asleep.

[0132] The detection results for the crew member's target behavior include two types: cases where the target behavior was detected and cases where it was not detected.

[0133] In the embodiments of this disclosure, target behaviors are classified into two types, and two detection policies are established. Target behaviors include the movement of a target body part by an occupant and the use of a target item by an occupant.

[0134] Specifically, the step of determining the verification result for the sleep state based on the detection result for the occupant's target behavior in the target image includes: determining the detection result for the movement of the occupant's target body part in the target image and / or the use of the target item within a predetermined second period; determining whether the detection result for the movement of the target body part and / or the use of the target item satisfies the corresponding sleep false detection suppression conditions; and, if the detection result for the movement of the target body part and / or the use of the target item satisfies the corresponding sleep false detection suppression conditions, determining the verification result for the sleep state as sleep false detection suppression.

[0135] The aforementioned target parts refer to parts of the vehicle that can move significantly and are easily detectable. Examples include head rotation and arm movement. In contrast, leg movement is undetectable, and the torso cannot move significantly on its own (when the torso moves, the arms also move).

[0136] Therefore, in the embodiments of this disclosure, the target areas are specifically the hands and the head.

[0137] Specifically, the step of determining the detection result of the movement of the target part of the occupant in the target image within a predetermined second period includes the step of determining a face rotation angle parameter representing the rotation angle of the occupant's face in the target image within the predetermined second period, and the step of determining the distance traveled by the occupant's arm in the target image.

[0138] In the embodiments of this disclosure, the face rotation angle parameter specifically represents the change in the included angle corresponding to the included angle formed by the orientation of the face and the orientation of the camera. Specifically, it is determined based on longitude and latitude.

[0139] Specifically, when calculating longitude and latitude to determine eye reliability, the longitude and latitude are added to the head posture time window to determine whether a significant change in head posture has occurred within the head posture time window. If a set of predetermined third thresholds, th_rotate_lat and th_rotate_long, is set, then if latitude > th_rotate_lat or longitude > th_rotate_long, it indicates that a significant change in head posture has occurred.

[0140] The distance the occupant's arm moves in the target image is determined, and the movement information of the person's arm is used to suppress false detection of sleep. The position of the arm is added to the arm movement time window, and it is determined whether the arm has moved significantly within a predetermined second period.

[0141] Specifically, if a predetermined fourth threshold th_front_hand_distance is set for the front arm and a predetermined fourth threshold th_back_hand_distance is set for the back arm, then if the distance traveled by the front arm > th_front_hand_distance or the distance traveled by the back arm > th_back_hand_distance, it indicates that the arm has moved significantly within the predetermined second period.

[0142] The reason why the arms in the front row and the arms in the back row correspond to different predetermined fourth thresholds is that the distances from the camera to the front row arms and the arms in the back row are different, resulting in different sizes in the image. Therefore, even if the same distance is moved in real space, the corresponding distance in the image will be different.

[0143] The step of determining whether the detection result of movement of the target part satisfies the corresponding sleep false detection suppression conditions includes the step of determining whether the rotation angle parameter of the face is greater than a predetermined third threshold, and the step of determining whether the distance traveled by the occupant's arm is greater than a predetermined fourth threshold, wherein the predetermined fourth threshold differs depending on the vehicle type and the seat area.

[0144] Furthermore, the position of the occupant's arm in the target image is determined based on skeletal points of the human body corresponding to the arm. For example, it is determined based on the bones of the occupant's elbow and wrist. In other words, the recognized skeletal points of the human body are used not only for recognizing the sleep state but also for detecting the occupant's target behavior in the target image, enabling multiple functions to be achieved with a single detection.

[0145] The step of determining whether the detection result of movement of the target part satisfies the corresponding sleep false detection suppression conditions includes the step of determining whether the rotation angle parameter of the face is greater than a predetermined third threshold, and the step of determining whether the distance traveled by the occupant's arm is greater than a predetermined fourth threshold, wherein the predetermined fourth threshold differs depending on the vehicle type and the seat area.

[0146] If any one of the judgment results is "YES", the verification result for the sleep state is confirmed to be suppression of false sleep detection. If all judgment results are "NO", the verification result for the sleep state is confirmed to be non-suppression of false sleep detection.

[0147] In embodiments of this disclosure, sleep false detection is suppressed based on the use of a target item by a person.

[0148] Specifically, in the above-mentioned method for recognizing sleep inside an automobile, the step of determining whether the detection result of using a target item satisfies the corresponding sleep false detection suppression conditions includes: detecting whether a target item exists in the image; if present, determining whether the target item is being held in the hand based on the contact relationship between the target item and the hand, and the contact relationship between the target item and a predetermined reference object; and if it is determined that the target item is being held in the hand, confirming that the determination result is sleep false detection suppression.

[0149] The target items may include mobile phones, food, books, tablets, and the like.

[0150] The aforementioned reference object may be the seat surface.

[0151] Based on the contact relationship between the target item and the hand, it is determined whether the target item is being held in the hand. Based on the contact relationship between the target item and a designated reference object, it is determined whether the target item is placed on the designated reference object and in contact with the hand, or whether the item is being used by the hand.

[0152] Therefore, based on the contact relationship between the target item and the hand, and the contact relationship between the target item and a designated reference object, it is possible to more accurately determine whether the target item is being held in the hand, that is, whether the crew member is using the target item.

[0153] The contact relationship between the target item and the hand is determined based on the position of the hand's skeletal points and the position of the target item. If the positional difference between the target item and the hand is less than a predetermined fifth threshold, it is determined that the target item and the hand are in contact.

[0154] The contact relationship between the target item and the designated reference object is determined based on the positions of the designated reference object and the target item. If the positional difference between the target item and the designated reference object is less than a predetermined sixth threshold, it is determined that the target item and the designated reference object are in contact.

[0155] For example, when using information about a person operating a mobile phone to suppress false sleep detection, first, it is detected whether or not a mobile phone is being held (contact between the hand and the mobile phone) in the target image. If the result is YSE, it is determined whether or not the held mobile phone is placed on the seat, and if the mobile phone is not placed on the seat, false sleep detection is suppressed.

[0156] In step S104, the result of recognizing the sleep state of the occupants inside the automobile is determined based on the verification results regarding the sleep state.

[0157] Specifically, the step of determining the recognition result of the sleep state of an occupant in the vehicle cabin based on the verification results for the sleep state includes, if the verification result is suppression of false sleep detection, updating the recognition result of the occupant's sleep state to a non-sleep state, and if the verification result is not suppression of false sleep detection, maintaining the recognition result of the occupant's sleep state to a sleep state.

[0158] Based on the verification of the sleep state, it is possible to suppress some false sleep detections and improve the accuracy of sleep state recognition.

[0159] In the vehicle interior sleep recognition method according to the embodiments of this disclosure, the result of recognizing the sleep state of the occupant in the vehicle interior is used to adjust the state of at least one device in the vehicle interior according to a predetermined sleep scene.

[0160] The device includes the following: seat angle, volume of the in-car audio system, interior lighting, temperature control device (e.g., in-car air conditioner), etc.

[0161] Specifically, if the sleep state recognition result determines that the occupant is asleep, a sleep signal is generated and transmitted to the device control module. After receiving the sleep signal, the device control module adjusts the state of the device according to a predetermined sleep scene. For example, it adjusts the seat angle, the volume of the in-car audio system, the interior lighting, and the temperature of the temperature control device.

[0162] Based on the same inventive concept, embodiments of this disclosure further provide an in-car sleep recognition device corresponding to an in-car sleep recognition method. Since the principle of solving the problem of the device according to embodiments of this disclosure is similar to that of the in-car sleep recognition method according to embodiments of this disclosure, the implementation of the device can be referred to the embodiments of the method, and the explanation of overlapping parts will be omitted.

[0163] Figure 9 shows a schematic configuration diagram of an in-car sleep recognition device according to an embodiment of the present disclosure. The device includes the following modules.

[0164] The acquisition module 901 acquires target images of the car's interior, including the occupants.

[0165] The determination module 902 determines whether or not the occupant is in a sleep state using a sleep state recognition method that is suitable for the determination result of the eye reliability of the occupant in the target image. The eye reliability represents the reliability when using the eyes for sleep recognition. If the first confirmation module 903 determines that the occupant is in a sleep state, it confirms the verification result for the occupant's sleep state based on the state of the target object in the target image.

[0166] The second confirmation module 904 confirms the recognition result of the sleep state of the occupant inside the automobile based on the verification results for the sleep state.

[0167] In some embodiments, in the automobile interior sleep recognition method, the determination module determines whether or not the occupant is in a sleep state using a sleep state recognition method that conforms to the determination result of eye reliability. Specifically, if the determination result of the occupant's eye reliability in the target image is satisfactory, the determination module determines whether or not the occupant is in a sleep state based on the state of the occupant's eyes in the target image. If the determination result of the occupant's eye reliability in the target image is unsatisfactory, the determination module determines whether or not the occupant is in a sleep state based on the occupant's body posture in the target image.

[0168] In some embodiments, in the in-car sleep recognition device, the determination module, when determining whether an occupant is in a sleep state based on the state of the occupant's eyes in the target image, specifically determines the cumulative duration of closed-eye images in a predetermined first period based on the state of the occupant's eyes in the target image, and determines whether the cumulative duration of closed-eye images satisfies a predetermined closed-eye duration requirement. If it does, it is determined that the occupant is in a sleep state.

[0169] In some embodiments, the vehicle interior sleep recognition device, when determining whether an occupant is in a sleep state based on the occupant's body posture in the target image, the determination module specifically recognizes the occupant's target skeletal point in the target image, determines the occupant's head-to-torso length ratio and trunk inclination based on the occupant's target skeletal point, and determines whether the occupant is in a sleep state based on the head-to-torso length ratio and trunk inclination. The head-to-torso length ratio represents the ratio between the length of the occupant's head and the length of their torso in the target image. The trunk inclination represents the inclination of the occupant's trunk in the target image with respect to the horizontal direction of the image.

[0170] In some embodiments, the in-car sleep recognition device determines whether an occupant is in a sleeping state based on the head-to-torso length ratio and trunk inclination. Specifically, the determination module determines the occupant's seat area in the vehicle based on the occupant's target skeletal point. Here, different seat areas correspond to different head-to-torso length ratio requirements and trunk inclination requirements. The determination is made whether the occupant's head-to-torso length ratio satisfies the head-to-torso length ratio requirement corresponding to the seat area, and whether the occupant's trunk inclination satisfies the trunk inclination requirement corresponding to the seat area. If both requirements are met, it is determined that the occupant is in a sleeping state.

[0171] In some embodiments, the in-car sleep recognition device determines the different head-torso length proportionality requirements and trunk tilt degree requirements corresponding to different seating areas based on images of the occupant lying down or tilting their torso in different vehicle types.

[0172] In some embodiments, the in-car sleep recognition device has different trunk inclination requirements corresponding to different seating areas, where the trunk inclination requirement for the right-hand occupant in the first row is that the torso is inclined to the left, the trunk inclination requirement for the left-hand occupant in the first row is that the torso is inclined to the right, and the trunk inclination requirement for the second-row occupant is that the trunk inclination is within a predetermined threshold range.

[0173] In some embodiments, the vehicle interior sleep recognition device has the following trunk tilt requirement for the right-side occupant in the first row: the tangent value of the trunk tilt is less than 0; the trunk tilt requirement for the left-side occupant in the first row: the tangent value of the trunk tilt is greater than 0; and the trunk tilt requirement for the second-row occupant: the tangent value of the trunk tilt is greater than a predetermined first threshold and less than a predetermined second threshold.

[0174] In some embodiments, in the in-car sleep recognition device, the determination module further determines the eye reliability determination result based on the angle between the face orientation and the camera's imaging direction, before determining whether the occupant is in a sleep state using a sleep state recognition method that matches the eye reliability determination result, based on the determination result of the occupant's eye area in the target image. The smaller the angle between the face orientation and the camera's imaging direction, the higher the reliability when using the eyes for sleep recognition, and the larger the angle between the face orientation and the camera's imaging direction, the lower the reliability when using the eyes for sleep recognition.

[0175] In some embodiments, in the in-car sleep recognition device, the determination module, when determining the eye reliability determination result based on the angle formed by the orientation of the face and the imaging direction of the camera, specifically determines an angle parameter corresponding to the angle formed by the orientation of the face and the imaging direction of the camera, and determines the eye reliability determination result as pass if the angle parameter satisfies a predetermined reliability condition, and determines the eye reliability determination result as fail if the angle parameter does not satisfy the predetermined reliability condition.

[0176] In some embodiments, in the in-car sleep recognition device, the determination module, when determining the included angle parameter corresponding to the included angle between the orientation of the face and the orientation of the camera, specifically defines the coordinates of the origin of the camera coordinate system in the human head coordinate system. It calculates the first included angle between the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system and the z-axis of the human head coordinate system, determines the included angle parameter corresponding to the first included angle, calculates the second included angle between the projection of the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system in the xOy plane of the human head coordinate system and the x-axis of the human head coordinate system, and determines the included angle parameter corresponding to the second included angle. The included angle parameter corresponding to the first included angle is used to determine the eye reliability.

[0177] In some embodiments, in the in-car sleep recognition device, the first confirmation module, when determining the verification result for the sleep state based on the detection result for the occupant's target behavior in the target image, specifically determines the detection result for the movement of the occupant's target part in the target image and / or the detection result for the use of the target item within a predetermined second period, determines whether the detection result for the movement of the target part and / or the detection result for the use of the target item satisfies the corresponding sleep false detection suppression condition, and determines that the verification result for the sleep state is sleep false detection suppression if the detection result for the movement of the target part and / or the detection result for the use of the target item satisfies the corresponding sleep false detection suppression condition.

[0178] In some embodiments, in the in-car sleep recognition device, the first determination module, when determining the detection result of the movement of the target part of the occupant in the target image within a predetermined second period, specifically determines a face rotation angle parameter representing the rotation angle of the occupant's face in the target image within a predetermined second period, and determines the distance traveled by the occupant's arm in the target image within a predetermined second period.

[0179] In some embodiments, the in-car sleep recognition device, when the first determination module determines whether the detection result of movement of a target part satisfies the corresponding sleep false detection suppression conditions, specifically determines whether the rotation angle parameter of the face is greater than a predetermined third threshold and whether the distance traveled by the occupant's arm is greater than a predetermined fourth threshold. Different vehicle models and different seating areas have different predetermined fourth thresholds.

[0180] In some embodiments, in the in-car sleep recognition device, the first confirmation module, when determining whether the detection result of using a target item satisfies the corresponding sleep false detection suppression conditions, specifically detects whether a target item exists in the image, and if it exists, determines whether the target item is being held in the hand based on the contact relationship between the target item and the hand, and the contact relationship between the target item and a predetermined reference object, and if it determines that the target item is being held in the hand, the determination result is confirmed to be sleep false detection suppression.

[0181] In some embodiments, in the in-vehicle sleep recognition device, the second confirmation module, based on the verification results for the sleep state, determines the recognition result of the sleep state of an occupant in the vehicle cabin. Specifically, if the verification result is sleep false detection suppression, it updates the recognition result of the occupant's sleep state to a non-sleep state. If the verification result is sleep false detection non-suppression, it maintains the recognition result of the occupant's sleep state as a sleep state.

[0182] In some embodiments, the vehicle interior sleep recognition device uses the recognition result of the sleep state of the occupant in the vehicle interior to adjust the state of at least one device in the vehicle interior according to a predetermined sleep scene.

[0183] Based on the same inventive concept, embodiments of this disclosure further provide electronic devices corresponding to the in-car sleep recognition method. Since the principle of solving the problem of the electronic devices according to embodiments of this disclosure is similar to that of the in-car sleep recognition method according to embodiments of this disclosure, the implementation of the electronic devices can be referenced to the embodiments of the method, and the explanation of overlapping parts will be omitted.

[0184] Figure 10 shows a schematic configuration diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 1000 includes a processor 1002, a memory 1001 that stores machine-readable commands that the processor 1002 can execute, and a bus. When the electronic device 1000 is operating, the processor 1002 and the memory 1001 communicate via the bus, and when the machine-readable commands are executed by the processor 1002, the steps of the automobile interior sleep recognition method are performed.

[0185] Based on the same inventive concept, embodiments of this disclosure further provide a computer-readable storage medium corresponding to a car interior sleep recognition method. Since the principle of solving the problem of the computer-readable storage medium according to embodiments of this disclosure is similar to that of the car interior sleep recognition method according to embodiments of this disclosure, the implementation of the computer-readable storage medium can be referenced to the embodiments of the method, and the explanation of redundant parts will be omitted.

[0186] This computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the car interior sleep recognition method are performed.

[0187] Those skilled in the art will be able to refer to the corresponding processes in the embodiments of the methods described above for the working processes of the systems and apparatus described above; therefore, for the sake of clarity and brevity, such explanations are omitted in this disclosure. In some embodiments provided in this disclosure, the systems, apparatus and methods described may be implemented in other ways. The embodiments of the apparatus described above are illustrative only. For example, the division of modules is merely a logical functional division, and in actual implementation, different divisions may exist. For example, multiple modules or components may be combined or integrated into another system, or some features may be omitted or not implemented. Furthermore, the mutual coupling, direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via several interfaces, devices or modules, and may be an electrical, mechanical or other type of connection.

[0188] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in the same location or distributed across multiple network units. To achieve the objectives of this embodiment, it is possible to select some or all of the units according to actual requirements.

[0189] Furthermore, each functional unit in each embodiment of this disclosure may be integrated into a single processing unit, function as an independent physical entity, or two or more units may be integrated into a single unit.

[0190] The aforementioned functions can be implemented in the form of a software function unit, which, when sold or used as an independent product, can be stored in a processor-executable, non-volatile, computer-readable storage medium. From this understanding, the technical proposal of the present invention itself, or a portion of it that can contribute to the prior art, or a part of the technical proposal, can be implemented in the form of a software product. This computer software product is stored in a storage medium and includes a plurality of commands for a computer device (such as a personal computer, server, or network device) to perform all or part of the steps of the above method in each embodiment of the present disclosure. The storage medium includes various media capable of storing program code, such as USB disks, portable hard disks, ROMs, RAMs, magnetic disks, or optical disks.

[0191] The foregoing describes only specific embodiments of the Disclosure, and the scope of protection of the Disclosure is not limited thereto. Any modifications or substitutions made by a person skilled in the art within the technical scope described in the Disclosure are included in the scope of protection of the Disclosure. Therefore, the scope of protection of the Disclosure is equivalent to the claims.

[0192] Industrial applicability Embodiments of this disclosure provide a method, apparatus, electronic device, and medium for recognizing sleep inside an automobile cabin. The automobile cabin sleep recognition method first acquires a target image of the automobile cabin, and then, based on the reliability of using the occupant's eyes in the target image for sleep recognition, determines whether the occupant is in a sleep state using a different sleep state recognition method, and suppresses false detection of the sleep state according to the detection result of the occupant's target behavior in the image. This expands the determination range of sleep recognition by using different detection methods, improves the detection accuracy of sleep recognition, and further prevents false detection of the sleep state based on the detection result of the occupant's target behavior, thereby further improving the detection accuracy of sleep recognition.

[0193] Furthermore, the vehicle interior sleep recognition method, apparatus, electronic equipment, and media according to the embodiments of this disclosure are feasible and applicable to various industrial applications. For example, the vehicle interior sleep recognition method, apparatus, electronic equipment, and media according to the embodiments of this disclosure are applicable to the vehicle service sector.

Claims

1. The steps include: acquiring a target image of the car interior, including the occupants; Based on the determination result of the eye reliability of the occupant in the target image, the step of determining whether or not the occupant is in a sleep state using a sleep state recognition method that is suitable for the determination result of the eye reliability, which represents the reliability when the eye is used for sleep recognition, If it is determined that the occupant is in a sleep state, the step is to determine the verification result of the occupant's sleep state based on the detection result of the occupant's target behavior in the target image, The step of determining the result of recognizing the sleep state of the occupants in the vehicle cabin based on the verification results regarding the sleep state. A method for recognizing sleep inside an automobile, characterized by the features described above.

2. The step of determining whether the occupant is in a sleep state using a sleep state recognition method that conforms to the determination result of the eye reliability is as follows: If the reliability of the occupant's eyes in the target image is deemed satisfactory, the step of determining whether the occupant is asleep or not based on the state of the occupant's eyes in the target image is to be performed. If the determination result of the reliability of the occupant's eyes in the target image is unsatisfactory, the step includes determining whether or not the occupant is in a sleeping state based on the occupant's body posture in the target image. The method for recognizing sleep inside an automobile according to feature 1.

3. The step of determining whether or not a occupant is asleep based on the state of the occupant's eyes in the aforementioned target image is: Based on the state of the occupant's eyes in the target image, the step of determining the cumulative time length of the closed-eye image in which the eyes are closed during a predetermined first period, The steps include determining whether the cumulative time length of the closed-eye images satisfies a predetermined closed-eye time length requirement, If the conditions are met, the step of determining that the occupant is in a sleeping state includes: The method for recognizing sleep inside an automobile according to feature 2.

4. The step of determining whether or not the occupant is in a sleeping state based on the occupant's physical posture in the aforementioned target image is: The steps include recognizing the target human skeletal point of the occupant in the aforementioned target image, A step of determining the head-to-torso length ratio and trunk inclination of the occupant based on the target human skeletal point of the occupant, The step includes determining whether the occupant is in a sleeping state based on the head-to-torso length ratio and trunk inclination, The head-to-torso length ratio represents the ratio between the length of the occupant's head and the length of their torso in the target image, and the trunk inclination represents the degree of inclination of the occupant's trunk in the target image relative to the horizontal direction of the image. The method for recognizing sleep inside an automobile according to feature 2.

5. The step of determining whether the occupant is in a sleeping state based on the head-to-torso length ratio and trunk inclination is as follows: Based on the target skeletal points of the occupant, the step of determining the seating area of ​​the occupant within the vehicle, The steps include determining whether the head-to-body length ratio of the occupant satisfies the head-to-body length ratio requirement corresponding to the seat area, and whether the trunk inclination of the occupant satisfies the trunk inclination requirement corresponding to the seat area, If both requirements are met, the process includes the step of determining that the crew is in a sleep state, Different seating areas correspond to different head-to-body length proportionality requirements and trunk inclination requirements. The method for recognizing sleep inside an automobile according to feature 4.

6. Different head-to-torso length ratio requirements and trunk tilt requirements corresponding to different seating areas are determined based on images of occupants lying down or with their torso tilted in different vehicle types. The method for recognizing sleep inside an automobile according to feature 5.

7. Different trunk inclination requirements for different seating areas are: The trunk inclination requirement for the occupant on the right side of the first row is that the fuselage is tilted to the left. The trunk inclination requirement for the occupant on the left side of the first row is that the fuselage is tilted to the right. The trunk inclination requirement for the second-row occupants is that the trunk inclination is within a predetermined threshold range. The method for recognizing sleep inside an automobile according to feature 5.

8. The trunk inclination requirement for the occupant on the right side of the first row is that the tangent value of the trunk inclination is less than 0. The trunk inclination requirement for the left-hand occupant in the first row is that the tangent value of the trunk inclination is greater than 0. The trunk inclination requirement for the second-row occupants is that the tangent value of the trunk inclination is greater than a predetermined first threshold and less than a predetermined second threshold. The method for recognizing sleep inside an automobile according to claim 5 or 7.

9. Before the step of determining whether or not the occupant is in a sleep state using a sleep state recognition method that is suitable for the eye reliability determination result of the target image, the car interior sleep recognition method is: The process further includes a step of determining the eye confidence level based on the angle formed by the orientation of the face and the imaging direction of the camera, The smaller the angle between the face orientation and the camera's imaging direction, the higher the reliability of using the eye area for sleep recognition. Conversely, the larger the angle between the face orientation and the camera's imaging direction, the lower the reliability of using the eye area for sleep recognition. The method for recognizing sleep inside an automobile according to feature 1.

10. The step of determining the eye reliability result based on the angle formed by the orientation of the face and the imaging direction of the camera is: The steps include determining the included angle parameter corresponding to the included angle formed by the orientation of the face and the imaging direction of the camera, If the aforementioned angle parameter satisfies a predetermined reliability condition, the step of determining the reliability of the eye section as passing, The process includes the step of determining the eye reliability result as a failure if the aforementioned angle parameter does not satisfy a predetermined reliability condition. The method for recognizing sleep inside an automobile according to feature 9.

11. The step of determining the included angle parameter corresponding to the included angle formed by the orientation of the face and the orientation of the camera is: The steps include defining the coordinates of the origin of the camera coordinate system in the human head coordinate system, The steps include: calculating the first included angle formed by the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system, and the z-axis of the human head coordinate system, and determining the included angle parameter corresponding to the first included angle; The process includes the steps of: calculating the second included angle formed by the projection of the line connecting the origin of the camera coordinate system and the origin of the human head coordinate system onto the xOy plane of the human head coordinate system and the x-axis of the human head coordinate system; and determining the included angle parameter corresponding to the second included angle. The included angle parameter corresponding to the first included angle is used to determine the reliability of the eye. The method for recognizing sleep inside an automobile according to feature 10.

12. The step of determining the verification result for the sleep state based on the detection result for the occupant's target behavior in the target image is as follows: A step of determining the detection result of the movement of the target part of the occupant in the target image and / or the detection result of the use of the target item within a predetermined second period, A step of determining whether the detection result of movement of the target area and / or the detection result of use of the target item satisfies the corresponding sleep false detection suppression conditions, The step includes determining that the verification result for the sleep state is sleep false detection suppression if the detection result for the movement of the target area and / or the detection result for the use of the target item satisfies the corresponding sleep false detection suppression conditions. The method for recognizing sleep inside an automobile according to feature 1.

13. The step of determining the detection result of the movement of the target part of the occupant in the target image within a predetermined second period is: A step of determining a face rotation angle parameter that represents the rotation angle of the occupant's face in the target image within a predetermined second period, The process includes the step of determining the distance traveled by the occupant's arm in the target image within a predetermined second period. The method for recognizing sleep inside an automobile according to feature 12.

14. The step of determining whether the detection result of movement of the target area satisfies the corresponding sleep misdetection suppression conditions is: The steps include determining whether the rotation angle parameter of the face is greater than a predetermined third threshold, The step includes determining whether the distance traveled by the occupant's arm is greater than a predetermined fourth threshold, The predetermined fourth threshold differs depending on the vehicle model and the seat area. The method for recognizing sleep inside an automobile according to feature 13.

15. The step of determining whether the detection result of using the target item satisfies the corresponding sleep false detection suppression conditions is: A step of detecting whether or not a target object exists in the image, If present, the step of determining whether the target item is being held in the hand based on the contact relationship between the target item and the hand, and the contact relationship between the target item and a predetermined reference object, The process includes the step of determining that the determination result is sleep false detection suppression if it is determined that the target item is being held in the hand. The method for recognizing sleep inside an automobile according to feature 13.

16. Based on the verification results regarding sleep states, the step of determining the recognition result of the sleep state of occupants in the vehicle cabin is: If the verification result indicates suppression of false sleep detection, the steps include updating the recognition result of the occupant's sleep state to a non-sleep state, If the verification result is that sleep misdetection is not suppressed, the step includes maintaining the recognition result of the occupant's sleep state as a sleep state. The method for recognizing sleep inside an automobile according to feature 1.

17. The results of recognizing the sleep state of the occupants in the vehicle cabin are used to adjust the state of at least one device in the vehicle cabin according to a predetermined sleep scene. The method for recognizing sleep inside an automobile according to feature 1.

18. An acquisition module that acquires target images of the vehicle interior, including the occupants, A determination module that determines whether or not a occupant is in a sleep state using a sleep state recognition method that conforms to the determination result of eye reliability, which represents the reliability when the eye area is used for sleep recognition, based on the determination result of the eye area reliability in the target image, If it is determined that the occupant is in a sleep state, a first confirmation module confirms the verification result regarding the occupant's sleep state based on the state of the target object in the target image, The system includes a second confirmation module that determines the recognition result of the sleep state of the occupants in the vehicle cabin based on the verification results for the sleep state. A car interior sleep recognition device characterized by the following features.

19. An electronic device including a processor, a memory storing machine-readable commands that the processor can execute, and a bus, When the electronic device is operating, the processor and the memory communicate via the bus, and when the machine-readable command is executed by the processor, the steps of the car interior sleep recognition method according to any one of claims 1 to 17 are performed. An electronic device characterized by the following features.

20. A computer-readable storage medium that stores computer programs, When the computer program is executed on the processor, the steps of the car interior sleep recognition method according to any one of claims 1 to 17 are performed. A computer-readable storage medium characterized by the following features.

Images