Posture detection method and device, computer readable medium and electronic device

Abstract

The present disclosure provides a posture detection method and device, a computer readable medium and an electronic device, and relates to the technical field of computers. The method comprises: acquiring a front-facing face image; determining first angle data of a face contained in the front-facing face image in a device coordinate system; acquiring device sensor data, and determining second angle data of a terminal device in a world coordinate system through the device sensor data; determining real angle data of the face in the world coordinate system through the first angle data and the second angle data, and determining a real posture corresponding to the face according to the real angle data. The present disclosure can determine real angle data of a face of a user using a mobile device through collected images and sensors of the device, improve the accuracy of the real angle data of the face of the user, and timely remind of an unhealthy posture, ensure the health of the user, and improve the use experience of the user.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically to an attitude detection method, attitude detection device, computer-readable medium, and electronic device. Background Technology

[0002] As people's living standards continue to improve, smartphones, tablets, and other electronic devices are becoming increasingly indispensable tools. However, when using smartphones or tablets for work or entertainment, people are prone to unhealthy postures such as looking down, lying on their side, or lying down. Maintaining such unhealthy postures for a long time can easily lead to health problems.

[0003] Currently, relevant posture detection solutions either use specific posture detection devices, such as desk lamps, chairs, and school desks, but these solutions have high hardware costs; or they use cameras in fixed positions, but these solutions have limited applicability and the angle of the face captured is relative to the camera, not the actual angle, resulting in low accuracy in posture detection. Summary of the Invention

[0004] The purpose of this disclosure is to provide a posture detection method, posture detection device, computer-readable medium, and electronic device, thereby improving the accuracy of user face angle recognition and expanding the scope of application, at least to some extent, while reducing hardware costs.

[0005] According to a first aspect of this disclosure, an attitude detection method is provided, comprising:

[0006] Acquire the frontal face image;

[0007] Determine the first angle data of the face contained in the front face image, wherein the first angle data is the angle data of the face relative to the terminal device in the device coordinate system;

[0008] Acquire device sensor data and determine second angle data using the device sensor data, wherein the second angle data is the angle data of the terminal device in the world coordinate system;

[0009] The true angle data of the face is determined by the first angle data and the second angle data. The true angle data is the angle data of the face in the world coordinate system. The true pose of the face is determined based on the true angle data.

[0010] Wherein, the first angle data, the second angle data, and the actual angle data are Euler angles.

[0011] According to a second aspect of this disclosure, an attitude detection device is provided, comprising:

[0012] Image acquisition module, used to acquire front-facing face images;

[0013] The first angle determination module is used to determine the first angle data of the face contained in the front face image. The first angle data is the angle data of the face relative to the terminal device in the device coordinate system.

[0014] The second angle determination module is used to acquire device sensor data and determine second angle data through the device sensor data. The second angle data is the angle data of the terminal device in the world coordinate system.

[0015] The pose recognition module is used to determine the true angle data of the face through the first angle data and the second angle data, wherein the true angle data is the angle data of the face in the world coordinate system, and to determine the true pose of the face based on the true angle data.

[0016] Wherein, the first angle data, the second angle data, and the actual angle data are Euler angles.

[0017] According to a third aspect of this disclosure, a computer-readable medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method described above.

[0018] According to a fourth aspect of this disclosure, an electronic device is provided, characterized in that it comprises:

[0019] Processor; and

[0020] Memory is used to store one or more programs, which, when executed by one or more processors, cause the one or more processors to perform the methods described above.

[0021] One embodiment of the pose detection method disclosed herein determines a first angle data of the face contained in a captured front-facing face image relative to the terminal device in the device coordinate system. Then, it determines a second angle data of the terminal device in the world coordinate system using device sensor data. Finally, it determines the true angle data of the face in the world coordinate system using both the first and second angle data, and determines the true pose of the face based on the true angle data. On one hand, by jointly determining the true angle data of the user's face using the first angle data of the face contained in the front-facing face image and the second angle data determined by the device sensor data, the accuracy of true pose recognition can be effectively improved. On the other hand, by using a front-facing camera and pose sensor, which are present in most current terminal devices, pose detection can effectively reduce hardware costs and expand the applicability of pose detection.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0024] Figure 1 A schematic diagram of an exemplary system architecture to which embodiments of the present disclosure may be applied is shown;

[0025] Figure 2 The schematic diagram illustrates a flowchart of an attitude detection method according to an exemplary embodiment of the present disclosure;

[0026] Figure 3 This illustration schematically shows a flowchart of a method for detecting the user operation status of a terminal device in an exemplary embodiment of the present disclosure;

[0027] Figure 4 This schematic diagram illustrates a real angle data of a user's face in an exemplary embodiment of the present disclosure;

[0028] Figure 5 This schematic diagram illustrates a true posture of looking down in an exemplary embodiment of the present disclosure;

[0029] Figure 6 This diagram illustrates a real-world posture, either a head-tilting posture or a side-lying posture, in an exemplary embodiment of this disclosure.

[0030] Figure 7 This schematic diagram illustrates a true squinting posture in an exemplary embodiment of the present disclosure.

[0031] Figure 8 This schematic diagram illustrates a real-world posture, which is a lying posture, in an exemplary embodiment of this disclosure.

[0032] Figure 9 This schematic diagram illustrates the network structure of a face detection model in an exemplary embodiment of the present disclosure.

[0033] Figure 10 This schematic diagram illustrates a method for determining an output face region in an exemplary embodiment of the present disclosure.

[0034] Figure 11 This schematic diagram illustrates a process for determining true angle data in an exemplary embodiment of the present disclosure.

[0035] Figure 12 This schematically illustrates a flowchart of detecting an undesirable posture in an exemplary embodiment of the present disclosure;

[0036] Figure 13 This schematic diagram illustrates the composition of an attitude detection device in an exemplary embodiment of the present disclosure.

[0037] Figure 14 A schematic diagram of an electronic device to which embodiments of the present disclosure may be applied is shown. Detailed Implementation

[0038] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0039] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0040] Figure 1 A schematic diagram of a system architecture for an exemplary application environment in which an attitude detection method and apparatus according to embodiments of the present disclosure can be applied is shown.

[0041] like Figure 1 As shown, system architecture 100 may include one or more of terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables. Terminal devices 101, 102, and 103 may be various electronic devices with image processing capabilities and artificial intelligence processors, including but not limited to mobile devices such as smartphones, tablets, and smart glasses. It should be understood that... Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.

[0042] The posture detection method provided in this embodiment is generally executed by terminal devices 101, 102, and 103, and correspondingly, the posture detection device is generally disposed in terminal devices 101, 102, and 103. However, it is readily understood by those skilled in the art that the posture detection method provided in this embodiment can also be executed by server 105, and correspondingly, the posture detection device can also be disposed in server 105. This exemplary embodiment does not impose any special limitations on this.

[0043] For example, in one exemplary embodiment, the user may upload the current front face image and device sensor data collected at the moment to the server 105 through terminal devices 101, 102, and 103. After the server generates the real pose using the pose detection method provided in this embodiment, it transmits the real pose to the terminal devices 101, 102, and 103.

[0044] The following describes the attitude detection method and attitude detection device of this disclosure in detail, taking the execution of the method by a terminal device as an example.

[0045] Figure 2 The flowchart of an attitude detection method in this exemplary embodiment is shown, which may include the following steps S210 to S240:

[0046] In step S210, the front face image is acquired.

[0047] In an exemplary embodiment, the front-facing face image refers to the face image of the user in front of the screen acquired through the front-facing camera of the terminal device.

[0048] In step S220, the first angle data of the face contained in the front face image is determined.

[0049] In an exemplary embodiment, the first angle data refers to the angle data of the face in the front face image relative to the screen of the terminal device. The first angle data is the angle data of the face in the front face image relative to the terminal device in the device coordinate system. The first angle data can be represented by Euler angles, specifically including pitch angle, roll angle, and yaw angle.

[0050] In step S230, device sensor data is acquired, and second angle data is determined using the device sensor data.

[0051] In an exemplary embodiment, device sensor data refers to data output by the terminal device through an attitude sensor. For example, the attitude sensor may be a gravity sensor and a magnetic sensor, and the device sensor data may be gravitational acceleration data output by the gravity sensor and magnetic data output by the magnetic sensor. The attitude sensor may also be an inertial sensor (IMU), and the device sensor data may be acceleration data output by the inertial sensor. This example embodiment does not impose any special limitations on this.

[0052] The second angle data refers to the device angle data of the terminal device itself. The device angle data is the angle data of the terminal device in the world coordinate system. The second angle data can also be represented by Euler angles, which can include pitch angle, roll angle, and yaw angle.

[0053] In step S240, the true angle data is determined by the first angle data and the second angle data, and the true pose corresponding to the face is determined based on the true angle data.

[0054] In an exemplary embodiment, the true angle data refers to the angle data of the user's face in the world coordinate system. Specifically, the true angle data of the user's face in the world coordinate system can be determined by combining the first angle data of the user's face in the device coordinate system and the second angle data of the terminal device in the world coordinate system. Based on this true angle data, the true pose corresponding to the user's face can be determined, effectively improving the accuracy of true pose recognition.

[0055] Steps S210 to S240 will be described in detail below.

[0056] In one exemplary embodiment, when the terminal device is detected to be in a user operation state, the acquisition of the user's face image in front of the screen, i.e., the front face image, can be started. This can effectively avoid unnecessary energy consumption caused when the user is not using the terminal device and reduce misidentification.

[0057] User operation state refers to the state of the terminal device when the user is operating it. For example, the user operation state can be the state where the terminal device's display screen is always on, or the state where it is interacting with the user's interactive operation information. This example embodiment does not make any special limitations on this.

[0058] Optionally, step S210 can be performed by... Figure 3 The steps in the process of determining the user's operation status on the terminal device are referenced. Figure 3 As shown, it can specifically include:

[0059] Step S310: Obtain system operation data, which includes interactive operation information and screen illumination time;

[0060] Step S320: In response to detecting the interactive operation information and / or in response to the screen lighting time being greater than or equal to a time threshold, determine that the current state is a user operation state.

[0061] System operation data refers to the data generated by the terminal device during user operation. For example, system operation data can be interactive operation information or screen illumination time. Of course, those skilled in the art will understand that as long as the system operation data that can be identified as being in the user operation state of the terminal device is within the protection scope of this embodiment. No special limitation is made on the type of system operation data here.

[0062] Interactive operation information refers to the data generated during the interaction between the user and the terminal device. For example, interactive operation information may be the user's touch operation on the screen, the pressing operation of physical buttons, or the input of voice commands, etc. This embodiment is not limited to these.

[0063] The time threshold refers to a pre-set threshold used to avoid misidentification. For example, the time threshold can be 1 minute or 5 minutes. It can be customized according to the actual use case, or a relevant setting interface can be provided for users to set themselves. This example embodiment does not make any special limitations on this.

[0064] Assuming the time threshold can be 1 minute, then when interactive operation information is continuously detected, or the screen is continuously lit for a duration greater than or equal to 1 minute, it can be considered that the user is continuously using the terminal device, confirming that the terminal device is in a user operation state, and then proceeding with subsequent processes, thus avoiding misidentification operations performed by the user when temporarily opening the terminal device.

[0065] In an exemplary embodiment, after step S240 detects the user's actual posture, it can determine whether the user is using the terminal device in an unhealthy posture based on the detected actual posture, and when it is determined that the posture is unhealthy, it can remind the user so that the user can adjust their sitting posture in time to protect the user's health.

[0066] Specifically, a posture angle threshold can be preset. The posture angle threshold is a preset threshold used to detect whether the user's posture is an undesirable posture. For example, the posture angle threshold can be 50°, that is, when the user's actual posture exceeds 50° in a certain direction, the user can be considered to be in an undesirable posture. Of course, the posture angle threshold can also be 30°, etc. The specific settings can be customized according to the user's habits or detection sensitivity parameters in actual use. This example embodiment does not impose any special limitations on this.

[0067] When detection is required, a pre-set attitude angle threshold can be obtained. When the actual attitude angle is detected to be greater than or equal to the attitude angle threshold, the actual attitude can be identified as an undesirable attitude. At the same time, a warning message corresponding to the undesirable attitude can be generated to remind the user to adjust the undesirable attitude.

[0068] The warning information can be a vibration reminder, or it can be a display of poor posture and correction methods on the screen. Of course, it can also be a voice reminder. The voice content can include the duration of the poor posture, the degree of cervical fatigue, or the corresponding adjustment and rest methods, etc. This example embodiment is not limited to these.

[0069] The warning message can be output once or multiple times periodically. In one possible embodiment, to avoid excessive interference, a pop-up window can be provided to the user with options to remind them later or not to remind them again. If the user repeatedly selects to remind them later, and the system detects that the user is in a poor posture, a blocking reminder, such as a full-screen flashing reminder, will be issued. This embodiment is not limited to this.

[0070] Of course, the terminal device can also continuously record data related to the user's poor posture and provide health reports to the user periodically, which is also within the scope of protection of this embodiment.

[0071] In an exemplary embodiment, an undesirable posture may include one or more combinations of head-down posture, head-tilt posture, side-lying posture, front-lying posture, and squinting posture. Of course, the examples given here are only common undesirable postures, and specific undesirable postures can be customized according to actual usage. This embodiment is not limited to this.

[0072] One possible implementation is to display all types of undesirable postures to the user, who can then select a specific type for detection and identification based on their own situation. This avoids repeated misidentifications caused by the user's own actions and ensures a better user experience.

[0073] In one exemplary embodiment, the true attitude angles may include the true pitch angle, the true yaw angle, and the true roll angle, with reference to... Figure 4 As shown, the world coordinate system of the user's face is 401. Then, the rotation of the user's face along the XOZ plane (i.e., the rotation about the Y axis) is the user's true pitch angle 402, the rotation of the user's face along the YOZ plane (i.e., the rotation about the X axis) is the user's true yaw angle 403, and the rotation of the user's face along the XOY plane (i.e., the rotation about the Z axis) is the user's true roll angle 404.

[0074] Optionally, if the actual pitch angle of the user's face is detected to be greater than or equal to the pitch angle threshold, the actual posture can be determined as a head-down posture.

[0075] The pitch angle threshold refers to a pre-set threshold used to determine whether the user's actual pitch angle is in a head-down posture. For example, the pitch angle threshold could be 30° or 50°, etc. (Reference) Figure 5 As shown, relevant studies indicate that when the human body is in an upright posture (501), the weight borne by the cervical spine is the same as the weight of the head (approximately 5 kg). However, when the head is tilted, the force on the cervical spine changes accordingly. Since the weight of the head is constant, the additional weight borne by the cervical spine comes entirely from the tension generated by the muscles. When the head tilt angle (i.e., the pitch angle) reaches 30° in the head-down posture (502), the pressure on the cervical spine is 2.5 times that of the upright posture (501) (approximately equivalent to 18 kg); when the head tilt angle (i.e., the pitch angle) reaches 50° in the head-down posture (503), the pressure on the cervical spine is 5.5 times that of the upright posture (501) (approximately equivalent to 27 kg). Therefore, in practical applications, the pitch angle threshold can be set with reference to relevant research results; this embodiment is not limited to this.

[0076] Optionally, when the detected actual roll angle of the user's face is greater than or equal to the roll angle threshold, the actual posture can be determined as a head-tilt posture or a side-lying posture. The roll angle threshold is a pre-set threshold used to determine whether the user's actual roll angle is in a head-tilt or side-lying posture. For example, the roll angle threshold can be 15° or 30°, etc. Of course, in practical applications, the roll angle threshold can be customized based on human health data; this example embodiment does not impose any special limitations on this.

[0077] refer to Figure 6As shown, when the human body is in a side-lying position 601 or a head-tilting position 602, the head and cervical spine are not in a straight line, which leads to a significant increase in pressure on the cervical spine and affects human health.

[0078] Optionally, if the actual yaw angle of the user's face is detected to be greater than or equal to the yaw angle threshold, the actual pose can be determined as a squint pose.

[0079] The yaw angle threshold is a pre-set threshold used to determine whether the user's actual yaw angle is in a squint posture. For example, the yaw angle threshold can be 30° or 45°. Of course, in practical applications, the roll angle threshold can be customized based on human health data. This example embodiment does not impose any special limitations on this. (Reference) Figure 7 As shown, when the user's actual yaw angle is greater than or equal to the yaw angle threshold, the human body is in a strabismic posture 701. Continuously using the terminal device in a strabismic posture will have a significant impact on eye development and vision, causing health problems.

[0080] Optionally, if the actual pitch angle of the user's face is detected to be greater than or equal to the pitch angle threshold, and the second angle data is greater than or equal to the device angle threshold, then the actual posture can be determined as a lying posture.

[0081] The device angle threshold refers to a pre-set threshold used to help determine whether the user's face is in a lying position. For example, the device angle threshold can be 60° or 90°, and can be customized according to the actual situation. This example embodiment does not impose any special limitations on this. (Reference) Figure 8 As shown, when the actual pitch angle of the user's face is greater than or equal to the pitch angle threshold, and the second angle data corresponding to the terminal device is greater than or equal to the device angle threshold, it can be determined that the human body posture is in a lying posture 801.

[0082] In one exemplary embodiment, the face region in the foreground face image can be determined by inputting the foreground face image into a face detection model, and the face region can be cropped. Then, the cropped face region can be input into a face angle model to output the first angle data of the face in the face region.

[0083] Here, the face detection model refers to a deep learning model used to identify and detect face regions appearing in an image. For example, the face detection model may include, but is not limited to, the Retinaface network. Of course, other deep learning models that can achieve face detection are also within the scope of protection of this embodiment.

[0084] refer to Figure 9As shown, the face detection model 900 for detecting face regions in a foreground face image may include at least a feature extraction pyramid network 901, a context module 902, and a multi-task loss function 903. The feature extraction pyramid network 901 is mainly used for global detection of face regions of different sizes in the foreground face image. In this embodiment, MobileNet is used as the feature extraction pyramid network 901, which can effectively improve the computational speed of the face detection model 900 on mobile devices and reduce the computational load. Multiple independent context modules 902 are mainly used to improve the modeling capability of the face detection model 900. The multi-task loss function 903 is mainly used to distinguish between multiple identified face regions. Finally, multiple bounding boxes are used to label the identified face regions in the foreground face image. Specifically, the structure of the context module 902 can be a network structure 904, and in this embodiment, the input image size of the face detection model 900 can be 320*320, which can further improve the recognition and detection speed of the face detection model 900.

[0085] Optionally, the foreground face image is input into the face detection model to obtain a foreground face image with at least one labeled face bounding box. Then, the area of ​​each face bounding box can be calculated, and the face bounding box with the largest area can be taken as the face region in the foreground face image.

[0086] refer to Figure 10 As shown, when the foreground face image 1000 is input into the face detection model, face regions 1001, 1002, 1003, and 1004 are identified and detected. Then, if true pose detection is performed on all face regions 1001, 1002, 1003, and 1004, the area of ​​the corresponding bounding box can be calculated for each region. Assuming that the bounding box area of ​​face region 1001 is the largest, face region 1001 can be used as the final face region for true pose detection. This not only effectively reduces the computational load but also lowers false recognition rates and improves the accuracy of true pose detection.

[0087] A face angle model refers to a deep learning model used to identify the angle of a face in an image relative to the screen of a terminal device. For example, a face angle model may include, but is not limited to, MLPMixer. MLPMixer can effectively reduce the number of model parameters, enabling face angle recognition to be applied to mobile devices. Of course, other deep learning models that can achieve face detection are also within the scope of protection of this embodiment.

[0088] Optionally, the face angle model can be trained using a pre-collected sample dataset. This sample dataset can include both real and synthetic facial landmark datasets. Depending on the angle of the landmark annotations, a larger dataset allows the face angle model to recognize a wider range of face angles. Additionally, a masking layer (simulating the presence of masks or other obstructions) can be added to both the real and synthetic facial landmark datasets to enable the face angle model to recognize faces wearing masks or with obstructions. Furthermore, random angle flipping or mirroring operations can be performed on the sample data to allow the face angle model to support a roll angle range of -180° to +180°, accurately recognizing real faces at any angle within 360°.

[0089] In an exemplary embodiment, when the terminal device is in a user operation state, a front image can be captured first and then input into a face detection model to determine whether there is a face in front of the screen. Pose detection can be performed when the terminal device is in a user operation state and a face is detected in front of the screen, which can further ensure the accuracy of pose recognition results and avoid misidentification.

[0090] In one exemplary embodiment, it can be achieved through Figure 11 The steps described in the document determine the true angle data using the first angle data and the second angle data. (Refer to...) Figure 11 As shown, it can specifically include:

[0091] Step S1110: Obtain the preset angle fitting coefficients;

[0092] Step S1120: Fit the first angle data and the second angle data using the angle fitting coefficient to obtain the true angle data.

[0093] Among them, the angle fitting coefficient refers to the weighting coefficient obtained by fitting a pre-defined linear regression model with a large amount of sample data.

[0094] For example, a pre-defined linear regression model can be represented by equation (1):

[0095] A * Face Angle + B * Terminal Device Angle = Actual Face Angle

[0096] Where A and B are the preset angle fitting coefficients, the face angle is the output of the face angle model, the terminal device angle is the sensor data output by the terminal device's sensor, and the real face angle is the real angle of the face measured by tools such as a level and protractor, i.e., the label data. By collecting a sample set of {relative face angle of the face relative to the terminal device, terminal device angle, and real face angle}, the angle fitting coefficients can be obtained by performing linear regression calculation. In use, the obtained first angle data (i.e., relative face angle) and second angle data (i.e., terminal device angle) are substituted into the relation (1) to obtain the real angle data of the user's face (i.e., the real face angle).

[0097] Of course, the linear regression model in this embodiment is only an illustrative example. In actual use, linear regression calculation can also be performed by artificial intelligence models. This example embodiment does not impose any special limitations on this.

[0098] Figure 12 The illustration schematically shows a flowchart of detecting an undesirable posture in an exemplary embodiment of the present disclosure.

[0099] refer to Figure 12 As shown, in step S1210, a front-facing image is acquired through the front-facing camera, and the front-facing image is detected through a face detection model.

[0100] Step S1220: Determine whether there is a face in the front image. If there is a face region in the front image, proceed to step S1230; otherwise, return to step S1210 to re-detect.

[0101] Step S1230: Input the face region into the face angle model to obtain the first angle data;

[0102] Step S1240: Determine the second angle data using the device sensor data;

[0103] Step S1250: Perform linear regression fitting on the first angle data and the second angle data using the angle fitting coefficient to obtain the true angle data, and determine the true attitude based on the true angle data.

[0104] Step S1260: Determine whether the actual pose is an undesirable pose. If the actual pose is an undesirable pose, proceed to step S1270; otherwise, end the current process.

[0105] Step S1270: Generate and display a warning message to remind the user to adjust the poor posture and end the current process.

[0106] In summary, this exemplary embodiment can determine the first angle data of the face contained in the acquired front-facing face image relative to the terminal device in the device coordinate system. Then, the second angle data of the terminal device in the world coordinate system can be determined through device sensor data. Furthermore, the true angle data of the face in the world coordinate system can be determined using both the first and second angle data, and the true pose of the face can be determined based on the true angle data. On one hand, by jointly determining the true angle data of the user's face using the first angle data of the face contained in the front-facing face image and the second angle data determined by the device sensor data, the accuracy of true pose recognition can be effectively improved. On the other hand, using the front-facing camera and pose sensor, which are present in most current terminal devices, for pose detection can effectively reduce hardware costs and expand the applicability of pose detection.

[0107] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0108] Further reference Figure 13 As shown, this example embodiment also provides a posture detection device 1300, which may include an image acquisition module 1310, a first angle determination module 1320, a second angle determination module 1330, and a posture recognition module 1340. Wherein:

[0109] Image acquisition module 1310 is used to acquire front face images;

[0110] The first angle determination module 1320 is used to determine the first angle data of the face contained in the front face image, wherein the first angle data is the angle data of the face relative to the terminal device in the device coordinate system.

[0111] The second angle determination module 1330 is used to acquire device sensor data and determine second angle data through the device sensor data. The second angle data is the angle data of the terminal device in the world coordinate system.

[0112] The pose recognition module 1340 is used to determine the true angle data of the face through the first angle data and the second angle data, wherein the true angle data is the angle data of the face in the world coordinate system, and to determine the true pose of the face based on the true angle data; wherein the first angle data, the second angle data and the true angle data are Euler angles.

[0113] In an exemplary embodiment, the attitude detection device 1300 may further include a poor attitude warning module, which can be used for:

[0114] Obtain the attitude angle threshold;

[0115] If the attitude angle of the true attitude is greater than or equal to the attitude angle threshold, then the true attitude is determined to be an undesirable attitude; and

[0116] Generate warning information corresponding to the poor posture.

[0117] In an exemplary embodiment, an undesirable attitude may include a head-down attitude, a head-tilt attitude, a side-lying attitude, and a squint attitude; the true attitude angle may include the true pitch angle, the true yaw angle, and the true roll angle; and the attitude angle threshold may include the pitch angle threshold, the yaw angle threshold, and the roll angle threshold.

[0118] The poor posture warning module can be used for:

[0119] If the actual pitch angle is greater than or equal to the pitch angle threshold, then the actual attitude is determined to be the head-down attitude; or

[0120] If the actual roll angle is greater than or equal to the roll angle threshold, then the actual posture is determined to be either the head-tilt posture or the side-lying posture; or

[0121] If the actual yaw angle is greater than or equal to the yaw angle threshold, then the actual attitude is determined as the squint attitude.

[0122] In an exemplary embodiment, poor posture may further include a lying posture, and the posture angle threshold may further include a device angle threshold. The poor posture warning module may be used to:

[0123] If the actual pitch angle is greater than or equal to the pitch angle threshold, and the second angle data is greater than or equal to the device angle threshold, then the actual posture is determined as the lying posture.

[0124] In an exemplary embodiment, the first angle determination module 1320 may be used to:

[0125] The foreground face image is input into the face detection model to determine the face region in the foreground face image;

[0126] The face region is cropped, and the cropped face region is input into the face angle model, and the first angle data of the face in the face region is output.

[0127] In an exemplary embodiment, the first angle determination module 1320 may be used to:

[0128] The foreground face image is input into the face detection model, and multiple face bounding boxes are marked in the foreground face image;

[0129] The area of ​​the face frame is determined, and the face frame with the largest area is taken as the face region in the foreground face image.

[0130] In one exemplary embodiment, the pose recognition module 1340 can be used to:

[0131] Obtain the preset angle fitting coefficients;

[0132] The first angle data and the second angle data are fitted and calculated using the angle fitting coefficients to obtain the true angle data.

[0133] In one exemplary embodiment, the image acquisition module 1310 can be used to:

[0134] In response to user interaction, the system retrieves the foreground face image; and

[0135] Acquire system operation data, which includes interactive operation information and screen illumination time;

[0136] If the system detects the interactive operation information and / or the screen illumination time is greater than or equal to a time threshold, it determines that the user is currently in an operation state.

[0137] The specific details of each module in the above-mentioned device have been described in detail in the method section of the implementation. For any undisclosed details, please refer to the implementation content of the method section, and therefore will not be repeated here.

[0138] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0139] An exemplary embodiment of this disclosure provides an electronic device for implementing an attitude detection method, which may be a terminal device or a server. The electronic device includes at least a processor and a memory, the memory storing executable instructions of the processor, and the processor configured to execute the attitude detection method by executing the executable instructions.

[0140] The following is based on Figure 14 Taking electronic device 1400 as an example, the construction of electronic device in this disclosure will be described by way of example. Figure 14The electronic device 1400 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0141] like Figure 14 As shown, the electronic device 1400 is presented in the form of a general-purpose computing device. The components of the electronic device 1400 may include, but are not limited to: at least one processing unit 1410, at least one storage unit 1420, a bus 1430 connecting different system components (including storage unit 1420 and processing unit 1410), and a display unit 1440.

[0142] The storage unit 1420 stores program code, which can be executed by the processing unit 1410, causing the processing unit 1410 to execute the attitude detection method in this specification.

[0143] Storage unit 1420 may include readable media in the form of volatile storage units, such as random access memory (RAM) 1421 and / or cache memory 1422, and may further include read-only memory (ROM) 1423.

[0144] Storage unit 1420 may also include a program / utility 1424 having a set (at least one) of program modules 1425, such program modules 1425 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0145] Bus 1430 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0146] Electronic device 1400 can also communicate with one or more external devices 1470 (e.g., sensor devices, Bluetooth devices, etc.), and with one or more devices that enable users to interact with electronic device 1400, and / or with any device that enables electronic device 1400 to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interface 1450. Furthermore, electronic device 1400 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 1460. As shown, network adapter 1460 communicates with other modules of electronic device 1400 via bus 1430. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and sensor modules (e.g., gyroscope sensors, magnetometers, accelerometers, distance sensors, proximity sensors, etc.).

[0147] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0148] Exemplary embodiments of this disclosure also provide a computer-readable storage medium having a program product stored thereon capable of implementing the methods described above in this specification. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0149] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0150] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0151] Furthermore, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0152] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0153] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A posture detection method characterized by comprising: The method comprises: obtaining a front-facing face image; determining first angle data of a face contained in the front-facing face image, the first angle data being angle data of the face in a device coordinate system relative to a terminal device; obtaining device sensor data and determining second angle data from the device sensor data, the second angle data being angle data of the terminal device in a world coordinate system; determining real angle data of the face from the first angle data and the second angle data, the real angle data being angle data of the face in the world coordinate system, and determining a real pose corresponding to the face according to the real angle data; wherein the first angle data, the second angle data and the real angle data are Euler angles; the determination of the first angle data of the face contained in the front-facing face image comprises inputting the front-facing face image into a face detection model to determine a face region in the front-facing face image, and inputting the face region into a face angle model to output the first angle data of the face in the face region; wherein the inputting of the front-facing face image into the face detection model to determine the face region in the front-facing face image comprises inputting the front-facing face image into the face detection model to label multiple face boxes in the front-facing face image, and determining the area of the face boxes to take the face box with the largest area as the face region in the front-facing face image; the determination of the real angle data from the first angle data and the second angle data comprises obtaining a preset angle fitting coefficient, and performing fitting calculation on the first angle data and the second angle data by using the angle fitting coefficient to obtain the real angle data; wherein the angle fitting coefficient is dynamically generated by using a linear regression model.

2. The method of claim 1, wherein, The method further comprises: obtaining a pose angle threshold value; in response to the real pose angle being greater than or equal to the pose angle threshold value, determining the real pose as a bad pose; and generating warning information corresponding to the bad pose.

3. The method of claim 2, wherein, The bad pose includes a head-down pose, a head-tilt pose, a side-lying pose and a squinting pose, the real pose angle includes a real pitch angle, a real yaw angle and a real roll angle, and the pose angle threshold value includes a pitch angle threshold value, a yaw angle threshold value and a roll angle threshold value; the determination of the real pose as the bad pose in response to the real pose angle being greater than or equal to the pose angle threshold value comprises: in response to the real pitch angle being greater than or equal to the pitch angle threshold value, determining the real pose as the head-down pose; or in response to the real roll angle being greater than or equal to the roll angle threshold value, determining the real pose as the head-tilt pose or the side-lying pose; or in response to the real yaw angle being greater than or equal to the yaw angle threshold value, determining the real pose as the squinting pose.

4. The method of claim 3, wherein, The adverse posture includes a lying posture, the posture angle threshold includes a device angle threshold, and the real posture is determined as the adverse posture in response to the real posture angle being greater than or equal to the posture angle threshold. The real posture is determined as the lying posture in response to the real pitch angle being greater than or equal to the pitch angle threshold and the second angle data being greater than or equal to a device angle threshold.

5. The method of claim 1, wherein, The front-facing face image includes: The front-facing face image is acquired in response to the terminal device being in a user operation state. The terminal device being in the user operation state includes: System running data is acquired, the system running data including interactive operation information and screen lighting time; The terminal device is determined to be in the user operation state in response to the interactive operation information being detected and / or in response to the screen lighting time being greater than or equal to a time threshold.

6. A posture detection apparatus characterized by comprising: It includes: An image acquisition module is configured to acquire a front-facing face image; A first angle determination module is configured to determine first angle data of a face included in the front-facing face image, the first angle data being angle data of the face in a device coordinate system relative to a terminal device; A second angle determination module is configured to acquire device sensor data and determine second angle data from the device sensor data, the second angle data being angle data of the terminal device in a world coordinate system; A posture recognition module is configured to determine real angle data of the face from the first angle data and the second angle data, the real angle data being angle data of the face in the world coordinate system, and determine a real posture corresponding to the face according to the real angle data; The first angle data, the second angle data, and the real angle data are Euler angles; The first angle data of the face included in the front-facing face image is determined by inputting the front-facing face image into a face detection model to determine a face region in the front-facing face image, cropping the face region, and inputting the cropped face region into a face angle model to output the first angle data of the face in the face region; The front-facing face image is input into the face detection model to determine the face region in the front-facing face image, a plurality of face boxes are labeled in the front-facing face image, the area of the face box is determined, and the face box with the largest area is taken as the face region in the front-facing face image; The real angle data is determined by the first angle data and the second angle data, including: obtaining a preset angle fitting coefficient; fitting and calculating the first angle data and the second angle data by the angle fitting coefficient to obtain the real angle data; and the angle fitting coefficient is dynamically generated by a linear regression model.

7. A computer readable medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the method of any one of claims 1 to 5.

8. An electronic device, comprising: It includes: A processor; and A memory for storing executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 5 by executing the executable instructions. The processor is configured to perform the method of any one of claims 1 to 5 by executing the executable instructions.

Images