Sitting posture detection method and device, electronic equipment, storage medium and program product

Abstract

The application provides a sitting posture detection method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a first image of a target object collected by an image collection device; determining whether the first image is a key frame; performing key point detection on the first image according to a key point detection mode corresponding to the result of whether the first image is a key frame, to obtain the coordinates of the key points of the target object in the first image; acquiring a sitting posture detection result of the target object according to the coordinates of the key points of the target object in the first image; and if the sitting posture detection result indicates that the target object has a sitting posture problem, outputting an alarm information, which is used to prompt the target object to adjust the sitting posture. The application improves the accuracy and efficiency of the sitting posture detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the image processing technology field, and particularly relates to a sitting posture detection method and device, electronic equipment, a storage medium and a program product. BACKGROUND

[0002] Compared with a traditional desk lamp which is only used as a lighting tool, an intelligent desk lamp is more and more popular due to its various functions. Taking an intelligent desk lamp with a sitting posture detection function as an example, a camera can be installed on the intelligent desk lamp to collect an image including a user. Whether the user's sitting posture is standard can be determined according to the image.

[0003] At present, a common sitting posture detection method mainly uses a deep learning model to perform face key point detection on an image including a user frame by frame, and determines whether the user's sitting posture is standard according to a face key point detection result. In addition, in some related technologies, an infrared detection device can also be installed on the above-mentioned desk lamp to detect a user's sitting posture contour. Then, the user's sitting posture is determined according to the user's sitting posture contour.

[0004] However, the above-mentioned existing sitting posture detection method has problems of low detection accuracy and low detection efficiency. SUMMARY

[0005] The present application provides a sitting posture detection method, device, electronic equipment, storage medium and program product to improve the accuracy and efficiency of sitting posture detection.

[0006] In a first aspect, the present application provides a sitting posture detection method, which comprises the following steps.

[0007] obtaining a first image of a target object collected by an image collection device;

[0008] determining whether the first image is a key frame;

[0009] performing key point detection on the first image according to a key point detection mode corresponding to a result of whether the first image is a key frame, to obtain coordinates of key points of the target object in the first image;

[0010] obtaining a sitting posture detection result of the target object according to the coordinates of the key points of the target object in the first image;

[0011] if the sitting posture detection result represents that the target object has a sitting posture problem, outputting an alarm information, wherein the alarm information is used to prompt the target object to adjust the sitting posture.

[0012] Optionally, the step of performing key point detection on the first image according to a key point detection mode corresponding to a result of whether the first image is a key frame, to obtain coordinates of key points of the target object in the first image, comprises the following steps.

[0013] if the first image is not a key frame, obtaining an optical flow change vector between the first image and a first key frame, and predicting coordinates of key points of the target object in the first image according to the optical flow change vector and coordinates of key points of the target object in the first key frame; the first key frame is a key frame closest to a first image acquisition time in a key frame cache pool and collected by the image acquisition device; the optical flow change vector is used to represent a motion direction and a motion speed of the target object;

[0014] if the first image is a key frame, performing key point detection on the first image by using a deep learning algorithm to obtain the coordinates of the key points of the target object in the first image.

[0015] Optionally, the obtaining of the optical flow change vector between the first image and the first key frame comprises:

[0016] the optical flow change vector is obtained by using a sparse optical flow field algorithm.

[0017] Optionally, the obtaining of the coordinates of the key points of the target object in the first image by performing key point detection on the first image by using the deep learning algorithm comprises:

[0018] the key point detection on the first image is performed by using the deep learning algorithm.

[0019] if the coordinates of the key points of the target object are successfully detected, the first image and the coordinates of the key points of the target object in the first image are cached into the key frame cache pool;

[0020] if the coordinates of the key points of the target object are not successfully detected, the coordinates of the key points of the target object in the first image are predicted according to the optical flow change vector between the first image and the first key frame and the coordinates of the key points of the target object in the first key frame.

[0021] Optionally, the caching of the first image and the coordinates of the key points of the target object in the first image into the key frame cache pool comprises:

[0022] the first image and the coordinates of the key points of the target object in the first image are cached into the key frame cache pool in a key-value pair manner; wherein a key of the key-value pair is an identifier of a key frame, and a value of the key-value pair comprises the first image and the coordinates of the key points of the target object in the first image.

[0023] Optionally, the determining of whether the first image is a key frame comprises:

[0024] If the key frame cache pool is empty, it is determined that the first image is a key frame;

[0025] If the key frame cache pool caches a key frame, it is determined whether the first image is a key frame according to a brightness change between the first image and the first key frame.

[0026] Optionally, the determining whether the first image is a key frame according to the brightness change between the first image and the first key frame comprises:

[0027] obtaining an absolute value of a brightness difference between the first image and the first key frame;

[0028] If the absolute value of the brightness difference is less than or equal to a preset threshold, it is determined that the first image is not a key frame;

[0029] Or, if the absolute value of the brightness difference is greater than the preset threshold, it is determined that the first image is a key frame.

[0030] Optionally, the coordinates of the key points are coordinates in an image coordinate system, and the obtaining of the pose detection result of the target object according to the coordinates of the key points of the target object in the first image comprises:

[0031] obtaining a chest orientation vector of the target object in a three-dimensional coordinate system according to the coordinates of the key points of the target object in the first image; the origin of the three-dimensional coordinate system is a chest center point of the target object;

[0032] calibrating the coordinates of the key points of the target object in the first image according to the chest orientation vector of the target object to obtain calibrated coordinates of the key points of the target object; a two-dimensional coordinate system in which the calibrated coordinates of the key points of the target object are located is parallel to a two-dimensional coordinate system in which the image acquisition device is located;

[0033] obtaining the pose detection result of the target object using the calibrated coordinates of the key points of the target object.

[0034] Optionally, the key points comprise a left shoulder, a right shoulder and a chest center point, and the obtaining of the chest orientation vector of the target object in the three-dimensional coordinate system according to the coordinates of the key points of the target object in the first image comprises:

[0035] obtaining a shoulder vector of the target object and coordinates of a shoulder center point in the first image according to the coordinates of the left shoulder and the coordinates of the right shoulder of the target object in the first image;

[0036] According to the coordinates of the shoulder center point and the coordinates of the chest center point of the target object in the first image, a torso vector of the target object is obtained;

[0037] According to the shoulder vector and the torso vector of the target object in the first image, a chest orientation vector of the target object is obtained.

[0038] Optionally, the calibration of the coordinates of the key points of the target object in the first image according to the chest orientation vector of the target object to obtain the calibrated coordinates of the key points of the target object includes:

[0039] According to the chest orientation vector, a projection vector of the chest orientation vector on a vertical plane of the three-dimensional coordinate system is obtained;

[0040] According to the projection vector, a rotation matrix of the two-dimensional coordinate system is obtained;

[0041] The coordinates of the key points of the target object in the first image are calibrated using the rotation matrix to obtain the calibrated coordinates of the key points of the target object.

[0042] Optionally, the obtaining of the sitting posture detection result of the target object using the calibrated coordinates of the key points of the target object includes:

[0043] The detection parameters of the target object are obtained using the calibrated coordinates of the key points of the target object, and the detection parameters include at least one of the following: head left-right inclination angle, upper body left-right inclination angle, shoulder-jaw difference, and head-to-body ratio;

[0044] According to the detection parameters of the target object, the sitting posture detection result of the target object is obtained.

[0045] Optionally, if the sitting posture detection result indicates that the target object has a sitting posture problem, the sitting posture detection result of the target object includes that the target object has a sitting posture problem and a category to which the sitting posture problem belongs.

[0046] The alarm information further includes the category to which the sitting posture problem belongs.

[0047] Optionally, before the obtaining of the first image of the target object collected by the image collection device, the method further includes:

[0048] A sitting posture detection instruction is received;

[0049] Alternatively, when it is determined according to the image collected by the image collection device that the target object exists in the collection range, the sitting posture detection function is started.

[0050] In a second aspect, the present application provides a sitting posture detection device, the device comprising:

[0051] an acquisition module configured to acquire a first image of a target object captured by an image acquisition device;

[0052] a processing module configured to determine whether the first image is a key frame, perform key point detection on the first image according to a key point detection manner corresponding to a result of whether the first image is a key frame, to obtain coordinates of key points of the target object in the first image, and acquire a sitting posture detection result of the target object according to the coordinates of the key points of the target object in the first image;

[0053] an output module configured to output an alarm information when the sitting posture detection result indicates that the target object has a sitting posture problem, the alarm information being used to prompt the target object to adjust the sitting posture.

[0054] In a third aspect, the present application provides an electronic device, comprising at least one processor, a memory, and

[0055] the memory stores computer-executable instructions;

[0056] the at least one processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the method of any one of the first aspect.

[0057] In a fourth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium storing computer-executable instructions, when the computer-executable instructions are executed by a processor, the method of any one of the first aspect is implemented.

[0058] In a fifth aspect, the present application provides a computer program product, comprising a computer program, when the computer program is executed by a processor, the method of any one of the first aspect is implemented.

[0059] The sitting posture detection method, device, electronic device, storage medium and program product provided by the present application can determine the key point detection manner of the first image of the target object by judging whether the first image is a key frame. The coordinates of the key points obtained by detecting the first image can be used to obtain the sitting posture detection result of the target object. Through the above method, the electronic device can adopt different key point detection manners for key frames and non-key frames, thereby ensuring the accuracy and efficiency of key point recognition, and improving the accuracy of judging whether the target object has a sitting posture problem based on the coordinates of the key points. When the sitting posture detection result indicates that the target object has a sitting posture problem, the electronic device can output an alarm information to prompt the target object to adjust the sitting posture. Through the above method, the accuracy of the electronic device outputting the alarm information is improved, thereby improving the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor.

[0061] FIG. 1 It is a structural schematic diagram of an intelligent table lamp;

[0062] FIG. 2 It is a flowchart of a sitting posture detection method provided by the application;

[0063] FIG. 3 It is a flowchart of a method for obtaining a sitting posture detection result of a target object according to coordinates of key points of the target object provided by the application;

[0064] FIG. 4 It is a schematic diagram of a three-dimensional coordinate system in which a chest orientation vector is located provided by the application;

[0065] FIG. 5 It is a scene schematic diagram in which an image acquisition device acquires a first image provided by the application;

[0066] FIG. 6 It is a flowchart of another sitting posture detection method provided by the application;

[0067] FIG. 7 It is a structural schematic diagram of a sitting posture detection device provided by the application;

[0068] FIG. 8 It is a structural schematic diagram of an electronic device provided by the application.

[0069] Through the above-mentioned accompanying drawings, the specific embodiments of the application have been shown, and more detailed descriptions will be given in the following. These accompanying drawings and textual descriptions are not intended to limit the scope of the concept of the application by any means, but to illustrate the concept of the application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solutions and advantages of the application more clear, the technical solutions in the application will be clearly and completely described in the following by combining the accompanying drawings in the application. Obviously, the described embodiments are some embodiments of the application, but not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0071] Firstly, the following explains some concepts related to the present application:

[0072] Key frame: the key frame is the frame where the key action of the target object motion change is located. The target object in the non-key frame usually does not have motion change, or the motion change is small.

[0073] Optical flow change vector: the optical flow change vector is a two-dimensional vector. The optical flow change vector includes the speed and direction of the pattern motion in the time-varying image.

[0074] Key value pair (key value, kv): each key corresponds to a value.

[0075] Image coordinate system: a rectangular coordinate system with the upper left corner of the image as the origin and with pixels as the unit.

[0076] RGB image: refers to a three-primary-color light (Red Green Blue, RGB) image.

[0077] RGBD image: refers to an RGB image and a depth (Depth) image corresponding to the RGB image.

[0078] Exemplary, FIG. 1 is a structural schematic diagram of an intelligent desk lamp. As FIG. 1 shown, the intelligent desk lamp can include lighting, cameras, display screens and other components. Among them, the camera can be used to collect images including the user's body. In some embodiments, the intelligent desk lamp can use a pre-stored posture detection algorithm in the intelligent desk lamp to detect the posture of the image to determine whether the user's posture is standard.

[0079] It should be understood that FIG. 1 only for exemplary description of the part of the structure of the intelligent desk lamp related to the present application, the present application does not limit the shape of the intelligent desk lamp, the distribution of components, and whether it also includes other components.

[0080] At present, the existing posture detection method mainly includes: user posture detection based on deep learning algorithm. When using this method to detect the user's posture, for each single frame image collected by the camera, the pre-trained deep learning model is used to recognize the face key points, such as left eye, right eye, nose and other key points. If all the key points in the frame image can be recognized, it means that the user does not have a serious low head, and the user's posture is determined to be standard. If all the key points cannot be recognized from the frame image, it means that the user has a serious low head, which leads to the failure of key point recognition, and the user's posture is determined to be non-standard.

[0081] That is, the above method can only be used to detect whether the user has a serious low head posture. However, in fact, the inventors have found through research that the posture of the user can also include tilting the head left and right, tilting the upper body left and right, being too close to the table, and being too close to the display screen of the smart table lamp. For example, if the user's upper body is tilted severely, causing the head to be outside the image capture range, but the shoulders and chest are still within the image capture range, the above method cannot detect it. Therefore, the current posture detection method has the problem of poor accuracy. In addition, the method needs to use a deep learning model for image recognition for each frame of image, and the structure of the deep learning model is usually complex, which may cause the posture detection to be inefficient.

[0082] In addition, some related technologies also propose installing an infrared device on a smart table lamp, using the infrared device to obtain an infrared image including the user, and extracting the user's posture contour from the infrared image. Then, the obtained multiple user posture contours are compared to determine whether the user's posture is standard. However, the infrared image usually has poor imaging accuracy due to interference light, which may cause the accuracy of the user's posture detection to be poor. In addition, the user's posture contour cannot be extracted until a series of image processing such as image enhancement and binarization processing are performed on the infrared image, and the user's posture cannot be determined to be standard until multiple user posture contours are obtained. Therefore, this method also has the problem of low recognition efficiency.

[0083] In view of the above problems of low accuracy and low efficiency of the existing posture detection method, the present application proposes a posture detection method based on RGB image to determine whether the image is a key frame, and different key point detection methods are used for key frames and non-key frames to ensure the accuracy of key point detection in the case of large motion changes in the key frame, and to improve the efficiency of key point detection in the case of non-key frame (i.e. small motion changes of the target object), thereby ensuring the accuracy of posture detection while improving the efficiency of posture detection through the above method.

[0084] It should be understood that the subject of the posture detection method provided by the present application can be any electronic device with processing function. Optionally, the image capture device for capturing the above-mentioned RGB image and the device for performing posture detection processing can be integrated in the same electronic device. For example, the electronic device can be a smart table lamp or other smart home device with an image capture device, or a camera with processing function. In some embodiments, the image capture device and the device for performing posture detection processing can also be deployed in different electronic devices, which is not limited by the present application.

[0085] The technical solutions of the present application will be described in detail below in combination with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments.

[0086] FIG. 2 A flowchart of a sitting posture detection method provided by the present application is shown in FIG. 1. As shown in FIG. 1, the method comprises the following steps: FIG. 2

[0087] S101, acquiring a first image of a target object collected by an image collection device.

[0088] For example, the image collection device can be a camera. For example, as shown in FIG. 2, the image collection device can be a camera installed on a smart desk lamp. FIG. 1

[0089] The target object can be at least one user that can be photographed by the image collection device within the collection range. When the first image includes at least one user, the electronic device can regard each user in the first image as a target object. Alternatively, the electronic device can identify a target object from the at least one user after acquiring the first image. In this implementation, it should be understood that the present application does not limit how the electronic device identifies a target object from the at least one user.

[0090] S102, determining whether the first image is a key frame.

[0091] In some embodiments, the electronic device can perform key frame identification on each frame of the first image to determine whether the frame image is a key frame. Alternatively, any existing key frame identification method can be referred to to determine whether the first image is a key frame, which will not be described herein.

[0092] In some embodiments, the electronic device can also determine whether the first image is a key frame according to the first image and a neighboring frame image of the frame of the first image after acquiring the first image. Alternatively, the electronic device can acquire a motion change amount between the first image and a target frame, and then determine whether the first image is a key frame according to the motion change amount. For example, the target frame can be a previous frame image of the first image, or a key frame determined by the electronic device in advance.

[0093] S103, performing key point detection on the first image according to a key point detection mode corresponding to a result of determining whether the first image is a key frame, to obtain coordinates of key points of the target object in the first image.

[0094] ​​Exemplarily, the key points can include at least one of the following: left eye, right eye, nose, chin, left shoulder, right shoulder, center point of chest, and the like. The coordinates of the key points of the target object in the first image can be coordinates in a two-dimensional coordinate system, so as to improve the efficiency of the sitting posture detection based on the coordinates.

[0095] Exemplarily, the electronic device can generate an identifier indicating that the first image is a key frame when determining that the first image is a key frame, and generate an identifier indicating that the first image is not a key frame when determining that the first image is not a key frame. The electronic device can pre-store a mapping relationship between the identifiers of the results of whether the first image is a key frame and the key point detection manners. Optionally, the electronic device can determine the key point detection manner corresponding to the result of whether the first image is a key frame according to the identifier of the result of whether the first image is a key frame and the mapping relationship.

[0096] Optionally, the key point detection manner corresponding to the first image being a key frame can be a key point detection manner with higher detection accuracy, so as to ensure the accuracy of the key point detection of the target object in the case that the target object has large motion changes in the key frame. The key point detection manner corresponding to the first image not being a key frame can be a key point detection manner with higher detection efficiency, so as to reduce the calculation redundancy and improve the efficiency of the key point detection of the target object in the case that the target object has small motion changes in the non-key frame.

[0097] S104. Obtain a sitting posture detection result of the target object according to the coordinates of the key points of the target object in the first image.

[0098] Optionally, the sitting posture detection result of the target object can be used to indicate that the target object has no sitting posture problem or that the target object has a sitting posture problem. Further, in some embodiments, the sitting posture detection result of the target object can also be used to indicate what kind of sitting posture problem the target object has. Exemplarily, the kind of the sitting posture problem can include at least one of the following: the target object has a serious left (or right) head tilt, the target object has a serious left (or right) upper body tilt, the target object is close to a desktop, the target object is close to a display screen, and the like.

[0099] As a possible implementation manner, the electronic device can directly determine at least one parameter for judging whether the sitting posture of the target object has a problem based on the coordinates of the key points after obtaining the coordinates of the key points of the target object, and obtain the sitting posture detection result of the target object according to the at least one parameter.

[0100] As another possible implementation, the electronic device can also determine an orientation of the target object relative to the aforementioned image capture device based on the coordinates of the key points of the target object, and calibrate the coordinates of the aforementioned key points based on the orientation, so as to improve the accuracy of the determination of the sitting posture detection result of the target object based on the coordinates of the key points.

[0101] S105: Determine whether the sitting posture detection result is used to represent that the target object has a sitting posture problem.

[0102] Optionally, the electronic device can generate an identifier used to represent the sitting posture detection result after obtaining the sitting posture detection result of the target object. The identifier corresponding to different sitting posture detection results is different. Then, the electronic device can determine whether the sitting posture detection result is used to represent that the target object has a sitting posture problem based on the identifier corresponding to the sitting posture detection result.

[0103] If the sitting posture detection result is used to represent that the target object has a sitting posture problem, the electronic device can execute step S106 to output alarm information.

[0104] If the sitting posture detection result is used to represent that the target object does not have a sitting posture problem, the electronic device can optionally continue to return to execute steps S101-S105 to continuously detect the sitting posture of the target object based on the next frame of the first image, and execute step S106 to output alarm information when the target object has a sitting posture problem.

[0105] S106: Output alarm information used to prompt the target object to adjust the sitting posture.

[0106] The electronic device can display the alarm information through a display device, or report the alarm information through a voice output device. For example, a terminal such as the smart table lamp shown in FIG. 1 can display the alarm information through a display screen. For example, the smart table lamp can display "Please pay attention to the sitting posture!" through the display screen. Alternatively, the smart table lamp can also report the alarm information through a voice output device, for example, broadcast the voice "Please pay attention to the sitting posture". FIG. 1

[0107] In some embodiments, the electronic device can also execute the step S106 when it is determined for a continuous preset number of times that the sitting posture detection result is used to represent that the target object has a sitting posture problem, so as to improve the accuracy of the output of the alarm information by the electronic device. The aforementioned preset number can be pre-stored in the electronic device by a user.

[0108] ​Alternatively, the electronic device can perform the sitting posture detection at a preset frequency. In this implementation, the electronic device can further perform this step S106 when each of the sitting posture detection results determined in a preset time length indicates that the target object has a sitting posture problem, so as to improve the accuracy of the alarm information output by the electronic device. The preset frequency and the preset time length can be pre-stored in the electronic device by the user. For example, the electronic device can perform the sitting posture detection once per second, and output the alarm information when each of the sitting posture detection results obtained within 10 seconds indicates that the target object has a sitting posture problem.

[0109] Alternatively, the sitting posture detection result of the target object can indicate that the target object has a sitting posture problem, and the alarm information can further include the category of the sitting posture problem. For example, when the category of the sitting posture problem is that the target object is close to the desktop, the smart desk lamp can output "You are currently close to the desktop, please work in the standard sitting posture" through the display screen or the voice output device.

[0110] In this embodiment, the manner of performing the key point detection on the first image is determined by judging whether the first image of the target object is a key frame. The sitting posture detection result of the target object can be obtained by detecting the coordinates of the key points of the first image. Through the above method, the electronic device can adopt different key point detection manners for key frames and non-key frames, so as to ensure the accuracy and efficiency of the key point recognition, and further improve the accuracy of determining whether the target object has a sitting posture problem based on the coordinates of the key points. When the sitting posture detection result indicates that the target object has a sitting posture problem, the electronic device can output an alarm information to prompt the target object to adjust the sitting posture. Through the above method, the accuracy of the alarm information output by the electronic device is improved, and the user experience is improved.

[0111] The timing at which the electronic device starts the sitting posture detection is described below. As a possible implementation, the electronic device can obtain the first image of the target object captured by the image capturing device after receiving the sitting posture detection instruction.

[0112] It should be understood that the present application does not limit how the electronic device receives the sitting posture detection instruction. Alternatively, for example, the terminal including a touch display screen can receive the sitting posture detection instruction input by the user through the touch display screen. Alternatively, for example, the terminal including a microphone can receive the voice instruction input by the user to start the sitting posture detection as the sitting posture detection instruction through the microphone.

[0113] As another possible implementation, the electronic device can also automatically detect whether to start the sitting posture detection, so as to improve the user experience. Optionally, the electronic device can start the sitting posture detection function when it is determined according to the image collected by the image collection device that there is a target object in the collection range, and then perform the foregoing step S101 to start the sitting posture detection on the target object. In this implementation, it should be understood that the present application does not limit how the electronic device determines whether there is a target object in the collection range according to the image collected by the image collection device. Optionally, any existing target detection method can be used to detect whether there is a target object, which will not be described here.

[0114] As another possible implementation, the electronic device can also receive a sitting posture detection time period input by the user, and start the sitting posture detection when the time period is reached, and stop the sitting posture detection after the time period.

[0115] Alternatively, in some embodiments, after starting the sitting posture detection, the electronic device can also receive a sitting posture detection termination instruction to stop the sitting posture detection. Further alternatively, the electronic device can also turn off the sitting posture detection function when it is determined according to the image collected by the image collection device that there is no target object in the collection range, so as to save computing resources and save power consumption.

[0116] In the present embodiment, the electronic device can start the sitting posture detection on the target object after receiving the sitting posture detection instruction, so that the electronic device can perform the sitting posture detection when the user needs, thereby improving the accuracy of the sitting posture detection. Alternatively, the electronic device can automatically detect whether to start the sitting posture detection, thereby improving the automation of the sitting posture detection, and further improving the efficiency of the sitting posture detection.

[0117] The following describes in detail how the electronic device determines whether the first image is a key frame:

[0118] As a possible implementation, the electronic device can first determine whether the first image is a key frame according to whether the key frame cache pool is empty.

[0119] Optionally, if the key frame cache pool is empty, it means that the present sitting posture detection is started and no image has been determined as a key frame, and therefore the electronic device can determine that the first image is a key frame.

[0120] If the key frame cache pool stores the key frame, the electronic device can determine whether the first image is a key frame according to a luminance change between the first image and the first key frame. The first key frame is a key frame closest to the first image acquisition time in the key frame cache pool. By taking the key frame closest to the first image acquisition time as the first key frame, the luminance change is determined by comparing the first image with the latest key frame, which avoids error accumulation and further improves the accuracy of the sitting posture detection.

[0121] Optionally, the luminance change between the first image and the first key frame may, for example, refer to a luminance difference between the first image and the first key frame, or an absolute value of the luminance difference between the first image and the first key frame.

[0122] It should be understood that the present application does not limit how the electronic device obtains the luminance change between the first image and the first key frame. For example, when the luminance change between the first image and the first key frame refers to the absolute value of the luminance difference between the first image and the first key frame, the electronic device may, for example, obtain the luminance difference between the first image and the first key frame by frame difference method, and then obtain the absolute value of the luminance difference between the first image and the first key frame.

[0123] Still taking the luminance change between the first image and the first key frame as the absolute value of the luminance difference between the first image and the first key frame as an example, in this implementation, optionally, after obtaining the absolute value of the luminance difference between the first image and the first key frame, the electronic device may, for example, determine whether the first image is a key frame by the following manner:

[0124] The electronic device can determine whether the absolute value of the luminance difference is less than or equal to a preset threshold. Optionally, the preset threshold may, for example, be pre-stored in the electronic device by a user.

[0125] If the absolute value of the luminance difference is less than or equal to the preset threshold, it indicates that the motion change of the target object included in the first image relative to the target object included in the first key frame is small. That is, the first image is not the frame in which the key action of the motion change of the target object is located. Therefore, the electronic device can determine that the first image is not a key frame.

[0126] If the absolute value of the luminance difference is greater than the preset threshold, it indicates that the motion change of the target object included in the first image relative to the target object included in the first key frame is large. That is, the first image is the frame in which the key action of the motion change of the target object is located. Therefore, the electronic device can determine that the first image is a key frame.

[0127] In this embodiment, when the key frame cache pool is empty, the first image is directly taken as a key frame, thereby improving the accuracy of the electronic device in determining whether the first image is a key frame. When a key frame is cached in the key frame cache pool, whether the first image is a key frame is determined according to the brightness change between the first image and the first key frame. Through the above method, whether the motion change of the target object between the first image and the first key frame is large can be determined, and then whether the first image is a key frame can be determined.

[0128] The following describes how the electronic device performs key point detection on the first image when the first image is not a key frame to obtain the coordinates of the key points of the target object in the first image in detail.

[0129] As a possible implementation manner, the electronic device can obtain an optical flow change vector between the first image and the first key frame, and predict the coordinates of the key points of the target object in the first image according to the optical flow change vector and the coordinates of the key points of the target object in the first key frame.

[0130] As described above, the first key frame is a key frame closest to the first image acquisition time in the key frame cache pool. The optical flow change vector is used to represent the motion direction and motion speed of the target object.

[0131] Optionally, the number of key frames cached in the key frame cache pool is not limited in the present application. For example, the key frame cache pool always caches a frame image closest to the first image acquisition time, and the electronic device can read the frame image from the cache pool and directly take the frame image as the first key frame.

[0132] Then, the electronic device can obtain an optical flow change vector between the first image and the first key frame. Optionally, the electronic device can use a sparse optical flow field algorithm to obtain the optical flow change vector. By using the sparse optical flow field algorithm, the electronic device only needs to obtain the optical flow change vector based on a small number of pixel points in the first image and a small number of pixel points in the first key frame, thereby improving the efficiency of obtaining the optical flow change vector and the efficiency of the sitting posture detection.

[0133] In some embodiments, the electronic device can also use a dense optical flow field algorithm or other algorithms that can be used to calculate the optical flow change vector between two frame images to obtain the optical flow change vector between the first image and the first key frame.

[0134] After obtaining the optical flow change vector between the first image and the first key frame, the electronic device can first obtain the key point coordinates of the target object in the first key frame, and then predict the coordinates of the key points of the target object in the first image according to the optical flow change vector and the key point coordinates of the target object in the first key frame. For example, the key frame cache pool can also store the key point coordinates of the target object in the first key frame. That is, the electronic device can read the key point coordinates of the target object in the first key frame from the key frame cache pool.

[0135] Optionally, for each key point coordinate of the target object in the first key frame, the electronic device can obtain the coordinates of the key points of the target object in the first image according to the sum of the key point coordinate in the first key frame and the optical flow change vector. For example, taking the nose of the target object in the first key frame as an example, the electronic device can take the sum of the coordinates of the nose of the target object in the first key frame and the optical flow change vector as the coordinates of the nose of the target object in the first image.

[0136] In this embodiment, because the sitting posture detection is usually continuous detection in a period of time, there is a strong spatio-temporal correlation between adjacent frames captured by the image acquisition device. By determining the coordinates of the key points of the target object through the optical flow change vector between the first image and the first key frame, the above information can be maximally utilized, the accuracy of key point coordinate prediction is improved, and the accuracy of sitting posture detection is further improved.

[0137] The following describes how the electronic device performs key point detection on the first image when the first image is a key frame to obtain the coordinates of the key points of the target object in the first image.

[0138] As a possible implementation manner, when the first image is a key frame, the electronic device can perform key point detection on the first image by using a deep learning algorithm to obtain the coordinates of the key points of the target object in the first image. Because the key frame is usually an image corresponding to a large change in the action of the target object, performing key point detection on the key frame by using a deep learning algorithm with high accuracy can improve the accuracy of detecting the key points in the key frame, and further improve the accuracy of the sitting posture detection based on the coordinates of the key points.

[0139] The deep learning algorithm can be a trained key point detection model pre-stored in the electronic device by the user. It should be understood that the present application does not limit how to train the key point detection model to obtain the deep learning algorithm, and the execution subject of training the key point detection model. In addition, the present application does not limit the key point detection model. Optionally, any existing deep learning model capable of being used as a key point detection model can be referred to, and details are not described herein.

[0140] In this implementation, considering that the key point detection on the first image by using the deep learning algorithm can be successful or failed, the electronic device can determine whether the coordinates of the key points of the target object are successfully detected after the key point detection on the first image by using the deep learning algorithm.

[0141] In some embodiments, the electronic device can determine that the coordinates of the key points of the target object are successfully detected when the coordinates of all the key points of the target object in the first image are obtained by using the deep learning algorithm. Alternatively, the electronic device can also determine that the coordinates of the key points of the target object are not successfully detected when the coordinates of all the key points of the target object in the first image are not obtained by using the deep learning algorithm.

[0142] If the coordinates of the key points of the target object are not successfully detected, the electronic device can obtain the coordinates of the key points of the target object in the first image by using another key point detection method. Optionally, the electronic device can predict the coordinates of the key points of the target object in the first image according to the optical flow change vector between the first image and the first key frame, and the coordinates of the key points of the target object in the first key frame. Optionally, the electronic device can obtain the coordinates of the key points of the target object according to the optical flow change vector in the manner described in the foregoing embodiments, which will not be described here.

[0143] In this implementation, when the coordinates of the key points are not successfully detected by using the deep learning algorithm, the key points in the first image are predicted by using the optical flow change vector, which effectively reduces the probability of missing detection of the key points. Through the above method, even if part of the key points of the target object are out of the range of the image acquisition device, the coordinates of the key points can be estimated by the optical flow change of the surrounding pixels, which improves the robustness of the sitting posture detection.

[0144] If the coordinates of the key points of the target object are successfully detected, the electronic device can further cache the first image and the coordinates of the key points of the target object in the first image into the key frame cache pool. Because the accuracy of the deep learning algorithm is usually high, caching the coordinates of the key points obtained based on the deep learning algorithm into the key frame cache pool ensures the accuracy of the coordinates of the key points of the key frames in the key frame cache pool, thereby avoiding error accumulation and further improving the accuracy of the sitting posture detection.

[0145] In this implementation, optionally, the electronic device can cache the first image and the coordinates of the key points of the target object in the first image into the key frame cache pool in the form of a key-value pair. The key of the key-value pair can be an identifier of the key frame, and the value of the key-value pair can include the first image and the coordinates of the key points of the target object in the first image. For example, the identifier of the key frame can be a timestamp corresponding to the key frame. Through the above method, the electronic device can quickly obtain the key frame and the coordinates of the key points of the target object in the key frame according to the identifier of the key frame, thereby improving the efficiency of the posture detection of the electronic device.

[0146] In some embodiments, after successfully obtaining the coordinates of the key points of the first image by the deep learning algorithm, the electronic device can replace the original key frame and the coordinates of the key points of the original key frame in the key frame cache pool with the first image and the coordinates of the key points of the first image, thereby updating the key frame cache pool. Through the above method, the storage resources of the electronic device are saved, and the efficiency of the posture detection of the electronic device based on the key frames in the key frame cache pool is improved.

[0147] In this embodiment, when the first image is a non-key frame, the coordinates of the key points of the target object in the first image are predicted using the optical flow field algorithm; when the first image is a key frame, the coordinates of the key points of the target object in the first image are obtained by performing key point detection on the first image using the deep learning algorithm. The efficiency of key point detection by the optical flow field algorithm is generally higher than that by the deep learning algorithm, so the efficiency of posture detection of the electronic device is improved by the above method. In addition, the motion of the target object in the key frame changes greatly, so the accuracy of key point detection of the key frame can be ensured by using the deep learning algorithm with high accuracy. Because the motion of the target object in the non-key frame changes slightly, the accuracy of key point detection can also be ensured by using the optical flow field algorithm combined with the information above. Therefore, by the above method, the accuracy of key point detection is ensured while the efficiency of key point detection is improved, thereby improving the accuracy and efficiency of posture detection.

[0148] The following describes how the electronic device obtains the posture detection result of the target object according to the coordinates of the key points of the target object in the first image:

[0149] FIG. 3 A flowchart of a method for obtaining a posture detection result of a target object according to coordinates of key points of the target object provided by the present application is shown in FIG. 10. As a possible implementation, the foregoing step S104 can include the following steps: FIG. 3 ​

[0150] S201, obtain, according to the coordinates of the key points of the target object in the first image, a chest orientation vector of the target object in a three-dimensional coordinate system.

[0151] The coordinates of the key points of the target object in the first image are coordinates in an image coordinate system. The origin of the three-dimensional coordinate system is a chest center point of the target object. In some embodiments, the starting point of the chest orientation vector can coincide with the origin of the three-dimensional coordinate system. An exemplary FIG. 4 is a schematic diagram of a three-dimensional coordinate system in which a chest orientation vector is provided in the present application. As FIG. 4 indicated, the vector C therein is a chest orientation vector. The origin of the three-dimensional coordinate system can be a chest center point of the target object.

[0152] Optionally, the electronic device may, for example, obtain, according to the coordinates of the key points of the target object in the first image, a chest orientation vector of the target object in a three-dimensional coordinate system in the following manner:

[0153] First, the electronic device may, for example, obtain, according to the coordinates of the left shoulder of the target object in the first image and the coordinates of the right shoulder, a shoulder vector of the target object in the first image and the coordinates of a shoulder center point. An exemplary FIG. 5 is a schematic diagram of a scene in which an image acquisition device acquires a first image provided in the present application. As FIG. 5 indicated, the shoulder vector of the target object may, for example, be as FIG. 5 indicated by the vector S in

[0154] The electronic device may, for example, subtract the coordinates of the left shoulder from the coordinates of the right shoulder to obtain the shoulder vector of the target object. The electronic device may, for example, divide the sum of the x-axis direction coordinate in the coordinates of the right shoulder of the target object and the x-axis direction coordinate in the coordinates of the left shoulder by 2 to obtain the x-axis direction coordinate in the coordinates of the shoulder center point. The electronic device may, for example, divide the sum of the y-axis direction coordinate in the coordinates of the right shoulder of the target object and the y-axis direction coordinate in the coordinates of the left shoulder by 2 to obtain the y-axis direction coordinate in the coordinates of the shoulder center point.

[0155] Then, the electronic device may, for example, obtain, according to the coordinates of the shoulder center point and the coordinates of the chest center point of the target object in the first image, a torso vector of the target object. An exemplary FIG. 5 the torso vector may, for example, be as indicated by the vector T in

[0156] After obtaining the shoulder vector and the torso vector of the target object, the electronic device can obtain a chest orientation vector of the target object according to the shoulder vector and the torso vector of the target object in the first image. For example, the electronic device can obtain the chest orientation vector of the target object by using the following formula (1):

[0157] C = T x S (1)

[0158] wherein T represents the torso vector of the target object, S represents the shoulder vector of the target object, and C represents the chest orientation vector of the target object. Therefore, formula (1) indicates that the electronic device can obtain the normal vector of the torso vector of the target object and the shoulder vector of the target object by cross-multiplying the torso vector of the target object and the shoulder vector of the target object, and take the normal vector as the chest orientation vector of the target object.

[0159] S202. Calibrate the coordinates of the key points of the target object in the first image according to the chest orientation vector of the target object, to obtain the calibrated coordinates of the key points of the target object.

[0160] wherein the two-dimensional coordinate system in which the calibrated coordinates of the key points of the target object are located is parallel to the two-dimensional coordinate system in which the image acquisition device is located.

[0161] Optionally, the electronic device can calibrate the coordinates of the key points by using the following method to obtain the calibrated coordinates of the key points of the target object:

[0162] As a possible implementation manner, the electronic device can convert the coordinates of the key points of the target object in the first image to a two-dimensional coordinate system parallel to the two-dimensional coordinate system in which the image acquisition device is located based on the chest orientation vector.

[0163] Firstly, the electronic device can obtain a projection vector of the chest orientation vector on a vertical plane in a three-dimensional coordinate system according to the chest orientation vector of the target object. For example, when the chest orientation vector is (x c , y c , z c ) and the vertical plane is the xoz plane of the three-dimensional coordinate system, the electronic device can set the coordinate of the chest orientation vector on the x-axis direction to zero to obtain the projection vector of the chest orientation vector on the vertical plane in the three-dimensional coordinate system (0, y c , z c ).

[0164] Then, the electronic device can obtain a rotation matrix of the two-dimensional coordinate system according to the projection vector. Optionally, the electronic device can first calculate the included angle between the projection vector and a first axis, and obtain the rotation matrix of the two-dimensional coordinate system according to the included angle. Wherein the first axis is the axis of the three-dimensional coordinate system perpendicular to the two-dimensional coordinate system in which the image acquisition device is located, for exampleFIG. 4 The y-axis shown in the middle.

[0165] Optionally, the angle between the projection vector and the first axis can be the angle between the projection vector and the positive direction of the first axis, or the angle between the projection vector and the negative direction of the first axis, which is not limited in the present application.

[0166] Taking the angle between the projection vector and the first axis as the angle between the projection vector and the positive direction of the first axis as an example, the electronic device can obtain the rotation matrix of the two-dimensional coordinate system through the following formula (2):

[0167]

[0168] Wherein, R represents the rotation matrix of the two-dimensional coordinate system, and θ represents the angle between the projection vector and the positive direction of the first axis.

[0169] Wherein Wherein

[0170] After obtaining the rotation matrix of the two-dimensional coordinate system, the electronic device can use the rotation matrix to calibrate the coordinates of the key points of the target object in the first image to obtain the coordinates of the calibrated key points of the target object. Optionally, for the coordinates of each key point of the target object in the first image, the electronic device can multiply the coordinates of the key point by the above-mentioned rotation matrix, and the result is taken as the coordinates of the calibrated key points of the target object.

[0171] For example, assuming that the left eye key point coordinates of the target object are (x, z), the electronic device can obtain the calibrated left eye key point coordinates through the following formula (3).

[0172]

[0173] Wherein, The calibrated left eye key point coordinates are (x', z').

[0174] S203, using the coordinates of the calibrated key points of the target object, obtaining the pose detection result of the target object.

[0175] Optionally, the electronic device may, for example, obtain the sitting posture detection result of the target object by judging whether the coordinates of the key points of each calibrated target object are all within a preset coordinate range. The preset coordinate ranges corresponding to different key points can be different. The preset coordinate ranges corresponding to each key point can be pre-stored in the electronic device by the user. For example, the electronic device can determine that the sitting posture detection result of the target object is used to represent that the target object does not have a sitting posture problem when the coordinates of the key points of each calibrated target object are all within the preset coordinate range. If the coordinates of the key points of the calibrated target object are not within the preset coordinate range, the electronic device can determine that the sitting posture detection result of the target object is used to represent that the target object has a sitting posture problem.

[0176] Alternatively, the electronic device can also first obtain the detection parameters of the target object using the coordinates of the key points of the calibrated target object, and then obtain the sitting posture detection result of the target object according to the detection parameters of the target object. The detection parameters can include at least one of the following: head left-right tilt angle, upper body left-right tilt angle, shoulder-jaw difference, and head-to-body ratio.

[0177] Optionally, the electronic device can determine whether the target object has a sitting posture problem based on the above-mentioned detection parameters. Alternatively, if the sitting posture detection result represents that the target object has a sitting posture problem, the electronic device can also determine the category to which the sitting posture problem of the target object belongs based on the above-mentioned detection parameters.

[0178] It should be understood that the present application does not limit how the electronic device obtains the sitting posture detection result of the target object based on the above-mentioned detection parameters.

[0179] For example, taking the detection parameters including: head left-right tilt angle, upper body left-right tilt angle, shoulder-jaw difference, and head-to-body ratio as an example, the electronic device can determine that the sitting posture detection result of the target object is used to represent that the target object does not have a sitting posture problem when the head left-right tilt angle of the target object is within a preset head tilt angle range, the upper body left-right tilt angle of the target object is within a preset upper body tilt angle range, the shoulder-jaw difference is greater than or equal to a first preset shoulder-jaw difference threshold, and the head-to-body ratio is less than or equal to a preset ratio threshold.

[0180] The electronic device can determine that the category to which the sitting posture problem of the target object belongs is that the left and right head tilts of the target object are serious when the left and right head tilts of the target object are outside a preset head tilt range. The electronic device can determine that the category to which the sitting posture problem of the target object belongs is that the left and right upper body tilts of the target object are serious when the left and right upper body tilts of the target object are outside a preset upper body tilt range. The electronic device can determine that the category to which the sitting posture problem of the target object belongs is that the target object is looking at the desktop when the shoulder-jaw difference of the target object is less than or equal to a second preset shoulder-jaw difference threshold. The electronic device can determine that the category to which the sitting posture problem of the target object belongs is that the target object is looking at the screen when the proportion of the head of the target object relative to the body is greater than a preset proportion threshold.

[0181] Optionally, the preset head tilt range, the preset upper body tilt range, the first preset shoulder-jaw difference threshold, the second preset shoulder-jaw difference threshold, and the preset proportion threshold can be pre-stored in the electronic device by the user. The first preset shoulder-jaw difference threshold is greater than the second preset shoulder-jaw difference threshold.

[0182] The following exemplary describes how the electronic device obtains the values of the above-mentioned detection parameters:

[0183] Taking the head left and right tilt as an example, the electronic device may, for example, determine the center points of the left and right eyes based on the coordinates of the left and right eye key points. Then, according to the center points of the left and right eyes and the coordinates of the nose key point, the center line of the head of the target object is determined. Then, the electronic device can obtain the angle of the center line of the head of the target object deviating from the vertical direction to the right (or to the left) as the head left and right tilt.

[0184] Taking the upper body left and right tilt as an example, the electronic device may, for example, determine the center points of the left and right shoulders based on the coordinates of the left and right shoulder key points. Then, according to the center points of the left and right shoulders and the coordinates of the chest center key point, the center line of the upper body of the target object is determined. Then, the electronic device can obtain the angle of the center line of the upper body of the target object deviating from the vertical direction to the right (or to the left) as the upper body left and right tilt.

[0185] Taking the shoulder-jaw difference as an example, the electronic device may, for example, determine the straight line where the left shoulder and the right shoulder are located according to the coordinates of the left shoulder and the coordinates of the right shoulder. Then, the electronic device can obtain the distance from the chin key point to the straight line where the left shoulder and the right shoulder are located according to the coordinates of the chin key point and the straight line where the left shoulder and the right shoulder are located. The electronic device can take the distance as the shoulder-jaw difference of the target object. By taking the shoulder-jaw difference as a detection parameter, the nearsighted sitting posture in the case where the eyes, the desktop, etc. cannot be photographed can be effectively detected, thereby improving the diversity of the sitting posture detection and further improving the accuracy of the sitting posture detection.

[0186] For example, the proportion of the head relative to the body can be determined by the electronic device according to the coordinates of the key points on the face, and the area of the face coordinate frame can be obtained. The face coordinate frame can be the coordinates of the vertices of the detection frame determined by the face detection. Then, the electronic device can determine the area of the body of the target object according to the coordinates of the left shoulder, the right shoulder, the center of the chest, and other key points. Then, the electronic device can obtain the quotient of the area of the face coordinate frame and the area of the body of the target object as the proportion of the head relative to the body.

[0187] In some embodiments, the detection parameters can further include, for example, the proportion of the head relative to the display screen, and the present application does not limit the same.

[0188] In the present embodiment, the electronic device can obtain the chest orientation vector for calibrating the coordinates of the key points according to the coordinates of the key points of the target object, so that the electronic device does not need to obtain other parameters to calibrate the coordinates of the key points. For example, the present application does not need to require the first image to be an RGBD image, nor to perform human body reconstruction such as 3-dimension mesh (3D mesh), but directly maps and transforms the coordinates of the key points, thereby reducing the amount of calculation of the electronic device for coordinate calibration and further improving the efficiency of the sitting posture detection.

[0189] In addition, by calibrating the coordinates of the key points, the two-dimensional coordinate system in which the calibrated coordinates of the key points are located is parallel to the two-dimensional coordinate system in which the image acquisition device is located, which is equivalent to adjusting the human body so that the human body is directly opposite the image acquisition device, reducing the possibility that the accuracy of the key point coordinates is low due to the image acquisition angle, and further improving the accuracy of the sitting posture detection. In addition, by the above method, the image acquisition device or the terminal device equipped with the image acquisition device can be used flexibly without being limited to being directly opposite the target object.

[0190] For example, the above electronic device is a smart table lamp, and the above image acquisition device is a camera mounted on the smart table lamp, FIG. 6 Another method for detecting a sitting posture is provided in the present application. As shown in FIG. 6 The method can include the following steps:

[0191] Step 1, the smart table lamp receives an instruction input by the target object through the display screen, the instruction being used to indicate starting to write a task.

[0192] Step 2, the smart table lamp initializes a key frame cache pool, and acquires an RGB first image of the target object from the camera.

[0193] Step 3, determining whether the first image is a key frame.

[0194] Step 3.1, when the key frame cache pool is empty, determining the first image as a key frame.

[0195] Step 3.2, when the key frame cache pool is not empty, using the inter-frame difference method to difference the first image and the first key frame, to obtain the absolute value of the brightness difference of the two frames of images. When the absolute value of the brightness difference is greater than a preset threshold, determining the first image as a key frame. Otherwise, determining the first image as a non-key frame.

[0196] If the first image is a key frame, step 4.1 is executed. If the first image is a non-key frame, step 4.2 is executed.

[0197] Step 4.1, the first image is a key frame, then using the skeletal key point detection algorithm in the deep learning algorithm to detect the key points. The key points include 7 key points of left eye, right eye, nose, chin, left shoulder, right shoulder, and chest center point.

[0198] Step 4.1.1, if the skeletal key point detection is successful, the first image and the corresponding coordinates of each skeletal key point are stored in the cache pool.

[0199] Step 4.1.2, if the skeletal key point detection fails, the optical flow change vector of the first image and the first key frame is obtained by the sparse optical flow field algorithm, and then the skeletal key point coordinates predicted by the optical flow method are obtained based on the optical flow change vector.

[0200] Step 4.2, if the first image is a non-key frame, the optical flow change vector of the first image and the first key frame is obtained by the sparse optical flow field algorithm, and then the skeletal key point coordinates predicted by the optical flow method are obtained based on the optical flow change vector.

[0201] Step 5, according to the coordinates of the left shoulder key point, the coordinates of the right shoulder key point, and the coordinates of the chest center point of the target object, the chest orientation vector of the target object is obtained, and based on the chest orientation vector, the coordinates of the key points and the coordinates of the face detection coordinate frame are calibrated to obtain the calibrated coordinates of the key points.

[0202] The specific implementation of step 5 can refer to the method described in the foregoing embodiments, which will not be repeated here.

[0203] Step 6, using the calibrated coordinates of the key points of the target object, the detection parameters of the target object are obtained.

[0204] Step 7, based on the detection parameters of the target object, the sitting posture detection result of the target object is obtained.

[0205] Exemplarily, the mapping relationship between the detection parameter of the target object and the sitting posture detection result of the target object can be as shown in Table 1 as follows:

[0206] Table 1

[0207] Sitting posture detection result Corresponding judgment standard of detection parameter No sitting posture problem L1∈[-15°, 15°] and L2∈[-5°, 5°] and d≥-5 and r≤0.2 Severe left and right tilt of head L1≥15° or L1≤-15° Severe left and right tilt of upper body L2≥5° or L2≤-5° Myopic desktop d≤-8 Myopic screen r≥0.2

[0208] Wherein, L1 represents the left-right tilt angle of the head of the target object, L2 represents the left-right tilt angle of the upper body of the target object, d represents the shoulder-jaw difference of the target object, and r represents the proportion of the head relative to the body of the target object.

[0209] Step 8, judging whether the sitting posture detection result represents that the target object has a sitting posture problem.

[0210] If yes, step 9 is executed to output an alarm information.

[0211] If no, the real-time monitoring is continued, and steps 1-9 are cycled to perform the sitting posture detection on the target object.

[0212] Step 9, the intelligent table lamp outputs an alarm information to prompt the target object to adjust the sitting posture.

[0213] In the embodiment, whether the current frame is a key frame is judged based on the inter-frame difference method, the key points are predicted for the key frame by using the deep learning algorithm, the optical flow field is calculated for the non-key frame to estimate the positions of the key points, and the subsequent sitting posture type discrimination is combined with the corresponding discrimination rules. Through the above method, the deep learning algorithm does not need to be executed for each frame of image, the non-key frame missing detection situation can be effectively reduced, and the stability of the continuous frame sitting posture detection is improved. Through the above method, the bad sitting posture in the reading, writing and other states can be fed back, supervised and prompted to the target object to restore the normal sitting posture and protect the physical and mental health of the target object.

[0214] FIG. 7 A structure schematic diagram of a sitting posture detection device provided in the present application is shown in FIG. 1. FIG. 7 As shown in the figure, the device comprises an acquisition module 31, a processing module 32, and an output module 33. Wherein,

[0215] The acquisition module 31 is configured to acquire a first image of a target object collected by an image collection device.

[0216] The processing module 32 is configured to determine whether the first image is a key frame, perform key point detection on the first image according to a key point detection mode corresponding to the result of whether the first image is a key frame, obtain the coordinates of the key points of the target object in the first image, and acquire a sitting posture detection result of the target object according to the coordinates of the key points of the target object in the first image.

[0217] The output module 33 is configured to output an alarm information when the sitting posture detection result indicates that the target object has a sitting posture problem. The alarm information is used to prompt the target object to adjust the sitting posture.

[0218] Optionally, the processing module 32 is specifically configured to obtain an optical flow change vector between the first image and a first key frame when the first image is a non-key frame, and predict the coordinates of the key points of the target object in the first image according to the optical flow change vector and the coordinates of the key points of the target object in the first key frame. The first key frame is a key frame closest to the first image acquisition time in the key frame cache pool. The optical flow change vector is used to represent the motion direction and speed of the target object.

[0219] Alternatively, the processing module 32 is specifically configured to perform key point detection on the first image by using a deep learning algorithm when the first image is a key frame, and obtain the coordinates of the key points of the target object in the first image.

[0220] Optionally, the processing module 32 is specifically configured to obtain the optical flow change vector by using a sparse optical flow field algorithm.

[0221] Optionally, the processing module 32 is specifically configured to perform key point detection on the first image by using a deep learning algorithm, cache the first image and the coordinates of the key points of the target object in the first image into the key frame cache pool when the coordinates of the key points of the target object are successfully detected, and predict the coordinates of the key points of the target object in the first image according to the optical flow change vector between the first image and the first key frame and the coordinates of the key points of the target object in the first key frame when the coordinates of the key points of the target object are not successfully detected.

[0222] Optionally, the processing module 32 is specifically configured to cache the first image and the coordinates of the key points of the target object in the first image into the key frame cache pool by using a key-value pair. The key of the key-value pair is the identifier of the key frame, and the value of the key-value pair includes the first image and the coordinates of the key points of the target object in the first image.

[0223] Optionally, the processing module 32 is specifically configured to determine that the first image is a key frame when the key frame cache pool is empty, and determine whether the first image is a key frame according to the brightness change between the first image and the first key frame when there is a key frame in the key frame cache pool.

[0224] Optionally, the processing module 32 is specifically configured to: obtain an absolute value of a brightness difference between the first image and the first key frame; when the absolute value of the brightness difference is less than or equal to a preset threshold, determine that the first image is a non-key frame; or when the absolute value of the brightness difference is greater than the preset threshold, determine that the first image is a key frame.

[0225] Optionally, the coordinates of the key points are coordinates in an image coordinate system. Optionally, the processing module 32 is specifically configured to: obtain a chest orientation vector of the target object in a three-dimensional coordinate system according to the coordinates of the key points of the target object in the first image; calibrate the coordinates of the key points of the target object in the first image according to the chest orientation vector of the target object, to obtain calibrated coordinates of the key points of the target object; and obtain the pose detection result of the target object by using the calibrated coordinates of the key points of the target object. The origin of the three-dimensional coordinate system is a chest center point of the target object. A two-dimensional coordinate system in which the calibrated coordinates of the key points of the target object are located is parallel to a two-dimensional coordinate system in which the image acquisition device is located.

[0226] Optionally, the key points include a left shoulder, a right shoulder, and a chest center point. Optionally, the processing module 32 is specifically configured to: obtain a shoulder vector of the target object in the first image and coordinates of a shoulder center point according to the coordinates of the left shoulder and the coordinates of the right shoulder of the target object in the first image; obtain a torso vector of the target object according to the coordinates of the shoulder center point and the coordinates of the chest center point of the target object in the first image; and obtain a chest orientation vector of the target object according to the shoulder vector and the torso vector of the target object in the first image.

[0227] Optionally, the processing module 32 is specifically configured to: obtain a projection vector of the chest orientation vector on a vertical plane of the three-dimensional coordinate system according to the chest orientation vector of the target object; obtain a rotation matrix of the two-dimensional coordinate system according to the projection vector; and calibrate the coordinates of the key points of the target object in the first image by using the rotation matrix, to obtain calibrated coordinates of the key points of the target object.

[0228] Optionally, the processing module 32 is specifically configured to: obtain a detection parameter of the target object by using the calibrated coordinates of the key points of the target object; and obtain the pose detection result of the target object according to the detection parameter of the target object. The detection parameter includes at least one of the following: a left-right tilt angle of the head, a left-right tilt angle of the upper body, a shoulder-jaw difference, and a proportion of the head relative to the body.

[0229] Optionally, when the sitting posture detection result represents that the target object has a sitting posture problem, the sitting posture detection result of the target object comprises: the target object has a sitting posture problem, and a category to which the sitting posture problem belongs. In this implementation manner, the alarm information further comprises: the category to which the sitting posture problem belongs.

[0230] Optionally, the apparatus can further comprise a receiving module 34 configured to receive a sitting posture detection instruction before the first image of the target object is acquired by the image acquisition device.

[0231] Alternatively, the processing module 32 is further configured to start the sitting posture detection function when it is determined that the target object exists in the acquisition range according to the image acquired by the image acquisition device before the first image of the target object is acquired by the image acquisition device.

[0232] The sitting posture detection apparatus provided in the present application is used to execute the sitting posture detection method provided in the foregoing embodiments, and has similar implementation principles and technical effects, which will not be described herein again.

[0233] FIG. 8 An electronic device structure schematic diagram is provided in the present application. For example, the electronic device can be a smart table lamp, a smart camera, etc. FIG. 8 As shown in the figure, the electronic device 400 can comprise at least one processor 401 and a memory 402.

[0234] The memory 402 is used to store programs. Specifically, the programs can comprise program codes, and the program codes comprise computer operation instructions.

[0235] The memory 402 can comprise a high-speed RAM memory, and can also comprise a non-volatile memory, for example, at least one disk memory.

[0236] The processor 401 is used to execute the computer operation instructions stored in the memory 402, so as to implement the sitting posture detection method described in the foregoing method embodiments. The processor 401 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0237] Taking the smart table lamp as an example, the smart table lamp can further comprise an image acquisition device. The image acquisition device can be used to acquire the first image of the target object. Optionally, the smart table lamp can further comprise a display screen, a voice broadcast device, etc.

[0238] Optionally, the electronic device 400 can further include a communication interface 403. In a specific implementation, if the communication interface 403, the memory 402, and the processor 401 are independently implemented, the communication interface 403, the memory 402, and the processor 401 can be connected to each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0239] Optionally, in a specific implementation, if the communication interface 403, the memory 402, and the processor 401 are integrated on a chip, the communication interface 403, the memory 402, and the processor 401 can complete communication through an internal interface.

[0240] The application further provides a computer readable storage medium, which can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes. Specifically, the computer readable storage medium stores program instructions, and the program instructions are used for the method in the above embodiments.

[0241] The application further provides a program product, which includes execution instructions stored in a readable storage medium. At least one processor of an electronic device can read the execution instructions from the readable storage medium, and the at least one processor executes the execution instructions to enable the electronic device to implement the sitting posture detection method provided in the various embodiments.

[0242] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. A sitting posture detection method characterized by comprising: The method comprises: acquiring a first image of a target object collected by an image collection device; determining whether the first image is a key frame; if the first image is not a key frame, acquiring an optical flow change vector between the first image and a first key frame, and predicting coordinates of key points of the target object in the first image according to the optical flow change vector and coordinates of the key points of the target object in the first key frame; the first key frame is a key frame collected by the image collection device closest to the first image collection time in a key frame cache pool, and the optical flow change vector is used to represent a motion direction and a motion speed of the target object; if the first image is a key frame, performing key point detection on the first image by using a deep learning algorithm to acquire the coordinates of the key points of the target object in the first image; acquiring a pose detection result of the target object according to the coordinates of the key points of the target object in the first image; if the pose detection result indicates that the target object has a pose problem, outputting an alarm information, and the alarm information is used to prompt the target object to adjust the pose.

2. The method of claim 1, wherein, The method comprises: acquiring the optical flow change vector by using a sparse optical flow field algorithm.

3. The method of claim 1, wherein, The method comprises: performing key point detection on the first image by using a deep learning algorithm; if the coordinates of the key points of the target object are successfully detected, caching the first image and the coordinates of the key points of the target object in the first image into the key frame cache pool; if the coordinates of the key points of the target object are not successfully detected, predicting the coordinates of the key points of the target object in the first image according to the optical flow change vector between the first image and the first key frame and the coordinates of the key points of the target object in the first key frame.

4. The method of claim 3, wherein, The method comprises: caching the first image and the coordinates of the key points of the target object in the first image into the key frame cache pool in the form of a key-value pair; wherein the key of the key-value pair is an identifier of a key frame, and the value of the key-value pair comprises the first image and the coordinates of the key points of the target object in the first image.

5. The method of claim 1, wherein, The method comprises: if the key frame cache pool is empty, determining that the first image is a key frame; if there is a key frame cached in the key frame cache pool, determining whether the first image is a key frame according to a brightness change between the first image and the first key frame.

6. The method of claim 5, wherein, The method comprises: acquiring an absolute value of a brightness difference between the first image and the first key frame; If the absolute value of the luminance difference is less than or equal to a preset threshold, it is determined that the first image is a non-key frame. Or, if the absolute value of the luminance difference is greater than the preset threshold, it is determined that the first image is a key frame.

7. The method according to any one of claims 1 to 6, characterized in that, The coordinates of the key points are coordinates in an image coordinate system, and the obtaining of the pose detection result of the target object according to the coordinates of the key points of the target object in the first image comprises: obtaining a chest orientation vector of the target object in a three-dimensional coordinate system according to the coordinates of the key points of the target object in the first image, wherein an origin of the three-dimensional coordinate system is a chest center point of the target object; calibrating the coordinates of the key points of the target object in the first image according to the chest orientation vector of the target object to obtain calibrated coordinates of the key points of the target object, wherein a two-dimensional coordinate system in which the calibrated coordinates of the key points of the target object are located is parallel to a two-dimensional coordinate system in which the image acquisition device is located; and obtaining the pose detection result of the target object by using the calibrated coordinates of the key points of the target object.

8. The method of claim 7, wherein, The key points include a left shoulder, a right shoulder, and a chest center point, and the obtaining of the chest orientation vector of the target object in the three-dimensional coordinate system according to the coordinates of the key points of the target object in the first image comprises: obtaining a shoulder vector of the target object in the first image and coordinates of a shoulder center point according to the coordinates of the left shoulder and the coordinates of the right shoulder of the target object in the first image; obtaining a torso vector of the target object according to the coordinates of the shoulder center point and the coordinates of the chest center point of the target object in the first image; obtaining the chest orientation vector of the target object according to the shoulder vector and the torso vector of the target object in the first image.

9. The method of claim 8, wherein, The calibrating of the coordinates of the key points of the target object in the first image according to the chest orientation vector of the target object to obtain the calibrated coordinates of the key points of the target object comprises: obtaining a projection vector of the chest orientation vector on a vertical plane of the three-dimensional coordinate system according to the chest orientation vector of the target object; obtaining a rotation matrix of the two-dimensional coordinate system according to the projection vector; calibrating the coordinates of the key points of the target object in the first image by using the rotation matrix to obtain the calibrated coordinates of the key points of the target object.

10. The method of claim 7, wherein, The obtaining of the pose detection result of the target object by using the calibrated coordinates of the key points of the target object comprises: obtaining detection parameters of the target object by using the calibrated coordinates of the key points of the target object, wherein the detection parameters include at least one of a left-right tilt angle of a head, a left-right tilt angle of an upper body, a shoulder-jaw difference, and a proportion of the head relative to a body; and obtaining the pose detection result of the target object according to the detection parameters of the target object.

11. The method of claim 10, wherein, If the pose detection result indicates that the target object has a pose problem, the pose detection result of the target object includes that the target object has the pose problem and a category to which the pose problem belongs. The alarm information further comprises: a category to which the sitting posture problem belongs.

12. The method according to any one of claims 1 to 6, characterized in that, Before the first image of the target object collected by the image collection device is acquired, the method further comprises: receiving a sitting posture detection instruction; Alternatively, when it is determined according to the image collected by the image collection device that the target object exists in the collection range, the sitting posture detection function is started.

13. A sitting posture detection apparatus characterized by comprising: The device comprises: an acquisition module configured to acquire a first image of a target object collected by an image collection device; a processing module configured to determine whether the first image is a key frame; if the first image is not a key frame, acquire an optical flow change vector between the first image and a first key frame, and predict coordinates of key points of the target object in the first image according to the optical flow change vector and coordinates of key points of the target object in the first key frame; the first key frame is a key frame collected by the image collection device in a key frame cache pool and closest to a first image collection time, and the optical flow change vector is used to represent a motion direction and a motion speed of the target object; if the first image is a key frame, perform key point detection on the first image by using a deep learning algorithm to acquire the coordinates of the key points of the target object in the first image; and acquire a sitting posture detection result of the target object according to the coordinates of the key points of the target object in the first image; an output module configured to output an alarm information when the sitting posture detection result represents that the target object has a sitting posture problem, the alarm information being used to prompt the target object to adjust the sitting posture.

14. An electronic device, comprising: comprise: at least one processor, a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, so that the electronic device executes the method of any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by the processor, the method of any one of claims 1-12 is implemented.

16. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-12.

Images