Learning and accompanying method, device and robot combined with abnormal sitting posture dynamic detection
By pre-training the object detection model and monocular depth detection technology, combining video data and depth information, students' abnormal sitting postures are identified, which solves the problems of inaccurate and high cost in the existing technology, and realizes accurate and economical student sitting posture monitoring. , improves the health and learning effect of the learning environment.
Patent Information
- Application Number
- CN202311775638.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-21
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2043-12-21
AI Technical Summary
The existing technology has problems of inaccurate and high cost in student learning and accompanying scenarios, especially portable or cheap sensors have limitations in accuracy and stability, and the design and manufacturing of depth sensors require professional technology, which increases the cost. and affected by environmental conditions.
Using pre-trained object detection model and monocular depth detection technology, by obtaining real-time video data of students' learning scenes, decomposing them into multi-frame real-time images, extracting key point information and depth information, combining abnormal sitting posture parameters to identify abnormal sitting postures, and to users Send a reminder message.
It realizes accurate detection of students' abnormal sitting posture without affecting the learning effect, reduces detection costs, improves the accuracy and efficiency of detection, provides immediate feedback, and helps students and guardians take timely measures to prevent potential health problems.
Smart Images

Figure CN117746505B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of learning and accompanying technology, and in particular to a learning and accompanying method, device and robot combined with dynamic detection of abnormal sitting posture. Background Art
[0002] Students usually need to maintain a relatively fixed sitting posture for a long time during the long learning process, especially in class or when doing homework. Bad sitting posture may cause discomfort in various parts of the body, such as cervical and lumbar spine problems. By testing students' learning sitting posture, schools and parents can discover bad habits in time and take measures to improve them, thereby maintaining students' physical health; at the same time, proper sitting posture helps to keep students focused and improve learning effects. Correct sitting posture helps promote blood circulation and increase oxygen supply to the brain, thereby improving learning efficiency. On the contrary, bad sitting posture may cause students to become tired and distracted, affecting their academic performance; bad sitting posture Studying sitting posture may cause eye fatigue and discomfort, increasing the risk of vision problems. By paying attention to students' sitting posture, the quality of the learning environment can be improved and an atmosphere conducive to students' physical and mental health and academic development can be created. Existing technologies use sensors (such as gyroscopes, accelerometers, etc.) to monitor students' sitting posture. The precision and stability of these sensors are crucial to accuracy. However, some portable or cheap sensors may have limitations in accuracy and stability, resulting in inaccurate posture detection; some technologies may require students to wear equipment for a long time or maintain a specific posture during the learning process, which may affect students' comfort and thus affect learning outcomes.
[0003] The existing Chinese patent CN107169456A discloses a sitting posture detection method based on sitting posture depth image, which specifically discloses obtaining key points by segmenting the sitting posture image, the key points include the head vertex, the head center point, the left shoulder point, the right shoulder point, the shoulder center point, and the body center point; determining the sitting posture according to the key points; obtaining the sitting posture depth image of the human body through the depth sensor, if the angle between the head center point-shoulder center point connection and the horizontal direction is greater than 105°, and the angle between the shoulder center point-body center point connection and the horizontal direction is 80°-100°, it is a left head tilt; if the angle between the head center point-shoulder center point connection and the horizontal direction is less than 75°, and the shoulder center point-body center point connection is less than 80°-100°, it is a left head tilt; If the angle between the point connection line and the horizontal direction is 80°-100°, it is a right head tilt; if the depth distance value of the head vertex is more than 20mm greater than the depth distance value of the head center, it is an upward head tilt; if the depth distance value of the head vertex is more than 20mm less than the depth distance value of the head center, it is a downward head tilt; if the angle between the shoulder center point-body center point connection line and the horizontal direction is <80°, the body is tilted to the left; if the angle between the shoulder center point-body center point connection line and the horizontal direction is >100°, the body is tilted to the right; according to the human body contour, the circumscribed rectangle of the contour is obtained. If the aspect ratio of the rectangle is less than 0.6, it is lying down; if the depth distance value from the shoulder center point to the body center point shows a decreasing trend, it is leaning back. Although the above patents also judge the sitting posture based on the depth, in order to obtain high-quality depth images, a specially designed depth sensor is usually required. These sensors use different technologies, such as structured light, time of flight, or stereo vision, to measure the distance from an object to the sensor. The design and manufacture of such special sensors often require highly specialized technology and engineering knowledge, which increases costs. At the same time, the performance of depth sensors is usually affected by environmental conditions, such as lighting, occlusion, etc. In order to ensure stable operation in various environments, additional hardware or complex calibration systems may be required, which also increases costs.
[0004] Therefore, how to accurately detect abnormal sitting postures in student learning scenarios without affecting learning outcomes and reduce detection costs is an urgent problem to be solved. Summary of the invention
[0005] In view of this, the present invention provides a learning care method, device and robot combined with dynamic detection of abnormal sitting posture, so as to solve the problems of inaccurate abnormal sitting posture detection and high cost in the existing technology of student learning care scenario.
[0006] The technical solution adopted by the present invention is:
[0007] In a first aspect, the present invention provides a learning and accompanying method combined with abnormal sitting posture dynamic detection, the method comprising:
[0008] S1: Acquire real-time video data in a student learning and accompanying scenario, and decompose the real-time video data into multiple frames of real-time images;
[0009] S2: Inputting each of the real-time images into a pre-trained target detection model, and extracting a student learning image when the learning supplies intersect with the hand according to the detected preset learning supplies position information and hand position information;
[0010] S3: performing feature extraction and monocular depth detection on the student learning image, and outputting key point information and depth information related to the student learning;
[0011] S4: Analyze the student's real-time sitting posture according to the key point information and the depth information, and output the real-time sitting posture analysis result;
[0012] S5: identifying abnormal sitting posture according to the real-time sitting posture analysis result and in combination with preset abnormal sitting posture parameters;
[0013] S6: When an abnormal sitting posture is identified, a reminder message corresponding to the sitting posture detection result is sent to the user, reminding the user to adjust the student's sitting posture.
[0014] Preferably, S3 includes:
[0015] S31: inputting multiple frames of the student learning image into a human key point detection model to identify the key point information, wherein the human key points include at least: left eye, right eye, nose, left shoulder and right shoulder;
[0016] S32: Performing monocular depth detection on the student learning image to obtain a relative depth map corresponding to the student learning image;
[0017] S33: filtering the relative depth maps corresponding to the continuous multiple frames of student images, and outputting the depth maps after filtering as the depth information.
[0018] Preferably, the S4 includes:
[0019] S41: Determine the coordinate positions of key points in a plurality of consecutive frames of student learning images according to the key point information;
[0020] S42: acquiring a depth value corresponding to the key point coordinate position according to the key point coordinate position and the depth information;
[0021] S43: Determine the coordinate position of the first target key point and the corresponding first depth value, and determine the coordinate position of the second target key point and the corresponding second depth value according to the key point coordinate position and the corresponding depth value, wherein the first target key point includes the left eye, the right eye and the nose, and the second target key point includes the left shoulder and the right shoulder;
[0022] S44: determining the degree of head deviation of the student according to the coordinate position of the first target key point and the corresponding first depth value, and outputting a head deviation determination result;
[0023] S45: determining the degree of body deviation of the student according to the coordinate position of the second target key point and the corresponding second depth value, and outputting a body deviation determination result;
[0024] S46: Outputting the real-time sitting posture analysis result based on the head deviation determination result and the body deviation determination result.
[0025] Preferably, the S44 includes:
[0026] S441: acquiring a left eye coordinate position, a right eye coordinate position, a left eye depth value, and a right eye depth value according to the first target key point coordinate position and the first depth value;
[0027] S442: Compare the left eye coordinate position and the right eye coordinate position in each frame of the learning image to determine whether the head is offset to the left or to the right;
[0028] S443: Using the left eye depth value and the right eye depth value as eye depth values, comparing the eye depth values in the depth maps of adjacent frames, and determining whether the head is offset forward or backward.
[0029] Preferably, the S45 includes:
[0030] S451: Acquire a left shoulder coordinate position, a right shoulder coordinate position, a left shoulder depth value, and a right shoulder depth value according to the second target key point coordinate position and the second depth value;
[0031] S452: comparing the shoulder coordinate positions in the adjacent frame learning images according to the left shoulder coordinate position and the right shoulder coordinate position, and determining whether the body is shifted to the left or to the right;
[0032] S453: According to the left shoulder depth value and the right shoulder depth value, the shoulder depth values in the adjacent frame depth maps are compared to determine whether the body is offset forward or backward.
[0033] Preferably, the S5 includes:
[0034] S51: If it is determined that the head is offset to the left or to the right, then according to the coordinate position of the first target key point, the left eye key point and the nose key point are connected to obtain a first straight line, and the right eye key point and the nose key point are connected to obtain a second straight line;
[0035] S52: Obtain a first angle between the first straight line and the vertical direction and a second angle between the second straight line and the vertical direction;
[0036] S53: comparing the angle difference between the first angle and the second angle with a preset angle difference threshold, and identifying a first abnormal sitting posture when the angle difference in the continuous multiple frames of images is greater than the angle difference threshold;
[0037] S54: If it is determined that the head is shifted forward or backward, the absolute value of the eye depth change in the adjacent frame depth map is calculated. When the absolute value of the eye depth value change calculated in multiple consecutive frames is greater than a preset depth threshold, it is identified as a second abnormal sitting posture.
[0038] Preferably, the S5 further includes:
[0039] S55: If it is determined that the body is shifted to the left or to the right, the left shoulder coordinate position and the right shoulder coordinate position are obtained according to the coordinate position of the second target key point;
[0040] S56: Calculate the difference between the longitudinal distances of the left shoulder and the right shoulder according to the left shoulder coordinate position and the right shoulder coordinate position;
[0041] S57: When the distance difference in the continuous multiple frames of images is greater than a preset distance threshold, identifying it as a third abnormal sitting posture;
[0042] S58: If it is determined that the body is shifted forward or backward, the absolute value of the shoulder depth change in the adjacent frame depth map is calculated. When the absolute value of the shoulder depth value change calculated in multiple consecutive frames is greater than a preset depth threshold, it is identified as the fourth abnormal sitting posture.
[0043] In a second aspect, the present invention provides a learning accompanying device combined with abnormal sitting posture dynamic detection, the device comprising:
[0044] An image acquisition module is used to acquire real-time video data in a student learning and accompanying scenario, and decompose the real-time video data into multiple frames of real-time images;
[0045] A learning image extraction module, used to input each of the real-time images into a pre-trained target detection model, and extract a student learning image when the learning supplies intersect with the hand based on the detected preset learning supplies position information and hand position information;
[0046] An information extraction module, used to perform feature extraction and monocular depth detection on the student learning image, and output key point information and depth information related to the student learning;
[0047] A real-time sitting posture analysis module is used to analyze the real-time sitting posture of the student based on the key point information and depth information, and output the real-time sitting posture analysis result;
[0048] An abnormal sitting posture recognition module, used to recognize abnormal sitting posture according to the real-time sitting posture analysis result and in combination with preset abnormal sitting posture parameters;
[0049] The user reminder module is used to send a reminder message corresponding to the sitting posture detection result to the user when an abnormal sitting posture is identified, so as to remind the user to adjust the student's sitting posture.
[0050] In a third aspect, an embodiment of the present invention further provides a learning companion robot, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method of the first aspect in the above-mentioned embodiment.
[0051] In a fourth aspect, an embodiment of the present invention further provides a storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method of the first aspect in the above-mentioned embodiment is implemented.
[0052] In summary, the beneficial effects of the present invention are as follows:
[0053] The present invention provides a learning care method, device and robot combined with dynamic detection of abnormal sitting posture, the method comprising: acquiring real-time video data in a student learning care scenario, and decomposing the real-time video data into multiple frames of real-time images; inputting each of the real-time images into a pre-trained target detection model, and extracting a student learning image when the learning supplies intersect with the hands based on the detected preset learning supplies position information and hand position information; performing feature extraction and monocular depth detection on the student learning image, and outputting key point information and depth information related to the student learning; analyzing the student's real-time sitting posture based on the key point information and depth information, and outputting a real-time sitting posture analysis result; identifying an abnormal sitting posture based on the real-time sitting posture analysis result and in combination with preset abnormal sitting posture parameters; when an abnormal sitting posture is identified, sending a reminder message corresponding to the sitting posture detection result to the user, reminding the user to adjust the student's sitting posture. The present invention adopts a pre-trained target detection model, which can more accurately identify the learning supplies and hand positions in the student learning scene and thus accurately extract the student learning image; the comprehensive use of key point information and monocular depth detection helps to more comprehensively understand the student's sitting posture, and multi-level information fusion also improves the accuracy of abnormal sitting posture detection; the use of pre-trained target detection models and deep learning technology can give full play to the advantages of computer vision in image analysis. Compared with traditional hardware equipment, this software-based solution usually requires lower costs. At the same time, the present invention can also provide instant feedback and reminders in real-time video data, so that students and guardians can take timely action, which helps to prevent potential health problems and improve the efficiency of responding to abnormal sitting postures. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for use in the embodiment of the present invention will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.
[0055] Figure 1 It is a schematic diagram of the overall working process of the learning and accompanying method combined with abnormal sitting posture dynamic detection in Example 1 of the present invention;
[0056] Figure 2 This is a schematic diagram of the process of extracting student learning images in Example 1 of the present invention;
[0057] Figure 3 This is a schematic diagram of a process for analyzing a student's real-time sitting posture in Embodiment 1 of the present invention;
[0058] Figure 4 This is a schematic diagram of a process for determining the degree of deviation of a student's head in Embodiment 1 of the present invention;
[0059] Figure 5 This is a schematic diagram of a process for determining the degree of body deviation of a student in Embodiment 1 of the present invention;
[0060] Figure 6 This is a schematic diagram of a process for identifying the first and second abnormal sitting postures in Embodiment 1 of the present invention;
[0061] Figure 7 This is a schematic diagram of a process for identifying the third and fourth abnormal sitting postures in Embodiment 1 of the present invention;
[0062] Figure 8 It is a structural block diagram of a learning accompanying device combined with abnormal sitting posture dynamic detection in Example 2 of the present invention;
[0063] Fig. 9 This is a schematic diagram of the structure of the learning accompanying robot in Example 3 of the present invention. DETAILED DESCRIPTION
[0064] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described in conjunction with the drawings in the embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. In the description of the present invention, it should be understood that the orientation or position relationship indicated by the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. In the absence of further restrictions, the elements defined by the phrase "comprising..." do not exclude the existence of other identical elements in the process, method, article or device comprising the elements. If there is no conflict, the embodiments of the present invention and the various features in the embodiments can be combined with each other, all within the protection scope of the present invention.
[0065] Example 1
[0066] See also Figure 1 Embodiment 1 of the present invention discloses a learning and accompanying method combined with abnormal sitting posture dynamic detection, the method comprising:
[0067] S1: Acquire real-time video data in a student learning and accompanying scenario, and decompose the real-time video data into multiple frames of real-time images;
[0068] Specifically, use the camera or other video acquisition equipment on the accompanying robot to obtain real-time video data of the student learning accompanying scene, decompose the continuous video data into a series of still image frames, and implement it through video processing technology, such as using the frame extraction function in the computer vision library; further process each frame of the image, including image enhancement, denoising, and face detection, which helps to improve the accuracy and effectiveness of subsequent analysis; extract key features from each image, and apply the extracted features to the analysis of the learning accompanying scene to obtain information about the student's learning status and needs, which is used to adjust teaching strategies, provide personalized suggestions, etc.
[0069] S2: Inputting each of the real-time images into a pre-trained target detection model, and extracting a student learning image when the learning supplies intersect with the hand according to the detected preset learning supplies position information and hand position information;
[0070] Specifically, the real-time video data is input into a pre-trained target detection model, wherein the target detection model includes a deep learning model based on the YoloV8s structure, and detects books on the desktop and students' hands in the learning scene; extracts the preset learning supplies position information and hand position information, extracts the preset learning supplies (such as books) area position information and the student hand area position information from the output of the target detection model, and obtains the coordinates of each detected area by analyzing the output of the model.
[0071] For each frame, determine whether the position information of the extracted learning supplies area and the hand area intersects, and use geometric calculation methods, such as rectangular frame overlap determination, to determine whether they have intersecting areas; with every N frames as an analysis unit, count the number of intersecting frames to determine whether the student is learning. If the intersection of learning supplies and hands is detected in more than N / 2 frames, it can be considered that the student is learning. When it is determined that the student is learning, extract the corresponding student learning images, including the student's posture when studying and information on the use of learning supplies. These student learning images are further used to analyze the student's learning status or other related information.
[0072] S3: performing feature extraction and monocular depth detection on the student learning image, and outputting key point information and depth information related to the student learning;
[0073] Specifically, when it is determined that students are learning, the student learning images are deeply analyzed. By extracting features from the student learning images, key point information, such as the student's posture and gesture, is captured. At the same time, monocular depth detection technology is applied to obtain the depth information of each key point in the student learning image, so as to construct the student posture and learning environment in three-dimensional space. This deep analysis process is achieved through modern computer vision technology and deep learning models. Key point information provides the location of key points related to body posture, while depth information can provide the distance of these key points relative to the camera. The comprehensive analysis of this information helps to more comprehensively understand the student's behavior and environment during the learning process, and further provide more refined learning analysis, such as information on students' concentration, coordination of movements during learning, etc. This provides educators and researchers with more data support to optimize teaching methods, personalize learning experiences, and gain a deeper understanding of students' learning process.
[0074] In one embodiment, see Figure 2 , said S3 comprises:
[0075] S31: inputting multiple frames of the student learning image into a human key point detection model to identify the key point information, wherein the human key points include at least: left eye, right eye, nose, left shoulder and right shoulder;
[0076] Specifically, the MediaPipe_Pose model is used to detect human key points on multiple consecutive frames of the student learning images. The model can detect the key points of the human body, where the human key points include at least: left eye, right eye, nose, left shoulder and right shoulder; the detection of these key points can provide detailed information about human posture and facial expressions, providing a basis for subsequent depth detection.
[0077] S32: Performing monocular depth detection on the student learning image to obtain a relative depth map corresponding to the student learning image;
[0078] Specifically, the MobileDepth model is used to perform monocular depth detection on multiple consecutive frames of the student learning images. The MobileDepth model can generate a depth field composed of relative values, indicating the depth of each point in the image relative to the camera. This depth field can provide relative information about the distance of objects in the image from the camera. The MobileDepth model can give full play to the advantages of computer vision in image analysis. Compared with traditional hardware equipment, this software-based solution usually requires lower costs.
[0079] S33: filtering the relative depth maps corresponding to the continuous multiple frames of student images, and outputting the depth maps after filtering as the depth information.
[0080] Specifically, for N consecutive image frames, the depth map of each frame is calculated. In order to obtain a complete depth map, the relative depth map of each frame is first median filtered, which helps to deal with outliers in the depth map, making the depth map of a single image more complete and accurate. Subsequently, the relative depth maps of N consecutive frames are smoothed and mean filtered. The purpose of this step is to obtain a continuous and stable depth map. By averaging the depth information of multiple frames, the noise and fluctuations that may exist in the single-frame depth map are reduced, making the final depth map smoother and more reliable. The goal of the entire process is to obtain accurate and continuous depth information through human key point detection and monocular depth detection, combined with the filtering processing of the depth map, to provide high-quality data support for subsequent student learning behavior analysis.
[0081] S4: Analyze the student's real-time sitting posture according to the key point information and the depth information, and output the real-time sitting posture analysis result;
[0082] Specifically, based on the extracted human key point information and depth information, the real-time sitting posture of the student is analyzed, which includes a comprehensive analysis of the position and relative depth of the detected key points (such as eyes, nose, shoulders, etc.). Through this analysis process, the system is able to output detailed results about the student's current sitting posture, such as head tilt, shoulder position, body posture and other information.
[0083] In one embodiment, see Figure 3 , said S4 comprises:
[0084] S41: Determine the coordinate positions of key points in a plurality of consecutive frames of student learning images according to the key point information;
[0085] S42: acquiring a depth value corresponding to the key point coordinate position according to the key point coordinate position and the depth information;
[0086] Specifically, for each frame of the depth map, the previously mentioned human key point set K (including left eye p1, right eye p2, nose p3, left shoulder point p4, and left shoulder point p5) is used to find the corresponding depth value on the depth map. These depth values represent the depth of each key point relative to the camera. For N frames of images, the depth values of the key points in each frame are constructed into a temporal depth coordinate value D in chronological order. For example, D1(p1, p2, ..., p5) represents the depth value of the key point in the first frame, D2(p1, p2, ..., p5) represents the depth value of the key point in the second frame, and so on, until DN(p1, p2, ..., p5) represents the depth value of the key point in the Nth frame.
[0087] S43: Determine the coordinate position of the first target key point and the corresponding first depth value, and determine the coordinate position of the second target key point and the corresponding second depth value according to the key point coordinate position and the corresponding depth value, wherein the first target key point includes the left eye, the right eye and the nose, and the second target key point includes the left shoulder and the right shoulder;
[0088] S44: determining the degree of head deviation of the student according to the coordinate position of the first target key point and the corresponding first depth value, and outputting a head deviation determination result;
[0089] In one embodiment, see Figure 4 , the S44 comprises:
[0090] S441: acquiring a left eye coordinate position, a right eye coordinate position, a left eye depth value, and a right eye depth value according to the first target key point coordinate position and the first depth value;
[0091] S442: Compare the left eye coordinate position and the right eye coordinate position in each frame of the learning image to determine whether the head is offset to the left or to the right;
[0092] Specifically, for each frame of the learning image, the coordinate position information of the left eye (p1) and the right eye (p2) is extracted, and the y-axis coordinates of the left eye and the right eye are compared. Through observation and description, if the y-axis coordinate of the left eye in multiple consecutive frames of images is continuously greater than the y-axis coordinate of the right eye p2, since the camera captures mirror data, it is considered that the head is bent to the left. Conversely, if the y-axis coordinate sequence of the left eye p1 in multiple consecutive frames of images is smaller than the y-axis coordinate sequence of the right eye p2, it is considered that the head is bent to the right.
[0093] S443: Using the left eye depth value and the right eye depth value as eye depth values, comparing the eye depth values in the depth maps of adjacent frames, and determining whether the head is offset forward or backward.
[0094] Specifically, the left eye depth value and the right eye depth value are merged into the eye depth value. The merging method is achieved by taking the average of the depth values of the two. For continuous depth map frames, the eye depth values in adjacent frames are compared. If the eye depth value continues to increase over a period of time, it is considered that the head is shifting forward. Conversely, if the eye depth value continues to decrease over a period of time, it is considered that the head is shifting backward.
[0095] S45: determining the degree of body deviation of the student according to the coordinate position of the second target key point and the corresponding second depth value, and outputting a body deviation determination result;
[0096] In one embodiment, see Figure 5 , the S45 comprises:
[0097] S451: Acquire a left shoulder coordinate position, a right shoulder coordinate position, a left shoulder depth value, and a right shoulder depth value according to the second target key point coordinate position and the second depth value;
[0098] S452: comparing the shoulder coordinate positions in the adjacent frame learning images according to the left shoulder coordinate position and the right shoulder coordinate position, and determining whether the body is shifted to the left or to the right;
[0099] Specifically, for each frame of the learning image, the coordinate position information of the left shoulder and the right shoulder is extracted. For continuous image frames, the coordinate positions of the left shoulder and the right shoulder in adjacent frame images are compared. Specifically, the coordinate changes of the left shoulder and the right shoulder on the x-axis are observed, and the deviation direction of the body is determined according to the comparison results. If the x-axis coordinate of the left shoulder continues to increase over a period of time, while the x-axis coordinate of the right shoulder is relatively small, since the camera captures mirror data, it is considered that the body is deviating to the right. Conversely, if the x-axis coordinate of the left shoulder is relatively small over a period of time, while the x-axis coordinate of the right shoulder continues to increase, it is considered that the body is deviating to the left.
[0100] S453: According to the left shoulder depth value and the right shoulder depth value, the shoulder depth values in the adjacent frame depth maps are compared to determine whether the body is offset forward or backward.
[0101] Specifically, for each depth map frame, the depth values of the left shoulder and the right shoulder are extracted, and for consecutive depth map frames, the depth values of the left shoulder and the right shoulder in adjacent frames are compared. The changes in the depth values of the left and right shoulders can be observed, and based on the comparison results, the system can determine the direction of the body's deviation. For example, if the left shoulder depth value continues to decrease over a period of time, while the right shoulder depth value is relatively large, it is considered that the body is deviating forward; conversely, if the left shoulder depth value continues to increase, while the right shoulder depth value is relatively small, it is considered that the body is deviating backward.
[0102] S46: Outputting the real-time sitting posture analysis result based on the head deviation determination result and the body deviation determination result.
[0103] Specifically, according to the previous steps, the offset direction of the head is obtained, that is, the head is offset to the left or to the right; similarly, the offset direction of the body is obtained, that is, the body is offset forward or backward; the judgment results of the head offset and the body offset are combined to form a comprehensive result of the real-time sitting posture analysis, and the comprehensive real-time sitting posture analysis result is output to indicate the student's sitting posture.
[0104] S5: identifying abnormal sitting posture according to the real-time sitting posture analysis result and in combination with preset abnormal sitting posture parameters;
[0105] In one embodiment, see Figure 6 , the S5 comprises:
[0106] S51: If it is determined that the head is offset to the left or to the right, then according to the coordinate position of the first target key point, the left eye key point and the nose key point are connected to obtain a first straight line, and the right eye key point and the nose key point are connected to obtain a second straight line;
[0107] Specifically, if the head is determined to be offset to the left or to the right, assuming that in a certain frame of the learning image, the left eye key point position is (x1, y1), the right eye key point position is (x2, y2), and the nose key point position is (x3, y3), through these coordinate information, it can be obtained that the first straight line is the straight line from the left eye to the nose, that is, straight line L1, and the second straight line is the straight line from the right eye to the nose, that is, straight line L2. The straight line connecting the left and right eyes to the nose helps to form the reference axis of the head, which is convenient for the subsequent analysis of the angular offset of the head. By connecting the straight lines, the posture changes of the head can be better described, which provides a basis for the subsequent angle calculation.
[0108] S52: Obtain a first angle between the first straight line and the vertical direction and a second angle between the second straight line and the vertical direction;
[0109] Specifically, suppose that through calculation, the angle between the straight line L1 and the vertical direction is θ1, and the angle between the straight line L2 and the vertical direction is θ2. Obtaining the angle information between the left and right eyes and the nose helps to quantify the degree of head deviation. This angle information can be used to determine whether the head is deviated left or right, providing a more detailed head posture analysis.
[0110] S53: comparing the angle difference between the first angle and the second angle with a preset angle difference threshold, and identifying a first abnormal sitting posture when the angle difference in the continuous multiple frames of images is greater than the angle difference threshold;
[0111] Specifically, the angle difference between the first angle and the second angle is compared with a preset angle difference threshold. In the continuous multi-frame image, if the angle difference between θ1 and θ2 is greater than the preset angle difference threshold, the first abnormal sitting posture is identified as the left-right deviation of the head. Such a determination helps to timely discover students' bad learning postures, provide real-time feedback, and promote correct sitting postures.
[0112] S54: If it is determined that the head is shifted forward or backward, the absolute value of the eye depth change in the adjacent frame depth map is calculated. When the absolute value of the eye depth value change calculated in multiple consecutive frames is greater than a preset depth threshold, it is identified as a second abnormal sitting posture.
[0113] Specifically, if the head is determined to be shifted forward or backward, the depth map is calculated to determine that the absolute value of the change in the eye depth value of the adjacent frames is greater than the preset depth threshold, and the second abnormal sitting posture is identified as the forward and backward shift of the head through the change in the eye depth value. Such a determination helps monitor the changes in the students' body posture, provide timely feedback, and promote the correct sitting posture.
[0114] In one embodiment, see Figure 7 , the S5 further includes:
[0115] S55: If it is determined that the body is shifted to the left or to the right, the left shoulder coordinate position and the right shoulder coordinate position are obtained according to the coordinate position of the second target key point;
[0116] Specifically, if the body is determined to be offset to the left or to the right, in a frame of learning image, according to the coordinate position of the second target key point, the coordinate position of the left shoulder key point is obtained as (x4, y4), and the coordinate position of the right shoulder key point is obtained as (x5, y5). By obtaining the coordinate positions of the left and right shoulders, the position information of the shoulders can be accurately captured, providing a basis for subsequent analysis.
[0117] S56: Calculate the difference between the longitudinal distances of the left shoulder and the right shoulder according to the left shoulder coordinate position and the right shoulder coordinate position;
[0118] Specifically, according to the coordinate positions of the left shoulder and the right shoulder, the difference in longitudinal distance is calculated, that is, Δy=|y4-y5|. By calculating the distance difference of the shoulder on the y-axis, it is possible to understand whether the shoulder is offset in the vertical direction, which helps to determine whether the body is bent to the left or right.
[0119] S57: When the distance difference in the continuous multiple frames of images is greater than a preset distance threshold, identifying it as a third abnormal sitting posture;
[0120] Specifically, in the continuous multi-frame images, it is observed that the difference Δy in the longitudinal distance of the shoulders continues to increase and is greater than the preset distance threshold, and the third abnormal sitting posture is identified as the left and right bending of the body. By judging the change in the difference in the longitudinal distance of the shoulders, the left and right bending abnormal sitting posture of the body can be identified. This helps to timely discover the bad sitting posture that students may have when studying and provide timely feedback.
[0121] S58: If it is determined that the body is shifted forward or backward, the absolute value of the shoulder depth change in the adjacent frame depth map is calculated. When the absolute value of the shoulder depth value change calculated in multiple consecutive frames is greater than a preset depth threshold, it is identified as the fourth abnormal sitting posture.
[0122] Specifically, when the body is shifted forward or backward, by calculating the absolute value of the shoulder depth change in the depth map of adjacent frames, it is observed that the depth value change in multiple consecutive frames is greater than the preset depth threshold, and the fourth abnormal sitting posture is identified as the forward and backward deviation of the body. By observing the change in the shoulder depth value, it is determined whether the body is shifted forward and backward, which helps to monitor the changes in students' body posture during the learning process and provide real-time feedback.
[0123] S6: When an abnormal sitting posture is identified, a reminder message corresponding to the sitting posture detection result is sent to the user, reminding the user to adjust the student's sitting posture.
[0124] Specifically, if the student's head is detected to be deflected to the left, a reminder message is sent to the user, such as: "Note that the student's head is deflected to the left, please consider adjusting the sitting posture"; this reminder helps to correct the student's bad sitting posture due to habitual head posture, and avoid discomfort caused by maintaining the same head deviation for a long time; if the student's head is detected to be deflected forward, a reminder message can be sent to the user, such as: "Note that the student's head is leaning forward, it is recommended to adjust the sitting posture to maintain the correct learning posture." Such a reminder helps to prevent students from being in an inappropriate head posture for a long time, and helps to maintain good neck and spine health. If the student's body is detected to be bent to the left, a reminder message can be sent to the user, such as: Reminder, the student's body is bent to the left, it is recommended to adjust to maintain a uniform sitting posture, this reminder helps to correct the student's inappropriate body bending due to habitual body posture, and prevent potential health problems; if the system detects that the student's body is deflected forward, a reminder message is sent to the user, such as: "Please note that the student's body is leaning forward, it is recommended to adjust to maintain the correct sitting posture." Such a reminder helps to prevent students from being in an inappropriate forward body posture for a long time, helps to maintain the health of the spine, and reduces possible discomfort. By sending users specific reminders and providing timely and personalized feedback, it helps users correct poor learning postures, which not only helps students maintain good health habits, but also improves learning outcomes and comfort.
[0125] Example 2
[0126] See also Figure 8 Embodiment 2 of the present invention further provides a learning accompanying device combined with abnormal sitting posture dynamic detection, the device comprising:
[0127] An image acquisition module is used to acquire real-time video data in a student learning and accompanying scenario, and decompose the real-time video data into multiple frames of real-time images;
[0128] A learning image extraction module, used to input each of the real-time images into a pre-trained target detection model, and extract a student learning image when the learning supplies intersect with the hand based on the detected preset learning supplies position information and hand position information;
[0129] An information extraction module, used to perform feature extraction and monocular depth detection on the student learning image, and output key point information and depth information related to the student learning;
[0130] In one embodiment, the information extraction module includes:
[0131] A key point information extraction submodule is used to input multiple frames of the student learning image into a human key point detection model to identify the key point information, wherein the human key points include at least: left eye, right eye, nose, left shoulder and right shoulder;
[0132] A monocular depth detection submodule is used to perform monocular depth detection on the student learning image to obtain a relative depth map corresponding to the student learning image;
[0133] The filtering processing submodule is used to filter the relative depth maps corresponding to multiple consecutive frames of student images, and output the depth maps after filtering as the depth information.
[0134] A real-time sitting posture analysis module is used to analyze the real-time sitting posture of the student based on the key point information and depth information, and output the real-time sitting posture analysis result;
[0135] Preferably, the real-time sitting posture analysis module includes:
[0136] A key point coordinate position extraction submodule is used to determine the key point coordinate positions in multiple consecutive frames of student learning images based on the key point information;
[0137] A depth value acquisition submodule, used to acquire a depth value corresponding to the key point coordinate position according to the key point coordinate position and the depth information;
[0138] a depth value determination submodule, configured to determine a first target key point coordinate position and a corresponding first depth value, and determine a second target key point coordinate position and a corresponding second depth value according to the key point coordinate position and the corresponding depth value, wherein the first target key point includes the left eye, the right eye and the nose, and the second target key point includes the left shoulder and the right shoulder;
[0139] A deviation degree determination submodule, used to determine the deviation degree of the student's head according to the coordinate position of the first target key point and the corresponding first depth value, and output a head deviation determination result;
[0140] In one embodiment, the offset degree determination submodule includes:
[0141] an eye coordinate position and depth value acquisition unit, configured to acquire a left eye coordinate position, a right eye coordinate position, a left eye depth value, and a right eye depth value according to the first target key point coordinate position and the first depth value;
[0142] A head left-right deviation determination unit, used for comparing the left eye coordinate position and the right eye coordinate position in each frame of learning image to determine whether the head is deviated to the left or to the right;
[0143] The head forward and backward offset determination unit is used to use the left eye depth value and the right eye depth value as eye depth values, compare the eye depth values in the adjacent frame depth maps, and determine whether the head is offset forward or backward.
[0144] A body deviation determination submodule, used to determine the degree of body deviation of the student according to the coordinate position of the second target key point and the corresponding second depth value, and output a body deviation determination result;
[0145] Preferably, the body displacement determination submodule comprises:
[0146] A left and right shoulder coordinate position and depth value acquisition unit, configured to acquire a left shoulder coordinate position, a right shoulder coordinate position, a left shoulder depth value, and a right shoulder depth value according to the second target key point coordinate position and the second depth value;
[0147] A body left-right deviation determination unit, used to compare the shoulder coordinate positions in adjacent frame learning images according to the left shoulder coordinate position and the right shoulder coordinate position, and determine whether the body is deviated to the left or the right;
[0148] The body forward and backward offset determination unit is used to compare the shoulder depth values in the adjacent frame depth maps according to the left shoulder depth value and the right shoulder depth value to determine whether the body is offset forward or backward.
[0149] The sitting posture analysis result output unit is used to output the real-time sitting posture analysis result according to the head deviation determination result and the body deviation determination result.
[0150] An abnormal sitting posture recognition module, used to recognize abnormal sitting posture according to the real-time sitting posture analysis result and in combination with preset abnormal sitting posture parameters;
[0151] Preferably, the abnormal sitting posture recognition module includes:
[0152] A straight line acquisition submodule, for connecting the left eye key point and the nose key point to obtain a first straight line according to the coordinate position of the first target key point, and connecting the right eye key point and the nose key point as a second straight line if it is determined that the head is offset to the left or to the right;
[0153] An angle acquisition submodule, used to acquire a first angle between the first straight line and the vertical direction and a second angle between the second straight line and the vertical direction;
[0154] A first abnormal sitting posture recognition submodule is used to compare the angle difference between the first angle and the second angle with a preset angle difference threshold, and when the angle difference in a plurality of consecutive frames of images is greater than the angle difference threshold, recognize it as a first abnormal sitting posture;
[0155] The second abnormal sitting posture recognition submodule is used to calculate the absolute value of the eye depth change in the adjacent frame depth map if it is determined that the head is offset forward or backward. When the absolute value of the eye depth value change calculated in multiple consecutive frames of images is greater than a preset depth threshold, it is identified as the second abnormal sitting posture.
[0156] Preferably, the abnormal sitting posture recognition module further includes:
[0157] A coordinate position acquisition submodule, for acquiring a left shoulder coordinate position and a right shoulder coordinate position according to the coordinate position of the second target key point if it is determined that the body is offset to the left or to the right;
[0158] A submodule for calculating the difference in longitudinal distance between the left shoulder and the right shoulder, used for calculating the difference in longitudinal distance between the left shoulder and the right shoulder according to the left shoulder coordinate position and the right shoulder coordinate position;
[0159] A third abnormal sitting posture recognition submodule, configured to recognize a third abnormal sitting posture when the distance difference between the consecutive multiple frames of images is greater than a preset distance threshold;
[0160] The fourth abnormal sitting posture recognition submodule is used to calculate the absolute value of the shoulder depth change in the adjacent frame depth map if it is determined that the body is offset forward or backward. When the absolute value of the shoulder depth value change calculated in multiple consecutive frames of images is greater than a preset depth threshold, it is identified as the fourth abnormal sitting posture.
[0161] The user reminder module is used to send a reminder message corresponding to the sitting posture detection result to the user when an abnormal sitting posture is identified, so as to remind the user to adjust the student's sitting posture.
[0162] Specifically, the learning accompanying device combined with dynamic detection of abnormal sitting posture provided in Example 2 of the present invention is adopted, and the device includes: an image acquisition module, which is used to acquire real-time video data in a student learning accompanying scene, and decompose the real-time video data into multiple frames of real-time images; a learning image extraction module, which is used to input each of the real-time images into a pre-trained target detection model, and extract the student learning image when the learning supplies intersect with the hands based on the detected preset learning supplies position information and hand position information; an information extraction module, which is used to perform feature extraction and monocular depth detection on the student learning image, and output key point information and depth information related to the student learning; a real-time sitting posture analysis module, which is used to analyze the student's real-time sitting posture based on the key point information and depth information, and output the real-time sitting posture analysis result; an abnormal sitting posture recognition module, which is used to identify the abnormal sitting posture based on the real-time sitting posture analysis result and in combination with preset abnormal sitting posture parameters; a user reminder module, which is used to send a reminder message corresponding to the sitting posture detection result to the user when an abnormal sitting posture is identified, and remind the user to adjust the student's sitting posture. This device uses a pre-trained target detection model, which can more accurately identify the learning supplies and hand positions in the student learning scene and then accurately extract the student learning image; the comprehensive use of key point information and monocular depth detection helps to have a more comprehensive understanding of the student's sitting posture, and multi-level information fusion also improves the accuracy of abnormal sitting posture detection; the use of pre-trained target detection models and deep learning technology can give full play to the advantages of computer vision in image analysis. Compared with traditional hardware equipment, this software-based solution usually requires lower costs. At the same time, this device can also provide instant feedback and reminders in real-time video data, so that students and guardians can take timely action, which helps to prevent potential health problems and improve the efficiency of responding to abnormal sitting postures.
[0163] Example 3
[0164] In addition, combined Figure 1 The learning and accompanying method combined with dynamic detection of abnormal sitting posture described in Example 1 of the present invention can be implemented by a learning and accompanying robot. Fig. 9 A schematic diagram of the hardware structure of the learning and accompanying robot provided in Example 3 of the present invention is shown.
[0165] The learning companion robot may include a processor and a memory storing computer program instructions.
[0166] Specifically, the processor may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present invention.
[0167] The memory may include a large capacity memory for data or instructions. By way of example and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory may include a removable or non-removable (or fixed) medium. Where appropriate, the memory may be inside or outside a data processing device. In a particular embodiment, the memory is a non-volatile solid-state memory. In a particular embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.
[0168] The processor reads and executes computer program instructions stored in the memory to implement any one of the learning and nursing methods combined with dynamic detection of abnormal sitting posture in the above embodiments.
[0169] In one example, the learning companion robot may further include a communication interface and a bus. Fig. 9 As shown, the processor, memory, and communication interface are connected via a bus and communicate with each other.
[0170] The communication interface is mainly used to implement communication between the modules, devices, units and / or equipment in the embodiments of the present invention.
[0171] The bus includes hardware, software or both, and the components of the learning accompanying robot are coupled to each other. For example, but not limitation, the bus may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industrial standard architecture (EISA) bus, a front-side bus (FSB), a hypertransport (HT) interconnection, an industrial standard architecture (ISA) bus, an infinite bandwidth interconnection, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus or other suitable buses or a combination of two or more of these. In appropriate cases, the bus may include one or more buses. Although the embodiments of the present invention describe and illustrate specific buses, the present invention considers any suitable bus or interconnection.
[0172] Example 4
[0173] In addition, in combination with the learning and accompanying method combined with abnormal sitting posture dynamic detection in the above embodiment 1, embodiment 4 of the present invention can also provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the learning and accompanying methods combined with abnormal sitting posture dynamic detection in the above embodiments is implemented.
[0174] In summary, the embodiments of the present invention provide a learning and accompanying method, device and robot combined with dynamic detection of abnormal sitting posture.
[0175] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.
[0176] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.
[0177] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.
[0178] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.
Claims
1. A learning and nursing method combined with dynamic detection of abnormal sitting posture, characterized in that: The method comprises: S1: Acquire real-time video data in a student learning and accompanying scenario, and decompose the real-time video data into multiple frames of real-time images; S2: Inputting each of the real-time images into a pre-trained target detection model, and extracting a student learning image when the learning supplies intersect with the hand according to the detected preset learning supplies position information and hand position information; S3: performing feature extraction and monocular depth detection on the student learning image, and outputting key point information and depth information related to the student learning; S4: Analyze the student's real-time sitting posture according to the key point information and the depth information, and output the real-time sitting posture analysis result; S5: identifying abnormal sitting posture according to the real-time sitting posture analysis result and in combination with preset abnormal sitting posture parameters; S6: When an abnormal sitting posture is identified, a reminder message corresponding to the sitting posture detection result is sent to the user to remind the user to adjust the student's sitting posture; The S3 includes: S31: inputting multiple frames of the student learning image into a human key point detection model to identify the key point information, wherein the human key points include at least: left eye, right eye, nose, left shoulder and right shoulder; S32: Performing monocular depth detection on the student learning image to obtain a relative depth map corresponding to the student learning image; S33: filtering the relative depth maps corresponding to the continuous multiple frames of student images, and outputting the depth maps after filtering as the depth information; The S4 includes: S41: Determine the coordinate positions of key points in a plurality of consecutive frames of student learning images according to the key point information; S42: acquiring a depth value corresponding to the key point coordinate position according to the key point coordinate position and the depth information; S43: Determine the coordinate position of the first target key point and the corresponding first depth value, and determine the coordinate position of the second target key point and the corresponding second depth value according to the key point coordinate position and the corresponding depth value, wherein the first target key point includes the left eye, the right eye and the nose, and the second target key point includes the left shoulder and the right shoulder; S44: determining the degree of head deviation of the student according to the coordinate position of the first target key point and the corresponding first depth value, and outputting a head deviation determination result; S45: determining the degree of body deviation of the student according to the coordinate position of the second target key point and the corresponding second depth value, and outputting a body deviation determination result; S46: outputting the real-time sitting posture analysis result according to the head deviation determination result and the body deviation determination result; The S44 includes: S441: acquiring a left eye coordinate position, a right eye coordinate position, a left eye depth value, and a right eye depth value according to the first target key point coordinate position and the first depth value; S442: Compare the left eye coordinate position and the right eye coordinate position in each frame of the learning image to determine whether the head is offset to the left or to the right; S443: taking the left eye depth value and the right eye depth value as eye depth values, comparing the eye depth values in the depth maps of adjacent frames, and determining whether the head is offset forward or backward; The S5 includes: S51: If it is determined that the head is offset to the left or to the right, then according to the coordinate position of the first target key point, the left eye key point and the nose key point are connected to obtain a first straight line, and the right eye key point and the nose key point are connected to obtain a second straight line; S52: Obtain a first angle between the first straight line and the vertical direction and a second angle between the second straight line and the vertical direction; S53: comparing the angle difference between the first angle and the second angle with a preset angle difference threshold, and identifying a first abnormal sitting posture when the angle difference in the continuous multiple frames of images is greater than the angle difference threshold; S54: If it is determined that the head is shifted forward or backward, the absolute value of the eye depth change in the adjacent frame depth map is calculated. When the absolute value of the eye depth value change calculated in multiple consecutive frames is greater than a preset depth threshold, it is identified as a second abnormal sitting posture.
2. The learning and nursing method combined with abnormal sitting posture dynamic detection according to claim 1 is characterized in that: The S45 includes: S451: Acquire a left shoulder coordinate position, a right shoulder coordinate position, a left shoulder depth value, and a right shoulder depth value according to the second target key point coordinate position and the second depth value; S452: comparing the shoulder coordinate positions in the adjacent frame learning images according to the left shoulder coordinate position and the right shoulder coordinate position, and determining whether the body is shifted to the left or to the right; S453: According to the left shoulder depth value and the right shoulder depth value, the shoulder depth values in the adjacent frame depth maps are compared to determine whether the body is offset forward or backward.
3. The learning and accompanying method combined with abnormal sitting posture dynamic detection according to claim 2 is characterized in that: The S5 further includes: S55: If it is determined that the body is shifted to the left or to the right, the left shoulder coordinate position and the right shoulder coordinate position are obtained according to the coordinate position of the second target key point; S56: Calculate the difference between the longitudinal distances of the left shoulder and the right shoulder according to the left shoulder coordinate position and the right shoulder coordinate position; S57: When the distance difference in the continuous multiple frames of images is greater than a preset distance threshold, identifying it as a third abnormal sitting posture; S58: If it is determined that the body is shifted forward or backward, the absolute value of the shoulder depth change in the adjacent frame depth map is calculated. When the absolute value of the shoulder depth value change calculated in multiple consecutive frames is greater than a preset depth threshold, it is identified as the fourth abnormal sitting posture.
4. A learning and accompanying device combined with abnormal sitting posture dynamic detection, characterized in that: The device comprises: An image acquisition module is used to acquire real-time video data in a student learning and accompanying scenario, and decompose the real-time video data into multiple frames of real-time images; A learning image extraction module, used to input each of the real-time images into a pre-trained target detection model, and extract a student learning image when the learning supplies intersect with the hand based on the detected preset learning supplies position information and hand position information; An information extraction module, used to perform feature extraction and monocular depth detection on the student learning image, and output key point information and depth information related to the student learning; A real-time sitting posture analysis module is used to analyze the real-time sitting posture of the student based on the key point information and depth information, and output the real-time sitting posture analysis result; An abnormal sitting posture recognition module, used to recognize abnormal sitting posture according to the real-time sitting posture analysis result and in combination with preset abnormal sitting posture parameters; A user reminder module is used to send a reminder message corresponding to the sitting posture detection result to the user when an abnormal sitting posture is identified, so as to remind the user to adjust the student's sitting posture; The performing feature extraction and monocular depth detection on the student learning image and outputting key point information and depth information related to the student learning comprises: Inputting multiple frames of the student learning images into a human key point detection model to identify the key point information, wherein the human key points include at least: left eye, right eye, nose, left shoulder and right shoulder; Performing monocular depth detection on the student learning image to obtain a relative depth map corresponding to the student learning image; Performing filtering on relative depth maps corresponding to a plurality of consecutive frames of student images, and outputting the depth maps after filtering as the depth information; The analyzing the student's real-time sitting posture based on the key point information and the depth information and outputting the real-time sitting posture analysis result includes: Determine the coordinate positions of key points in a plurality of consecutive frames of student learning images based on the key point information; According to the key point coordinate position and the depth information, obtaining a depth value corresponding to the key point coordinate position; According to the key point coordinate positions and the corresponding depth values, determine the first target key point coordinate position and the corresponding first depth value, and determine the second target key point coordinate position and the corresponding second depth value, wherein the first target key point includes the left eye, the right eye and the nose, and the second target key point includes the left shoulder and the right shoulder; Determine the degree of head deviation of the student according to the coordinate position of the first target key point and the corresponding first depth value, and output the head deviation determination result; Determine the degree of body deviation of the student according to the coordinate position of the second target key point and the corresponding second depth value, and output the body deviation determination result; Outputting the real-time sitting posture analysis result according to the head deviation determination result and the body deviation determination result; The step of determining the degree of head deviation of the student based on the coordinate position of the first target key point and the corresponding first depth value, and outputting the head deviation determination result includes: According to the first target key point coordinate position and the first depth value, obtain a left eye coordinate position, a right eye coordinate position, a left eye depth value, and a right eye depth value; Comparing the left eye coordinate position and the right eye coordinate position in each frame of the learning image to determine whether the head is offset to the left or to the right; The left eye depth value and the right eye depth value are used as eye depth values, and the eye depth values in the depth maps of adjacent frames are compared to determine whether the head is offset forward or backward; The identifying of abnormal sitting posture according to the real-time sitting posture analysis result and in combination with preset abnormal sitting posture parameters includes: If it is determined that the head is offset to the left or to the right, the left eye key point and the nose key point are connected to obtain a first straight line according to the coordinate position of the first target key point, and the right eye key point and the nose key point are connected to obtain a second straight line; Obtaining a first angle between the first straight line and the vertical direction and a second angle between the second straight line and the vertical direction; Comparing the angle difference between the first angle and the second angle with a preset angle difference threshold, and identifying a first abnormal sitting posture when the angle difference in a plurality of consecutive frames of images is greater than the angle difference threshold; If it is determined that the head is offset forward or backward, the absolute value of the eye depth change in the depth map of adjacent frames is calculated. When the absolute value of the eye depth value change calculated in multiple consecutive frames of images is greater than the preset depth threshold, it is identified as the second abnormal sitting posture.
5. A learning and nursing robot, characterized in that: include: At least one processor, at least one memory and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, implement the method according to any one of claims 1 to 3.
6. A storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 3 is implemented.
