Human body side-lying posture recognition method and device and storage medium

Abstract

The invention discloses a human body side-lying posture recognition method and device and a storage medium. The method comprises the steps that a depth camera is fixed to an opposite side-lying position human body for shooting; comprises: adjusting a shooting state of a depth camera; adjusting the position height of the depth camera; simultaneously shooting to obtain a standard color image and a registration depth image; the two-dimensional coordinates of the two-dimensional human body key points and the new key points are obtained; comprising the following steps: identifying a standard color image through an attitude estimation model to obtain two-dimensional body key point coordinates; comparing the abscissa values of the key points of the right wrist and the right elbow to determine the orientation of the side-lying human body; constructing new key points according to the identified key points, and calculating two-dimensional coordinates of the new key points; the three-dimensional coordinates of the key points are calculated to obtain a human skeleton; the three-dimensional coordinates of the key point and the new key point are obtained through the depth value of the key point measured in the depth map and the known two-dimensional coordinates. According to the invention, the influence of shielding on depth identification is avoided.

Description

Technical Field

[0001] The present invention belongs to the field of visual recognition. More specifically, it relates to a method, device, and storage medium for recognizing the lateral lying posture of the human body. Background Art

[0002] Laparoscopic surgery has become the main means for the diagnosis and treatment of certain diseases at present. With the development of new technologies and new operations, the introduction of high-precision and advanced equipment, and the implementation of precision medicine and intelligent surgery, the placement of the surgical position also faces challenges of individuation, precision, and intelligence. The human body posture recognition technology can identify the positions of different body parts in the form of key points and show the human body characteristics in the form of a skeleton by connecting them, which provides convenience for human body posture analysis. Applying the human body posture recognition technology to the clinical recognition of the lateral lying posture can quickly and non-contact obtain the patient's posture and has practical value in intelligent surgery.

[0003] The existing human body posture recognition technology can relatively accurately identify the two-dimensional key point positions of the human body according to the captured color image or depth image, but still faces challenges including three-dimensional point prediction and handling occlusions. The existing research results have a low accuracy rate for the recognition of the lateral lying posture of the human body, and the joint angles calculated according to the recognition results have a large gap with the actual measured angles. Therefore, it is not suitable for the posture analysis of clinical data. At the same time, no research results have explained the image acquisition methods and means required for clinical lateral lying posture recognition. Summary of the Invention

[0004] In view of the above defects or improvement requirements of the existing technology, the present invention provides a method, device, and storage medium for recognizing the lateral lying posture of the human body. By using a depth camera to recognize the posture of the human body in the lateral lying position, it avoids directly measuring each human body angle before the operation, improves the efficiency of measuring the angle, and realizes the analysis of the lateral lying human body posture.

[0005] To achieve the above object, according to the first aspect of the present invention, a method for recognizing the lateral lying posture of the human body is provided, including:

[0006] Step 1, facing the human body in the lateral lying position, fix the depth camera for shooting; including: adjusting the shooting state of the depth camera; adjusting the position height of the depth camera; simultaneously shooting to obtain a standard color image, registering the depth image, and combining it with IMU data;

[0007] Step 2, obtaining the two-dimensional coordinates of the two-dimensional human key points and the new key points; including: identifying the standard color image through a pose estimation model to obtain the two-dimensional coordinates of the two-dimensional human key points; comparing the abscissa values of the right wrist and right elbow key points to determine the orientation of the lateral lying human body; constructing new key points according to the identified key points and calculating their two-dimensional coordinates;

[0008] Step 3: Calculate the three-dimensional coordinates of the key points and construct a three-dimensional human skeleton, including obtaining the three-dimensional coordinates of the key points and new key points from the depth values of the key points measured in the depth map and the known two-dimensional coordinates, and connecting the three-dimensional coordinates with straight line segments according to the human body structure to obtain a three-dimensional human skeleton.

[0009] Step 4: Calculate the joint angles according to the three-dimensional human skeleton using mathematical geometric relationships.

[0010] Step 5: Compensate the hip joint and knee joint to obtain the final angles.

[0011] Further, in the shooting state of the depth camera in Step 1, it includes aligning the depth camera lens with the sacral region so that the lens is on the horizontal plane of the human body.

[0012] Further, the adjustment of the position and height of the depth camera in Step 1 includes:

[0013] When the human body is lying with the lower leg straight and the upper leg bent, keep the vertical height range between the depth camera and the bed surface at 50 - 85 cm. At this time, it must be ensured that the front end of the lower leg knee and the upper shoulder are exposed. If it can be ensured that the patella region of the leg knee is not blocked by the upper leg, the rising range is 50 - 125 cm.

[0014] When the human body is lying with the lower leg bent and the upper leg straight, make the medial longitudinal arch of the foot appear in the field of view of the downward-facing depth camera, the vertical height of the lens from the bed surface is greater than 80 cm, and ensure that the upper arm does not block the lower arm.

[0015] When the human body is lying with both legs bent or straight, the shooting position of the depth camera is the same as when the lower leg is straight and the upper leg is bent.

[0016] Further, when there are other people in the captured image in Step 1, add a black mask to the background outside the human body candidate box, and then input the image with the mask into the pose estimation model (taking the open-source model MMPose as an example) to identify the human key points.

[0017] Further, the method for determining the lying orientation of the human body by comparing the abscissa values of the right wrist and right elbow key points in Step 2 is as follows: It is required that the two arms of the human body are straight, do not block each other, and are placed on the side where the body flips. Taking the left side of each picture as the starting point and establishing a horizontal x-axis to the right, take out the two-dimensional key point coordinates representing the right wrist and right elbow of the human body from the two-dimensional key point coordinates of the human body recognized by the pose estimation model. If the abscissa value of the right wrist is greater than that of the right elbow, it is considered that the human body is lying on the left side; if the abscissa value of the right wrist is less than that of the right elbow, it is considered that the human body is lying on the right side.

[0018] Further, in Step 2, constructing new key points according to the recognized key points and calculating their two-dimensional coordinates includes:

[0019] Construct new key points at the lower calf in two dimensions: Move the abscissa of the identified lower ankle key point β pixels in the direction of the human body's side lying position, where 10 > β > 3, that is, add β to the abscissa value when lying on the left side and subtract β from the abscissa value when lying on the right side. If the lower leg of the human body is straight, construct a new lower calf key point at the midpoint of the line connecting the lower knee key point and the moved lower ankle key point; if the lower leg is bent, construct a new two-dimensional lower calf key point at the 3 / 10 position on the line connecting the lower knee key point and the moved lower ankle key point.

[0020] Construct a new key point for the two-dimensional lower knee when lying on the side: Assume that the coordinates of the lower knee key point obtained through image recognition are (x, y);

[0021] When the lower leg is straight:

[0022] When lying on the left side, find the depth values at the coordinates (x - 1, y), (x, y),

[0023] (x + 1, y), (x + 2, y), (x + 3, y) in the registered depth map, and select the coordinate (x’, y’) corresponding to the minimum depth value among them;

[0024] When lying on the right side, find the depth values at the coordinates (x - 3, y), (x - 2, y), (x - 1, y), (x, y), (x + 1, y) in the registered depth map, and select the coordinate (x’, y’) corresponding to the minimum depth value among them;

[0025] When the lower leg is bent:

[0026] Set the compensation rate in the direction of the line connecting the lower knee and lower hip key points to 0.04, that is, the direction is from the two-dimensional lower knee key point to the two-dimensional lower hip key point, and the range starts from the two-dimensional lower knee key point, 0.04 * the length of the line segment between the two key points. Let the coordinates of the lower knee key point vary within the range of 0.01 - 0.04 in this direction, with a compensation rate of 0.01 each time, and the coordinates after each change are expressed as (x”, y”);

[0027] When lying on the left side, each time the compensation rate changes, find the depth values at the coordinates (x” - 1, y”), (x”, y”), (x” + 1, y”), (x”

[0028] + 2, y”), (x” + 3, y”) in the registered depth map; finally, select the coordinate (x’, y’) corresponding to the minimum depth value among all the search results at different compensation rates;

[0029] When lying on the right side, each time the compensation rate changes, find the depth values at the coordinates (x” - 3, y”), (x” - 2, y”), (x” - 1, y”),

[0030] The depth values at (x”, y”) and (x” + 1, y”); finally, select the coordinates (x’, y’) corresponding to the minimum depth value from all the search results with different compensation rates.

[0031] Construct a new two-dimensional lower knee key point at the coordinates (x’, y’) obtained in the above steps.

[0032] Construct a new two-dimensional shoulder key point located above when lying on the side: Extend the line connecting the two-dimensional upper elbow key point and the two-dimensional upper shoulder key point to 10% of the length of this line segment to construct a new key point.

[0033] Construct a new two-dimensional shoulder key point located below when lying on the side: Take the point closer to the two-dimensional lower shoulder key point at 3 / 10 of the length of the line segment connecting the two-dimensional lower elbow key point and the two-dimensional lower shoulder key point as the new two-dimensional lower shoulder key point.

[0034] Further, the specific method of step 3 is as follows:

[0035] Find the depth value z corresponding to this coordinate on the depth map according to the two-dimensional coordinates of the key point and the new key point, and combine to obtain the three-dimensional coordinates (x, y, z) of the key point and the new key point.

[0036] Connect the original hip key point, the new knee key point, and the lower leg key point on the same side as it, that is, when lying on the left side, connect the left hip key point, the new lower knee key point, and the new lower leg key point in sequence with a straight line segment, or when lying on the right side, connect the right hip key point, the new lower knee key point, and the new lower leg key point in sequence.

[0037] Connect the new upper shoulder key point with the elbow key point and the hip key point on the same side, that is, when lying on the left side, connect the right elbow key point, the new upper shoulder key point, and the right hip key point in sequence with a straight line segment, or when lying on the right side, connect the left elbow key point, the new upper shoulder key point, and the left hip key point in sequence.

[0038] Connect the lower shoulder key point with the elbow joint on the same side, that is, when lying on the left side, connect the left elbow key point and the new lower shoulder key point in sequence with a straight line, or when lying on the right side, connect the right elbow key point and the new lower shoulder key point in sequence.

[0039] Further, the angles finally identified and calculated in step 4 include:

[0040] Lying on the left side: lower shoulder joint ∠7 - 5 - 11, lower elbow joint ∠20 - 7 - 9, lower hip joint ∠5 - 11 - 17, lower knee joint ∠11 - 17 - 18, upper shoulder joint ∠8 - 19 - 12, upper elbow joint ∠6 - 8 - 10, upper hip joint ∠6 - 12 - 14, upper knee joint ∠12 - 14 - 16;

[0041] Right lateral lying: Lower axis joint ∠20 - 8 - 10, lower shoulder joint ∠8 - 6 - 12, lower hip joint ∠6 - 12 - 17, lower knee joint ∠12 - 17 - 18, upper shoulder joint ∠7 - 19 - 11, upper axis joint ∠5 - 7 - 9, upper hip joint ∠5 - 11 - 13, upper knee joint ∠11 - 13 - 15.

[0042] Further, in step 5, the hip joint and the knee joint are compensated to obtain the final angle, which is obtained from the formula θ ′ = θ * k, where θ ′ is the transformed angle, θ is the calculated angle, and k is the compensation coefficient determined according to the difference between the recognition result and the actual angle;

[0043] When calculating the angle of the hip joint on the upper side during lateral lying, k = 0.926

[0044] When calculating the angle of the knee joint on the upper side, k = 0.943

[0045] When calculating the angle of the knee joint on the lower side, k = 1.05

[0046] When the lower leg is bent:

[0047] When calculating the angle of the upper hip joint, k = 0.943

[0048] When calculating the angle of the lower hip joint, k = 0.9

[0049] When calculating the angle of the lower knee joint, k = 0.943.

[0050] According to the second aspect of the present invention, a human body lateral lying posture recognition device is provided, including:

[0051] A shooting unit; used to adjust the shooting state of the depth camera, adjust the position of the depth camera, and simultaneously shoot to obtain a standard color map, register the depth map and IMU data;

[0052] A coordinate acquisition unit; used to identify the two-dimensional human body key point coordinates by the posture estimation model from the standard color map; compare the abscissa values of the right wrist and right elbow key points to determine the orientation of the lateral lying human body; construct new key points according to the identified key points and calculate their two-dimensional coordinates;

[0053] A three-dimensional human body skeleton construction unit: used to obtain the three-dimensional coordinates of the key points and new key points from the depth values of the key points measured in the depth map and the known two-dimensional coordinates; connect the three-dimensional coordinates with straight line segments according to the human body structure to obtain a three-dimensional human body skeleton;

[0054] A joint angle calculation unit: calculates the joint angle according to the three-dimensional human body skeleton using mathematical geometric relationships;

[0055] A compensation calculation unit: used to compensate the hip joint and the knee joint to obtain the final angle.

[0056] According to the third aspect of the present invention, there is provided a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to cause a computer to execute the above-mentioned method for photographing and recognizing the lateral lying posture of a human body.

[0057] Generally speaking, compared with the prior art by the above technical solutions conceived by the present invention, the following beneficial effects can be achieved:

[0058] 1. The method for recognizing the lateral lying posture of a human body according to the present invention uses a depth camera to recognize the posture of a human body in the lateral lying position, avoiding directly measuring various human body angles before surgery, improving the efficiency of measuring angles, and realizing the analysis of the lateral lying posture of a human body.

[0059] 2. The method for recognizing the lateral lying posture of a human body according to the present invention introduces new human key points and compensates the calculated angles, and can still accurately recognize the human joint angles even when there is occlusion in the lateral lying position, improving the accuracy of recognizing the human joint angles in the lateral lying position.

[0060] 3. The method for recognizing the lateral lying posture of a human body according to the present invention only uses a depth camera that can be held and photographed to collect human body data in a narrow operating room environment, making it possible to collect clinical data.

[0061] 4. The method for recognizing the lateral lying posture of a human body according to the present invention removes the influence of other people in the field of view on the recognition of the key points of the lateral lying human body through a human candidate box, expanding the application scenarios of the lateral lying posture recognition.

[0062] 5. The method for recognizing the lateral lying posture of a human body according to the present invention determines the position of the depth camera for photographing the human body in the lateral lying position, avoiding the influence of occlusion on depth recognition, opening up a technical route for using a single depth camera to recognize the lateral lying posture of a clinical human body, and having the potential to be extended to the recognition of the lateral lying posture of a human body in life applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is the flowchart of the recognition of the lateral lying posture of a human body in an embodiment of the present invention;

[0064] Figure 2 is the schematic diagram of the process for obtaining the two-dimensional coordinates of the human key points in an embodiment of the present invention;

[0065] Figure 3 is the schematic diagram of the right lateral lying skeleton in an embodiment of the present invention;

[0066] Figure 4 is the schematic diagram of the left lateral lying skeleton in an embodiment of the present invention;

[0067] Figure 5Schematic structural diagram of a human body lateral lying posture recognition device proposed for an embodiment of the present invention.

[0068] In all the drawings, the same reference numerals represent the same technical features, specifically: 5 - three-dimensional left shoulder key point, 6 - three-dimensional right shoulder key point, 7 - three-dimensional left elbow key point, 8 - three-dimensional right elbow key point, 9 - three-dimensional left wrist key point, 10 - three-dimensional right wrist key point, 11 - three-dimensional left hip key point, 12 - three-dimensional right hip key point, 13 - three-dimensional left knee key point, 14 - three-dimensional right knee key point, 15 - three-dimensional left ankle key point, 16 - three-dimensional right ankle key point, 17 - three-dimensional lower knee new key point (lower: located below when lying on the side), 18 - three-dimensional lower calf new key point, 19 - three-dimensional upper shoulder new key point (upper: located above when lying on the side), 20 - three-dimensional lower shoulder new key point, 21 - human body lateral lying posture recognition device, 22 - shooting unit; 23 - coordinate acquisition unit, 24 - three-dimensional human skeleton construction unit, 25 - joint angle calculation unit, 26 - compensation calculation unit. Specific implementation manner

[0069] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0070] Embodiment 1

[0071] Please refer to Figure 1 , a method for recognizing the lateral lying posture of a human body, including:

[0072] Step 1, shooting a human body in the contralateral lateral lying position with a fixed depth camera; including: adjusting the shooting state of the depth camera; adjusting the position height of the depth camera; simultaneously shooting to obtain a standard color image, registering the depth image and IMU data;

[0073] Specifically in Step 1, it further includes making the human body to be recognized lie flat on the bed, and the human body posture is the lateral lying position required during the operation.

[0074] The shooting state of the depth camera in Step 1 includes aligning the depth camera lens with the sacral region so that the lens is on the horizontal plane of the human body.

[0075] Adjusting the position height of the depth camera in Step 1 includes:

[0076] When the human body is lying with the lower leg straight and the upper leg bent, keep the vertical height range between the depth camera and the bed surface at 50 - 85 cm. At this time, it is necessary to ensure that the front end of the knee of the lower leg and the upper shoulder are exposed. If it can be ensured that the patella area of the knee of the leg is not blocked by the upper leg, the rising range is 50 - 125 cm;

[0077] When the human body is lying with the lower leg bent and the upper leg straight, make the medial longitudinal arch of the foot appear in the field of view of the downward-facing depth camera. The vertical height of the lens from the bed surface is greater than 80 cm, and ensure that the upper arm does not block the lower arm;

[0078] When the human body is lying with both legs bent or straight, the shooting position of the depth camera is the same as when the lower leg is straight and the upper leg is bent. Use a tripod / handheld to fix the shooting position of the depth camera.

[0079] After the human body stabilizes, take and save the color image, depth image, and IMU data without external objects blocking the front of the human body.

[0080] Please refer to Figure 2 , in step 1, when there are other people in the captured image, add a black mask to the background outside the human body candidate box, and then input the image with the mask into the pose estimation model (taking the open-source model MMPose as an example) to identify the human body key points.

[0081] Step 2, obtain the two-dimensional coordinates of the human body key points and the new key points; including: obtaining the two-dimensional human body key point coordinates by identifying the standard color image through the pose estimation model; comparing the abscissa values of the right wrist and right elbow key points to determine the orientation of the side-lying human body; constructing new key points based on the identified key points and calculating their two-dimensional coordinates;

[0082] Specifically, in step 2, compare the abscissa values of the right wrist and right elbow key points to determine the orientation of the side-lying human body; require the two arms of the human body to be straight, not blocking each other, and placed on the side where the body flips; establish a horizontal coordinate axis with the left side of each picture as the starting point and the direction horizontally to the right. Take out the two-dimensional key point coordinates representing the right wrist and right elbow of the human body from the two-dimensional key point coordinates of the human body identified by the pose estimation model. If the abscissa value of the right wrist is greater than that of the right elbow, it is considered that the human body is lying on the left side; if the abscissa value of the right wrist is less than that of the right elbow, it is considered that the human body is lying on the right side.

[0083] In step 2, construct new key points based on the identified key points and calculate their two-dimensional coordinates; including:

[0084] Construct new key points at the lower calf in two dimensions: Move the abscissa of the identified lower ankle key point β pixels in the direction of the human body's side lying position, where 10 > β > 3, that is, add β to the abscissa value when lying on the left side and subtract β from the abscissa value when lying on the right side. If the lower leg of the human body is straight, construct a new lower calf key point at the midpoint of the line connecting the lower knee key point and the moved lower ankle key point; if the lower leg is bent, construct a new two-dimensional lower calf key point at the 3 / 10 position on the line connecting the lower knee key point and the moved lower ankle key point;

[0085] Construct new key points of the two-dimensional lower knee when lying on the side: Assume that the coordinates of the lower knee key point obtained through image recognition are (x, y);

[0086] When the lower leg is straight:

[0087] When lying on the left side, find the depth values at the coordinates (x - 1, y), (x, y),

[0088] (x + 1, y), (x + 2, y), (x + 3, y) in the registered depth map, and select the coordinates (x', y') corresponding to the minimum depth value among them;

[0089] When lying on the right side, find the depth values at the coordinates (x - 3, y), (x - 2, y), (x - 1, y), (x, y), (x + 1, y) in the registered depth map, and select the coordinates (x', y') corresponding to the minimum depth value among them;

[0090] When the lower leg is bent:

[0091] Set the compensation rate in the direction of the line connecting the lower knee and lower hip key points to 0.04, that is, the direction is from the two-dimensional lower knee key point to the two-dimensional lower hip key point, and the range starts from the two-dimensional lower knee key point, 0.04 * the length of the line segment between the two key points. Let the coordinates of the lower knee key point vary within the range of 0.01 - 0.04 in this direction, with a compensation rate of 0.01 each time, and the coordinates after each change are represented as (x'', y'');

[0092] When lying on the left side, each time the compensation rate changes, find the depth values at the coordinates (x'' - 1, y''), (x'', y''), (x'' + 1, y''), (x''

[0093] + 2, y''), (x'' + 3, y'') in the registered depth map; finally, select the coordinates (x', y') corresponding to the minimum depth value among all the search results at different compensation rates;

[0094] When lying on the right side, each time the compensation rate changes, find the depth values at the coordinates (x'' - 3, y''), (x'' - 2, y''), (x'' - 1, y''),

[0095] The depth values at (x”, y”) and (x” + 1, y”); finally, select the coordinates (x’, y’) corresponding to the minimum depth value from all the search results with different compensation rates.

[0096] Construct a new two-dimensional lower knee key point at the coordinates (x’, y’) obtained in the above steps.

[0097] Construct a new two-dimensional shoulder key point located above when lying on the side: Extend the line connecting the two-dimensional upper elbow key point and the two-dimensional upper shoulder key point to 10% of the length of this line segment to construct a new key point.

[0098] Construct a new two-dimensional shoulder key point located below when lying on the side: Take the point closer to the two-dimensional lower shoulder key point at 3 / 10 of the length of the line segment connecting the two-dimensional lower elbow key point and the two-dimensional lower shoulder key point as the new two-dimensional lower shoulder key point.

[0099] In some preferred embodiments: Conduct targeted training and optimization on the pose estimation model (such as MMPose). Collect more human side-lying image data under different body types, ages, and poses for training to increase the model's recognition ability for special cases and details. For example, improve the recognition accuracy for obese people or cases where limbs are occluded. You can also try to introduce more advanced neural network architectures, such as networks based on the attention mechanism, to make the model more focused on key areas when recognizing key points and reduce interference from background and irrelevant information.

[0100] Step 3, calculate the three-dimensional coordinates of the key points and construct a three-dimensional human skeleton; including obtaining the three-dimensional coordinates of the key points and new key points from the depth values of the key points measured in the depth map and the known two-dimensional coordinates; connecting the three-dimensional coordinates with straight line segments according to the human body structure to obtain a three-dimensional human skeleton.

[0101] Please refer to Figure 3 , Figure 4 The specific method of the above step 3 is: Find the corresponding depth value z of the coordinate on the depth map according to the two-dimensional coordinates of the key points and new key points, and combine them to obtain the three-dimensional coordinates (x, y, z) of the key points and new key points.

[0102] Connect the original hip key point on the same side as it, the new knee key point 17, and the lower leg key point 18, that is, when lying on the left side, connect the left hip key point 11, the new lower knee key point 17, and the new lower leg key point 18 in sequence with a straight line segment, or when lying on the right side, connect the right hip key point 12, the new lower knee key point 17, and the new lower leg key point 18 in sequence.

[0103] Connect the new upper shoulder key point with the elbow key point and hip key point on the same side, that is, when lying on the left side, connect the right elbow key point 8, the new upper shoulder key point 19, and the right hip key point 12 in sequence with a straight line segment, or when lying on the right side, connect the left elbow key point 7, the new upper shoulder key point 19, and the left hip key point 11 in sequence.

[0104] The lower shoulder key point is connected to the elbow joint on the same side. That is, when lying on the left side, the left elbow key point 7 and the new lower shoulder key point 20 are connected in sequence by a straight line segment, or when lying on the right side, the right elbow key point 8 and the new lower shoulder key point 20 are connected in sequence.

[0105] Step 4: Calculate the joint angles based on the three-dimensional human skeleton using mathematical geometric relationships.

[0106] Please refer to Figure 3 , Figure 4 , in Step 4, the finally identified and calculated angles include:

[0107] When lying on the left side: the lower shoulder joint ∠7 - 5 - 11, the lower elbow joint ∠20 - 7 - 9, the lower hip joint ∠5 - 11 - 17, the lower knee joint ∠11 - 17 - 18, the upper shoulder joint ∠8 - 19 - 12, the upper elbow joint ∠6 - 8 - 10, the upper hip joint ∠6 - 12 - 14, the upper knee joint ∠12 - 14 - 16;

[0108] When lying on the right side: the lower axis joint ∠20 - 8 - 10, the lower shoulder joint ∠8 - 6 - 12, the lower hip joint ∠6 - 12 - 17, the lower knee joint ∠12 - 17 - 18, the upper shoulder joint ∠7 - 19 - 11, the upper axis joint ∠5 - 7 - 9, the upper hip joint ∠5 - 11 - 13, the upper knee joint ∠11 - 13 - 15.

[0109] The formula for calculating the joint angles based on the three-dimensional coordinates of the key points is as follows:

[0110]

[0111] where θ is the included angle between the QP and QR of the human skeleton, are the vectors formed by the key points Q and P, Q and R. Using this formula, the upper and lower elbow joints, upper and lower shoulder joints, upper and lower hip joints, and upper and lower knee joint angles of the side-lying human body are calculated.

[0112] Step 5: Compensate the hip joint and knee joint to obtain the final angles;

[0113] In Step 5, the hip joint and knee joint are compensated to obtain the final angles, which are obtained from the formula θ ′ = θ * k, where θ ′ is the transformed angle, θ is the calculated angle, and k is the compensation coefficient determined according to the difference between the recognition result and the actual angle;

[0114] When calculating the angle of the upper hip joint during side lying, k = 0.926

[0115] When calculating the angle of the upper knee joint, k = 0.943

[0116] When calculating the angle of the lower knee joint, k = 1.05

[0117] When the lower leg is bent:

[0118] When calculating the upper hip joint angle, k = 0.943

[0119] When calculating the lower hip joint angle, k = 0.9

[0120] When calculating the lower knee joint angle, k = 0.943.

[0121] In some preferred embodiments, when compensating the hip joint and the knee joint to obtain the final angle, the compensation coefficient k can be determined according to the difference between the recognition result and the actual angle. This can be optimized through experiments and machine learning techniques to improve the accuracy of angle prediction.

[0122] Please refer to Figure 5 , an embodiment of the present invention provides a human body lateral lying posture recognition device 21, including: a shooting unit 22; a coordinate acquisition unit 23, a three-dimensional human skeleton unit 24 is constructed, a joint angle calculation unit 25, and a compensation calculation unit 26.

[0123] The shooting unit 22; is used to adjust the shooting state of the depth camera, adjust the position of the depth camera, and simultaneously shoot to obtain a standard color map, register the depth map and IMU data;

[0124] The standard color map is used to input the pose model, identify the two-dimensional coordinates of the key points, and the registered depth map and IMU data are combined with the two-dimensional coordinates of the key points to obtain the three-dimensional coordinates of the key points and the new key points.

[0125] Inertial Measurement Unit (IMU) data. The IMU can measure the acceleration and angular velocity of the human body limbs. By fusing it with visual data, it can still accurately calculate the human body posture change when the depth camera's view is limited or there is a short-term occlusion, improving the stability and robustness of posture recognition.

[0126] The coordinate acquisition unit 23; is used to identify the two-dimensional human body key point coordinates from the standard color map through the pose estimation model; compare the abscissa values of the right wrist and right elbow key points to determine the orientation of the lateral lying human body; construct new key points according to the identified key points and calculate their two-dimensional coordinates;

[0127] The method for determining the orientation of the lateral lying human body is as follows: It is required that the two arms of the human body are straight, not blocked from each other, and placed on the side where the body flips. Taking the left side of each picture as the starting point, a horizontal x-axis is established to the right. Take out the two-dimensional key point coordinates representing the right wrist and right elbow of the human body from the two-dimensional key point coordinates of the human body recognized by the pose estimation model. If the abscissa value of the right wrist is greater than that of the right elbow, it is considered that the human body is lying on the left side; if the abscissa value of the right wrist is less than that of the right elbow, it is considered that the human body is lying on the right side.

[0128] The method for constructing new key points is the same as that described in the invention content.

[0129] Construct a three-dimensional human skeleton unit 24: used to obtain the three-dimensional coordinates of key points and new key points through the depth values of key points measured in the depth map and the known two-dimensional coordinates; use straight line segments to connect the three-dimensional coordinates according to the human body structure to obtain a three-dimensional human skeleton;

[0130] Calculate joint angle unit 25: calculate joint angles according to the three-dimensional human skeleton using mathematical geometric relationships;

[0131] Compensation calculation unit 26: used to compensate the hip joint and knee joint to obtain the final angle.

[0132] In addition, each functional module or unit in various embodiments of the present invention can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0133] If the above functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that makes contributions to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. And the aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs and other various media that can store program codes.

[0134] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for recognizing a side-lying posture of a human body, characterized in that: The method comprises: Step 1, facing the side-lying human body, fix the depth camera to shoot; including: adjusting the shooting state of the depth camera; adjusting the position and height of the depth camera; shooting to obtain a standard color image, and registering the depth image and IMU data at the same time; Step 2, obtaining the two-dimensional coordinates of the two-dimensional human body key points and the new key points; including: obtaining the two-dimensional coordinates of the human body key points by recognizing the standard color map through the posture estimation model; comparing the horizontal coordinate values ​​of the right wrist and right elbow key points to determine the orientation of the side-lying human body; constructing new key points based on the recognized key points and calculating their two-dimensional coordinates; Step 3, calculating the three-dimensional coordinates of the key points and constructing a three-dimensional human skeleton; including obtaining the three-dimensional coordinates of the key points and new key points through the depth values ​​of the key points measured in the depth map and the known two-dimensional coordinates; and obtaining the three-dimensional human skeleton by connecting the three-dimensional coordinates using straight line segments according to the human body structure; Step 4, calculating the joint angles using mathematical geometric relationships based on the three-dimensional human skeleton; Step 5: Compensate the hip and knee joints to obtain the final angles.

2. The method for recognizing a side-lying posture of a human body according to claim 1, characterized in that: Adjusting the depth camera position height in step 1 includes: When the human body lies down with the lower leg straight and the upper leg bent, the vertical height range between the depth camera and the bed surface is maintained at 50-85cm. At this time, the front end of the lower knee and the upper shoulder must be exposed. If the patella area of ​​the knee is not blocked by the upper leg, the rising range is 50-125cm. When the person lies down with the lower leg bent and the upper leg straight, the inner longitudinal arch of the foot appears in the depth camera field of view, the vertical height of the lens from the bed surface is greater than 80cm, and the upper arm does not cover the lower arm; When a person lies down with both legs bent or straightened at the same time, the depth camera's shooting position is the same as when the lower leg is straightened and the upper leg is bent.

3. The method for recognizing a side-lying posture of a human body according to claim 1, characterized in that: In the step 1, when there are other persons in the captured image, a black mask is added to the background outside the human candidate frame, and then the masked image is input into the posture estimation model to identify key points of the human body.

4. The method for recognizing a side-lying posture of a human body according to claim 1, characterized in that: The method for comparing the horizontal coordinate values ​​of the right wrist and right elbow key points in step 2 to determine the orientation of the side-lying human body is as follows: the two arms of the human body are required to be straight, not blocking each other, and placed on the side where the body is turned over, the left side of each picture is used as the starting point, and the horizontal axis is established horizontally to the right, and the two-dimensional coordinates of the key points of the human body obtained by the posture estimation model are taken to represent the coordinates of the right wrist and right elbow of the human body. If the horizontal coordinate value of the right wrist is greater than the horizontal coordinate value of the right elbow, it is considered that the human body is lying on the left side; if the horizontal coordinate value of the right wrist is less than the horizontal coordinate value of the right elbow, it is considered that the human body is lying on the right side.

5. The method for recognizing a side-lying posture of a human body according to claim 1, characterized in that: The method of constructing new key points according to the identified key points in step 2 and calculating their two-dimensional coordinates includes: Construct a new two-dimensional lower leg key point: move the identified lower ankle key point's horizontal coordinate by β pixels in the direction of the human body lying on the side, 10>β>3, that is, the horizontal coordinate value is +β when lying on the left side, and the horizontal coordinate value is -β when lying on the right side. If the human body's lower leg is straight, a new lower leg key point is constructed at the midpoint of the line connecting the lower knee key point and the moved lower ankle key point; if the lower leg is bent, a new two-dimensional lower leg key point is constructed at 3 / 10 of the line connecting the lower knee key point and the moved lower ankle key point; Construct a new two-dimensional key point of the lower knee when lying on the side: Assume that the coordinates of the key point of the lower knee obtained through image recognition are (x, y); If the lower leg is straight: If lying on the left side, the coordinates to be found in the registered depth map are (x-1,y), (x,y), The depth values ​​at (x+1,y), (x+2,y), (x+3,y) are as follows: select the coordinate (x',y') corresponding to the minimum depth value; If the person is lying on the right side, search for the depth values ​​at coordinates (x-3, y), (x-2, y), (x-1, y), (x, y), (x+1, y) in the registered depth map, and select the coordinate (x', y') corresponding to the minimum depth value; If the lower leg is bent: The compensation rate in the direction of the line connecting the lower knee and lower hip key points is set to 0.04, that is, the direction is from the 2D lower knee key point to the 2D lower hip key point, and the range is from the 2D lower knee key point, 0.04*the length of the line connecting the two key points, so that the coordinates of the lower knee key point are within the range of 0.01-0.04 in this direction, and the compensation rate is changed by 0.01 each time, and the coordinates after each change are expressed as (x", y"); If lying on the left side, each time the compensation rate changes, search for the depth value at the coordinates (x”-1, y”), (x”, y”), (x”+1, y”), (x”+2, y”), (x”+3, y”) in the registered depth map; finally, select the coordinate (x', y') corresponding to the minimum depth value from all the search results at different compensation rates; If lying on the right side, each time the compensation rate changes, the coordinates to be found in the depth map after registration are (x”-3, y”), (x”-2, y”), (x”-1, y”), (x”, y”), (x”+1, y”); finally, select the coordinate (x', y') corresponding to the minimum depth value from all search results with different compensation rates; Construct a new two-dimensional lower knee key point at the coordinates (x', y') obtained in the above steps; Construct a new 2D shoulder key point located at the top when lying on the side: extend the line connecting the 2D upper elbow key point and the 2D upper shoulder key point to 10% of the length of the line segment and construct a new key point; Construct a new 2D shoulder key point located at the bottom when lying on the side: take the point closer to the 2D lower shoulder key point at 3 / 10 of the length of the line segment connecting the 2D lower elbow key point and the 2D lower shoulder key point, and construct it as a new 2D lower shoulder key point.

6. The method for recognizing a side-lying posture of a human body according to claim 1, characterized in that: The specific method of step 3 is: According to the two-dimensional coordinates of the key point and the new key point, find the depth value z corresponding to the coordinate on the depth map, and combine them to obtain the three-dimensional coordinates (x, y, z) of the key point and the new key point; Connect the original hip key point and the new knee key point (17) and the calf key point (18) on the same side, that is, when lying on the left side, use a straight line segment to connect the left hip key point (11), the new lower knee key point (17), and the new lower calf key point (18) in sequence, or when lying on the right side, use a straight line segment to connect the right hip key point (12), the new lower knee key point (17), and the new lower calf key point (18) in sequence; The new upper shoulder key point is connected to the elbow key point and hip key point on the same side. That is, when lying on the left side, a straight line is used to connect the right elbow key point (8), the new upper shoulder key point (19), and the right hip key point (12) in sequence; or when lying on the right side, a straight line is used to connect the left elbow key point (7), the new upper shoulder key point (19), and the left hip key point (11) in sequence. The lower shoulder key point is connected to the elbow joint on the same side, that is, when lying on the left side, a straight line segment is used to connect the left elbow key point (7) and the lower shoulder new key point (20) in sequence, or when lying on the right side, the right elbow key point (8) and the lower shoulder new key point (20) are connected in sequence.

7. The method for recognizing a side-lying posture of a human body according to claim 1, characterized in that: The joint angles that need to be calculated in step 4 include: upper and lower elbow joints, upper and lower shoulder joints, upper and lower hip joints, and upper and lower knee joints.

8. The method for recognizing a side-lying posture of a human body according to claim 1, characterized in that: In step 5, the hip joint and the knee joint are compensated to obtain the final angle, which is given by the formula θ ′ =θ*k, where θ ′ is the transformed angle, θ ′ is the calculated angle, k is the compensation coefficient determined according to the difference between the recognition result and the actual angle; When calculating the angle of the hip joint located on the upper side when lying on the side, k = 0.926 In the above knee joint angle calculation, k = 0.943 The following knee angle calculation uses k = 1.05 With the lower leg bent: When calculating the upper hip joint angle, k = 0.943 When calculating the lower hip joint angle, k = 0.9 When calculating the lower knee joint angle, k=0.

943.

9. A device for recognizing a side-lying posture of a human body, characterized in that: The device comprises: Shooting unit: used to adjust the shooting state of the depth camera, adjust the position of the depth camera, simultaneously shoot and obtain standard color images, and align the depth map and IMU data; Coordinate acquisition unit; used to obtain the coordinates of two-dimensional human body key points through the posture estimation model recognition standard color map; compare the horizontal coordinate values ​​of the right wrist and right elbow key points to determine the orientation of the side-lying human body; construct new key points based on the recognized key points and calculate their two-dimensional coordinates; Constructing a 3D human skeleton unit: used to obtain the 3D coordinates of key points and new key points through the depth values ​​of key points measured in the depth map and the known 2D coordinates; using straight line segments to connect the 3D coordinates to obtain a 3D human skeleton according to the human body structure; Joint angle calculation unit: calculates joint angles based on the three-dimensional human skeleton using mathematical geometric relationships; Compensation calculation unit: used to compensate the hip joint and knee joint to obtain the final angle.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the shooting and human side-lying posture recognition method described in any one of claims 1 to 8.

Images