Pose determination method and apparatus

Abstract

The application discloses a pose determination method and device, and relates to the technical field of terminals; the method comprises the following steps: acquiring a first image comprising a target object; when a target part of the target object is missing in the first image, sending an acquisition request for acquiring first pose data collected by a wearable device worn by the target object to the wearable device, wherein the first pose data comprises pose data of the target part; receiving the first pose data sent by the wearable device; and determining a complete pose of the target object in the first image according to the first pose data and the first image, and the usability is relatively strong.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to attitude determination methods and apparatus. Background Technology

[0002] With increasing focus on health, numerous fitness aids have emerged. The development of artificial intelligence has spurred the application of posture recognition technology in fitness movement recognition. Users urgently need fitness aids that can recognize their fitness postures and provide guidance based on those postures. Current posture recognition methods in fitness scenarios first acquire images or videos of the user's movements using a camera. Then, image recognition-based posture recognition algorithms identify the user's posture in the images or videos, comparing it with the correct fitness posture to generate guidance. However, in real-world fitness scenarios, due to the limited field of view of cameras, some body parts (such as arm joints) often extend beyond the camera's field of view. When these body parts are not captured by the camera, image recognition-based posture recognition methods cannot identify the posture of these uncaptured body parts, thus failing to provide accurate guidance.

[0003] Current posture recognition methods are only applicable to scenarios where the camera can capture all parts of the human body, resulting in poor usability. Summary of the Invention

[0004] This application provides an attitude determination method and apparatus, which improves the usability of the attitude determination method.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, a posture determination method is provided, the method comprising: acquiring a first image including a target object; when a target part of the target object is missing in the first image, sending an acquisition request to a wearable device worn by the target object for requesting to acquire first posture data collected by the wearable device, the first posture data including posture data of the target part; receiving the first posture data sent by the wearable device; and determining the complete posture of the target object in the first image based on the first posture data and the first image.

[0007] In the first aspect, when the first image of the target object obtained by the terminal is missing a part of the target object, an acquisition request is sent to the wearable device worn by the target object to obtain the pose data of the target part. Then, the complete pose of the target object is determined based on the first pose data and the first image. Even if the terminal cannot obtain an image that includes the complete target object, the complete pose of the target object can still be determined, which has strong usability.

[0008] In one possible implementation, the method further includes: identifying limb key points of the target object in the first image; if the number of identified limb key points is less than a preset number, determining the target part of the target object missing in the first image, the target part including the missing limb key points from the preset number of limb key points, the preset number of limb key points being used to depict the complete posture of the target object.

[0009] In this implementation, the key points of the limbs of the target object in the first image are identified. If the number of identified key points is less than a preset number, an acquisition request for obtaining the posture data of the target part is sent to the wearable device worn by the target object. This provides a mechanism for issuing acquisition requests, which only issues acquisition requests under preset conditions, thus saving terminal computing power.

[0010] In one possible implementation, determining the complete pose of a target object in the first image based on the first pose data and the first image includes: determining a first pose feature map of the target object based on the first image; determining a second pose feature map based on the first pose data and initial pose data, wherein the initial pose data is used to characterize the complete pose of the target object determined based on the initial position when the target object begins to move, and the second pose feature map is used to characterize the pose of the target part; and fusing the first pose feature map and the second pose feature map to obtain the complete pose of the target object in the first image.

[0011] In this implementation, a first pose feature map including the pose of the target object in the first image and a second pose feature map including the missing target parts in the first image are first determined. Then, the first pose feature map and the second pose feature map are fused to obtain the complete pose of the target object. Even if the terminal cannot obtain an image including the complete target object, the complete pose of the target object can still be determined, which has strong usability.

[0012] In one possible implementation, the first posture data includes a first posture angle acquired by the wearable device when the target object begins to move and a second posture angle acquired by the wearable device when the target part of the target object is missing in the first image; determining the second posture feature map based on the first posture data and the initial posture data includes: determining the rotation angle of the second posture angle relative to the first posture angle; determining the second posture feature map based on the rotation angle and the second posture data, wherein the second posture data is the posture data of the target part in the initial posture data.

[0013] In this implementation, a second pose feature map is determined based on the rotation angle of the target part and the data of the target part in the initial pose data of the target part. This second pose feature map can be used to fuse with the first pose feature map to obtain the complete pose of the target object.

[0014] In one possible implementation, the target part includes key points of the target limb; the second posture data includes the initial coordinates of the key points of the target limb; a second posture feature map is determined based on the rotation angle and the second posture data, wherein the second posture data is the posture data of the target part in the initial posture data, including: determining the target coordinates of the key points of the target limb after the target part has moved by the rotation angle and the initial coordinates; determining the vector of the target part based on the target coordinates; and constructing the second posture feature map based on the vector of the target part.

[0015] In this implementation, a second pose feature map is constructed based on the vector of the target part, and the second pose feature map is fused with the first pose feature map to obtain the complete pose of the target object.

[0016] In one possible implementation, the first image is an image frame in the target video, and the method further includes: acquiring a second image, which is an image frame in the target video located after the first image; when the second image includes the complete target object, sending a stop command to the wearable device, the stop command being used to instruct the cessation of sending the pose data collected by the wearable device; and determining the complete pose of the target object in the second image based on the second image.

[0017] In this implementation, a stop command is sent to the wearable device when the target object is fully visible in the acquired second image, which saves communication resources between the terminal and the wearable device.

[0018] Secondly, this application provides an attitude determination device, which can be a terminal or a chip or system-on-a-chip within a terminal. This attitude determination device can implement the functions performed by the terminal in the first aspect or possible implementations thereof. These functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions. For example, the attitude determination device includes:

[0019] The acquisition module is used to acquire a first image, which includes the target object;

[0020] The transceiver module is used to send an acquisition request to the wearable device worn by the target object when the target part of the target object is missing in the first image. The acquisition request is used to request the acquisition of the first posture data collected by the wearable device. The first posture data includes the posture data of the target part.

[0021] The transceiver module is also used to receive the first posture data sent by the wearable device;

[0022] The determination module is used to determine the complete pose of the target object in the first image based on the first pose data and the first image.

[0023] In one possible implementation, the device further includes a recognition module for recognizing limb key points of the target object in the first image;

[0024] The determination module is also used to determine the target part of the target object missing in the first image when the number of identified limb key points is less than a preset number. The target part includes the missing limb key points among the preset number of limb key points, and the preset number of limb key points are used to depict the complete posture of the target object.

[0025] In one possible implementation, a module is defined specifically for:

[0026] Determine the first pose feature map of the target object based on the first image;

[0027] The second posture feature map is determined based on the first posture data and the initial posture data. The initial posture data is used to characterize the complete posture of the target object determined based on the initial position when the target object starts to move, and the second posture feature map is used to characterize the posture of the target part.

[0028] By fusing the first pose feature map and the second pose feature map, the complete pose of the target object in the first image is obtained.

[0029] In one possible implementation, the first pose data includes a first pose angle acquired by the wearable device when the target object begins to move and a second pose angle acquired by the wearable device when the target part of the target object is missing in the first image.

[0030] The module is specifically used for:

[0031] Determine the rotation angle of the second attitude angle relative to the first attitude angle;

[0032] Based on the rotation angle and the second attitude data, a second attitude feature map is determined, wherein the second attitude data is the attitude data of the target part in the initial attitude data.

[0033] In one possible implementation, the target body part includes key points of the target limb; the second pose data includes the initial coordinates of the key points of the target limb.

[0034] The module is specifically used for:

[0035] Based on the rotation angle and initial coordinates, determine the target coordinates of the key points of the target limb after the target part has been rotated.

[0036] Determine the vector of the target location based on the target coordinates;

[0037] A second pose feature map is constructed based on the vectors of the target region.

[0038] In one possible implementation, the first image is an image frame in the target video, and the acquisition module is also used to acquire a second image, which is an image frame in the target video that is located after the first image;

[0039] The transceiver module is also used to send a stop command to the wearable device when the second image includes the complete target object. The stop command is used to indicate that the posture data collected by the wearable device should be stopped. The determination module is also used to determine the complete posture of the target object in the second image based on the second image.

[0040] Thirdly, this application provides an electronic device including a processor and a transceiver, the processor and transceiver being configured to support the electronic device in performing the method of the first aspect. Furthermore, the electronic device may also include a memory storing computer instructions, which the processor can execute to perform the method of the first aspect.

[0041] Fourthly, this application provides a computer-readable storage medium that stores computer instructions, wherein the method of the first aspect is executed when the computer instructions are run.

[0042] Fifthly, this application provides a computer program product containing instructions that, when run on a computer, enable the computer to perform the method described in the first aspect.

[0043] In a sixth aspect, this application provides a chip including a processor and a transceiver, the processor and transceiver being used to support an attitude determination apparatus in performing the method of the first aspect.

[0044] The beneficial effects described in aspects two through six of this application can be referred to in the analysis of the beneficial effects of aspect one, and will not be repeated here. Attached Figure Description

[0045] Figure 1 A flowchart illustrating an attitude determination method provided in an embodiment of this application;

[0046] Figure 2 A flowchart illustrating another attitude determination method provided in an embodiment of this application;

[0047] Figure 3 A schematic diagram of key limb points provided in an embodiment of this application;

[0048] Figure 4 Another schematic diagram of key limb points provided in this application embodiment;

[0049] Figure 5 A flowchart illustrating another attitude determination method provided in an embodiment of this application;

[0050] Figure 6 A schematic diagram illustrating the conversion of a vector into an attitude feature map, provided as an embodiment of this application;

[0051] Figure 7 A schematic diagram of a target region provided in an embodiment of this application;

[0052] Figure 8 A flowchart illustrating another attitude determination method provided in an embodiment of this application;

[0053] Figure 9 A flowchart illustrating another attitude determination method provided in an embodiment of this application;

[0054] Figure 10 A vector diagram illustrating a target region provided in an embodiment of this application;

[0055] Figure 11 A schematic diagram of a second pose feature map provided in an embodiment of this application;

[0056] Figure 12 A flowchart illustrating another attitude determination method provided in an embodiment of this application;

[0057] Figure 13 This is a schematic diagram of the structure of an attitude determination device provided in an embodiment of this application;

[0058] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0059] Figure 15 This is a schematic diagram of another attitude determination device provided in an embodiment of this application. Detailed Implementation

[0060] The network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0061] One method for posture recognition in a fitness scenario identifies a user's posture by collecting posture angles from wearable devices. However, this method can only determine the posture angles of the parts of the user wearing the wearable device, and cannot measure the posture angles of the user's whole body, thus failing to generate a complete human posture.

[0062] In another posture recognition method for fitness scenarios, the camera first captures images or videos of the user's movements. Then, an image recognition-based posture recognition algorithm identifies the user's movements in the images or videos to obtain the body posture. This posture is then compared with the correct fitness posture to generate guidance. However, in real-world fitness scenarios, due to the limited field of view of cameras, some body parts (such as arm joints) often extend beyond the camera's field of view. When these body parts are not captured by the camera, image recognition-based posture recognition methods cannot identify the posture of these uncaptured body parts, thus failing to provide accurate guidance to the user.

[0063] It is evident that current pose recognition methods have poor usability.

[0064] To address the aforementioned technical problems, this application provides a posture determination method. When a first image of a target object acquired by a terminal lacks a portion of the target object, a request is sent to a wearable device worn by the target object to obtain posture data of the target portion. Then, the complete posture of the target object is determined based on the first posture data and the first image. Even if the terminal cannot acquire an image including the complete target object, the complete posture of the target object can still be determined, resulting in high usability. The method provided by this application embodiment is described below with reference to the accompanying drawings.

[0065] The attitude determination method provided in this application can be executed by an electronic device with data processing capabilities, such as a terminal equipment or a computer. The terminal can be a user equipment (UE), a mobile station (MS), or a mobile terminal (MT). Specifically, the terminal can be a mobile phone, a tablet computer, or a computer with wireless transceiver capabilities. It can also be a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal in industrial control, a wireless terminal in autonomous driving, a wireless terminal in telemedicine, a wireless terminal in a smart grid, a wireless terminal in a smart city, a smart home, or a vehicle-mounted terminal. In this application embodiment, the device used to implement the functions of the terminal device can be the terminal itself, or a device capable of supporting the terminal in implementing that function, such as a chip system. This device can be installed in the terminal or used in conjunction with the terminal.

[0066] The following embodiments of the attitude determination method provided in this application use a terminal as an example to describe the method provided in this application.

[0067] Figure 1 A flowchart illustrating the attitude determination method provided in an embodiment of this application is shown. Figure 1 As shown, the method may include the following steps:

[0068] S110, acquire the first image.

[0069] The first image includes a target object. The target object is an object whose posture needs to be determined; it can be a living organism or device capable of movement. For example, the target object can be a person, an animal, a robot, etc. The first image can be acquired using a video input device such as a camera. This embodiment uses a person as an example to illustrate the solution provided by this embodiment.

[0070] S120, when the target part of the target object is missing in the first image, a request to obtain it is sent to the wearable device worn by the target object.

[0071] Here, "target area" does not refer to a specific part of the target object, but rather to the part of the target object that is missing from the first image. For example, assuming the target object is a person, and the first image does not include the person's left forearm, the target area in this scenario is the left forearm. When the target area of ​​the target object is missing from the first image, it means that the target object's pose in the first image is incomplete, and the complete pose of the target object needs to be completed based on the pose data of the target area. In this case, the terminal sends an acquisition request to the wearable device worn by the target object to request the first pose data of the target area to be collected by the wearable device. For example, when the target area is the left forearm, the acquisition request is sent to the wearable device worn on the left forearm.

[0072] Wearable devices are electronic devices that have the function of collecting the motion posture of an object wearing the wearable device and can communicate with a terminal. For example, a wearable device can be a sports watch.

[0073] S130 receives the first posture data sent by the wearable device.

[0074] Upon receiving an acquisition request from the terminal, the wearable device responds by sending the first posture data of the target area collected by the wearable device to the terminal. Correspondingly, the terminal receives the first posture data of the target area from the wearable device.

[0075] S140, Based on the first pose data and the first image, determine the complete pose of the target object in the first image.

[0076] Once the first pose data and the first image are obtained, the complete pose of the target object can be determined based on the first pose data and the first image.

[0077] In this embodiment, when the first image of the target object obtained by the terminal is missing a part of the target object, an acquisition request is sent to the wearable device worn by the target object to obtain the posture data of the target part. Then, the complete posture of the target object is determined based on the first posture data and the first image. Even if the terminal cannot obtain an image that includes the complete target object, the complete posture of the target object can still be determined, which has strong usability.

[0078] In one embodiment, such as Figure 2 As shown, prior to S120, the method may further include:

[0079] S210, Identify the key points of the limbs of the target object in the first image.

[0080] In identifying limb keypoints, a pre-trained keypoint detection model can be applied. When training the keypoint detection model, multiple images containing the target object (e.g., multiple frames from a motion video containing the target object) and the coordinates of the corresponding limb keypoints can be used as training samples to generate the keypoint detection model. The trained keypoint detection model can then identify limb keypoints and their coordinates in an image.

[0081] The way limb keypoints are set is crucial when training a keypoint detection model. Different settings result in different preset numbers of limb keypoints, which are sufficient to depict the complete pose of the target object. Since the required accuracy of the target object's complete pose varies in different application scenarios, the limb keypoints can also be set differently. In scenarios with lower accuracy requirements for the target object's complete pose, fewer limb keypoints can be set; in scenarios with higher accuracy requirements, more limb keypoints can be set. For example, ... Figure 3 As shown, the following 17 keypoints can be set as limb keypoints: left eye 1, right eye 2, nose 0, left ear 3, right ear 4, left shoulder 5, right shoulder 6, left elbow 7, right elbow 8, left wrist 9, right wrist 10, left hip 11, right hip 12, left knee 13, right knee 14, left ankle 15, and right ankle 16. In another example, as... Figure 4 As shown, the following 18 key points can be set as limb key points: nose 0, neck 1, right shoulder 2, right elbow 3, right wrist 4, left shoulder 5, left elbow 6, left wrist 7, right hip 8, right knee 9, right ankle 10, left hip 11, left knee 12, left ankle 13, right eye 14, left eye 15, right ear 16, left ear 17. The above two methods of setting limb key points are only examples, and the embodiments of this application do not limit the methods of setting limb key points.

[0082] S220, if the number of identified limb key points is less than the preset number, determine the target part of the first image that is missing the target object.

[0083] The target area includes missing limb keypoints from a preset number of limb keypoints. The preset number of limb keypoints are used to depict the complete posture of the target object. When the number of limb keypoints is less than the preset number, it means that the posture of the target object in the first image is incomplete. Therefore, the target area of ​​the target object missing in the first image is determined to be used to determine the complete posture of the target object in the first image.

[0084] The preset number is the number of limb keypoints corresponding to the complete pose of the target object. For example, such as... Figure 3 In the example scenario shown, the preset quantity is 17. However, as... Figure 4 In the example scenario shown, the preset quantity is 18. The first image lacks a target part of the target object, which includes missing limb keypoints from a preset quantity of limb keypoints. This preset quantity of limb keypoints, used to depict the complete pose of the target object, can be determined based on the recognition result of S210. For example, as... Figure 4 In the scene shown, there are 18 complete limb key points. The S210 recognition result for the first image is: nose, neck, right shoulder, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, and left ear, totaling 16 limb key points. The right elbow and right wrist are missing. Therefore, the target area is the right arm, and the target limb key points included in the right arm are the corresponding limb key points of the right elbow and right wrist.

[0085] Correspondingly, when the number of identified limb key points is equal to the preset number, the pose of the target object in the first image is complete. In this case, there is no need to send the above acquisition request, but the complete pose of the target object in the first image can be determined directly based on the first image.

[0086] In this embodiment, the key points of the limbs of the target object in the first image are identified. If the number of identified key points of the limbs is less than a preset number, an acquisition request for requesting to obtain the posture data of the target part is sent to the wearable device worn by the target object. This provides a mechanism for issuing acquisition requests, which only issues acquisition requests under preset conditions, thus saving terminal computing power.

[0087] In one embodiment, such as Figure 5 As shown, S140 (determining the complete pose of the target object in the first image based on the first pose data and the first image) may include:

[0088] S310, Determine the first pose feature map of the target object based on the first image.

[0089] Referring to the explanation in S210, a pre-trained keypoint detection model can be applied to identify the first image. First, the limb keypoints and their coordinates contained in the first image are obtained; assuming N limb keypoints are obtained. The limb keypoints and their coordinates in the first image can be used to determine the first pose feature map of the target object. Specifically: the N limb keypoints are connected into N-1 vectors according to the object structure of the target object, and for each vector, its corresponding vector v is calculated. n If the direction is specified, then the pose of the target object in the first image is represented as V = v1, b2, ..., v N-1 .like Figure 6 As shown, for each vector v n , with v n Centered on the first image, a rectangular region is defined with a height equal to the limb width and a length equal to the distance between the limb's keypoints. All pixels within this rectangular region have the same direction vector. Then, v is used to... n The corresponding X and Y coordinates are used as the values ​​of each pixel within the rectangular area (pixels outside the rectangular area have a value of 0), resulting in the X pose feature map and Y pose feature map. This process is repeated for all vectors, resulting in 2(N-1) pose feature maps (i.e., the first pose feature map), represented as T = ((X1,Y1),(X2,Y2)....(X... N-1 ,Y N-1 The first pose feature map combination contains the pose information of the target object in the first image.

[0090] In one embodiment, a pre-trained keypoint connectivity model can be applied to determine the pose feature map, for example, to determine a first pose feature map of a target object. Specifically, the coordinates of limb keypoints contained in the first image are input into the keypoint connectivity model, and the first pose feature map is output.

[0091] During training, the keypoint connectivity model can be trained using the coordinates of limb keypoints and their corresponding pose feature maps from multiple images containing the target object (e.g., multiple frames in a motion video containing the target object) as training samples. The trained keypoint connectivity model can then generate a corresponding first pose feature map based on the coordinates of the limb keypoints in the image.

[0092] S320, determine the second attitude feature map based on the first attitude data and the initial attitude data.

[0093] The first posture data includes a first posture angle acquired by the wearable device when the target object begins to move, and a second posture angle acquired by the wearable device when the target part of the target object is missing from the first image. The initial posture data is the complete posture determined based on the initial position of the target object when it begins to move, and the second posture feature map is the posture feature map of the target part. The second posture feature map can be determined based on the first posture data and the initial posture data. For example, such as... Figure 7 As shown, assuming the target part is the right forearm, the dashed line represents the position of the right forearm at the start of the movement. The first posture angle is acquired at this position. The solid line representing the position of the right forearm represents the position of the target part when the target object is missing in the first image. The second posture angle is acquired at this position.

[0094] The specific process by which the terminal determines the second attitude feature map based on motion sensing data and initial complete attitude data can be referred to in the description of S410 to S420 below, and will not be repeated here.

[0095] S330, fuses the first pose feature map and the second pose feature map to obtain the complete pose of the target object.

[0096] The first pose feature map includes the pose of the target object in the first image, while the second pose feature map includes the pose feature map of the missing target parts in the first image. By fusing the first pose feature map and the second pose feature map, the complete pose of the target object can be obtained.

[0097] In this embodiment, a first pose feature map including the pose of the target object in the first image and a second pose feature map including the missing target part in the first image are first determined. Then, the first pose feature map and the second pose feature map are fused to obtain the complete pose of the target object. Even if the terminal cannot obtain an image including the complete target object, the complete pose of the target object can be determined, which has strong usability.

[0098] In one embodiment, such as Figure 8 As shown, S320 (determining the second attitude feature map based on the first attitude data and the initial attitude data) may include:

[0099] S410, determine the rotation angle of the second attitude angle relative to the first attitude angle.

[0100] The rotation angle of the second pose angle relative to the first pose angle reflects the pose change of the target part of the target object from the start of movement to the acquisition of the first image. For example, as shown... Figure 7As shown, the target part is the right forearm. The dashed line indicates the position of the right forearm at the start of the movement. The first posture angle is acquired from this position. The solid line at the position of the right forearm indicates the position of the target part when the target object is missing in the first image. The second posture angle is acquired from this position. α represents the rotation angle of the second posture angle relative to the first posture angle.

[0101] S420, determine the second attitude feature map based on the rotation angle and the second attitude data.

[0102] The second pose data is the pose data of the target body part from the initial pose data. The target body part includes key points of the target limb; the second pose data includes the initial coordinates of the key points of the target limb. For example, when the target body part is the left forearm, the key points of the target limb include the left wrist and left elbow. In another example, when the target body part is the entire left arm, the key points of the target limb include the left wrist, left elbow, and left shoulder. The rotation angle can reflect the pose change of the target body part from the start of movement to the acquisition of the first image, while the initial pose data records the initial position of the target object when it starts to move. Based on the rotation angle and the second pose data of the target body part in the initial pose data, the target coordinates of the target body part after the rotation angle can be calculated, and then the second pose feature map can be determined based on the target coordinates.

[0103] The specific process by which the terminal determines the second attitude feature map based on the rotation angle and the second attitude data will be further described in the following descriptions of S510 to S530.

[0104] In this embodiment of the application, a second posture feature map is determined based on the rotation angle of the target part and the data of the target part in the initial posture data of the target part. The second posture feature map can be used to fuse with the first posture feature map to obtain the complete posture of the target object.

[0105] In one embodiment, such as Figure 9 As shown, S420 (determining the second attitude feature map based on the rotation angle and the second attitude data) may include:

[0106] S510, based on the rotation angle and initial coordinates, determine the target coordinates of the key points of the target limb after the target part has been moved and rotated.

[0107] The terminal can determine the target coordinates after moving the initial coordinate reference by a rotation angle.

[0108] S520, determine the vector of the target part based on the target coordinates.

[0109] There may be one or more target limb key points, and correspondingly, there may be one or more target coordinates. Referring to the description of vectors in S310, for each target limb key point, the adjacent limb key points are determined according to the object structure rules of the target object (this limb key point may be another target limb key point or a non-target limb key point, depending on which target limb key points are included in the target part), and a vector is generated based on the target coordinates of the target limb key point and the coordinates of the adjacent limb key points, thus obtaining the vector of the target part.

[0110] For example, such as Figure 10 As shown, the target body part includes target limb keypoints A and B, with coordinates (x3, y3) and (x4, y4) respectively. The limb keypoint adjacent to target limb keypoint A is target limb keypoint B; the limb keypoints adjacent to target limb keypoint B include target limb keypoints A and C, with coordinates (x5, y5) for limb keypoint C. The vectors generated sequentially include: the vector calculated based on (x3, y3) and (x4, y4), the vector calculated based on (x3, y3) and (x4, y4), and the vector calculated based on (x4, y4) and (x5, y5). Multiple vectors obtained from the same calculation basis are not counted repeatedly; one vector calculated based on (x3, y3) and (x4, y4) is removed. Therefore, the vectors of the target body part include the vector calculated based on (x3, y3) and (x4, y4) and the vector calculated based on (x4, y4) and (x5, y5).

[0111] S530, constructs a second pose feature map based on the vector of the target part.

[0112] The second attitude feature map includes a first sub-map and a second sub-map. Refer to the description of the X-attitude feature map and Y-attitude feature map in S310. The first sub-map is an X-attitude feature map constructed based on the vectors of the target part, and the second sub-map is a Y-attitude feature map constructed based on the vectors of the target part. The first sub-map is the region formed by extending the width of the target part's vectors, and the pixel value in the first sub-map is the projection length of the target part's vectors on a preset X-axis; the second sub-map is the region formed by extending the width of the target part's vectors, and the pixel value in the second sub-map is the projection length of the target part's vectors on the Y-axis (perpendicular to the preset X-axis); the second attitude feature map also includes a matrix containing the first sub-map (in which pixels other than those in the first sub-map are 0) and a matrix containing the second sub-map (in which pixels other than those in the second sub-map are 0). For example, as shown... Figure 11 As shown, continue to use Figure 10In the example shown, the pixel values ​​in the first sub-image are x4-x3 and x5-x4, and the pixel values ​​in the second sub-image are y4-y3 and y5-y4.

[0113] In this embodiment of the application, a second pose feature map is constructed based on the vector of the target part, and the second pose feature map is fused with the first pose feature map to obtain the complete pose of the target object.

[0114] In one embodiment, such as Figure 12 As shown, the method may further include:

[0115] S610, acquire the second image.

[0116] In this context, the second image is a frame in the target video that follows the first image. In some motion scenarios (e.g., fitness scenarios), it is necessary to capture multiple motion images of the user in real time. In such scenarios, a target video is obtained by capturing images of the target object. This target video includes multiple image frames, including the first image and the second image.

[0117] S620, when the second image includes the complete target object, sends a stop command to the wearable device.

[0118] Specifically, when the target object is fully represented in the second image, it means that the complete pose of the target object in the second image can be determined based on the second image. At this point, a stop command is sent to the wearable device to instruct it to stop transmitting the pose data collected by the wearable device.

[0119] S630, determine the complete pose of the target object in the second image based on the second image.

[0120] In determining the complete pose of the target object in the second image, the key point detection model and key point connection model introduced in S310 can be applied to determine the complete pose of the target object in the second image. The specific determination process is the same as the principle of determining the first pose feature map in S310. Please refer to the introduction in S310, and it will not be repeated here.

[0121] In this embodiment of the application, when the target object is fully visible in the acquired second image, a stop command is sent to the wearable device, which can save communication resource consumption between the terminal and the wearable device.

[0122] The above mainly describes the solution provided by the embodiments of this application from the perspective of the execution logic of each step. It is understood that, in order to achieve the above functions, the execution entity of the embodiments of this application includes the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should readily recognize that, in conjunction with the algorithm steps of the various examples described in the embodiments disclosed herein, the method of the embodiments of this application can be implemented in hardware, software, or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software-driven hardware manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0123] This application embodiment can divide the execution subject of this application embodiment into functional modules according to the above method examples. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0124] In one embodiment, this embodiment is a fitness posture recognition scenario, in which the target object is a user. For example... Figure 13 As shown, the attitude determination device provided in this application embodiment may include the following modules:

[0125] The initial positioning calibration module 001 is used to guide the user to adjust the initial angle of the terminal to ensure that the terminal is placed at the correct angle. Then, it guides the user to adjust the initial position, records the coordinates of each key point under the user's initial position to generate initial attitude data, as well as the first attitude angle of the wearable device under the initial position.

[0126] The pose recognition module 002 is used to perform the actions described in this embodiment. Figures 1-11 The methods described are used to obtain the user's complete posture.

[0127] The motion assessment module 003 compares the user's complete posture with a standard motion posture to obtain motion assessment results, and can evaluate the standardization of the user's movements.

[0128] The user interaction feedback module 004 provides specific fitness guidance on how to adjust the user's movements based on the motion assessment results, helping the user to adjust their exercise movements.

[0129] In specific implementation, the execution subject of this application embodiment, such as a terminal, may be adopted. Figure 14 The shown composition or includes Figure 14 The components shown. Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. When the electronic device has the functions of the terminal described in the embodiment of this application, the electronic device can be a terminal or a chip or system-on-a-chip in a terminal.

[0130] like Figure 14 As shown, the electronic device may include a processor 801, a communication line 802, a transceiver 803, and a memory 804. The processor 801, memory 804, and transceiver 803 are connected via the communication line 802. In one example, the processor 801 may include one or more CPUs, for example... Figure 14 CPU0 and CPU1 in the CPU.

[0131] As an optional implementation, the electronic device includes multiple processors, for example, besides Figure 14 In addition to processor 801, it may also include processor 807.

[0132] The processor 801 can be a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 801 can also be other devices with processing capabilities, such as circuits, devices, or software modules.

[0133] Communication line 802 is used to transmit information between the components included in an electronic device.

[0134] Transceiver 803 is used to communicate with other devices or other communication networks. These other communication networks can be Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc. Transceiver 803 can be an interface circuit, pins, RF module, transceiver, or any device capable of enabling communication.

[0135] Furthermore, the electronic device may also include a memory 804. The memory 804 is used to store instructions, which may be computer programs.

[0136] The memory 804 can be a read-only memory (ROM) or other type of static storage device that can store static information and / or instructions; it can also be a random access memory (RAM) or other type of dynamic storage device that can store information and / or instructions; it can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage, magnetic disk storage medium or other magnetic storage device; optical disc storage includes compressed optical discs, laser discs, optical discs, digital universal optical discs, or Blu-ray discs, etc.

[0137] It should be noted that the memory 804 can exist independently of the processor 801, or it can be integrated with the processor 801. The memory 804 can be used to store instructions, program code, or some data, etc. The memory 804 can be located inside or outside the electronic device, without limitation. When the processor 801 executes the instructions stored in the memory 804, it can implement the method provided in the embodiments of this application.

[0138] The electronic device also includes an output device 805 and an input device 806. For example, the input device 806 may be a device such as a camera, keyboard, mouse, microphone or joystick, and the output device 805 may be a device such as a display screen or speaker.

[0139] It should be noted that electronic devices can be desktop computers, laptops, network servers, mobile phones, tablets, wireless terminals, embedded devices, chip systems, or other devices. Figure 14 Equipment with a similar structure. Furthermore... Figure 14 The structural composition shown herein does not constitute a limitation on the electronic device, except... Figure 14 In addition to the components shown, the electronic device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0140] In this embodiment of the application, the chip system may be composed of chips or may include chips and other discrete devices.

[0141] Figure 15 A structural diagram of another attitude determination device is shown. Figure 15 Each module in the device shown has the ability to implement Figures 1-11The corresponding steps in the process are implemented to achieve their respective technical effects. The beneficial effects of each module's execution steps can be found in the corresponding step descriptions, which will not be repeated here. The functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. This attitude determination device can be a terminal, a chip within a terminal, or a system-on-a-chip. For example, this attitude determination device includes:

[0142] The acquisition module 141 is used to acquire a first image, which includes the target object.

[0143] The transceiver module 142 is used to send an acquisition request to the wearable device worn by the target object when the target part of the target object is missing in the first image. The acquisition request is used to request the acquisition of the first posture data collected by the wearable device. The first posture data includes the posture data of the target part.

[0144] The transceiver module 142 is also used to receive first posture data sent by the wearable device.

[0145] The determination module 143 is used to determine the complete pose of the target object in the first image based on the first pose data and the first image.

[0146] In one possible implementation, the device also includes a recognition module for recognizing limb key points of the target object in the first image.

[0147] The determination module 143 is further configured to determine the target part of the target object missing in the first image when the number of identified limb key points is less than a preset number. The target part includes the missing limb key points among the preset number of limb key points. The preset number of limb key points are used to depict the complete posture of the target object.

[0148] In one possible implementation, module 143 is specifically used for:

[0149] The first pose feature map of the target object is determined based on the first image.

[0150] A second posture feature map is determined based on the first posture data and the initial posture data. The initial posture data is used to characterize the complete posture of the target object determined based on the initial position when the target object starts to move, and the second posture feature map is used to characterize the posture of the target part.

[0151] By fusing the first pose feature map and the second pose feature map, the complete pose of the target object in the first image is obtained.

[0152] In one possible implementation, the first posture data includes a first posture angle acquired by the wearable device when the target object begins to move and a second posture angle acquired by the wearable device when the target part of the target object is missing in the first image.

[0153] Module 143 is specifically used for:

[0154] Determine the rotation angle of the second attitude angle relative to the first attitude angle.

[0155] Based on the rotation angle and the second attitude data, a second attitude feature map is determined, wherein the second attitude data is the attitude data of the target part in the initial attitude data.

[0156] In one possible implementation, the target body part includes key points of the target limb. The second pose data includes the initial coordinates of the key points of the target limb.

[0157] Module 143 is specifically used for:

[0158] Based on the rotation angle and initial coordinates, determine the target coordinates of the key points of the target limb after the target part has been moved and rotated.

[0159] The vector of the target location is determined based on the target coordinates.

[0160] A second pose feature map is constructed based on the vectors of the target region.

[0161] In one possible implementation, the acquisition module 141 is also used to acquire a second image, which is an image frame in the target video that is located after the first image.

[0162] The transceiver module 142 is also configured to send a stop command to the wearable device when the second image includes the complete target object, wherein the stop command is used to instruct the wearable device to stop sending posture data.

[0163] The determination module 143 is also used to determine the complete pose of the target object in the second image based on the second image.

[0164] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be implemented by a computer program instructing related hardware. This program can be stored in the computer-readable storage medium, and when executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be a terminal device of any of the foregoing embodiments, such as an internal storage unit including a data sending end and / or a data receiving end, such as a hard disk or memory of the terminal device. The computer-readable storage medium can also be an external storage device of the terminal device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device. Further, the computer-readable storage medium can include both the internal storage unit and the external storage device of the terminal device. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0165] This application also provides computer instructions. All or part of the processes in the above method embodiments can be executed by computer instructions to related hardware (such as computers, processors, network devices, and terminals). The program can be stored in the aforementioned computer-readable storage medium.

[0166] This application also provides a chip system. The chip system can be composed of chips or may include chips and other discrete devices, without limitation. The chip system includes a processor and a transceiver. All or part of the processes in the above method embodiments can be completed by this chip system, such as the chip system being used to implement the functions performed by the terminal in the above method embodiments.

[0167] In one possible design, the chip system further includes a memory for storing program instructions and / or data. When the chip system is running, the processor executes the program instructions stored in the memory to enable the chip system to perform the functions performed by the terminal in the above method embodiments.

[0168] In the embodiments of this application, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0169] In the embodiments of this application, the memory can be non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), or it can be volatile memory, such as random-access memory (RAM). Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures, and accessible by a computer, but is not limited thereto. The memory in the embodiments of this application can also be a circuit or any other device capable of implementing storage functions, used to store instructions and / or data.

[0170] It should be noted that the terms "first" and "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0171] It should be understood that in the embodiments of this application, "at least one (item)" refers to one or more, "more than one" refers to two or more, "at least two (items)" refers to two or three or more, and "and / or" is used to describe the association relationship of related objects, indicating that there can be three relationships. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple. It should be understood that in the embodiments of this application, "B corresponding to A" means that B is associated with A. For example, B can be determined based on A. It should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information. Furthermore, the term "connection" in the embodiments of this application refers to various connection methods, such as direct or indirect connections, to achieve communication between devices; the embodiments of this application do not impose any limitations on this.

[0172] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0174] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0175] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device, such as a microcontroller, chip, or processor, to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0176] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for determining attitude, characterized in that, include: Acquire a first image, the first image including the target object; When the target part of the target object is missing in the first image, a request to acquire is sent to the wearable device worn by the target object. The request to acquire the first posture data collected by the wearable device includes the posture data of the target part, the first posture angle collected by the wearable device when the target object starts to move, and the second posture angle collected by the wearable device when the target part of the target object is missing in the first image. Receive the first posture data sent by the wearable device; A first pose feature map of the target object is determined based on the first image; Determine the rotation angle of the second posture angle relative to the first posture angle, and determine a second posture feature map based on the rotation angle and the second posture data. The second posture data is the posture data of the target part in the initial posture data, and the second posture feature map is used to characterize the posture of the target part. The initial posture data is used to characterize the complete posture of the target object determined based on the initial position when the target object starts to move.

2. The attitude determination method according to claim 1, characterized in that, The method further includes: Identify key limb points of the target object in the first image; If the number of identified limb key points is less than a preset number, it is determined that the target part of the target object is missing in the first image. The target part includes the missing limb key points from the preset number of limb key points, which are used to depict the complete posture of the target object.

3. The attitude determination method according to claim 1, characterized in that, The target body part includes key points of the target limb; the second posture data includes the initial coordinates of the key points of the target limb. The second posture feature map is determined based on the rotation angle and the second posture data, wherein the second posture data is the posture data of the target part in the initial posture data, including: Based on the rotation angle and the initial coordinates, determine the target coordinates of the key points of the target limb after the target part has moved by the rotation angle; The vector of the target location is determined based on the target coordinates; The second pose feature map is constructed based on the vector of the target region.

4. The attitude determination method according to any one of claims 1-3, characterized in that, The first image is an image frame from the target video, and the method further includes: Acquire a second image, which is an image frame in the target video that follows the first image; When the second image includes the complete target object, a stop command is sent to the wearable device, the stop command being used to instruct the cessation of sending the posture data collected by the wearable device; and, based on the second image, the complete posture of the target object in the second image is determined.

5. An attitude determination device, characterized in that, include: The acquisition module is used to acquire a first image, wherein the first image includes a target object; The transceiver module is used to send an acquisition request to the wearable device worn by the target object when the target part of the target object is missing in the first image. The acquisition request is used to request the acquisition of the first posture data collected by the wearable device. The first posture data includes the posture data of the target part, the first posture angle collected by the wearable device when the target object starts to move, and the second posture angle collected by the wearable device when the target part of the target object is missing in the first image. The transceiver module is also used to receive the first posture data sent by the wearable device; The determining module is used to determine a first pose feature map of the target object based on the first image; Determine the rotation angle of the second posture angle relative to the first posture angle, and determine a second posture feature map based on the rotation angle and the second posture data. The second posture data is the posture data of the target part in the initial posture data, and the second posture feature map is used to characterize the posture of the target part. The initial posture data is used to characterize the complete posture of the target object determined based on the initial position when the target object starts to move.

6. The attitude determination device according to claim 5, characterized in that, The device further includes a recognition module for recognizing key limb points of the target object in the first image; The determining module is further configured to determine, when the number of identified limb key points is less than a preset number, the target part of the target object missing in the first image, the target part including the missing limb key points among the preset number of limb key points, the preset number of limb key points being used to depict the complete posture of the target object.

7. The attitude determination device according to claim 5, characterized in that, The target body part includes key points of the target limb; the second posture data includes the initial coordinates of the key points of the target limb. The determining module is specifically used for: Based on the rotation angle and the initial coordinates, determine the target coordinates of the key points of the target limb after the target part has moved by the rotation angle; The vector of the target location is determined based on the target coordinates; The second pose feature map is constructed based on the vector of the target region.

8. The attitude determination device according to any one of claims 5-7, characterized in that, The first image is an image frame in the target video, and the acquisition module is further configured to acquire a second image, which is an image frame in the target video that is located after the first image; The transceiver module is further configured to send a stop command to the wearable device when the second image includes the complete target object, the stop command being used to instruct the transmission of the posture data collected by the wearable device to stop; The determining module is further configured to determine the complete pose of the target object in the second image based on the second image.

9. An attitude determination device, characterized in that, The attitude determination device includes a processor and a transceiver, the processor and the transceiver being configured to support the attitude determination device in performing the method as described in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed, perform the method as described in any one of claims 1-4.

Images