Attitude determination method and device

Abstract

The invention discloses a posture determination method and device, and relates to the technical field of terminals. The method comprises the following steps: acquiring a first image comprising a target object; when the target part of the target object is lacked in the first image, an acquisition request used for requesting to acquire first posture data acquired by wearable equipment is sent to the wearable equipment worn by the target object, and the first posture data comprises posture data of the target part; receiving first attitude data sent by the wearable device; according to the first attitude data and the first image, the complete attitude of the target object in the first image is determined, and the usability is high.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a posture determination method and device. Background Art

[0002] As people pay more attention to health, many fitness auxiliary products have emerged one after another. The development of artificial intelligence technology has spawned the application of posture recognition technology in fitness action recognition. Users urgently need a fitness auxiliary product that can recognize the user's fitness posture and can provide guidance based on the recognized fitness posture. In the current posture recognition method in the fitness scene, the user's movement image or video is first obtained through the camera, and then the image or video of the user's movement is recognized by the posture recognition algorithm based on image recognition, and then the human posture in the image or video is obtained, and the obtained human posture is compared with the correct fitness posture to generate guidance. However, in the actual fitness scene, due to the limited shooting range of the camera, some parts of the human body (such as arm joints) often exceed the shooting range of the camera during fitness. When some parts of the human body are not captured by the camera, the posture recognition method based only on image recognition cannot recognize the posture of the uncaptured parts of the human body, so it cannot provide users with correct guidance.

[0003] The current posture recognition method is only applicable to scenarios where the camera can fully capture all parts of the human body, and has poor usability. Summary of the invention

[0004] The present application provides a posture determination method and device, which improves the usability of the posture determination method.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, a posture determination method is provided, the method comprising: acquiring a first image including a target object; when a target part of the target object is missing in the first image, sending a request for obtaining first posture data collected by the wearable device to a wearable device worn by the target object, the first posture data including posture data of the target part; receiving the first posture data sent by the wearable device; and determining the complete posture of the target object in the first image based on the first posture data and the first image.

[0007] In the first aspect, when the first image of the target object acquired by the terminal lacks a part of the target object, an acquisition request is sent to the wearable device worn by the target object to obtain the posture data of the target part, and then the complete posture of the target object is determined based on the first posture data and the first image. Even if the terminal cannot acquire an image including the complete target object, the complete posture of the target object can be determined, and the usability is relatively strong.

[0008] In one possible implementation, the method also includes: identifying limb key points of the target object in the first image; when the number of identified limb key points is less than a preset number, determining that a target part of the target object is missing in the first image, the target part includes limb key points that are missing from a preset number of limb key points, and the preset number of limb key points are used to depict the complete posture of the target object.

[0009] In this implementation, the limb key points of the target object in the first image are identified, and when the number of identified limb key points is less than a preset number, an acquisition request for requesting the posture data of the target part is sent to the wearable device worn by the target object, thereby providing a mechanism for issuing acquisition requests, in which acquisition requests are only issued under preset circumstances, saving terminal computing power.

[0010] In one possible implementation, a complete posture of a target object in a first image is determined based on first posture data and a first image, including: determining a first posture feature map of the target object based on the first image; determining a second posture feature map based on the first posture data and initial posture data, wherein the initial posture data is used to characterize the complete posture of the target object determined based on an initial position when the target object starts to move, and the second posture feature map is used to characterize the posture of the target part; and fusing the first posture feature map and the second posture feature map to obtain the complete posture of the target object in the first image.

[0011] In this implementation, a first posture feature map including the posture of the target object in the first image and a second posture feature map including the target part missing in the first image are first determined, and then the first posture feature map and the second posture feature map are fused to obtain the complete posture of the target object. Even if the terminal cannot obtain an image including the complete target object, the complete posture of the target object can be determined, and the usability is strong.

[0012] In one possible implementation, the first posture data includes a first posture angle collected by the wearable device when the target object starts to move and a second posture angle collected by the wearable device when the target part of the target object is missing in the first image; determining the second posture feature map based on the first posture data and the initial posture data includes: determining the rotation angle of the second posture angle relative to the first posture angle; determining the second posture feature map based on the rotation angle and the second posture data, wherein the second posture data is the posture data of the target part in the initial posture data.

[0013] In this implementation, a second posture feature map is determined according to the rotation angle of the target part and the data of the target part in the initial posture data of the target part. The second posture feature map can be used to merge with the first posture feature map to obtain the complete posture of the target object.

[0014] In one possible implementation, the target part includes the key points of the target limb; the second posture data includes the initial coordinates of the key points of the target limb; according to the rotation angle and the second posture data, a second posture feature map is determined, wherein the second posture data is the posture data of the target part in the initial posture data, including: according to the rotation angle and the initial coordinates, determining the target coordinates of the key points of the target limb of the target part after moving the rotation angle; determining the vector of the target part according to the target coordinates; and constructing a second posture feature map according to the vector of the target part.

[0015] In this implementation, a second posture feature map is constructed according to the vector of the target part, and the second posture feature map is fused with the first posture feature map to obtain the complete posture of the target object.

[0016] In one possible implementation, the first image is an image frame in a target video, and the method further includes: acquiring a second image, which is an image frame in the target video that is located after the first image; when the second image includes a complete target object, sending a stop instruction to the wearable device, the stop instruction is used to instruct to stop sending posture data collected by the wearable device; and determining the complete posture of the target object in the second image based on the second image.

[0017] In this implementation, when the target object in the acquired second image is complete, a stop instruction is sent to the wearable device, which can save the communication resource consumption between the terminal and the wearable device.

[0018] In a second aspect, the present application provides a posture determination device, which can be a terminal or a chip or system on chip in the terminal. The posture determination device can implement the functions performed by the terminal in the above-mentioned first aspect or the possible implementation of the first aspect, and the functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. For example, the posture determination device includes:

[0019] An acquisition module, configured to acquire a first image, wherein the first image includes a target object;

[0020] a transceiver module, configured to send an acquisition request to a wearable device worn by the target object when the target part of the target object is missing in the first image, wherein the acquisition request is used to request acquisition of first posture data collected by the wearable device, wherein the first posture data includes posture data of the target part;

[0021] The transceiver module is further used to receive the first posture data sent by the wearable device;

[0022] The determination module is used to determine the complete posture of the target object in the first image according to the first posture data and the first image.

[0023] In a possible implementation, the apparatus further includes a recognition module, configured to recognize key points of a limb of the target object in the first image;

[0024] The determination module is also used to determine the target part of the target object missing in the first image when the number of identified limb key points is less than a preset number, wherein the target part includes the limb key points missing from a preset number of limb key points, and the preset number of limb key points are used to describe the complete posture of the target object.

[0025] In a possible implementation, a module is determined, specifically for:

[0026] Determine a first posture feature map of the target object according to the first image;

[0027] Determine a second posture feature map according to the first posture data and the initial posture data, wherein the initial posture data is used to characterize a complete posture of the target object determined according to an initial position when the target object starts to move, and the second posture feature map is used to characterize the posture of the target part;

[0028] The first posture feature map and the second posture feature map are fused to obtain a complete posture of the target object in the first image.

[0029] In a possible implementation, the first posture data includes a first posture angle collected by the wearable device when the target object starts to move and a second posture angle collected by the wearable device when the target part of the target object is missing in the first image;

[0030] Identify modules, specifically for:

[0031] determining a rotation angle of the second posture angle relative to the first posture angle;

[0032] A second posture feature map is determined according to the rotation angle and the second posture data, wherein the second posture data is the posture data of the target part in the initial posture data.

[0033] In a possible implementation, the target part includes a target limb key point; the second posture data includes initial coordinates of the target limb key point;

[0034] Identify modules, specifically for:

[0035] According to the rotation angle and the initial coordinates, the target coordinates of the key points of the target limb of the target part after the target part moves the rotation angle are determined;

[0036] Determine the vector of the target part according to the target coordinates;

[0037] A second posture feature map is constructed according to the vector of the target part.

[0038] In a possible implementation, the first image is an image frame in the target video, and the acquisition module is further used to acquire a second image, where the second image is an image frame in the target video that is located after the first image;

[0039] The transceiver module is also used to send a stop instruction to the wearable device when the second image includes a complete target object, and the stop instruction is used to instruct to stop sending the posture data collected by the wearable device; the determination module is also used to determine the complete posture of the target object in the second image based on the second image.

[0040] In a third aspect, the present application provides an electronic device, the electronic device comprising a processor and a transceiver, the processor and the transceiver are used to support the electronic device to execute the method of the first aspect. Further, the electronic device may also include a memory, the memory stores computer instructions, and the processor can execute the computer instructions to execute the method of the first aspect.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed, the method of the first aspect is executed.

[0042] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method of the first aspect.

[0043] In a sixth aspect, the present application provides a chip comprising a processor and a transceiver, wherein the processor and the transceiver are used to support a posture determination device to execute the method of the first aspect.

[0044] Among them, the beneficial effects described in the second to sixth aspects of the present application can refer to the analysis of the beneficial effects of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a posture determination method provided in an embodiment of the present application;

[0046] Figure 2 A flowchart of another posture determination method provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of key points of a limb provided in an embodiment of the present application;

[0048] Figure 4 Another schematic diagram of key points of a limb provided in an embodiment of the present application;

[0049] Figure 5 A flowchart of another posture determination method provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of converting a vector into a posture feature map provided in an embodiment of the present application;

[0051] Figure 7 A schematic diagram of a target part provided in an embodiment of the present application;

[0052] Figure 8 A flowchart of another posture determination method provided in an embodiment of the present application;

[0053] Fig. 9 A flowchart of another posture determination method provided in an embodiment of the present application;

[0054] Fig.10 A vector diagram of a target part provided in an embodiment of the present application;

[0055] Fig.11 A schematic diagram of a second posture feature graph provided in an embodiment of the present application;

[0056] Fig.12 A flowchart of another posture determination method provided in an embodiment of the present application;

[0057] Fig.13 A schematic diagram of the structure of a posture determination device provided in an embodiment of the present application;

[0058] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0059] Fig.15 A schematic diagram of the structure of another posture determination device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0061] In a posture recognition method in a fitness scenario, the user's posture is recognized by the posture angle collected by a wearable device worn by the user. However, this method can only determine the posture angle of the part where the user wears the wearable device, and cannot measure the posture angle of the user's entire body, so it is impossible to generate a complete human body posture.

[0062] In another posture recognition method in a fitness scene, firstly, the user's motion image or video is obtained through a camera, and then the image or video of the user's motion is recognized through a posture recognition algorithm based on image recognition, so as to obtain the human body posture, and compare the obtained human body posture with the correct fitness posture to generate guidance. However, in actual fitness scenes, due to the limited shooting range of the camera, some parts of the human body (such as arm joints) often exceed the shooting range of the camera during fitness. When some parts of the human body are not captured by the camera, the posture recognition method based only on image recognition cannot recognize the posture of the parts of the human body that are not captured, so it cannot provide the user with correct guidance.

[0063] It can be seen that the current posture recognition methods have only poor usability.

[0064] In order to solve the above technical problems, an embodiment of the present application provides a posture determination method. When the first image of the target object obtained by the terminal lacks a part of the target object, an acquisition request is sent to the wearable device worn by the target object to obtain the posture data of the target part, and then the complete posture of the target object is determined according to the first posture data and the first image. Even if the terminal cannot obtain an image including the complete target object, the complete posture of the target object can be determined, and the usability is strong. The method provided by the embodiment of the present application is described below in conjunction with the drawings of the specification.

[0065] The execution subject of the posture determination method provided in the embodiment of the present application can be an electronic device with data processing capability, such as a terminal equipment, a computer, etc. The terminal can be a user equipment (UE), a mobile station (MS), or a mobile terminal (MT), etc. Specifically, the terminal can be a mobile phone, a tablet computer, or a computer with wireless transceiver function, or a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal in industrial control, a wireless terminal in unmanned driving, a wireless terminal in telemedicine, a wireless terminal in a smart grid, a wireless terminal in a smart city, a smart home, or a vehicle-mounted terminal, etc. In the embodiment of the present application, the device for realizing the function of the terminal device can be a terminal, or a device that can support the terminal to realize the function, such as a chip system, which can be installed in the terminal or used in combination with the terminal.

[0066] The following describes the method provided by the embodiment of the present application by taking the execution subject of the posture determination method provided by the embodiment of the present application as an example that the terminal is an example.

[0067] Figure 1 FIG. 1 is a flow chart of a method for determining a posture according to an embodiment of the present application. Figure 1 As shown, the method may include the following steps:

[0068] S110, acquiring a first image.

[0069] The first image includes a target object. The target object is an object whose posture is to be determined, and the target object may be a living being or a device with movement ability. Exemplarily, the target object may be a person, an animal, a robot, etc. When acquiring the first image, it may be acquired through a video input device such as a camera. In the embodiment of the present application, the solution provided in the embodiment of the present application is introduced by taking the target object as a person as an example.

[0070] S120: When the target part of the target object is missing in the first image, send an acquisition request to a wearable device worn by the target object.

[0071] Among them, the target part does not specifically refer to a specific part of the target object, but the part of the target object that is missing in the first image. Exemplarily, assuming that the target object is a person, the first image does not include the person's left forearm. In this scenario, the target part is the left forearm. When the target part of the target object is missing in the first image, it means that the posture of the target object in the first image is incomplete, and the complete posture of the target object needs to be completed based on the posture data of the target part. At this time, the terminal sends an acquisition request to the wearable device worn by the target object to request the first posture data of the target part collected by the wearable device. Exemplarily, when the target part is the left forearm, the acquisition request is sent to the wearable device worn on the left forearm.

[0072] A wearable device is a type of electronic device that has the function of collecting the motion posture of a subject wearing the wearable device and can be connected to a terminal for communication. For example, the wearable device can be a sports watch.

[0073] S130, receiving first posture data sent by the wearable device.

[0074] After receiving the acquisition request from the terminal, the wearable device sends the first posture data of the target part collected by the wearable device to the terminal in response to the acquisition request. Correspondingly, the terminal receives the first posture data of the target part from the wearable device.

[0075] S140: Determine a complete posture of the target object in the first image according to the first posture data and the first image.

[0076] After the first posture data and the first image are acquired, the complete posture of the target object can be determined according to the first posture data and the first image.

[0077] In an embodiment of the present application, when the first image of the target object acquired by the terminal lacks a part of the target object, an acquisition request is sent to the wearable device worn by the target object to acquire posture data of the target part, and then the complete posture of the target object is determined based on the first posture data and the first image. Even if the terminal cannot acquire an image including the complete target object, the complete posture of the target object can be determined, and the usability is relatively strong.

[0078] In one embodiment, if Figure 2 As shown, before S120, the method may further include:

[0079] S210, identifying key points of limbs of a target object in a first image.

[0080] When identifying limb key points, a pre-trained key point detection model can be used for identification. When training the key point detection model, multiple images containing the target object (e.g., multiple frames of images in a motion video containing the target object) and the coordinates of the corresponding limb key points can be used as training samples for training to generate a key point detection model. The trained key point detection model can identify the limb key points and the coordinates of the limb key points in the image.

[0081] When training a key point detection model, the setting method of the limb key points is more critical. Limb key points with different settings have different preset numbers of limb key points, which can depict the complete posture of the target object. Since the accuracy required for the complete posture of the target object in different application scenarios is different, the limb key points of the target object can also have different settings. In scenarios where the accuracy required for the complete posture of the target object is lower, fewer limb key points can be set; in scenarios where the accuracy required for the complete posture of the target object is higher, more limb key points can be set. For example, Figure 3 As shown in FIG. 1 , the following 17 key points can be set as limb key points: left eye 1, right eye 2, nose 0, left ear 3, right ear 4, left shoulder 5, right shoulder 6, left elbow 7, right elbow 8, left wrist 9, right wrist 10, left hip 11, right hip 12, left knee 13, right knee 14, left ankle 15, and right ankle 16. In another example, as Figure 4 As shown, the following 18 key points can be set as limb key points: nose 0, neck 1, right shoulder 2, right elbow 3, right wrist 4, left shoulder 5, left elbow 6, left wrist 7, right hip 8, right knee 9, right ankle 10, left hip 11, left knee 12, left ankle 13, right eye 14, left eye 15, right ear 16, left ear 17. The above two limb key point setting methods are only examples, and the embodiment of the present application does not limit the limb key point setting method.

[0082] S220: When the number of identified limb key points is less than a preset number, determine that a target part of the target object is missing in the first image.

[0083] The target part includes the missing limb key points in the preset number of limb key points, and the preset number of limb key points are used to describe the complete posture of the target object. When the number of limb key points is less than the preset number, it means that the posture of the target object in the first image is incomplete, and the target part of the target object missing in the first image is determined to be used to determine the complete posture of the target object in the first image.

[0084] The preset number is the number of limb key points corresponding to the complete posture of the target object. For example, Figure 3 In the example scenario shown, the preset number is 17. Figure 4 In the illustrated example scenario, the preset number is 18. The target part of the target object is missing in the first image, and the target part includes the missing limb key points among the preset number of limb key points. The preset number of limb key points is used to describe the complete posture of the target object and can be determined according to the recognition result of S210. Figure 4 In the scene shown, there are 18 complete limb key points, and the result of S210 recognizing the first image is: nose, neck, right shoulder, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear and left ear, a total of 16 limb key points, missing the right elbow and right wrist. Then the target part is the right arm, and the target limb key points included in the right arm are the limb key points corresponding to the right elbow and right wrist.

[0085] Correspondingly, when the number of identified limb key points is equal to a preset number, in this case the posture of the target object in the first image is complete, and there is no need to send the above-mentioned acquisition request. Instead, the complete posture of the target object in the first image can be determined directly based on the first image.

[0086] In an embodiment of the present application, limb key points of the target object in the first image are identified, and when the number of identified limb key points is less than a preset number, an acquisition request for requesting the posture data of the target part is sent to the wearable device worn by the target object, thereby providing a mechanism for issuing acquisition requests, in which acquisition requests are only issued under preset circumstances, thereby saving terminal computing power.

[0087] In one embodiment, if Figure 5 As shown, S140 (determining the complete posture of the target object in the first image according to the first posture data and the first image) may include:

[0088] S310, determining a first posture feature map of the target object according to the first image.

[0089] Wherein, referring to the description in S210, a pre-trained key point detection model can be applied to recognize the first image, and firstly, the limb key points and the coordinates of the limb key points contained in the first image are obtained, assuming that N limb key points are obtained. The limb key points and the coordinates of the limb key points contained in the first image can be used to determine the first posture feature map of the target object. Specifically: the N limb key points are connected into N-1 vectors according to the object structure rule of the target object, and for each vector, its corresponding vector v is calculated n The direction of the target object in the first image is expressed as V = v1, b2…v N-1 .like Figure 6 As shown, for each vector v n , with v n As the center, a rectangular area with a height equal to the limb width and a length equal to the distance between the limb key points is divided in the first image. The direction vectors of all pixel points in the rectangular area point to the same direction. Use v n The corresponding X coordinates and Y coordinates are used as the values ​​of each pixel in the rectangular area (the values ​​of pixels outside the rectangular area are 0), and the X posture feature map and the Y posture feature map are obtained. The above operation is performed on all vectors, so that 2(N-1) posture feature maps (i.e., the first posture feature map) are obtained, which is expressed as T = ((X1, Y1), (X2, Y2)… (X N-1 ,Y N-1 )). The first posture feature map combination includes posture information of the target object in the first image.

[0090] In one embodiment, a pre-trained key point connection model may be used to determine a posture feature map, for example, to determine a first posture feature map of a target object. Specifically, the coordinates of the limb key points contained in the first image are input into the key point connection model, and the first posture feature map is output.

[0091] When training the key point connection model, the coordinates of the limb key points in multiple images containing the target object (e.g., multiple frames of images in a motion video containing the target object) and the corresponding posture feature graphs can be used as training samples to train and generate the key point connection model. The trained key point connection model can generate the corresponding first posture feature graph according to the coordinates of the limb key points in the image.

[0092] S320, determining a second posture feature map according to the first posture data and the initial posture data.

[0093] The first posture data includes a first posture angle collected by the wearable device when the target object starts to move and a second posture angle collected by the wearable device when the target part of the target object is missing in the first image. The initial posture data is a complete posture determined according to the initial position of the target object when it starts to move, and the second posture feature map is a posture feature map of the target part. The second posture feature map can be determined based on the first posture data and the initial posture data. For example, Figure 7 As shown, assuming that the target part is the right forearm, the dotted line represents the position of the right forearm when the movement starts, and the first posture angle is collected at this position. The solid line of the right forearm position represents the position of the target part when the target part of the target object is missing in the first image, and the second posture angle is collected at this position.

[0094] Among them, the specific process of the terminal determining the second posture feature map based on the motion sensing data and the initial complete posture data can be referred to the description of S410 to S420 below, which will not be repeated here.

[0095] S330, fusing the first posture feature map and the second posture feature map to obtain a complete posture of the target object.

[0096] Among them, the first posture feature map includes the posture of the target object in the first image, and the second posture feature map includes the posture feature map of the target part missing in the first image. The first posture feature map and the second posture feature map are combined to obtain the complete posture of the target object.

[0097] In an embodiment of the present application, a first posture feature map including the posture of the target object in the first image and a second posture feature map including the target part missing in the first image are first determined, and then the first posture feature map and the second posture feature map are fused to obtain the complete posture of the target object. Even if the terminal cannot obtain an image including the complete target object, the complete posture of the target object can be determined, and the usability is strong.

[0098] In one embodiment, if Figure 8 As shown, S320 (determining a second posture feature graph according to the first posture data and the initial posture data) may include:

[0099] S410: Determine a rotation angle of the second posture angle relative to the first posture angle.

[0100] The rotation angle of the second posture angle relative to the first posture angle can reflect the posture change of the target part of the target object from the start of the movement to the acquisition of the first image. Figure 7As shown, the target part is the right forearm, the dotted line represents the position of the right forearm when the movement starts, and the first posture angle is acquired at this position. The solid line of the right forearm position represents the position of the target part when the target part of the target object is missing in the first image, and the second posture angle is acquired at this position. α represents the rotation angle of the second posture angle relative to the first posture angle.

[0101] S420, determining a second posture feature map according to the rotation angle and the second posture data.

[0102] Among them, the second posture data is the posture data of the target part in the initial posture data. The target part includes the key points of the target limb; the second posture data includes the initial coordinates of the key points of the target limb. Exemplarily, when the target part is the left forearm, the key points of the target limb include the left wrist and the left elbow. In another example, when the target part is the left whole arm, the key points of the target limb include the left wrist, the left elbow and the left shoulder. The rotation angle can reflect the posture change of the target part of the target object from the start of movement to the acquisition of the first image, and the initial posture data records the initial position of the target object when it starts to move. According to the rotation angle and the second posture data of the target part in the initial posture data, the target coordinates of the target part after moving the rotation angle can be calculated, and then the second posture feature map is determined according to the target coordinates.

[0103] The specific process of the terminal determining the second posture feature map according to the rotation angle and the second posture data will be further introduced in the description of S510 to S530 below.

[0104] In an embodiment of the present application, a second posture feature map is determined based on the rotation angle of the target part and the data of the target part in the initial posture data of the target part. The second posture feature map can be used to merge with the first posture feature map to obtain the complete posture of the target object.

[0105] In one embodiment, if Fig. 9 As shown, S420 (determining a second posture feature map according to the rotation angle and the second posture data) may include:

[0106] S510, determining the target coordinates of the key points of the target limb of the target part after the target part is moved by the rotation angle according to the rotation angle and the initial coordinates.

[0107] Among them, after the terminal moves the initial coordinate reference rotation angle, it can determine the target coordinate.

[0108] S520, determining the vector of the target part according to the target coordinates.

[0109] Among them, the target limb key point may be one or more, and correspondingly, the target coordinates may also be one or more. Referring to the description of the vector in S310, for each target limb key point, the limb key points adjacent to it are determined according to the object structure rule of the target object (the limb key point may be another target limb key point or a non-target limb key point, depending on which target limb key points are included in the target part), and a vector is generated according to the target coordinates of the target limb key point and the coordinates of the limb key points adjacent to the target limb key point, thereby obtaining the vector of the target part.

[0110] For example, Fig.10 As shown, the target part includes target limb key point A and target limb key point B, whose coordinates are (x3, y3) and (x4, y4) respectively. The limb key point adjacent to the target limb key point A is the target limb key point B; the limb key points adjacent to the target limb key point B include target limb key point A and limb key point C, and the coordinates of limb key point C are (x5, y5). Then the vectors generated in sequence include: the vector calculated according to (x3, y3) and (x4, y4), the vector calculated according to (x3, y3) and (x4, y4), and the vector calculated according to (x4, y4) and (x5, y5). Multiple vectors obtained by the same calculation basis are not repeated, and one vector calculated according to (x3, y3) and (x4, y4) is removed; then the vector of the target part includes the vector calculated according to (x3, y3) and (x4, y4) and the vector calculated according to (x4, y4) and (x5, y5).

[0111] S530: construct a second posture feature map according to the vector of the target part.

[0112] Wherein, the second posture feature map includes a first sub-map and a second sub-map. Refer to the description of the X posture feature map and the Y posture feature map in S310. The first sub-map is an X posture feature map constructed according to the vector of the target part, and the second sub-map is a Y posture feature map constructed according to the vector of the target part. The first sub-map is the area formed by the vector of the target part after extending the width of the target part, and the pixel value in the first sub-map is the projection length of the vector of the target part on the preset X-axis; the second sub-map is the area formed by the vector of the target part after extending the width of the target part, and the pixel value in the second sub-map is the projection length of the vector of the target part on the Y-axis (perpendicular to the preset X-axis); the second posture feature map also includes a matrix containing the first sub-map (the pixel values ​​in the matrix except the first sub-map are 0) and a matrix containing the second sub-map (the pixel values ​​in the matrix except the second sub-map are 0). Exemplarily, as Fig.11 As shown, continue to use Fig.10In the example shown, the pixel values ​​in the first sub-image are x4-x3 and x5-x4, and the pixel values ​​in the second sub-image are y4-y3 and y5-y4.

[0113] In the embodiment of the present application, a second posture feature map is constructed according to the vector of the target part, and the second posture feature map is fused with the first posture feature map to obtain the complete posture of the target object.

[0114] In one embodiment, if Fig.12 As shown, the method may also include:

[0115] S610, acquiring a second image.

[0116] The second image is an image frame in the target video that is located after the first image. In some sports scenes (e.g., fitness scenes), it is necessary to collect multiple sports images of the user in real time. In this scene, the target object is photographed to obtain a target video, which includes multiple image frames, and the multiple image frames include the first image and the second image.

[0117] S620: When the second image includes the complete target object, send a stop instruction to the wearable device.

[0118] When the part of the target object in the second image is complete, it means that the complete posture of the target object in the second image can be determined according to the second image. At this time, a stop instruction is sent to the wearable device, and the stop instruction is used to instruct to stop sending the posture data collected by the wearable device.

[0119] S630: Determine a complete posture of the target object in the second image based on the second image.

[0120] Among them, when determining the complete posture of the target object in the second image, the key point detection model and the key point connection model introduced in S310 can be applied to determine the complete posture of the target object in the second image. The specific determination process is the same as the principle of determining the first posture feature map in S310. Please refer to the introduction in S310 and will not be repeated here.

[0121] In an embodiment of the present application, when the part of the target object in the acquired second image is complete, a stop instruction is sent to the wearable device, which can save communication resource consumption between the terminal and the wearable device.

[0122] The above mainly introduces the scheme provided by the embodiment of the present application from the perspective of the execution logic of each step. It is understandable that, in order to realize the above functions, the execution subject of the embodiment of the present application includes the hardware structure and / or software module corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the algorithm steps of each example described in the embodiment disclosed herein, the method of the embodiment of the present application can be implemented in the form of hardware, software, or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0123] The embodiment of the present application can divide the execution subject of the embodiment of the present application into functional modules according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0124] In one embodiment, the embodiment is a fitness posture recognition scenario, in which the target object is a user. Fig.13 As shown, the posture determination device provided in the embodiment of the present application may include the following modules:

[0125] The initial position calibration module 001 is used to guide the user to adjust the initial angle of the terminal to ensure that the terminal is placed at the correct angle. Then the user is guided to adjust the initial position, and the coordinates of each key point in the user's initial position are recorded to generate initial posture data, as well as the first posture angle of the wearable device in the initial position.

[0126] The gesture recognition module 002 is used to execute the Figure 1-Figure 11 Introduced method to get the complete posture of the user.

[0127] The action evaluation module 003 compares the user's complete posture with the standard movement posture to obtain the action evaluation result, and can evaluate the standard degree of the user's action.

[0128] The user interaction feedback module 004 provides specific fitness guidance on how to adjust the user's movements based on the movement evaluation results, helping the user to adjust the movement.

[0129] In specific implementation, the execution subject of the embodiment of the present application, such as: the terminal can use Fig.14 The structure shown or includes Fig.14 Parts shown. Fig.14 A structural schematic diagram of an electronic device provided in an embodiment of the present application. When the electronic device has the function of the terminal described in the embodiment of the present application, the electronic device may be a terminal or a chip or system on chip in the terminal.

[0130] like Fig.14 As shown, the electronic device may include a processor 801, a communication line 802, a transceiver 803, and a memory 804. The processor 801, the memory 804, and the transceiver 803 may be connected via the communication line 802. In an example, the processor 801 may include one or more CPUs, such as Fig.14 CPU0 and CPU1 in.

[0131] As an optional implementation, the electronic device includes multiple processors, for example, Fig.14 In addition to the processor 801, a processor 807 may also be included.

[0132] The processor 801 may be a central processing unit (CPU), a general-purpose processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 801 may also be other devices with processing functions, such as circuits, devices, or software modules.

[0133] The communication line 802 is used to transmit information between the components included in the electronic device.

[0134] The transceiver 803 is used to communicate with other devices or other communication networks. The other communication networks may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The transceiver 803 may be an interface circuit, a pin, a radio frequency module, a transceiver or any device capable of achieving communication.

[0135] Furthermore, the electronic device may also include a memory 804. The memory 804 is used to store instructions, wherein the instructions may be computer programs.

[0136] Among them, the memory 804 can be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage, magnetic disk storage media or other magnetic storage devices, and the optical disc storage includes a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, or a Blu-ray disc, etc.

[0137] It should be noted that the memory 804 can exist independently of the processor 801, or can be integrated with the processor 801. The memory 804 can be used to store instructions or program codes or some data, etc. The memory 804 can be located inside the electronic device or outside the electronic device, without limitation. When the processor 801 executes the instructions stored in the memory 804, the method provided in the embodiment of the present application can be implemented.

[0138] The electronic device further includes an output device 805 and an input device 806. Exemplarily, the input device 806 may be a camera, a keyboard, a mouse, a microphone, a joystick, or the like, and the output device 805 may be a display screen, a speaker, or the like.

[0139] It should be noted that electronic devices can be desktop computers, portable computers, network servers, mobile phones, tablet computers, wireless terminals, embedded devices, chip systems or Fig.14 In addition, Fig.14 The components shown in the figure do not constitute a limitation on the electronic device, except Fig.14 In addition to the components shown, the electronic device may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0140] In the embodiment of the present application, the chip system may be composed of a chip, or may include a chip and other discrete devices.

[0141] Fig.15 A structural diagram of another posture determination device is shown. Fig.15 Each module in the device shown has the function of realizing Figure 1-Figure 11The functions of the corresponding steps in the figure can achieve the corresponding technical effects. The corresponding beneficial effects of the steps executed by each module can be referred to the description of the corresponding steps and will not be repeated here. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The posture determination device can be a terminal or a chip or system on chip in the terminal. For example, the posture determination device includes:

[0142] The acquisition module 141 is used to acquire a first image, where the first image includes a target object.

[0143] The transceiver module 142 is used to send an acquisition request to the wearable device worn by the target object when the target part of the target object is missing in the first image, and the acquisition request is used to request to obtain the first posture data collected by the wearable device, and the first posture data includes the posture data of the target part.

[0144] The transceiver module 142 is further configured to receive first posture data sent by the wearable device.

[0145] The determination module 143 is used to determine the complete posture of the target object in the first image according to the first posture data and the first image.

[0146] In a possible implementation, the device further includes a recognition module, configured to recognize limb key points of the target object in the first image.

[0147] The determination module 143 is also used to determine the target part of the target object missing in the first image when the number of identified limb key points is less than a preset number. The target part includes the limb key points missing from a preset number of limb key points, and the preset number of limb key points are used to describe the complete posture of the target object.

[0148] In a possible implementation, the determination module 143 is specifically configured to:

[0149] A first posture feature map of the target object is determined according to the first image.

[0150] A second posture feature map is determined based on the first posture data and the initial posture data, wherein the initial posture data is used to characterize the complete posture of the target object determined based on the initial position when the target object starts to move, and the second posture feature map is used to characterize the posture of the target part.

[0151] The first posture feature map and the second posture feature map are fused to obtain a complete posture of the target object in the first image.

[0152] In a possible implementation, the first posture data includes a first posture angle collected by the wearable device when the target object starts to move and a second posture angle collected by the wearable device when the target part of the target object is missing in the first image.

[0153] The determination module 143 is specifically used for:

[0154] A rotation angle of the second posture angle relative to the first posture angle is determined.

[0155] A second posture feature map is determined according to the rotation angle and the second posture data, wherein the second posture data is the posture data of the target part in the initial posture data.

[0156] In a possible implementation, the target part includes a target limb key point. The second posture data includes initial coordinates of the target limb key point.

[0157] The determination module 143 is specifically used for:

[0158] According to the rotation angle and the initial coordinates, the target coordinates of the key points of the target limb of the target part after moving the rotation angle are determined.

[0159] Determine the vector of the target part according to the target coordinates.

[0160] A second posture feature map is constructed according to the vector of the target part.

[0161] In a possible implementation, the acquisition module 141 is further configured to acquire a second image, where the second image is an image frame in the target video that is located after the first image.

[0162] The transceiver module 142 is further configured to send a stop instruction to the wearable device when the second image includes a complete target object, wherein the stop instruction is used to instruct the wearable device to stop sending the gesture data.

[0163] The determination module 143 is further configured to determine the complete posture of the target object in the second image based on the second image.

[0164] The embodiment of the present application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be completed by a computer program to instruct the relevant hardware, and the program can be stored in the above computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be a terminal device of any of the above embodiments, such as: an internal storage unit including a data sending end and / or a data receiving end, such as a hard disk or memory of a terminal device. The above computer-readable storage medium can also be an external storage device of the above terminal device, such as a plug-in hard disk equipped on the above terminal device, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. Further, the above computer-readable storage medium can also include both an internal storage unit of the above terminal device and an external storage device. The above computer-readable storage medium is used to store the above computer program and other programs and data required by the above terminal device. The above computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0165] The present application also provides a computer instruction. All or part of the process in the above method embodiment can be completed by computer instructions to instruct related hardware (such as computers, processors, network devices, and terminals, etc.). The program can be stored in the above computer-readable storage medium.

[0166] The embodiment of the present application also provides a chip system. The chip system can be composed of a chip, or can include a chip and other discrete devices, without limitation. The chip system includes a processor and a transceiver, and all or part of the processes in the above method embodiment can be completed by the chip system, such as the chip system can be used to implement the functions performed by the terminal in the above method embodiment.

[0167] In one possible design, the above-mentioned chip system also includes a memory, which is used to store program instructions and / or data. When the chip system is running, the processor executes the program instructions stored in the memory so that the chip system performs the functions performed by the terminal in the above-mentioned method embodiment.

[0168] In the embodiments of the present application, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0169] In the embodiment of the present application, the memory may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), etc., or a volatile memory (volatile memory), such as a random-access memory (RAM). The memory is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory in the embodiment of the present application may also be a circuit or any other device that can implement a storage function, for storing instructions and / or data.

[0170] It should be noted that the terms "first" and "second" in the specification, claims and drawings of the present application are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0171] It should be understood that in the embodiments of the present application, "at least one (item)" refers to one or more, "multiple" refers to two or more, "at least two (items)" refers to two or three and more than three, and "and / or" is used to describe the association relationship of the associated objects, indicating that there can be three relationships. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following (items)" or similar expressions refers to any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple. It should be understood that in the embodiments of the present application, "B corresponding to A" means that B is associated with A. For example, B can be determined based on A. It should also be understood that determining B based on A does not mean determining B based only on A, but B can also be determined based on A and / or other information. In addition, the "connection" in the embodiments of the present application refers to various connection modes such as direct connection or indirect connection to achieve communication between devices, and the embodiments of the present application do not impose any limitation on this.

[0172] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0173] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0174] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0175] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device, such as: a single-chip microcomputer, a chip, etc., or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, disks, or optical disks.

[0176] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for determining a posture, characterized in that: include: Acquire a first image, wherein the first image includes a target object; When the target part of the target object is missing in the first image, sending an acquisition request to a wearable device worn by the target object, the acquisition request is used to request to acquire first posture data collected by the wearable device, the first posture data including posture data of the target part; Receiving the first posture data sent by the wearable device; A complete posture of a target object in the first image is determined according to the first posture data and the first image.

2. The method for determining the posture according to claim 1, characterized in that: The method further comprises: Identifying key points of a limb of a target object in the first image; When the number of the identified limb key points is less than a preset number, it is determined that the target part of the target object is missing in the first image, and the target part includes the limb key points missing from the preset number of limb key points, and the preset number of limb key points are used to depict the complete posture of the target object.

3. The posture determination method according to claim 1 or 2, characterized in that: Determining the complete posture of the target object in the first image according to the first posture data and the first image includes: Determine a first posture feature map of the target object according to the first image; Determining a second posture feature graph according to the first posture data and the initial posture data, wherein the initial posture data is used to characterize the complete posture of the target object determined according to the initial position when the target object starts to move, and the second posture feature graph is used to characterize the posture of the target part; The first posture feature map and the second posture feature map are fused to obtain a complete posture of the target object in the first image.

4. The method for determining the posture according to claim 3, characterized in that: The first posture data includes a first posture angle collected by the wearable device when the target object starts to move and a second posture angle collected by the wearable device when the target part of the target object is missing in the first image; The determining of the second posture feature graph according to the first posture data and the initial posture data comprises: determining a rotation angle of the second posture angle relative to the first posture angle; The second posture feature map is determined according to the rotation angle and the second posture data, wherein the second posture data is the posture data of the target part in the initial posture data.

5. The method for determining the posture according to claim 4, characterized in that: The target part includes a target limb key point; the second posture data includes the initial coordinates of the target limb key point; Determining the second posture feature map according to the rotation angle and the second posture data, wherein the second posture data is the posture data of the target part in the initial posture data, comprises: Determine, according to the rotation angle and the initial coordinates, the target coordinates of the key points of the target limb after the target part is moved by the rotation angle; Determine the vector of the target part according to the target coordinates; The second posture feature map is constructed according to the vector of the target part.

6. The posture determination method according to any one of claims 1 to 5, characterized in that: The first image is an image frame in a target video, and the method further includes: Acquire a second image, where the second image is an image frame in the target video that is located after the first image; When the second image includes the complete target object, sending a stop instruction to the wearable device, wherein the stop instruction is used to instruct to stop sending the posture data collected by the wearable device; and determining the complete posture of the target object in the second image based on the second image.

7. A posture determination device, characterized in that: include: An acquisition module, configured to acquire a first image, wherein the first image includes a target object; a transceiver module, configured to send an acquisition request to a wearable device worn by the target object when the target part of the target object is missing in the first image, wherein the acquisition request is used to request acquisition of first posture data collected by the wearable device, wherein the first posture data includes posture data of the target part; The transceiver module is further used to receive the first posture data sent by the wearable device; A determination module is used to determine the complete posture of the target object in the first image according to the first posture data and the first image.

8. The posture determination device according to claim 7, characterized in that: The device also includes a recognition module for identifying limb key points of the target object in the first image; The determination module is also used to determine that the target part of the target object is missing in the first image when the number of the identified limb key points is less than a preset number, and the target part includes the limb key points that are missing from the preset number of limb key points, and the preset number of limb key points are used to depict the complete posture of the target object.

9. The posture determination device according to claim 7 or 8, characterized in that: The determining module is specifically used for: Determine a first posture feature map of the target object according to the first image; Determining a second posture feature graph according to the first posture data and the initial posture data, wherein the initial posture data is used to characterize the complete posture of the target object determined according to the initial position when the target object starts to move, and the second posture feature graph is used to characterize the posture of the target part; The first posture feature map and the second posture feature map are fused to obtain a complete posture of the target object in the first image.

10. The posture determination device according to claim 9, characterized in that: The first posture data includes a first posture angle collected by the wearable device when the target object starts to move and a second posture angle collected by the wearable device when the target part of the target object is missing in the first image; The determining module is specifically used for: determining a rotation angle of the second posture angle relative to the first posture angle; The second posture feature map is determined according to the rotation angle and the second posture data, wherein the second posture data is the posture data of the target part in the initial posture data.

11. The posture determination device according to claim 10, characterized in that: The target part includes a target limb key point; the second posture data includes the initial coordinates of the target limb key point; The determining module is specifically used for: Determine, according to the rotation angle and the initial coordinates, the target coordinates of the key points of the target limb after the target part is moved by the rotation angle; Determine the vector of the target part according to the target coordinates; The second posture feature map is constructed according to the vector of the target part.

12. The posture determination device according to any one of claims 7 to 11, characterized in that: The first image is an image frame in the target video, and the acquisition module is further used to acquire a second image, where the second image is an image frame in the target video that is located after the first image; The transceiver module is further configured to send a stop instruction to the wearable device when the second image includes the complete target object, wherein the stop instruction is used to instruct to stop sending the posture data collected by the wearable device; The determination module is further used to determine the complete posture of the target object in the second image based on the second image.

13. A posture determination device, characterized in that: The posture determination device comprises a processor and a transceiver, and the processor and the transceiver are used to support the posture determination device to execute the method according to any one of claims 1-6.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed, the method according to any one of claims 1 to 6 is executed.

Images