Action scoring device and method

Abstract

The embodiment of the application provides a kind of action scoring equipment and method, it is related to data processing technical field.The action scoring equipment includes: first, the skeleton information extraction is carried out to action image, obtains the first skeleton information corresponding to the action image, the first skeleton information includes the coordinate value of at least one skeleton key point;When the first skeleton information is matched with the template skeleton information of target action, it is determined whether the skeleton key point of the first skeleton information includes the occlusion point;If the first skeleton key point of the first skeleton information is the occlusion point, then the first region corresponding to the first skeleton key point is obtained;Second, judge whether second skeleton key point is located in the first region, if yes, then with the coordinate value of the first skeleton key point as first coordinate value, the action score of the action image is calculated.The embodiment of the application is used to solve the problem that when multiple occlusion points appear, action scoring is not accurate.

Description

Technical Field

[0001] This application relates to the field of data processing technology. More specifically, it relates to a motion scoring device and method. Background Technology

[0002] In the process of scoring movements in fitness software, when a large number of body parts are obscured, a human skeleton keypoint detection algorithm is usually used to evaluate the user's movements. This means treating the obscured points as unobscured points and then using the same scoring method as for unobscured points to evaluate the movement. This method is feasible when the obscured point is a single point. However, when the obscured points are a series of points, the skeleton point detection will have a large number of errors, which will affect the scoring of the obscured areas. This will result in users not being able to obtain accurate movement scores, and may even affect the entire scoring system, leading to a poor user experience. Summary of the Invention

[0003] An exemplary embodiment of this application provides an action scoring device and method to solve the problem of inaccurate action scoring when multiple occlusion points occur, thereby improving the user experience.

[0004] The technical solutions provided in this application are as follows:

[0005] In a first aspect, embodiments of this application provide a motion scoring device, including:

[0006] The controller is configured as follows:

[0007] Skeleton information is extracted from the motion image to obtain first skeleton information corresponding to the motion image; the first skeleton information includes the coordinate values ​​of at least one skeleton key point.

[0008] When the first skeleton information matches the template skeleton information of the target action, determine whether the skeleton key points of the first skeleton information contain occlusion points.

[0009] If the first skeleton key point of the first skeleton information is the occlusion point, then the first region corresponding to the first skeleton key point is obtained; the distance from any point in the first region to the first skeleton key point is less than the threshold distance corresponding to the first skeleton key point.

[0010] Determine whether the second skeleton key point is located in the first area. The second skeleton key point is the skeleton key point in the template skeleton information that corresponds to the first skeleton key point.

[0011] If so, the motion score of the motion image is calculated using the coordinates of the first skeleton key point as the first coordinate value; the first coordinate value is the coordinates of the second skeleton key point.

[0012] Secondly, embodiments of this application provide an action scoring method, including:

[0013] Skeleton information is extracted from the motion image to obtain first skeleton information corresponding to the motion image; the first skeleton information includes the coordinate values ​​of at least one skeleton key point.

[0014] When the first skeleton information matches the template skeleton information of the target action, determine whether the skeleton key points of the first skeleton information contain occlusion points.

[0015] If the first skeleton key point of the first skeleton information is the occlusion point, then the first region corresponding to the first skeleton key point is obtained; the distance from any point in the first region to the first skeleton key point is less than the threshold distance corresponding to the first skeleton key point.

[0016] Determine whether the second skeleton key point is located in the first area. The second skeleton key point is the skeleton key point in the template skeleton information that corresponds to the first skeleton key point.

[0017] If so, the motion score of the motion image is calculated using the coordinates of the first skeleton key point as the first coordinate value; the first coordinate value is the coordinates of the second skeleton key point.

[0018] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a computing device, causes the computing device to implement the display method described in any embodiment of the first aspect.

[0019] Fourthly, the present invention provides a computer program product that, when run on a computer, enables the computer to implement the action scoring method described in the second aspect or any embodiment of the second aspect.

[0020] As can be seen from the above technical solutions, the motion scoring device and method provided in this application embodiment, the controller, is configured to: extract skeleton information from a motion image to obtain first skeleton information corresponding to the motion image; the first skeleton information includes the coordinate values ​​of at least one skeleton key point; when the first skeleton information matches the template skeleton information of the target motion, determine whether the skeleton key point of the first skeleton information contains an occlusion point; if the first skeleton key point of the first skeleton information is the occlusion point, then obtain a first region corresponding to the first skeleton key point; the distance from any point in the first region to the first skeleton key point is less than the threshold distance corresponding to the first skeleton key point; determine whether a second skeleton key point is located in... Within the first region, the second skeleton key point is the skeleton key point in the template skeleton information corresponding to the first skeleton key point; if so, the coordinate value of the first skeleton key point is used as the first coordinate value to calculate the action score of the action image; the first coordinate value is the coordinate value of the second skeleton key point. Compared with the prior art, when there are a large number of occlusion points, the user cannot obtain an accurate action score. The technical solution of this application determines whether the occlusion point is within the threshold range of the second skeleton key point corresponding to the template skeleton. If it is, the coordinate value of the second skeleton key point is used to replace the occlusion point for action scoring. Therefore, the embodiment of this application can still obtain an accurate action score when there are multiple occlusion points, thus improving the user experience. Attached Figure Description

[0021] To more clearly illustrate the implementation methods in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0022] Figure 1 A scene architecture diagram of the action scoring method provided in an embodiment of the present invention is shown;

[0023] Figure 2 A hardware configuration block diagram of the control device provided in an embodiment of the present invention is shown;

[0024] Figure 3 A hardware configuration block diagram of the motion scoring device provided in an embodiment of the present invention is shown;

[0025] Figure 4 The diagram shows the software configuration of the motion scoring device provided in an embodiment of the present invention;

[0026] Figure 5 A flowchart illustrating the steps of the action scoring method provided in an embodiment of the present invention is shown;

[0027] Figure 6 A schematic diagram of the skeleton information of an action scoring method provided by an embodiment of the present invention is shown;

[0028] Figure 7 The following is a flowchart illustrating the steps of an action scoring method provided by an embodiment of the present invention;

[0029] Figure 8 This diagram illustrates a double-chain Hungarian matching logic in an action scoring method provided by an embodiment of the present invention.

[0030] Figure 9 A schematic diagram of skeleton information for another motion scoring method provided in an embodiment of the present invention is shown;

[0031] Figure 10 A schematic diagram of skeleton information for another motion scoring method provided in an embodiment of the present invention is shown;

[0032] Figure 11 A flowchart illustrating the steps of another action scoring method provided by an embodiment of the present invention is shown;

[0033] Figure 12 A schematic diagram of skeleton information for another motion scoring method provided in an embodiment of the present invention is shown;

[0034] Figure 13 A flowchart illustrating the steps of another action scoring method provided by an embodiment of the present invention is shown;

[0035] Figure 14 A flowchart illustrating the steps of another action scoring method provided by an embodiment of the present invention is shown. Detailed Implementation

[0036] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0037] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0038] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0039] Figure 1 This is a schematic diagram of the scene architecture for the action scoring method provided in an embodiment of this application. Figure 1 As shown in the embodiment of this application, the scenario architecture includes: a server 400 and an action scoring device 300.

[0040] The action scoring device 300 provided in this application embodiment can have various implementation forms, such as smart speakers, televisions, refrigerators, washing machines, air conditioners, smart curtains, routers, set-top boxes, mobile phones, personal computers (PCs), smart TVs, laser projection devices, monitors, electronic bulletin boards, wearable devices, in-vehicle devices, electronic tables, etc.

[0041] In some embodiments, a user can operate the motion scoring device 300 through a smart device 200 or a control device 100. When the motion scoring device 300 receives an audio signal, it can communicate with the server 400. The motion scoring device 300 can communicate with the server 400 through a local area network (LAN) or a wireless local area network (WLAN).

[0042] Server 400 can be a server that provides various services, such as a server that supports audio data collected by terminal devices. The server can analyze and process the received audio and other data, and feed back the processing results (such as endpoint information) to the terminal device. Server 400 can be a server cluster or multiple server clusters, and can include one or more types of servers.

[0043] The motion scoring device 300 can be either hardware or software. When the motion scoring device 300 is hardware, it can be various electronic devices with sound acquisition capabilities, including but not limited to smart speakers, smartphones, televisions, tablets, e-book readers, smartwatches, media players, computers, AI devices, robots, smart vehicles, etc. When the motion scoring device 300 is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software programs or software modules (e.g., used to provide sound acquisition services) or as a single software program or software module. No specific limitations are made here.

[0044] It should be noted that the action scoring method provided in this application embodiment can be executed by the server 400, the action scoring device 300, or by both the server 400 and the action scoring device 300. This application does not limit the execution of the method.

[0045] Figure 2A hardware configuration block diagram of the motion scoring device 300 according to an exemplary embodiment is shown. Figure 2 The action scoring device 300 shown includes at least one of the following: a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, an audio processor, RAM, ROM, and a first to an nth interface for input / output.

[0046] The communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The action scoring device 300 can establish the transmission and reception of control signals and data signals through the communicator 220 and the server 400.

[0047] User interface 280 can be used to receive external control signals.

[0048] Detector 230 is used to collect signals from the external environment or to interact with the external environment. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to collect external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0049] The sound acquisition device can be a microphone, also known as a "microphone" or "voice transducer," which can be used to receive the user's voice and convert the sound signal into an electrical signal. The motion scoring device 300 can be equipped with at least one microphone. In some embodiments, the motion scoring device 300 can be equipped with two microphones, which, in addition to acquiring sound signals, can also perform noise reduction. In other embodiments, the motion scoring device 300 can also be equipped with three, four, or more microphones, enabling sound signal acquisition, noise reduction, sound source identification, and directional recording functions, etc.

[0050] Furthermore, the microphone can be built into the motion scoring device 300, or it can be connected to the motion scoring device 300 via wired or wireless means. Of course, this embodiment does not limit the location of the microphone on the motion scoring device 300. Alternatively, the motion scoring device 300 may not include a microphone, meaning the microphone is not located within the motion scoring device 300. The motion scoring device 300 can connect an external microphone (also called a microphone) via an interface (such as USB interface 130). This external microphone can be fixed to the motion scoring device 300 using an external fastener (such as a camera bracket with a clip).

[0051] The controller 250 controls the operation of the motion scoring device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the motion scoring device 300.

[0052] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0053] In some embodiments, the operating system of the smart device is, for example, the Android system. Figure 3 As shown, the action scoring device 300 can be logically divided into an application layer (referred to as "application layer") 21, a kernel layer 22, and a hardware layer 23.

[0054] Among them, such as Figure 3 As shown, the hardware layer may include Figure 2 The controller 250, communicator 220, detector 230, etc., are shown. Application layer 21 includes one or more applications. These applications can be system applications or third-party applications. For example, application layer 21 includes a speech recognition application, which can provide an action scoring interface and services for connecting the action scoring device 300 to the server 400.

[0055] The kernel layer 22 serves as a software middleware between the hardware layer and the application layer 21, and is used to manage and control hardware and software resources.

[0056] In some embodiments, kernel layer 22 includes a detector driver, which is used to send voice data collected by detector 230 to a voice recognition application. For example, when the voice recognition application in action scoring device 300 is started and a communication connection is established between action scoring device 300 and server 400, the detector driver is used to send the user-input voice data collected by detector 230 to the voice recognition application. Subsequently, the voice recognition application sends query information containing this voice data to intent recognition module 102 in the server. Intent recognition module 102 is used to input the voice data sent by action scoring device 300 into the intent recognition model.

[0057] To clearly illustrate the embodiments of this application, the following description is provided in conjunction with... Figure 4 This application describes a speech recognition network architecture provided in its embodiments.

[0058] See Figure 4 As shown, Figure 4 This is a schematic diagram of an action scoring network architecture provided in an embodiment of this application. Figure 4 In this system, the action scoring device receives input information and outputs the processing result of that information. The speech recognition module deploys a speech recognition service to convert audio into text; the semantic understanding module deploys a semantic understanding service to perform semantic parsing on the text; the business management module deploys a business instruction management service to provide business instructions; the language generation module deploys a language generation service (NLG) to convert instructions instructing the action scoring device into text; and the speech synthesis module deploys a text-to-speech (TTS) service to process the text corresponding to the instructions and send it to a speaker for playback. In one embodiment, Figure 4 The architecture shown can contain multiple entity service devices with different business services deployed, or one or more entity service devices can combine one or more functional services.

[0059] In some embodiments, the following describes the basis Figure 4 The process of processing information from the input action scoring device in the architecture shown is described with an example, taking voice commands input via voice as an example:

[0060] [Speech Recognition]

[0061] The motion scoring device can perform noise reduction and feature extraction on the audio of the voice command after receiving a voice input. The noise reduction process may include steps such as removing echoes and environmental noise.

[0062] [Semantic understanding]

[0063] Using acoustic and language models, natural language understanding is performed on the identified candidate text and associated contextual information. The text is parsed into structured, machine-readable information, including business domain, intent, slots, and other semantic information. An executable intent confidence score is obtained, and the semantic understanding module selects one or more candidate executable intents based on the determined intent confidence score.

[0064] [Business Management]

[0065] Based on the semantic parsing results of the voice command text, the semantic understanding module issues execution instructions to the corresponding business management module to execute the operation corresponding to the voice command, complete the user's request for this operation, and provide feedback on the execution result of the operation corresponding to the voice command.

[0066] In some embodiments, the motion scoring device 300 extracts skeleton information from the motion image through the controller 250 to obtain first skeleton information corresponding to the motion image; the first skeleton information includes the coordinate values ​​of at least one skeleton key point; when the first skeleton information matches the template skeleton information of the target motion, it is determined whether the skeleton key point of the first skeleton information contains an occlusion point; if the first skeleton key point of the first skeleton information is the occlusion point, then a first region corresponding to the first skeleton key point is obtained; the distance from any point in the first region to the first skeleton key point is less than the threshold distance corresponding to the first skeleton key point; it is determined whether a second skeleton key point is located in the first region, the second skeleton key point being the skeleton key point in the template skeleton information corresponding to the first skeleton key point; if so, the motion score of the motion image is calculated using the coordinate value of the first skeleton key point as the first coordinate value; the first coordinate value is the coordinate value of the second skeleton key point.

[0067] In some embodiments, the motion scoring device 300 obtains a second coordinate value when the second skeleton key point is located outside the first region via the controller 250; the second coordinate value is the coordinate value of the point within the first region that is closest to the second skeleton key point; and the motion score of the motion image is calculated using the coordinate value of the first skeleton key point as the second coordinate value.

[0068] In some embodiments, the motion scoring device 300 obtains the distance between the second skeleton key point and the third skeleton key point through the controller 250. The third skeleton key point is the skeleton key point in the template skeleton information that is closest to the second skeleton key point. The threshold distance corresponding to the first skeleton key point is determined based on the distance between the second skeleton key point and the third skeleton key point. A circular area with the first skeleton key point as the center and the threshold distance corresponding to the first skeleton key point as the radius is determined as the first region corresponding to the first skeleton key point.

[0069] In some embodiments, the motion scoring device 300 calculates the motion score of the motion image based on the coordinate values ​​of each skeleton key point in the first skeleton information when the occlusion point is not included in the first skeleton information, through the controller 250.

[0070] In some embodiments, the motion scoring device 300 obtains the angles of each joint angle in the first skeleton information based on the coordinate values ​​of each skeleton key point in the first skeleton information through the controller 250; obtains the angle difference of each joint angle in the first skeleton information, wherein the angle difference of any joint angle is the difference between the angle of that joint angle and the angle of the corresponding joint angle in the template skeleton information; calculates the joint angle score of the motion image based on the angle difference of each joint angle; and calculates the motion score of the motion image based on the joint angle score of the motion image.

[0071] In some embodiments, the motion scoring device 300 calculates the joint angle score of the motion image using the controller 250 based on the angle difference of each joint angle and the following formula:

[0072]

[0073] Where S1 is the joint angle score of the motion image, N is the number of joint angles in the first skeleton information, and Δx i W represents the angle difference of the i-th joint angle. ai is the weighting coefficient for the i-th joint angle.

[0074] In some embodiments, the motion scoring device 300 calculates a basic score for the motion image using the controller 250 according to the following formula:

[0075] S2=min{[Peark*Wpk+min(Peara, Peak)*Wpa], 1)

[0076] Wherein, S2 is the base score of the motion image, Peark is the weighted Pearson product of the template skeleton information after weighting and normalization with the skeleton key points in the first skeleton information, Wpk is the weight coefficient of the skeleton key point Pearson product, Peara is the weighted Pearson product of the template skeleton information after weighting and normalization with the joint angles in the first skeleton information, and Wpa is the weight coefficient of the joint angle Pearson product.

[0077] The motion score of the motion image is calculated based on the base score and the joint angle score of the motion image.

[0078] In some embodiments, the motion scoring device 300 calculates the motion score of the motion image according to the following formula via the controller 250: S=w*(w1*S1+(1-w1)*S2)

[0079] Wherein, S is the motion score of the motion image; w is the scoring coefficient of the target motion; and w1 is the joint angle score coefficient.

[0080] In some embodiments, the motion scoring device 300 determines, via the controller 250, whether the first skeleton information matches the template skeleton information of the target motion according to the double-chain Hungarian matching algorithm.

[0081] Figure 5 An exemplary flowchart of the action scoring method provided in an embodiment of this application is shown, such as... Figure 5 As shown, the action scoring method provided in this application includes the following steps:

[0082] S501. Extract skeleton information from the action image to obtain the first skeleton information corresponding to the action image.

[0083] The first skeleton information includes the coordinate values ​​of at least one skeleton key point.

[0084] In some embodiments, the motion scoring device can be a terminal device such as a television or mobile phone. For example, when a user needs to exercise, they can follow the prompts in a fitness video played on the television and perform the corresponding fitness movements to achieve the desired fitness effect. In the above scenario, the motion image can be obtained by capturing the user's fitness movements in real time through an image acquisition device set on the television, forming video frame data, and then using the video frame data to obtain the motion image.

[0085] Reference Figure 6 The diagram shown is a schematic diagram of the key points and joint regions of the skeleton. In some embodiments, the distribution of human bones can be divided into at least one skeletal region, and the distribution of human joints can be divided into at least one joint region.

[0086] For example, the human skeleton can be divided into 5 skeletal regions, such as: trunk, left upper limb, right upper limb, left lower limb, and right lower limb. The human joints can be divided into 12 joint regions, such as: left trunk angle, right trunk angle, left elbow angle, left arm angle, right elbow angle, right arm angle, left knee angle, left hip angle, left foot angle, right knee angle, right hip angle, and right foot angle.

[0087] The torso includes: left and right eyes, left and right ears, nose, left and right shoulders, left and right hips; the left upper limb includes: left palm, left wrist, left elbow, and left shoulder; the right upper limb includes: right palm, right wrist, right elbow, and right shoulder; the left lower limb includes: left hip, left knee, and left ankle; and the right lower limb includes: right hip, right knee, and right ankle. Different first weight values ​​are assigned to these five skeletal regions based on the different body parts worked by different fitness movements in the standard movement template. Therefore, when matching each actual keypoint in the sequence to be matched with the theoretical keypoint in the standard movement template, the first weight value corresponding to the actual keypoint can be determined to be equal to the first weight value of the skeletal region to which the actual keypoint belongs.

[0088] Similarly, to more accurately describe the fitness movements in the standard movement template, we introduce the concept of joint regions, referring to... Figure 6 As shown, there are 12 joint angles in total, including A0 left trunk angle, A1 right trunk angle, A2 left elbow angle, A3 left arm angle, A4 right elbow angle, A5 right arm angle, A6 left knee angle, A7 left hip angle, A8 left foot angle, A9 right knee angle, A10 right hip angle, and A11 right foot angle.

[0089] Similarly, based on the five skeletal regions defined by the bone distribution, we also divide the joint angles into these five regions. The trunk includes: A0 (left trunk angle) and A1 (right trunk angle); the left upper limb includes: A2 (left elbow angle) and A3 (left arm angle); the right upper limb includes: A4 (right elbow angle) and A5 (right arm angle); the left lower limb includes: A6 (left knee angle), A7 (left hip angle), and A8 (left foot angle); and the right lower limb includes: A9 (right knee angle), A10 (right hip angle), and A11 (right foot angle). Specific angles are as follows... Figure 6 As shown.

[0090] For example, refer to Figure 7 As shown, the method for extracting skeleton information from an action image and obtaining the first skeleton information corresponding to the action image includes the following steps:

[0091] S5011. Preprocess the frame images in the video frame data to determine the preprocessed motion image.

[0092] In some examples, preprocessing includes at least one of denoising, scaling, distortion correction, and stereo correction.

[0093] S5012. Recognize the motion image and determine the actual detection box corresponding to the human body.

[0094] In some embodiments, the television can project a motion image onto the theoretical detection box corresponding to the current frame image, thereby determining the image within the theoretical detection box. Then, foreground and background separation is performed on the image within the theoretical detection box to obtain the corresponding foreground. Next, image recognition is performed on this foreground to determine if a human body is contained within it. Finally, by recognizing the human body, the smallest rectangle inscribed within it is determined, and this smallest rectangle is used as the actual detection box corresponding to the human body.

[0095] Alternatively, the action image can be separated into foreground and background to determine the foreground. Then, image recognition is performed on this foreground to identify any human figures contained within it. Finally, by recognizing the human figure, the smallest rectangle inscribed within it is determined, and this smallest rectangle is used as the actual detection box corresponding to that human figure.

[0096] Alternatively, the motion image can be input into the human body first target detection network to determine the actual detection box corresponding to the human body.

[0097] S5013. Based on the actual detection box, the frame action image is cropped to obtain the human body image.

[0098] In some embodiments, when cropping a frame image based on an actual detection box, the actual detection box and the frame image are first projected to determine the image of the frame image within the actual detection box. Then, foreground and background separation is performed on the image within the actual detection box to obtain the corresponding foreground. Next, image recognition is performed on this foreground to determine if a human body is contained within it. Finally, the human body contained in the foreground is cropped to obtain a human body image.

[0099] S5014. Recognize the human body image and determine the first skeleton information corresponding to the action image.

[0100] In some embodiments, within each decomposed movement of a fitness exercise, different decomposed movements correspond to different theoretical keypoints. When recognizing a human image, the theoretical image of the decomposed movement corresponding to the current fitness exercise is matched with the human image to determine the best-matching theoretical image. Then, the human image is projected onto the best-matching theoretical image to determine the location point on the human image corresponding to each theoretical keypoint in the theoretical image, and each location point is then used as a target keypoint. Alternatively, the human image is input into a human keypoint detection network to determine at least one target keypoint corresponding to the human body. The human keypoint detection network is trained on a second target detection network based on training sample images and training supervision images.

[0101] In some embodiments, the size of the human body image may differ from the size of the input image to the human body keypoint detection network. In this case, when inputting the human body image into the human body keypoint detection network, it is necessary to determine whether the size of the human body image is the same as the size of the input image. If they are the same, the human body image can be directly input into the human body keypoint detection network. If they are different, the size of the human body image needs to be converted to the same size as the input image of the human body keypoint detection network before the converted human body image is input into the human body first target detection network.

[0102] Specifically, the training process of the human keypoint detection network is as follows:

[0103] Acquire training sample images and training supervision images; both training sample images and training supervision images include human bodies and corresponding key points of the human body.

[0104] The second object detection network is trained based on the training sample images to obtain the pre-trained second object detection network.

[0105] The second object detection network includes at least one of the following: a heatmap-based human skeleton point detection network (High-Resolution Net, HRNet), Simple Baseline (a key point detection framework), or a non-heatmap-based network with argmax as the output.

[0106] The network parameters of the pre-trained second target detection network are adjusted based on the training supervised images until the pre-trained second target detection network converges, thus obtaining the human key point detection network.

[0107] S5015. Based on the theoretical detection box and the actual detection box, the first skeleton information is normalized to determine the coordinate values ​​of at least one skeleton key point.

[0108] In some embodiments, when normalizing the skeleton key points based on theoretical and actual detection frames, the theoretical and actual detection frames can be input into a normalization formula to obtain the skeleton key points. The normalization formula includes:

[0109]

[0110] Among them, the nth skeleton key point S k_n The coordinates are equal to (x k_n y k_n The actual detection box B0 equals [B 0_xmin B 0_ymin B 0_w B 0_h ], r b0 Equal to the actual detection box width B 0_w / Actual detection frame height B 0_h The nth actual key point S Nk_n The coordinates are equal to (x Nk_n y Nk_n ), B 0_xmin and B 0_ymin Let x and y be the coordinates of the top-left corner of the actual detection box B0 when it is mapped onto the frame image, respectively. s It is the normalization coefficient, and the theoretical detection box B1 = [B 1_xmin B 1_ymin B 1_w B 1_h ], a s =(a sx a sy ), a sx = Width B of the theoretical detection frame 1_w The width B of the actual detection frame 0_w The ratio, asy = The theoretical detection frame height B 1_h The actual height B of the detection frame 0_h The ratio of .

[0111] S502. When the first skeleton information matches the template skeleton information of the target action, determine whether the skeleton key points of the first skeleton information contain occlusion points.

[0112] The method for determining whether the first skeleton information matches the template skeleton information of the target action can be as follows:

[0113] The first skeleton information is determined to match the template skeleton information of the target action based on the double-chain Hungarian matching algorithm.

[0114] The bilinked Hungarian matching algorithm is based on the idea of ​​sufficiency proof in Hall's theorem (the theory of the existence of matching in bipartite graphs). It is the most common algorithm for matching bipartite graphs. The core of this algorithm is to find augmenting paths. It is an algorithm that uses augmenting paths to find the maximum matching in a bipartite graph.

[0115] In some embodiments, it is necessary to determine at least one set of matching sequences corresponding to the human body based on the skeletal key points. The actual key points corresponding to the motion image of each frame are pushed onto the stack in the key point buffer register Buffer. When the total number of motion images cached in the Buffer is greater than or equal to the total number of standard motion templates contained in the target sliding window (e.g., 4), the actual key points of the 4 frames cached in the Buffer are retrieved to form the matching sequence.

[0116] Alternatively, the skeleton key points corresponding to one frame image in every M frames of video frame data can be selected to form a sequence to be matched, which can greatly reduce the consumption of computing resources. M is an integer greater than or equal to 1.

[0117] When matching the first skeleton information with the template skeleton information through the target sliding window, it is necessary to count the matching results of each set of sequences to be matched. Since the difference in matching results between adjacent frame images is small, the update logic of the matching results can be set, such as: if the frame image of the current frame is matched with the matching window, if the matching degree of any standard action template in the matching window is greater than the matching degree of the standard action template in the previous frame that is adjacent and continuous to the current frame, and the matching degree of all matching window templates in the matching window before the standard action template is greater than the template matching threshold (e.g., 0.5), then update the matching degree, key point score, and key angle score of the sequence to be matched. If the matching degree of the first c (e.g., c equals 2) standard action templates in the matching window is greater than the template matching threshold, then update the matching window, pop the first two standard action templates in the matching window, and add d (e.g., d equals 2) standard action templates other than those contained in the matching window to the end of the queue. If the matching window contains only two standard action templates, and the matching degree of each standard action template is greater than the template matching threshold, then the sliding window matching is successful. Here, c and d are both integers greater than or equal to 0.

[0118] Specifically, when matching window 0 with the sequence to be matched, and simultaneously matching window 1 with the sequence to be matched, the corresponding doubly-linked Hungarian matching logic is as follows: Figure 8 As shown.

[0119] Step 1: First, at time t0, after obtaining the actual key point SNk_t0 of the current frame image, push the actual key point SNk_t0 of the current frame onto the key point buffer.

[0120] Step 2: When the number of buffered frames is greater than or equal to the matching window length nlen (e.g., 6), retrieve the actual keypoints of 6 frames from the keypoint buffers to form the matching sequence Seqs, i.e., Seqs = [SNk_t0, SNk_t1, SNk_t2, SNk_3, SNk_4, SNk_5]. Simultaneously, retrieve n (e.g., 6) standard action templates from the Action templates according to the order of the fitness movements appearing in the target video, forming matching window 0 and matching window 1. Matching window 0 (SeqT0) = Matching window 1 (SeqT1) = [SNTk_0, SNTk_1, SNTk_2, SNTk_3, SNTk_4, SNTk_5], where SNTk_0 represents the standard action template of frame 0.

[0121] Step 3: Perform a timeout check. If the timer 0 of the matching sliding window 0 does not time out, proceed to (Step 4). If sliding window timer 0 times out, check if sliding window timer 1 is enabled. If sliding window timer 1 is enabled, assign all values ​​of sliding window template SeqT1 to sliding window template SeqT0, reset sliding window template SeqT1, and clear sliding window timer 1. If sliding window timer 1 is not enabled, reset sliding window template SeqT0 and clear sliding window timer 0.

[0122] Step 4: When the 0th template in SeqT0 is equal to SNTk_0, use the KM algorithm (a computer algorithm) to match the sequences to be matched, Seqs and SeqT0, and update SeqT0 after matching.

[0123] Step 5: After SeqT0 is updated, if the state of the matching sliding window 0 changes, reset the sliding window timer 0 and start the timer again.

[0124] Step Six: When the 0th template in SeqT0 is not equal to SNTk_0, first check the state of sliding window timer 1. If sliding window timer 1 times out, reset the matching sliding window 1 and clear sliding window timer 1. If sliding window timer 1 does not time out, use the KM algorithm to match the sequence to be matched Seqs with SeqT0 and SeqT1 respectively. After matching, update SeqT0 and SeqT1 respectively, using the same update strategy as in (Step Four).

[0125] Step 7: After SeqT0 and SeqT1 are updated, if the state of matching window 0 or matching window 1 changes, reset window timer 0 and window timer 1 respectively and start timing. If the number of remaining standard templates in matching window 0 or matching window 1 is 0, it means that the matching window has matched successfully. Therefore, first check matching window 0. If it matches successfully, pass the value of matching window 1 to 0 and give a prompt: Matching window 0 matched successfully. If matching window 0 does not match successfully, but matching window 1 matches successfully, reset matching window 0, matching window 1, and related status registers and timers, and send a prompt that matching window 1 matched successfully.

[0126] In some embodiments, a third average value of the second Pearson product moment corresponding to each actual keypoint is determined based on the second Pearson product moment between each actual keypoint and each standard motion template in the sequence to be matched. Then, the third average value corresponding to each actual keypoint is used to determine a fourth average value corresponding to the sequence to be matched. Finally, the joint angle score is determined as the fourth average value corresponding to the sequence to be matched based on the fourth average value corresponding to the sequence to be matched.

[0127] S503. If the first skeleton key point of the first skeleton information is the occlusion point, then obtain the first region corresponding to the first skeleton key point.

[0128] Wherein, the distance from any point in the first region to the first skeleton key point is less than the threshold distance corresponding to the first skeleton key point.

[0129] In S503 above, the method for obtaining the first region corresponding to the first skeleton key point includes the following steps 1 to 3:

[0130] Step 1: Obtain the distance between the second skeleton key point and the third skeleton key point. The third skeleton key point is the skeleton key point in the template skeleton information that is closest to the second skeleton key point.

[0131] In some embodiments, refer to Figure 9 The image shown is a schematic diagram of the template skeleton information. Figure 9 The diagram shows: [the data includes] the first skeleton keypoint Pa(x) a y a The second skeleton key point P corresponding to the template skeleton information b (x b y b The third skeleton keypoint P, which is closest to the second skeleton keypoint, is... c (x c y c ).

[0132] Step 2: Determine the threshold distance corresponding to the first skeleton key point based on the distance between the second skeleton key point and the third skeleton key point.

[0133] Reference Figure 9 As shown, the distance between the second skeleton key point and the third skeleton key point is r. When W=1, the calculation formula for the threshold distance R is as follows:

[0134]

[0135] Wherein, W is a threshold multiple of the distance between the second skeleton keypoint and the third skeleton keypoint, which is set by the developer and is not limited in this application.

[0136] Step 3: Determine the first region corresponding to the first skeleton key point as the circular region centered on the first skeleton key point and with the threshold distance corresponding to the first skeleton key point as the radius.

[0137] Reference Figure 9 As shown in the figure, the first region corresponding to the first skeleton key point is defined by the first skeleton key point Pa(x). a ya The circular region centered at () and with a radius of the threshold distance R.

[0138] S504. Determine whether the key points of the second skeleton are located within the first region.

[0139] The second skeleton key point is the skeleton key point in the template skeleton information that corresponds to the first skeleton key point.

[0140] In the above S504, when it is determined that the key point of the second skeleton is located in the first region, the following S505 is executed:

[0141] S505. Using the coordinates of the first skeleton key points as the first coordinate values, calculate the motion score of the motion image.

[0142] Wherein, the first coordinate value is the coordinate value of the key point of the second skeleton.

[0143] In some embodiments, refer to Figure 10 The diagram shown illustrates the location of the second skeleton keypoint within the first region, and its relationship to the first skeleton keypoint Pa(x). a y a The second skeleton key point P corresponding to the template skeleton information b (x b y b At this point, the second skeleton key point P b Within the first region, the first coordinate value is (x b y b ).

[0144] As can be seen from the above technical solutions, the motion scoring device and method provided in this application embodiment, the controller, is configured to: extract skeleton information from a motion image to obtain first skeleton information corresponding to the motion image; the first skeleton information includes the coordinate values ​​of at least one skeleton key point; when the first skeleton information matches the template skeleton information of the target motion, determine whether the skeleton key point of the first skeleton information contains an occlusion point; if the first skeleton key point of the first skeleton information is the occlusion point, then obtain a first region corresponding to the first skeleton key point; the distance from any point in the first region to the first skeleton key point is less than the threshold distance corresponding to the first skeleton key point; determine whether a second skeleton key point is located in... Within the first region, the second skeleton key point is the skeleton key point in the template skeleton information corresponding to the first skeleton key point; if so, the coordinate value of the first skeleton key point is used as the first coordinate value to calculate the action score of the action image; the first coordinate value is the coordinate value of the second skeleton key point. Compared with the prior art, when there are a large number of occlusion points, the user cannot obtain an accurate action score. The technical solution of this application determines whether the occlusion point is within the threshold range of the second skeleton key point corresponding to the template skeleton. If it is, the coordinate value of the second skeleton key point is used to replace the occlusion point for action scoring. Therefore, the embodiment of this application can still obtain an accurate action score when there are multiple occlusion points, thus improving the user experience.

[0145] Figure 11 An exemplary flowchart of the action scoring method provided in an embodiment of this application is shown, such as... Figure 11 As shown, the action scoring method provided in this application includes the following steps:

[0146] S1101. Extract skeleton information from the action image to obtain the first skeleton information corresponding to the action image.

[0147] The first skeleton information includes the coordinate values ​​of at least one skeleton key point.

[0148] When the first skeleton information matches the template skeleton information of the target action, the following S1102 is executed:

[0149] S1102. Determine whether the skeleton key points of the first skeleton information contain occlusion points.

[0150] In S1102 above, if the skeleton key point of the first skeleton information includes an occlusion point, and the first skeleton key point of the first skeleton information is the occlusion point, then the following S1103 is executed:

[0151] S1103. Obtain the first region corresponding to the first skeleton key point.

[0152] Wherein, the distance from any point in the first region to the first skeleton key point is less than the threshold distance corresponding to the first skeleton key point.

[0153] S1104. Determine whether the key points of the second skeleton are located within the first region.

[0154] The second skeleton key point is the skeleton key point in the template skeleton information that corresponds to the first skeleton key point.

[0155] In S1104 above, when it is determined that the key point of the second skeleton is located in the first region, the following S1105 is executed:

[0156] S1105. Using the coordinates of the first skeleton key points as the first coordinate values, calculate the motion score of the motion image.

[0157] Wherein, the first coordinate value is the coordinate value of the key point of the second skeleton.

[0158] In S1104 above, if the second skeleton key point is located outside the first region, then S1106 is executed as follows:

[0159] S1106, Obtain the second coordinate value.

[0160] Wherein, the second coordinate value is the coordinate value of the point within the first region that is closest to the second skeleton key point.

[0161] In some embodiments, refer to Figure 12 The image shows the situation where the key points of the second skeleton are outside the first region. Figure 12 In the diagram, Pa' is the point within the first region that is closest to the second skeleton key point. The coordinates of Pa' are the second coordinates, and the formula for calculating the coordinates of Pa' is as follows:

[0162]

[0163] S1107. Using the coordinates of the first skeleton key points as the second coordinates, calculate the motion score of the motion image.

[0164] In S1102 above, if the skeleton key points of the first skeleton information do not include the occlusion points, then S1108 is executed as follows:

[0165] S1108. Calculate the motion score of the motion image based on the coordinate values ​​of each key point of the skeleton in the first skeleton information.

[0166] In the above S1108, refer to Figure 13As shown, the calculation of the motion score of the motion image based on the coordinate values ​​of each key point of the skeleton in the first skeleton information includes the following steps S1301 to S1304:

[0167] S1301. Obtain the angles of each joint angle in the first skeleton information based on the coordinate values ​​of each skeleton key point in the first skeleton information.

[0168] In some embodiments, the angle of the joint angle can be referenced. Figure 6 As shown, for example, Figure 6 Angle A2 in the figure represents the right elbow angle of the human body.

[0169] S1302. Obtain the angle difference of each joint angle in the first skeleton information, wherein the angle difference of any joint angle is the difference between the angle of the joint angle and the angle of the corresponding joint angle in the template skeleton information.

[0170] S1303. Calculate the joint angle score of the motion image of each joint angle based on the angle difference of each joint angle.

[0171] The formula for calculating the joint angle score of the motion image based on the angle difference of each joint angle is as follows:

[0172]

[0173] Where S1 is the joint angle score of the motion image, N is the number of joint angles in the first skeleton information, and Δx i W represents the angle difference of the i-th joint angle. ai is the weighting coefficient for the i-th joint angle.

[0174] S1304. Calculate the motion score of the motion image based on the joint angle score of the motion image.

[0175] Specifically, the base score of the motion image is first calculated according to the following formula:

[0176] S2=min{[Peark*Wpk+min(Peara, Peak)*Wpa], 1)

[0177] Wherein, S2 is the base score of the motion image, Peark is the weighted Pearson product of the template skeleton information after weighting and normalization with the skeleton key points in the first skeleton information, Wpk is the weight coefficient of the skeleton key point Pearson product, Peara is the weighted Pearson product of the template skeleton information after weighting and normalization with the joint angles in the first skeleton information, and Wpa is the weight coefficient of the joint angle Pearson product.

[0178] Secondly, based on the base score and the joint angle score of the motion image, the motion score of the motion image is calculated.

[0179] The motion score of the motion image is calculated according to the following formula:

[0180] S = w * (w1 * S1 + (1 - w1) * S2)

[0181] Wherein, S is the motion score of the motion image; w is the scoring coefficient of the target motion; and w1 is the joint angle score coefficient.

[0182] In some embodiments, refer to Figure 14 As shown, the above method for obtaining joint angle scores and baseline scores may further include the following steps S1401 to S1404:

[0183] S1401. Based on each standard motion template in the target sliding window, each actual key point in the sequence to be matched, and the first weight value of the skeletal region corresponding to the actual key point, perform bipartite graph matching to determine the first Pearson product moment between each actual key point and each standard motion template.

[0184] In some embodiments, the standard action template can be used as the left vertex X, the sequence to be matched can be used as the right vertex Y, and bipartite graph matching can be performed using the first weight value of the skeletal region corresponding to the actual key point, so that the sum of the Pearson product moments of each pair of left and right links XiYi is maximized. Finally, the edges with matching edge weights less than the template matching threshold are removed to obtain the first Pearson product moment of each actual key point and each standard action template.

[0185] Therefore, when matching each actual key point in the sequence to be matched with the theoretical key point in the standard motion template, the second weight value corresponding to the actual key point can be determined to be equal to the first weight value of the bone region corresponding to the joint region to which the actual key point belongs, based on the bone region to which the actual key point belongs.

[0186] S1402. Perform weighted bipartite graph matching based on the second weight value of each standard motion template in the target sliding window, each actual key point in the sequence to be matched, and the joint region corresponding to the actual key point, and determine the second Pearson product moment between each actual key point and each standard motion template.

[0187] In some embodiments, the standard action template can be used as the left vertex X, the sequence to be matched can be used as the right vertex Y, and weighted bipartite graph matching can be performed with the second weight value of the joint region corresponding to the actual key point, so that the sum of the Pearson product moments of each group of left and right links XiYj is maximized. Finally, the edges with matching edge weights less than the template matching threshold are removed to obtain the second Pearson product moment of each actual key point and each standard action template.

[0188] S1403. Determine the key point scores of the sequence to be matched based on the first Pearson product moment.

[0189] In some embodiments, a first average value of the first Pearson product moment corresponding to each actual keypoint is determined based on the first Pearson product moment between each actual keypoint and each standard motion template in the sequence to be matched. Then, a second average value corresponding to the sequence to be matched is determined based on the first average value corresponding to each actual keypoint. Finally, a keypoint score is determined as the second average value corresponding to the sequence to be matched based on the second average value corresponding to the sequence to be matched.

[0190] S1404. Determine the joint angle score of the sequence to be matched based on the second Pearson product moment.

[0191] In some embodiments, a third average value of the second Pearson product moment corresponding to each actual keypoint is determined based on the second Pearson product moment between each actual keypoint and each standard motion template in the sequence to be matched. Then, the third average value corresponding to each actual keypoint is used to determine a fourth average value corresponding to the sequence to be matched. Finally, the joint angle score is determined as the fourth average value corresponding to the sequence to be matched based on the fourth average value corresponding to the sequence to be matched.

[0192] In some embodiments, this application provides a computer-readable storage medium storing a computer program that, when executed by a computing device, causes the computing device to implement the action scoring method described in any of the above embodiments.

[0193] In some embodiments, this application provides a computer program product that, when run on a computer, causes the computer to implement the action scoring method described in the second aspect or any embodiment of the second aspect.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0195] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. An action scoring apparatus, characterized by, The method comprises the following steps: The controller is configured to: Skeleton information extraction is performed on the action image to obtain first skeleton information corresponding to the action image; The first skeleton information comprises coordinate values of at least one skeleton key point; When the first skeleton information matches template skeleton information of a target action, it is determined whether an occlusion point is contained in a skeleton key point of the first skeleton information; If the first skeleton key point of the first skeleton information is the occlusion point, a first region corresponding to the first skeleton key point is obtained; any point in the first region is less than a threshold distance corresponding to the first skeleton key point from the first skeleton key point; It is determined whether a second skeleton key point is located in the first region, the second skeleton key point being a skeleton key point corresponding to the first skeleton key point in the template skeleton information; If yes, a motion score of the action image is calculated by taking the coordinate value of the first skeleton key point as a first coordinate value, the first coordinate value being a coordinate value of the second skeleton key point; The controller is specifically configured to: A distance between the second skeleton key point and a third skeleton key point is obtained, the third skeleton key point being a skeleton key point in the template skeleton information closest to the second skeleton key point; The threshold distance corresponding to the first skeleton key point is determined according to the distance between the second skeleton key point and the third skeleton key point; A circular region with the first skeleton key point as a center and the threshold distance corresponding to the first skeleton key point as a radius is determined as the first region corresponding to the first skeleton key point.

2. The motion scoring device of claim 1, wherein, The controller is further configured to: In a case where the second skeleton key point is located outside the first region, a second coordinate value is obtained, the second coordinate value being a coordinate value of a point closest to the second skeleton key point in the first region; The motion score of the action image is calculated by taking the coordinate value of the first skeleton key point as the second coordinate value.

3. The motion scoring device of claim 1, wherein, The controller is further configured to: In a case where the first skeleton information does not comprise the occlusion point, the motion score of the action image is calculated according to coordinate values of each skeleton key point in the first skeleton information.

4. The movement scoring device of any one of claims 1-3, wherein, The controller is specifically configured to: An angle of each joint angle in the first skeleton information is obtained according to the coordinate values of each skeleton key point in the first skeleton information; An angle difference of each joint angle in the first skeleton information is obtained, the angle difference of any joint angle being a difference between an angle of the joint angle and an angle of a corresponding joint angle in the template skeleton information; The joint angle score of each joint angle in the action image is calculated according to the angle difference of each joint angle; The motion score of the action image is calculated according to the joint angle score of the action image.

5. The action scoring device of claim 4, wherein, The controller is specifically configured to: The joint angle score of the action image is calculated according to the angle difference of each joint angle and the following formula: wherein, is a joint angle score for the motion image, is a number of joint angles in the first skeleton information, is an angle difference of the th joint angle, is a weight coefficient of the th joint angle.

6. The action scoring device of claim 5, wherein, The controller is specifically configured to: The base score of the action image is calculated according to the following formula: wherein, a base score for the action image, a weighted Pearson product moment of the normalized template skeleton information and the skeleton joint in the first skeleton information after weighting, a skeleton joint Pearson product moment weight coefficient, a weighted Pearson product moment of the normalized template skeleton information and the joint angle in the first skeleton information after weighting; a joint angle Pearson product moment weight coefficient; The motion score of the action image is calculated according to the base score of the action image and the joint angle score of the action image.

7. The action scoring device of claim 6, wherein, The controller is specifically configured to: An action score of the action image is calculated according to a formula as follows: wherein, is a motion score for the motion image; is a score coefficient for the target motion, the is a joint angle score score coefficient.

8. The movement scoring apparatus of any of claims 1-3, wherein, The controller is specifically configured to: It is determined whether the first skeleton information matches the template skeleton information of the target action according to a double-chain Hungarian matching algorithm.

9. A method of action scoring, characterized by, It comprises: Skeleton information extraction is performed on the action image to obtain first skeleton information corresponding to the action image; The first skeleton information comprises coordinate values of at least one skeleton key point; When the first skeleton information matches the template skeleton information of the target action, it is determined whether an occlusion point is contained in the skeleton key points of the first skeleton information; If the first skeleton key point of the first skeleton information is the occlusion point, a first region corresponding to the first skeleton key point is obtained; any point in the first region is less than a threshold distance corresponding to the first skeleton key point from the first skeleton key point; It is determined whether a second skeleton key point is located in the first region, the second skeleton key point being a skeleton key point corresponding to the first skeleton key point in the template skeleton information; If yes, an action score of the action image is calculated with coordinate values of the first skeleton key point as first coordinate values; The first coordinate values are coordinate values of the second skeleton key point; The method further comprises: A distance between the second skeleton key point and a third skeleton key point is obtained, the third skeleton key point being a skeleton key point closest to the second skeleton key point in the template skeleton information; A threshold distance corresponding to the first skeleton key point is determined according to the distance between the second skeleton key point and the third skeleton key point; A circular region with the first skeleton key point as a center and the threshold distance corresponding to the first skeleton key point as a radius is determined as the first region corresponding to the first skeleton key point.

Images