Eight-section brocade teaching system based on human body posture recognition

By using user-side video recording and server-side three-dimensional reconstruction technology in the Baduanjin teaching system, the differences between user actions and standard actions are compared and personalized reports are generated, which solves the problem that existing resources are difficult to provide personalized guidance, and improves practitioners standardization and safety of action.

CN119049127APending Publication Date: 2024-11-29BEIFANG UNIV OF NATITIES

Patent Information

Application Number
CN202411158191.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing Baduanjin teaching resources are difficult to provide personalized and immediate guidance, resulting in irregular movements of practitioners, affecting the exercise effect and possibly causing unexpected injuries.

Method used

The user side collects the motion state video and sends it to the server side for three-dimensional motion state reconstruction. Compare the differences in the human key points of the standard Baduanjin movement and the user’s movement, generate a personalized teaching report to guide users to practice correctly.

Benefits of technology

The personalized Baduanjin teaching guidance has been realized, which improves the practitioner's movement normativeness, reduces the risk of injury, and meets the enthusiast's needs for accurate and immediate guidance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049127A_ABST
    Figure CN119049127A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of deep learning, and discloses an eight-segment brocade teaching system based on human body posture recognition, which comprises a user side and a server side, and is characterized in that the user side comprises a user information input module, an eight-segment brocade training guide module and a training video storage and uploading module; the server side comprises an eight-segment brocade action recognition module, a training evaluation module and a training report generation module. Eight-section brocade actions of a user are analyzed based on a user side and a server side, eight-section brocade teaching guidance is achieved, a motion state video of the user is collected at the user side, the collected video is sent to the server side for three-dimensional motion state reconstruction, differences of human body key point positions of the standard eight-section brocade actions and the user actions are compared, and the three-dimensional motion state reconstruction is achieved. And an eight-section brocade personalized teaching report is formed, so that the user is guided to correctly practice the eight-section brocade.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning, and in particular to an Ba ​​Duan Jin teaching system based on human body posture recognition. Background Art

[0002] As a widely acclaimed fitness method, Ba Duan Jin has a unique charm in that it combines physical and mental conditioning with gentle body stretching, and is adapted to the needs of people of different ages. However, in the process of promotion and practice, a problem that cannot be ignored is that many practitioners find it difficult to obtain effective personalized guidance when they learn on their own or practice in small groups.

[0003] The existing guidance resources, whether they rely on remote online platforms or occasionally organized on-site teaching activities, are often limited by time and space, and it is difficult to meet the urgent needs of the majority of enthusiasts for accurate and immediate guidance. This situation may not only lead to irregular movements of practitioners, affecting the exercise effect, but also may cause accidental injuries due to wrong postures, which violates the original intention of Ba Duan Jin to strengthen the body. Summary of the invention

[0004] In order to improve the effect of Ba Duan Jin training, the present invention analyzes the user's Ba Duan Jin movements based on the user end and the server end to implement Ba Duan Jin teaching guidance, collects the user's motion state video at the user end, sends the collected video to the server end for three-dimensional motion state reconstruction, compares the differences in the positions of key points of the human body between the standard Ba Duan Jin movements and the user's movements, and forms a personalized Ba Duan Jin teaching report, thereby guiding the user to correctly practice Ba Duan Jin. For this purpose, a Ba Duan Jin teaching system based on human posture recognition is provided.

[0005] In order to achieve the above-mentioned object of the invention, the embodiment of the present invention provides the following technical solutions:

[0006] Compared with the prior art, the present invention has the following beneficial effects: the present invention analyzes the user's Ba Duan Jin movements based on the user end and the server end to realize Ba Duan Jin teaching guidance, uses the front or rear camera of the mobile phone on the user end to record the user's motion state, sends the collected video to the server end for three-dimensional motion state reconstruction, and after normalizing the model, compares the differences in the positions of key points of the human body between the standard Ba Duan Jin movements and the user's movements to form a personalized Ba Duan Jin teaching report to guide Ba Duan Jin teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0008] Figure 1 It is a block diagram of the system module of the present invention;

[0009] Figure 2 Schematic diagram of the network structure of the target detection module of the present invention;

[0010] Figure 3 A schematic diagram of the network structure of a two-dimensional human posture estimation module of the present invention;

[0011] Figure 4 A schematic diagram of the network structure of a 3D human posture estimation module of the present invention;

[0012] Figure 5 This is a schematic diagram of a training report document of the present invention. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0014] It should be noted that similar numbers and letters represent similar items in the following figures, so once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance, or implying any such actual relationship or order between these entities or operations. In addition, the terms "connected", "connected", etc. can be directly connected between elements, or indirectly connected via other elements.

[0015] The present invention is achieved through the following technical solutions: Figure 1 As shown, a Ba Duan Jin teaching system based on human posture recognition includes a user end and a server end. The user end includes a user information entry module, a Ba Duan Jin training guidance module, and a training video saving and uploading module. The server end includes a Ba Duan Jin motion recognition module, a training evaluation module, and a training report generation module.

[0016] The user information entry module is used to record the user's basic personal information and quickly query the user's training records. Users who use the app for the first time need to register and enter their basic personal information, including name, age, height, weight, etc. After the basic personal information is saved, the user profile will be created in the user database and the user's login status will be saved. When the user uses the app again, there is no need to log in again. In addition to the user's basic personal information, the user database also includes the training report document returned by the server.

[0017] The Ba Duan Jin training guidance module is used to play the corresponding teaching audio and video according to the Ba Duan Jin selected by the user. The Ba Duan Jin training guidance module further includes a Ba Duan Jin action unit selection module and an audio and video playback module. Generally speaking, Ba Duan Jin is composed of eight action units, and the teaching audio and video of each action unit is stored in the audio and video database. After the user fixes the mobile phone, the user can select the Ba Duan Jin action to be trained by operating the Ba Duan Jin action unit selection module on the user end of the mobile phone, and the audio and video playback module starts to play the corresponding teaching audio and video for the user to learn.

[0018] The training video storage and uploading module is used to shoot and record the user's training movements in real time, and upload the training video to the server after recording is completed. When the user is training, the user end on the mobile phone will call the mobile phone camera to shoot and store the user's training video. After the user's training is completed, the user end on the mobile phone will upload the user's training video to the server, and the server will analyze the user's training video to generate guidance and training feedback.

[0019] The Ba Duan Jin action recognition module is used to extract the three-dimensional coordinates of the key points of the skeleton from the training video of the user stored on the server. The Ba Duan Jin action recognition module further includes a target detection module, a two-dimensional human posture estimation module, and a three-dimensional human posture estimation module. The target detection module uses a target detection algorithm to detect the human body position of each frame image where the user is located, and a rectangular frame is used to represent the range of the human body. If the input image width is 608, the height is 608, and it has three color channels of R, G, and B, then the input original image size can be 608*608*3.

[0020] See also Figure 2 , the execution process of the target detection algorithm is:

[0021] (1) The original image is input into the CBL layer, which includes a Conv (convolutional layer), a BN (batch normalization layer), and a Leaky ReLU (activation layer) connected in sequence. The CBL layer expands the channels of the original image, and the size of the original image becomes 608*608*32.

[0022] (2) The multi-channel image is downsampled through the downsampling layers Res1, Res2, Res8, Res8, and Res4 in sequence. After each downsampling process, the width of the image becomes half of the original and the number of channels becomes twice the original. After 5 downsampling processes, the image size obtained is 19*19*1024.

[0023] (3) After five downsampling processes, the image is processed through five CBL layers and then divided into two parts. One part passes through two Conv layers, adjusts the number of channels to 255, and outputs a prediction result of 19*19*255 (size 1); the other part passes through upsampling, doubles the width and height of the image, and adjusts the number of channels, resulting in an image size of 38*38*256.

[0024] (4) The image of size 38*38*256 is concatenated with the image of size 38*38*512 obtained after downsampling layers Res1, Res2, Res8, and Res8 in step (2), and then processed in two parts after passing through five CBL layers. One part passes through two layers of Conv to adjust the number of channels to 255, and outputs a prediction result of size 38*38*255 (size 2); the other part passes through upsampling to double the width and height of the image and adjust the number of channels, and the resulting image size is 76*76*128.

[0025] (5) The image of size 76*76*128 is concatenated with the image of size 76*76*256 obtained after downsampling layers Res1, Res2, and Res8 in step (2), and then passes through five CBL layers and two Conv layers to adjust the number of channels to 255, and outputs a prediction result of size 76*76*255 (size 3).

[0026] The target detection algorithm outputs three feature maps of different scales, and calculates the coordinates b of the bounding box. x 、b y , and width b w , high b h , the position of the bounding box is obtained, and the calculation process is as follows:

[0027] b x =σ(t x )+c x

[0028] b y =σ(t y )+c y

[0029] b w =p w e tw

[0030] b h =p h e th

[0031] Among them, c x 、c y They represent the number of horizontal and vertical grids from the upper left corner of the grid where a point is located to the upper left intersection; p w 、p h Respectively represent the width and height of the prior border; t x ,t y They represent the horizontal and vertical offsets of the target center point relative to the upper left corner of the grid where a certain point is located; t w ,t h They represent the width and height of the predicted bounding box respectively; σ is the activation function. Finally, the final bounding box is selected by sorting the predicted bounding box scores and suppressing non-maximum values.

[0032] The 2D human posture estimation module estimates the 2D coordinates of key points of human skeleton by using heat map detection. Figure 3 , take a 256*256*3 picture as input to explain the specific process:

[0033] (1) The image output by the object detection module is input into two CBL layers in sequence. Each CBL layer includes Conv, BN, and Leaky ReLU connected in sequence. The feature map size output by the second CBL layer is 64*48*64.

[0034] (2) A feature map with a size of 64*48*64 is input into the Layer layer. The Layer layer is stacked by bottleneck layers. The number of channels of the feature map is adjusted to 256, and the output feature map has a size of 64*48*256.

[0035] (3) The feature map output by the Layer layer is input into the Transition1 layer. The Transition1 layer includes two CBL layers. The first CBL layer adjusts the number of channels of the feature map to 32 and outputs a feature map with a size of 64*48*32. The other CBL layer downsamples the feature map and adjusts the number of channels to 64, outputting a feature map with a size of 32*24*64, thereby adding a new scale branch in the Transition1 layer.

[0036] (4) The two-scale feature maps output by the Transition1 layer are input into the Stage2 layer. The Stage2 layer includes two Basic Block layers and two ReLU layers. The small-scale image with a size of 32*24*64 is processed by the first Basic Block layer and output to the first ReLU layer, and is output to the second ReLU layer after upsampling. The large-scale image with a size of 64*48*32 is processed by the second Basic Block layer and output to the second ReLU layer, and is added with the feature map output by the first BasicBlock layer and upsampled. Finally, the second ReLU layer outputs the feature map F2. The large-scale image with a size of 64*48*32 is processed by the second Basic Block layer and then downsampled and output to the first ReLU layer, and is added with the feature map output by the first Basic Block layer to the first ReLU layer. Finally, the first ReLU layer outputs the feature map F1. Note: When each ReLU layer is added, the scales of the two feature maps after upsampling or downsampling are the same, but the scales of the feature maps output by the two ReLU layers are different.

[0037] (5) The two-scale feature maps F1 and F2 output by the Stage2 layer are input into the Transition2 layer. The Transition2 layer includes the CBL layer. The CBL layer downsamples the small-scale feature map F1 to obtain a smaller-scale feature map F3, so that the Transition2 layer outputs images of three scales.

[0038] (6) The feature maps F3, F1, and F2 of three scales output by the Transition2 layer are input into the Stage3 layer. The Stage3 layer includes three Basic Block layers and three ReLU layers. The feature map F3 of the smallest scale is processed by the first Basic Block layer to generate the feature map F3ˋ, which is output to the first ReLU layer. At the same time, it is upsampled to generate the feature map F3ˋˋ, which is output to the second ReLU layer. At the same time, it is upsampled to generate the feature map F3ˋˋ, which is output to the third ReLU layer. The feature map F1 of the second largest scale is processed by the second Basic Block layer to generate the feature map F1ˋ, which is output to the second ReLU layer. At the same time, F1ˋ is downsampled to generate the feature map F1ˋˋ, which is output to the first ReLU layer. At the same time, F1ˋ is upsampled to generate the feature map F1ˋˋˋ, which is output to the third ReLU layer. The feature map F2 of the smallest scale is processed by the third Basic Block layer. After the Block layer is processed, the feature map F2ˋ is generated and output to the third ReLU layer. At the same time, F2ˋ is downsampled to generate the feature map F2ˋˋˋ and output to the second ReLU layer. At the same time, F2ˋ is further downsampled to generate the feature map F2ˋˋ and output to the first ReLU layer. Then F3ˋ, F1ˋˋ, and F2ˋˋ are added in the first ReLU layer, and finally the first ReLU layer outputs the feature map F4. F3ˋˋ, F1ˋ, and F2ˋˋˋ are added in the second ReLU layer, and finally the first ReLU layer outputs the feature map F5. F3ˋˋˋ, F1ˋˋˋ, and F2ˋ are added in the third ReLU layer, and finally the first ReLU layer outputs the feature map F6. Note: When the three feature maps are summed in each ReLU layer, the scales after upsampling or downsampling are the same, while the scales of the feature maps F4, F5, and F6 output by the three ReLU layers are different.

[0039] (7) The three scale feature maps F4, F5, and F6 output by the Stage3 layer are input into the Transition3 layer. The Transition3 layer includes the CBL layer. The CBL layer downsamples the small scale feature map F4 to obtain a smaller scale feature map F7. Thus, the Transition3 layer outputs feature maps of four scales.

[0040] (8) Input the four scale feature maps F7, F4, F5, and F6 output by the Transition3 layer into the Stage4 layer. The Stage4 layer includes four Basic Block layers and four ReLU layers. Similar to steps (4) and (6), the first ReLU layer outputs feature map F8, the second ReLU layer outputs feature map F9, the third ReLU layer outputs feature map F10, and the fourth ReLU layer outputs feature map F11. The specific processing process is not repeated here. Note: When each ReLU layer is added, the scales of the four feature maps after upsampling or downsampling are the same, while the scales of the feature maps F8, F9, F10, and F11 output by the four ReLU layers are different.

[0041] (9) The Stage4 layer also includes 4 Basic Block layers and 1 ReLU layer. The feature maps F8, F9, F10, and F11 of different scales are respectively input into a Basic Block layer, and then the feature maps output by the 4 Basic Block layers are fused in the ReLU layer. A feature map output by the ReLU layer passes through a Conv with a convolution kernel size of 1*1 and 17 convolution kernels to obtain a feature map of size 64*48*17, which represents the heat map of each bone key point.

[0042] The two-dimensional spatial position of each bone key point is found by finding the maximum score value of the heat map. Finally, it is mapped back to the image of the human body area framed by the bounding box to obtain the coordinates of the bone key points on the image. The bone key points are connected in the order of joints to obtain the two-dimensional posture of the human body.

[0043] The 3D human posture estimation module obtains the 3D posture estimation result by using the estimation result output by the 2D human posture estimation module. Figure 4 The 3D human posture estimation module includes: a multi-hypothesis generator module, a self-hypothesis refinement module, and a cross-hypothesis interaction module. The specific process is as follows:

[0044] (1) The Multi-Hypothesis Generation (MHG) module relies on the cascade design of the Transformer to analyze the intrinsic layout of the skeleton key points in each frame. The Transformer Encoder consists of cascaded MLP (Multi-Layer Perceptron) units and MSA (Multi-head self-attention) units. The MHG works at different levels of the network to generate different hypotheses of human posture based on the original posture. The mathematical expression of the MHG unit is described as follows:

[0045]

[0046] in, represents the 2D skeleton coordinates X initial features of the mth hypothesis;

[0047] LN is the normalization layer; J is the number of key points of the human body, N is the number of video frames; X m Represents the initial input of the mth hypothesis. When m>1, its value is When m = 1, its value is X 1 It is the concatenated output of all 2D skeleton coordinates X;

[0048] represents the learnable spatial embedding feature in the m-th hypothesis;

[0049] represents the intermediate feature obtained after the 2D skeleton coordinate X of the mth hypothesis passes through the lth layer of Transformer Encoder unit, l∈[1,2,...,L1], L1 is the number of layers of Transformer Encoder unit, m∈[1,2,...,M], M is the number of hypotheses; represents the intermediate feature obtained after the m-th hypothesized 2D skeleton coordinate X passes through the l-1th layer Transformer Encoder unit;

[0050] MSA m represents the MSA unit of the mth hypothesis; MLP m represents the MLP unit of the mth hypothesis;

[0051] represents the intermediate features output by the MSA unit in the l-th layer Transformer Encoder unit in the m-th hypothesis; X l " m Represents X' l m Intermediate features output by the MLP unit;

[0052] represents the intermediate feature obtained after the m-th hypothesized 2D skeleton coordinate X passes through the L1-th layer Transformer Encoder unit; represents the intermediate features output by the MSA unit in the L1-th layer Transformer Encoder unit in the m-th hypothesis; express Intermediate features output by the MLP unit.

[0053] (2) Introduce the Self-Hypothesis Refinement (SHR) module to optimize the initial hypothesis. This process is divided into two parts to strengthen the consistency and detail processing of time series data. One part is multi-hypothesis self-attention (MH-SA), which sets up a series of parallel self-attention modules, each focusing on deepening the understanding of a single posture hypothesis to form self-hypothesis communication:

[0054]

[0055] in, represents the intermediate features output by the l-th layer SHR module in the m-th hypothesis, l∈[1,2,...,L2], L2 is the number of layers in the SHR module, m∈[1,2,...,M], M is the number of hypotheses; represents the intermediate features output by the l-1th layer SHR module in the mth hypothesis;

[0056] Represents the initial input of the SHR module; represents the output features of the l-1th layer SHR module in the mth hypothesis; Represents the final output of the multi-hypothesis generator module High-dimensional features obtained through transposition and linear embedding;

[0057] represents the learnable temporal embedding feature in the m-th hypothesis.

[0058] The other part is the Hypothesis-Mixing Multi-LayerPerceptron (MLP), which combines the features of all hypotheses and uses the Hypothesis-Mixing MLP to learn their comprehensive information. Then, the features are evenly divided into non-overlapping blocks along the channel dimension to form a more refined hypothesis representation:

[0059]

[0060] in, Represents the intermediate features obtained after multiple hypotheses at the lth layer are fully spliced; represents the intermediate features obtained by the l-th layer MLP in the first hypothesis; It represents the intermediate feature obtained by the Mth layer MLP in the lth hypothesis; Concat represents the concatenation operation; N is the number of video frames, C is the embedding dimension, M is the number of hypotheses, and C multiplied by M constitutes the size of the feature.

[0061]

[0062] in, represents the output feature of the l-th layer SHR module in the first hypothesis, represents the output features of the l-th layer SHR module in the M-th hypothesis; It represents the final output of the l-th layer SHR module; HMMLP represents the hypothesized hybrid multi-layer perceptron.

[0063] (3) The cross-hypothesis interaction (CHI) unit is used to explore and exploit the interactions between different pose hypotheses. Each level of this unit consists of a multi-hypothesis cross-attention (MH-CA) and an MLP applied again. MH-CA uses a set of parallel cross-attention units to synchronously capture the interdependencies between different hypotheses and promote interactive communication between hypotheses:

[0064]

[0065] Among them, Z' l m represents the intermediate features output by the MCA module in the l-th layer CHI unit in the m-th hypothesis, l∈[1,2,...,L3], L3 represents the number of CHI unit layers, m∈[1,2,...,M], M represents the number of hypotheses; MCA m represents the mth hypothesized MCA module; represents the output feature of the l-1th layer CHI unit in the mth hypothesis, and m1 and m2 represent the two hypotheses adjacent to m.

[0066] (4) Finally, by applying a linear transformation layer on the features generated by the CHI unit, the model is able to output the predicted 3D human pose results.

[0067] The training evaluation module is used to determine whether the action is standard according to the three-dimensional coordinates of the skeleton key points and generate an action evaluation. The training evaluation module further includes a size normalization module, a key action extraction module, an action scoring module, and an action correction module.

[0068] Since the distance between the user and the camera is not fixed, it is necessary to consider the height and posture differences between the user's action and the standard action, and standardize the body shape of the skeleton key points. The size normalization module uses the height ratio from the neck to the ground in the standing action of the two action sequences to scale the action in proportion. The calculation formula of the scaling factor K is:

[0069]

[0070] Among them, h neck h is the height of the neck key point from the ground during the user's action. l_ankleh is the height of the left ankle key point from the ground during the user's action. r_ankle The distance between the right ankle key point and the ground during the user's action

[0071] ``height; h neck h is the height of the key point of the neck from the ground in the standard action, l_ankle In standard action

[0072] `

[0073] The height of the left ankle key point from the ground, h r_ankle It is the height of the right ankle key point from the ground in the standard movement.

[0074] A Ba Duan Jin exercise will include hundreds of frames of action. The number of frames of standard action and user action is different, and there is a delay between user action and standard action. Analyzing each action will not only increase the complexity of the system but also reduce the robustness of the system. Therefore, the key action extraction module is used to extract the key actions, calculate the joint angles of the key points of the human skeleton in the user action and the standard action, and weight them to obtain the similarity evaluation index q:

[0075]

[0076] Among them, k is the index of the skeleton key point, x k is the joint angle, ω k is the weight of the key point k of the skeleton, which is adjusted according to different actions in Ba Duan Jin. Assuming there are 7 skeleton joints,

[0077] The dynamic time warping method (DTW) is used as the comparison method between user actions and standard actions. The calculation method is:

[0078]

[0079] in, is the standard action similarity evaluation index of the i-th frame, and the sequence length is m; is the standard action similarity evaluation index of the j-th frame, and the sequence length is n.

[0080] Taking the Ba Duan Jin action "Hands Lifting the Sky Like a Star" as an example, this action can be decomposed into three stages, namely "lifting up", "hands to the top of the head" and "falling down". The user action is divided into three segments according to the time ratio of the three parts of the standard action, corresponding to the three actions of "lifting up", "hands to the top of the head" and "falling down". In each subdivided action, the DTW method is used to calculate the key frame with the highest similarity between the user action and the standard action as the key action.

[0081] The action scoring module uses a weighted calculation method of the overall similarity of the action sequence and the difference in the angles of each joint for the normative analysis of key actions. On the basis of the calculation of the joint angles, the overall characteristics of the action are added to improve the accuracy of the calculated similarity. The expression of the overall characteristics of the action is expressed by calculating the key point similarity (OKS, ObjectKeypoint Similarity) between the action sequences. For the predicted key points of a pair of real key points, the Euclidean distance between them is calculated, and the Euclidean distance is converted into an OKS score to measure the accuracy of the predicted key points. The value range of OKS is 0 to 1, indicating that the accuracy of the predicted key points ranges from the worst to the best, and the calculation formula is:

[0082]

[0083] Among them, N is the number of human skeleton key points, k is the skeleton key point index; d k is the Euclidean distance between the skeleton key points of the user action and the skeleton key points of the standard action; s is the scale factor of the test action, and its value is the square root of the area of ​​the human body detection box; γ k is the scale factor of the skeleton key point k, which is used to express whether the labeled skeleton key points are true when measuring the model accuracy. When calculating the similarity of the skeleton key point sequence, γ can be used k Set to a constant, here set to 1.

[0084] When judging the standardization of human posture, the action scoring module cannot simply rely on the positional relationship of the key points of the human skeleton, but should also pay attention to whether the joint angles of the action are standard. This scheme uses cosine similarity to measure the standardization of each joint angle. Cosine similarity is usually used to measure the difference in direction between two vectors. Each action is assigned a feature, and the feature similarity C is used to evaluate the standardization of the action. The feature similarity C is:

[0085]

[0086] Among them, k is the skeleton key point index, r is the action index, C kr is the feature similarity of the kth skeleton key point in the rth action; d is the overall similarity of the action; Δcosα k is the cosine similarity of the skeleton key point k; λ∈[0,1] represents the proportion of the action joint angle feature in the action similarity. When λ=0, the feature similarity only expresses the overall feature of the action, and when λ=1, the feature similarity only expresses the angle feature of the action.

[0087] The scores of different joints of an action are weighted to obtain the final action score, which is expressed as:

[0088]

[0089] Among them, SUM r represents the weighted score of the rth action; ω k is the weight of the skeleton key point k.

[0090] The motion correction module first generates a user joint motion image with human skeleton key points; then it evaluates the score of each joint. If the joint score is less than 90 points, the joint is recorded as an incorrect motion joint; finally, the standard motion key points corresponding to the incorrect motion joint are drawn on the user joint motion image to provide the user with a reference for motion correction.

[0091] The training report generation module is used to generate a training report document based on the action evaluation, and transmit the training report document back to the user end for the user to view. Figure 5 As described above, the training report document contains the overall action score and action guidance pictures, "1" is the user's comprehensive score, "2" is the training time, "3" is the user's image screenshot, "4" is the user's action skeleton key points, "5" is the correction action key points, and "6" is the score of each joint.

[0092] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. The Ba Duan Jin teaching system based on human posture recognition includes a user end and a server end, and is characterized by: The user terminal comprises: User information entry module, used to record the user's basic personal information and quickly query the user's training records; Ba Duan Jin training guidance module, used to play the corresponding teaching audio and video according to the Ba Duan Jin selected by the user; The training video saving and uploading module is used to shoot and record the user's training movements in real time, and upload the training video to the server after recording is completed; The server side includes: The Ba Duan Jin action recognition module is used to extract the three-dimensional coordinates of the skeleton key points from the user's training video stored on the server; The training evaluation module is used to determine whether the action is standard based on the three-dimensional coordinates of the skeleton key points and generate action evaluation; The training report generation module is used to generate a training report document based on the action evaluation and transmit the training report document back to the user end for the user to view.

2. The Ba Duan Jin teaching system based on human posture recognition according to claim 1 is characterized by: The Ba Duan Jin action recognition module includes: The target detection module is used to detect the human body position of the user in each frame of the image using a target detection algorithm; The 2D human posture estimation module is used to estimate the 2D coordinates of key points of the human skeleton using thermal image detection. The three-dimensional human posture estimation module is used to obtain a three-dimensional posture estimation result using the estimation result output by the two-dimensional human posture estimation module.

3. The Ba Duan Jin teaching system based on human posture recognition according to claim 2 is characterized by: The execution process of the target detection algorithm is as follows: (1) The original image is input into the CBL layer, which includes Conv, BN and Leaky ReLU connected in sequence. The CBL layer expands the channels of the original image, and the size of the original image becomes 608*608*32; (2) The multi-channel image is downsampled through the downsampling layers Res1, Res2, Res8, Res8, and Res4 in sequence. After each downsampling process, the width of the image is halved and the number of channels is doubled. After five downsampling processes, the image size is 19*19*1024. (3) After 5 times of downsampling, the image is processed through 5 CBL layers and then divided into two parts. One part passes through two layers of Conv to adjust the number of channels to 255 and output a prediction result of 19*19*255. The other part passes through upsampling to double the width and height of the image and adjust the number of channels. The resulting image size is 38*38*256. (4) The image with a size of 38*38*256 is concatenated with the image with a size of 38*38*512 obtained after the downsampling layers Res1, Res2, Res8, and Res8 in step (2), and then processed in two parts after passing through five CBL layers. One part passes through two layers of Conv to adjust the number of channels to 255, and outputs a prediction result with a size of 38*38*255; the other part passes through upsampling to double the width and height of the image and adjust the number of channels, and the resulting image size is 76*76*128; (5) The image with a size of 76*76*128 is concatenated with the image with a size of 76*76*256 obtained after the downsampling layers Res1, Res2, and Res8 in step (2), and then passes through five CBL layers and two Conv layers to adjust the number of channels to 255, and output a prediction result with a size of 76*76*255; The target detection algorithm outputs three feature maps of different scales, and calculates the coordinates b of the bounding box. x 、b y , and width b w , high b h , the position of the bounding box is obtained, and the calculation process is as follows: b x =σ(t x )+c x b y =σ(t y )+c y Among them, c x 、c y They represent the number of horizontal and vertical grids from the upper left corner of the grid where a point is located to the upper left intersection; p w 、p h Respectively represent the width and height of the prior border; t x ,t y They represent the horizontal and vertical offsets of the target center point relative to the upper left corner of the grid where a certain point is located; t w ,t h Respectively represent the width and height of the predicted bounding box; σ is the activation function; Finally, the final bounding box is selected by sorting the predicted bounding box scores and suppressing non-maximum values.

4. The Ba Duan Jin teaching system based on human posture recognition according to claim 2 is characterized by: The three-dimensional human posture estimation module includes a multi-hypothesis generator unit, a self-hypothesis refinement unit, and a cross-hypothesis interaction unit.

5. The Ba Duan Jin teaching system based on human posture recognition according to claim 4 is characterized by: The multi-hypothesis generator unit relies on the cascade design of Transformer to analyze the internal layout of the skeleton key points in each frame and generate different hypotheses of human posture based on the original posture. The mathematical expression is as follows: in, represents the 2D skeleton coordinates X initial features of the mth hypothesis; LN is the normalization layer; J is the number of key points of the human body, N is the number of video frames; X m Represents the initial input of the mth hypothesis. When m>1, its value is When m=1, its value is X 1 It is the concatenated output of all 2D skeleton coordinates X; represents the learnable spatial embedding feature in the m-th hypothesis; represents the intermediate feature obtained after the 2D skeleton coordinate X of the mth hypothesis passes through the lth layer of Transformer Encoder unit, l∈[1,2,...,L1], L1 is the number of layers of Transformer Encoder unit, m∈[1,2,...,M], M is the number of hypotheses; represents the intermediate feature obtained after the m-th hypothesized 2D skeleton coordinate X passes through the l-1th layer Transformer Encoder unit; MSA m represents the MSA unit of the mth hypothesis; MLP m represents the MLP unit of the mth hypothesis; represents the intermediate features output by the MSA unit in the l-th layer Transformer Encoder unit in the m-th hypothesis; Represents X' l m Intermediate features output by the MLP unit; represents the intermediate feature obtained after the m-th hypothesized 2D skeleton coordinate X passes through the L1-th layer Transformer Encoder unit; represents the intermediate features output by the MSA unit in the L1-th layer Transformer Encoder unit in the m-th hypothesis; express Intermediate features output by the MLP unit.

6. The Ba Duan Jin teaching system based on human posture recognition according to claim 4 is characterized by: The self-hypothesis refinement unit optimizes the preliminary hypothesis by using two parts: multi-hypothesis self-attention and hypothesis mixing multi-layer perceptron; The multi-hypothesis self-attention setting has self-attention modules operating in parallel, each focusing on deepening the understanding of a single pose hypothesis, forming self-hypothesis communication: in, represents the intermediate features output by the l-th layer SHR module in the m-th hypothesis, l∈[1,2,...,L2], L2 is the number of layers in the SHR module, m∈[1,2,...,M], M is the number of hypotheses; represents the intermediate features output by the l-1th layer SHR module in the mth hypothesis; Represents the initial input of the SHR module; represents the output features of the l-1th layer SHR module in the mth hypothesis; Represents the final output of the multi-hypothesis generator module High-dimensional features obtained through transposition and linear embedding; represents the learnable temporal embedding feature in the m-th hypothesis; Hypothesis hybrid multilayer perceptron merges all hypothesized features and evenly divides the features into non-overlapping blocks along the channel dimension. The hypothesis formed is expressed as: in, Represents the intermediate features obtained after multiple hypotheses at the lth layer are fully spliced; represents the intermediate features obtained by the l-th layer MLP in the first hypothesis; represents the intermediate features obtained by the Mth layer MLP in the lth hypothesis; Concat represents the concatenation operation; N is the number of video frames, C is the embedding dimension, and M is the number of hypotheses; in, represents the output feature of the l-th layer SHR module in the first hypothesis, represents the output features of the l-th layer SHR module in the M-th hypothesis; It represents the final output of the l-th layer SHR module; HMMLP represents the hypothesized hybrid multi-layer perceptron.

7. The Ba Duan Jin teaching system based on human posture recognition according to claim 6 is characterized by: The cross-hypothesis interaction unit synchronously captures the interdependencies between different hypotheses through a set of parallel cross-attention units: in, represents the intermediate features output by the MCA module in the l-th layer CHI unit in the m-th hypothesis, l∈[1,2,...,L3], L3 represents the number of CHI unit layers, m∈[1,2,...,M], M represents the number of hypotheses; MCA m represents the mth hypothesized MCA module; represents the output feature of the l-1th layer CHI unit in the mth hypothesis, and m1 and m2 represent the two hypotheses adjacent to m.

8. The Ba Duan Jin teaching system based on human posture recognition according to claim 1 is characterized by: The training evaluation module includes a size normalization module, a key action extraction module, an action scoring module, and an action correction module.

9. The Ba Duan Jin teaching system based on human posture recognition according to claim 8 is characterized by: The key action extraction module extracts key actions, calculates and weights the joint angles of key points of human skeletons in user actions and standard actions, and obtains a similarity evaluation index q: Among them, k is the index of the skeleton key point, x k is the joint angle, ω k is the weight of the skeleton key point k, which is adjusted according to different movements in Ba Duan Jin. N is the total number of skeleton key points; Use the dynamic time warping method as a comparison method between user actions and standard actions. The calculation method is: Among them, DTW is the dynamic time warping method; is the standard action similarity evaluation index of the i-th frame, and the sequence length is m; is the standard action similarity evaluation index of the j-th frame, and the sequence length is n.

Citation Information

Patent Citations

  • Multi-person evaluation system based on machine vision skeletal tracking

    CN115482580A

  • Arthrophlogosis rehabilitation training guidance device and system

    CN116959664A

  • 3D-CNN motion recognition method based on skeleton

    CN117409475A

  • Method and device for intelligent estimation of human body movement posture based on convolutional neural network

    WO2022036777A1

Cited By

  • RLHF-based motion similarity score optimization method

    CN121686568A

  • Human body action feature symbol sequence coding method and action scoring method

    CN122637474A