A balance beam game feedback system and feedback method based on deep photography technology

Abstract

The application discloses a balance beam game feedback system and feedback method based on a depth photography technology, and the method comprises the following steps: Step 1, a depth camera is used to shoot photos of various actions of a gymnast during balance beam training, an action data set of the gymnast is prepared, and the data is subjected to enhancement processing; Step 2, an nnformer is used as a gymnast bonus action segmentation model, and the action segmentation model is trained by using supervised learning; Step 3, the supervised learning method is improved, and the improved model is trained; Step 4, the trained submodel is used for image segmentation; and Step 5, the action intelligent score correction is performed in combination with a game feedback system. The improved supervised learning method is proposed, computer vision is combined, the accuracy is higher, the inference time is not increased, and the practicability and generalization of the model are higher. In addition, the game feedback system is combined, the training interest of the gymnast can be increased, and the application in life can be realized.

Description

TECHNICAL FIELD

[0001] The application relates to a balance beam game feedback system and a feedback method based on a depth photography technology, and belongs to the technical field of intelligent systems. BACKGROUND

[0002] AI intelligence has begun to be integrated into people's life. In recent years, with the rise of sports, many artificial intelligence products have begun to be applied to the training of national athletes. Conventional training uses a large amount of manpower and financial resources, and cannot be processed in detail. The repetitive training is boring and will also reduce the enthusiasm of athletes. The traditional boring balance beam training method is not only difficult, but also requires strict action requirements, and a more accurate way is needed to quickly improve the athletes. And the training method based on the game feedback system needs to be integrated into the boring training. SUMMARY

[0003] In view of the problems existing in the prior art, the application provides a balance beam game feedback system and a feedback method based on a depth photography technology, so as to solve the above technical problems.

[0004] In order to achieve the above purpose, the technical scheme adopted by the application is: a balance beam game feedback system based on a depth photography technology, comprising an intelligent balance beam, a voice broadcast device assembled on the lower side of the balance beam, front and side depth cameras, a display screen and a computer; the intelligent balance beam is further provided with a pressure sensor connected with an induction lamp, which is used to display the correct center of gravity placement position; the voice broadcast device is a loudspeaker, which is used to broadcast the scoring points and the reward mechanism; the depth camera is used to shoot the scoring key points of the details of the athlete's action, and real-time data is returned to the computer; the display screen is used to real-time feedback to the athlete various data; the computer is used to process and analyze the images collected by the depth camera.

[0005] A method for a balance beam game feedback system based on a depth photography technology, comprising the following steps:

[0006] Step S1: photographing the photos of various actions of the gymnast during the balance beam training by the depth camera (and leg back handstand (small flip), back handstand two feet falling (back handstand), kicking leg back handstand, kicking, leg forward handstand, forward handstand two feet falling, etc.), making a data set of the gymnast's air action, and enhancing the data set;

[0007] Step S2, using nnformer as a gymnast scoring action segmentation model, and using supervised learning to train the action segmentation model;

[0008] Step S3, improving the supervised learning method, and training the improved model.

[0009] Step S4, using the trained sub-model for image segmentation.

[0010] Step S5, combined with the game feedback system, intelligent scoring and prompting.

[0011] Further, in step S1: making data sets, the data set label making method is to process the picture by mask method, fill the picture with 0, 1, etc., so that the picture presents a pure black state; in order to make the picture visible, assign the points with pixel value 1 to 255, that is, change the 0 and 1 picture to 1, 255 image, and the label becomes a black and white photo at this time; set the region of interest as the arm, leg and waist part to achieve higher prediction effect.

[0012] Further, in step S1: enhancing the data set, by setting random horizontal flip, randomly adjusting saturation, randomly adjusting brightness, randomly losing image area, image enlargement, and adding salt and pepper noise to the pixel dimension to realize the enhancement of the data set.

[0013] Further, randomly lose the image area, randomly set part of the area on the picture to 0, and then train, the random area loss can be similar to random noise, which can enhance the robustness of the model and avoid overfitting; enlarge the image, and the image enlargement ratio is appropriate 1.2-1.5.

[0014] Further, in step 3, the motion segmentation model is improved by supervised learning training, and the specific operation is to obtain the 3D action graph of the athlete by deep camera and make data set;

[0015] First, pre-process the image, send the pre-processed 3D image file into the model for training, calculate the segmentation score ADice and Score of the important parts such as arm, leg and waist that are concerned in training; then perform verification test, send the aerial action of the athlete into the trained model for ADice and Score calculation.

[0016] Further, in step S3, the trained model is deployed to the mobile end, and the athlete's action is photographed, the athlete's photo is input into the training network for testing, and the score is calculated according to formulas (1) and (2);

[0017]

[0018]

[0019] In formula (1), is a foreground score, formula (2) is M ADice average score, M is the number of input body part indicators, Score is the total score of the athlete, the scores of different parts are obtained according to ADice, and then voice prompts are given through a mobile terminal.

[0020] Further, the motion intervention closed-loop process module is further included, and the specific process of the module is that training actions are set in advance, different actions are added in each training, and the athlete trains each action in a game through a game score mode; the coach monitors the action indicator score of the athlete in real time through the computer management module, and realizes real-time recording and strengthening training; the athlete breaks through the game according to the broadcast system and the game mode.

[0021] Further, in step 3, the improved model is trained by adopting multi-fold training, the number of iterations of each fold training is 40000 times, the learning rate is reduced, the initial learning rate is 0.005, attenuation is performed once every 4000 iterations, and a multi-online cascade training mode is adopted.

[0022] The application has the advantages that: the improved supervised learning method is provided, combined with computer vision, compared with the traditional supervised model, the method has higher accuracy, does not increase the reasoning time, has higher practicability and generalization, and can make the athlete improve in a more accurate way based on the training mode given by the game feedback system, and improve the training enthusiasm of the athlete. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 It is a structural schematic diagram of the application;

[0024] Figure 2 It is an nnformer network architecture diagram of the application;

[0025] Figure 3 It is a system hardware schematic diagram of the application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the application clearer and more clear, the application is further described in detail below through the drawings and examples. However, it should be understood that the specific examples described herein are only used to explain the application, and are not used to limit the scope of the application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs, and the terms used in the specification of the application herein are only for the purpose of describing specific examples and are not intended to limit the application.

[0028] As Figure 1 , Figure 2 and Figure 3 indicated, a balance beam game feedback system based on depth photography technology includes a smart balance beam, a voice broadcast device assembled on the lower side of the balance beam, front and side depth cameras, a display screen, and a computer; a pressure sensor is further arranged on the smart balance beam, which is connected with an induction lamp for displaying the correct center of gravity placement position; the voice broadcast device is a loudspeaker for broadcasting score points and reward mechanisms; the depth camera is used for photographing scoring key points of detailed actions of athletes and real-time data feedback to the computer; the display screen is used for real-time feedback of various data to athletes; and the computer is used for processing and data analysis of images collected by the depth camera.

[0029] A feedback method of the balance beam game feedback system based on depth photography technology according to claim 1, comprising the following steps:

[0030] Step S1: photographing photos of various actions of a gymnast during balance beam training (such as back handspring with legs together, back handspring with legs falling one after another, kicking leg back handspring, kick, front handspring with legs together, and front handspring with legs falling one after another) by a depth camera, making a data set of the air actions of the gymnast, and enhancing the data set;

[0031] Step S2: using nnformer as a gymnast scoring action segmentation model, and training the action segmentation model by supervised learning; nnformer is a currently efficient image segmentation method, and although it is currently applied to the medical field more, its efficient segmentation method is applied to various segmentation fields;

[0032] Step S3: improving the supervised learning method and training the improved model; different image preprocessing and image enhancement methods are adopted for training. For random horizontal flipping of the image position, the flipping p is 0.5, and half of the images are randomly flipped. The saturation of the image is adjusted. factor=random.uniform(0.8,1.2) The parameter interval of the saturation setting is 0.8 to 1.2 times of the original image. Randomly increase the brightness factor=random.uniform(0.8,1.2), and the parameter variation interval of the brightness setting is 0.8 to 1.2 times of the original image. Randomly lose the region of the image, and the loss probability p is set to 0.06-0.08, and the size n of the lost part is set to 0.1% of the original image size. Pepper and salt noise is also used in the present application. The above data initialization and enhancement of the original image can avoid overfitting during training and enhance the robustness;

[0033] Step S4, using the trained sub-model for image segmentation.

[0034] Step S5, combined with the game feedback system, intelligent scoring and prompting.

[0035] Preferably in step S1 of the embodiment, the data set is made, and the data set labels are made by using the mask method to process the picture and filling the picture with 0, 1, etc., so that the picture presents a pure black state; in order to make the picture visible, the points with a pixel value of 1 are assigned a value of 255, that is, the 0 and 1 picture is changed into a 1, 255 image, and the labels become black and white photos at this time; the region of interest is set as the arm, leg, and waist part to achieve higher prediction effect.

[0036] Preferably in step S1 of the embodiment, the data set is enhanced by setting random horizontal flip, randomly adjusting saturation, randomly adjusting brightness, randomly losing a region of the image, enlarging the image, and adding salt and pepper noise to the pixel dimension to realize the enhancement of the data set.

[0037] Preferably in the embodiment, the image is randomly lost in a region, and a part of the region on the picture is randomly set to 0, and then trained; the random loss of the region can be similar to random noise, which can enhance the robustness of the model and avoid overfitting; the image is enlarged, and the image enlargement ratio is appropriately 1.2-1.5.

[0038] Preferably in step 3 of the embodiment, the motion segmentation model is improved by supervised learning training, and the specific operation is to obtain the 3D motion graph of the athlete by a depth camera and make a data set;

[0039] First, the image is preprocessed, and the preprocessed 3D image file is sent into the model for training to calculate the segmentation scores ADice and Score of the important parts such as the arm, leg, and waist that are concerned about training; then, the verification test is performed, and the athlete's in-air action is photographed and sent into the trained model to calculate ADice and Score.

[0040] Preferably in step S3 of the embodiment, the trained model is deployed to the mobile end, and the athlete's action is photographed, the athlete's photo is input into the trained network for testing, and the scores are calculated according to the formulas (1) and (2).

[0041]

[0042]

[0043] In formula (1), is a foreground score, formula (2) is M ADice average score, M is the number of input body part indicators, Score is the total score of the athlete, the scores of different parts are obtained according to ADice, and then voice prompts are given through a mobile terminal.

[0044] The embodiment preferably further comprises a sports intervention closed-loop process module, and the specific process of the module is: training actions are set in advance, different actions are added in each training, and the athlete trains each action in a game through game score mode; the coach monitors the action indicator score of the athlete in real time through the computer management module, and achieves real-time recording and strengthening training; the athlete breaks through the game according to the broadcast system and the game mode.

[0045] In the step 3, the improved model is trained by adopting multi-fold training, the iteration number of each fold training is 40000 times, the learning rate is reduced, the initial learning rate is 0.005, the learning rate is attenuated once every 4000 iterations, and a multi-online cascade training mode is adopted.

[0046] In the application, the athlete stands on the balance beam and makes corresponding actions, the front and side of the athlete have depth photography equipment, the actions of the athlete are recorded and input into the computer.

[0047] The photography equipment transmits data in real time, and the action picture of the athlete is input into the pre-trained model for prediction.

[0048] Through continuous training of the athlete, the results of each training are input into the computer and analyzed, and summarized into a database, and when the athlete exercises, the computer automatically analyzes the action of the athlete, such as the wrong position of the center of gravity, and the broadcast system will broadcast to remind the athlete.

[0049] When the athlete learns a new action, the database will also analyze the common errors that athletes make when doing this action, and the sensing lamp connected to the pressure sensor on the balance beam will display the correct center of gravity position, and tell the athlete where to place the center of gravity to do the action correctly.

[0050] For game mode specific analysis, the athlete or ordinary player can change the training mode to training game mode, and the training game mode will throw the game image and the 3D modeling portrait of the athlete on the large screen. In the game training, the experience value will be set. Initially, the athlete or ordinary player can only start from simple actions, such as stable execution, and each task has different difficulty, which is set according to the training needs of the athlete or ordinary player. For example, jumping forward, combined with the pressure sensor, the difficulty is increased, the computer sets the walking area to a narrow range, and the position area of the two feet of the athlete or ordinary player will also become smaller, and the training difficulty is increased. For example, back somersault, the athlete will see the curve set by the game in the large screen, and the athlete will perform back somersault and obtain experience value.

[0051] The above only describes the preferred embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement or improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A balance beam game feedback method based on depth photography technology, characterized in that, Comprise the following steps: Step S1: taking photos of various actions of a gymnast during balance beam training by a depth camera, wherein the actions include parallel leg backward handspring, backward handspring with feet falling in turn, kicking leg backward handspring, kicking, parallel leg forward handspring, and forward handspring with feet falling in turn, making a data set of the gymnast's actions in the air, and enhancing the data set; Step S2: using nnformer as a gymnast's bonus action segmentation model, and training the action segmentation model by supervised learning; Step S3: improving the supervised learning method, and training the improved model; In step S3, the action segmentation model is improved by supervised learning, and the specific operation is to obtain a 3D action image of the gymnast by a depth camera, and make a data set; First, the image is preprocessed, and the preprocessed 3D image file is sent into the model for training to calculate the segmentation scores ADice and Score of the arms, legs, and waist of the body parts concerned; then, the athlete's action in the air is tested by sending the image into the trained model to calculate ADice and Score; In step S3, the trained model is deployed to the mobile terminal, and the athlete's action is photographed, and the athlete's photo is input into the training network for testing, and the scores are calculated according to formulas (1) and (2); In formula (1), the foreground score is calculated, and in formula (2), M is the number of input body part indicators, Score is the total score of the athlete, and the scores of different parts are obtained according to ADice, and then the mobile terminal is used for voice prompting; Step S4: using the trained sub-model for image segmentation; Step S5: combining the game feedback system to score the action intelligently and prompt.

2. The balance beam game feedback method based on depth photography technology according to claim 1, characterized in that, In step S1: the data set is made, and the data set labels are made by using the mask method to process the picture, filling the picture with 0 and 1, so that the picture presents a pure black state; in order to make the picture visible, the points with pixel value of 1 are assigned a value of 255, that is, the 0 and 1 picture is changed into a 1, 255 image, and the labels become black and white photos; the regions of interest are set as arms, legs, and waist parts to achieve higher prediction effect.

3. The balance beam game feedback method based on depth photography technology according to claim 1, characterized in that, In step S1: the data set is enhanced by setting random horizontal flip, randomly adjusting saturation, randomly adjusting brightness, randomly losing regions of the image, enlarging the image, and adding salt and pepper noise to the pixel dimension to realize the enhancement of the data set.

4. The balance beam game feedback method based on depth photography technology according to claim 3, characterized in that, Randomly losing regions of the image, randomly setting part of the region on the picture to 0, and then training, the random region loss can be similar to random noise, which can enhance the robustness of the model and avoid overfitting; the image is enlarged, and the appropriate image enlargement ratio is 1.2-1.

5.

5. The balance beam game feedback method based on depth photography technology according to claim 1, characterized in that, The motion intervention closed loop process module comprises the following specific process: training actions are set in advance, different actions are added in each training, and athletes train each action through game score mode in a game; a coach monitors action index score of the athletes in real time through a computer management module, and records and strengthens training in real time; the athletes play a game according to a broadcast system and a game mode.

6. The balance beam game feedback method based on depth photography technology according to claim 1, characterized in that, In the step S3, the improved model is trained by adopting multi-fold training, each fold training iteration is 40000 times, the learning rate is reduced, the initial learning rate is 0.005, the learning rate is attenuated once every 4000 iterations, and a multi-online cascade training mode is adopted.

7. A balance beam game feedback system based on depth photography technology, adopting the balance beam game feedback method based on depth photography technology according to claim 1, characterized in that, The application relates to a smart balance beam, a voice broadcast device assembled on the lower side of the balance beam, front and side depth cameras, a display screen and a computer; a pressure sensor is further arranged on the smart balance beam and is connected with an induction lamp, which is used for displaying a correct center of gravity placement position; the voice broadcast device is a loudspeaker, which is used for broadcasting score points and a reward mechanism; the depth camera is used for shooting score key points of detailed actions of athletes and real-time data feedback to the computer; the display screen is used for real-time feedback of various data of the athletes; and the computer is used for image processing and data analysis of the depth camera.

Images