A hand action pattern training method for autistic children rehabilitation training

Abstract

The application discloses a hand action mode training method for autism child rehabilitation training and belongs to the technical field of reinforcement learning. The application uses a reinforcement learning model to perform real-time analysis on the hand action mode recognition result in the rehabilitation training of a child autism patient, and recommends different training contents to train the child autism patient, so that the effect of guiding and optimizing the hand action mode of the child autism patient is achieved. The experience of the reinforcement learning model is used to replace artificial experience, the hand action rule of the autism child is found, and through an optimal training content recommendation strategy, the child is guided to efficiently master the training content and make correct action modes.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of reinforcement learning, and particularly relates to a hand action mode training method for autism child rehabilitation training. BACKGROUND

[0002] Autism children are mainly characterized by social barriers, communication barriers and repetitive and limited interest behaviors, and are usually subjected to rehabilitation education by using a rehabilitation education method based on applied behavior analysis, including demonstration, behavior package, comprehensive behavior therapy, natural teaching and the like, so that the behaviors of the autism children are evaluated, interpreted and intervened to improve the behaviors of the autism children and help the autism children master basic behavior capabilities. With the development of graphic display and image processing technologies, equipment and systems for autism child rehabilitation training have been developed, and in terms of display, the display mode is not limited to two-dimensional image video display, and VR, naked-eye three-dimensional display and the like can be used to more realistically simulate the interaction between people and objects in the real world, reduce the hindrance of the real world cognition of the autism children and improve the rehabilitation training effect of the autism children; in terms of image processing, with the development of deep learning technology, autism child behaviors are recognized and analyzed through image and video data, and the behaviors of the autism children are recognized and analyzed in a non-contact manner.

[0003] How to realize efficient guidance and optimization of hand action rehabilitation training of autism children is a problem to be solved, and therefore, the hand action mode training method for autism child rehabilitation training is provided. SUMMARY

[0004] The technical problem to be solved by the application is how to realize efficient guidance and optimization of hand action rehabilitation training of autism children, and the hand action mode training method for autism child rehabilitation training is provided, a hand action mode reinforcement learning algorithm model for autism child rehabilitation training is constructed, the action mode of a child under current training content is recognized through a hand action recognition algorithm, and the next training content is generated based on the action mode of the child and the current training content based on the reinforcement learning algorithm model, so that the child is efficiently guided to perform hand action rehabilitation training.

[0005] The application solves the above technical problems through the following technical scheme, and the application comprises the following steps:

[0006] S1: current training content is displayed in front of a screen based on a naked-eye three-dimensional display interaction mode, and an autism child interacts with the displayed training content in a pseudo-reality mode;

[0007] S2: Construct a hand action pattern library, use a hand action recognition algorithm to recognize the action pattern of the autistic child under the current training content, and find the reward or punishment of the autistic child under the current training content in the hand action pattern library according to the action pattern;

[0008] S3: Construct a DQN reinforcement learning algorithm model, train the content recommendation based on the DQN reinforcement learning algorithm model according to the reward or punishment of the autistic child under the current training content and the current training content, generate the next training content, and guide the autistic child to perform hand action rehabilitation training.

[0009] Further, in the step S2, the hand action pattern library is constructed by defining multiple groups of actions and their corresponding rewards and punishments.

[0010] Further, in the step S2, the processing process of the hand action recognition algorithm is as follows:

[0011] S21: Data preprocessing

[0012] Obtain video image frames and perform normalization processing on them;

[0013] S22: Hand key point detection

[0014] Use the open source hand key point detection algorithm MediaPipe Hands model to detect the hand key points in each image frame, and the output hand key point coordinates are represented as (x i ,y i ,z i ), where i represents the index of the key point, and i ranges from 1 to 21;

[0015] S23: Action sequence generation

[0016] Extract the hand key points in each image frame, generate a fixed-length action sequence C, and convert it to a feature vector sequence;

[0017] S24: Action recognition

[0018] Use the LSTM network to classify the feature vector sequence, input the feature vector sequence into the LSTM network, and the output is the class y of the action.

[0019] Further, in the step S23, the action sequence C is represented as follows:

[0020] C=[L1,L2,L3,…,L N ]

[0021] Where N represents the total number of image frames, and L1 to L N each include 21 hand key point coordinates.

[0022] Further, in the step S24, before using the LSTM network, it needs to be trained, and the cross-entropy Loss function is used for training, which is expressed as follows:

[0023]

[0024] Wherein, K represents the number of categories, y k is the real label, is the predicted probability.

[0025] Further, in the step S2, after obtaining the action pattern, that is, the category y of the action, the reward or punishment of the autistic child under the current training content is obtained in the hand action pattern library.

[0026] Further, in the step S3, the DQN reinforcement learning algorithm model models the training content recommendation problem as a Markov decision process, denoted as (S, A, P, R, γ), wherein S represents the state of the autistic child, that is, the feedback result of the historical training content, which refers to the action sequence made by the autistic child for the historical training content; A represents the action, that is, the training content item recommended by the recommendation agent; P represents the state transition probability, P(s t+1 |s t ,a t ) represents the probability of moving to the next state s t after performing action a t in the current state s t+1 ; R represents the reward or punishment of the autistic child in this training, R(s t ,a t ) represents the reward or punishment obtained by performing action a t in the current state s t , that is, the reward or punishment of the autistic child under the current training content is obtained in the hand action pattern library; γ is the discount factor, which represents the discount degree of future rewards or punishments.

[0027] Further, in the step S3, the specific process of training content recommendation using the DQN reinforcement learning algorithm model is as follows:

[0028] S31: State representation

[0029] In the process of training content recommendation based on the DQN reinforcement learning algorithm model, the state of the autistic child is comprehensively represented by the completion of the current training content, the hand action recognition result of the autistic child, and the training time, and the state s t is represented as a multi-dimensional vector, s t = [x1, x2, x3…, x n], wherein each x i is a feature describing the state of the autistic child;

[0030] S32: Action selection

[0031] Action selection is performed using an epsilon-greedy strategy, and the action a t is recommended by the recommendation agent.

[0032] S33: Perform action and observe reward

[0033] After the recommendation agent performs the action a t , the autistic child interacts according to the recommended training content item, the performance of the autistic child is recorded, and the reward or punishment r t of the autistic child is calculated according to the hand action pattern library, and the recommendation agent moves to the next state s t+1 according to the performance of the autistic child.

[0034] S34: Q value update

[0035] The recommendation agent uses experience replay and Q value update to train the model, stores past experiences (s t , a t , r t , s t+1 ) in the experience replay pool, and then randomly samples from the experience replay pool to update the Q network.

[0036] S35: Experience replay

[0037] Past experiences (s t , a t , r t , s t+1 , d t ) are stored in the experience replay buffer, and every set batch period, data is randomly sampled from the experience replay pool for training to prevent the model from falling into a local optimal solution and enhance the generalization ability of the model, wherein d t indicates whether it is a terminal state.

[0038] Further, in the step S32, a random action is selected with a probability of epsilon, and an action that maximizes the Q value is selected with a probability of 1-epsilon, and the action selection formula is as follows:

[0039]

[0040] wherein Q(s t , a; theta) is the expected reward value generated by performing action a in current state s t , and theta is the parameter of the Q network.

[0041] Further, in the step S34, the Q value update formula is as follows:

[0042] Q(s t ,a t )←Q(s t ,a t )+α(r t +γmax a′ Q(s t+1 ,a′)-Q(s t ,a t ))

[0043] Wherein, alpha is the learning rate, the DQN reinforcement learning algorithm model uses the neural network Q(s, a|theta) to parameterize the Q function, theta represents the parameters of the neural network, the target network Q'(s, a|theta - ) is introduced, and the parameter theta - is a delayed copy of theta, and is synchronized once every set number of steps.

[0044] Compared with the prior art, the autism child rehabilitation training hand action mode training method has the following advantages: the autism child rehabilitation training hand action mode training method uses a reinforcement learning model to analyze in real time according to a hand action mode recognition result in autism child rehabilitation training, and recommends different training contents to train the autism child, so as to guide and optimize the hand action mode of the autism child, replace artificial experience with experience of the reinforcement learning model, find the hand action rule of the autism child, and guide the child to efficiently master the training content and make correct action modes through an optimal training content recommendation strategy. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a flowchart of the autism child rehabilitation training hand action mode training method based on reinforcement learning in the embodiment of the application;

[0046] Figure 2 is a training device structure schematic diagram based on a naked-eye 3D display in the embodiment of the application. DETAILED DESCRIPTION

[0047] The embodiments of the application will be described in detail below, and the embodiments are implemented on the premise of the technical scheme of the application, and detailed implementation modes and specific operation processes are given, but the protection scope of the application is not limited to the following embodiments.

[0048] As Figure 1As shown, the hand action mode training method for autism rehabilitation training based on reinforcement learning provided by the application provides an interaction recommendation agent between an autistic child and training content, the recommendation agent, that is, the agent in reinforcement learning, carries out multi-round interaction with the autistic child, in each interaction process, the recommendation agent recommends training content to the autistic child according to the current recommendation strategy, then the autistic child feeds back the recommended training content, for example, the recommended training content is a kitten, the autistic child interacts with the kitten, the hand gesture of the autistic child is recognized through a camera based on a gesture action recognition algorithm, including touching, tapping, hitting, etc., and then the feedback is input to the recommendation agent, the recommendation agent calculates the reward according to the user feedback, that is, the reward in reinforcement learning, the recommendation agent updates the training content recommendation strategy according to the interaction record of the autistic child, that is, the state transition in reinforcement learning, and carries out the next round of training content recommendation according to the updated recommendation strategy and the changed interaction record of the autistic child.

[0049] The process of the training content recommendation problem is mathematically described, the set of autistic children is represented as U (|U| = M), the set of training content is represented as I (|I| = N), the time is t, the recommendation agent observes the state of the target autistic child u t , which is established by the interaction history of the autistic child, then according to the state s t of the autistic child and the recommendation strategy π, the training content i t ∈I is recommended to the target autistic child u t , the target autistic child u t feeds back the recommended training content according to its own behavior ability, the gesture action recognition algorithm recognizes the gesture action of the target autistic child u t , and judges the correctness of the gesture action, and feeds back to the recommendation agent, the recommendation agent calculates the reward r t obtained according to the feedback, and observes the interaction record of the target autistic child u t , updates the state of the target autistic child u t , then the recommendation agent carries out the next round of training content recommendation, the goal of the training content recommendation problem is to learn a recommendation strategy π: S→I to maximize the expected cumulative reward in the whole recommendation process:

[0050]

[0051] Where π * refers to the optimal training content recommendation strategy, Π represents the set of all possible recommendation strategies, T represents the length of the training content recommendation process this time, t represents the time in T, E represents the expectation, and r t refers to the reward or punishment obtained by the recommendation agent at time t.

[0052] I. Reward and punishment mechanism based on gesture recognition algorithm

[0053] In the present application, a hand action pattern library is constructed, which is realized by defining a set of actions and their corresponding rewards and punishments. An example of the hand action pattern library is shown in the following table.

[0054] Table 1 Hand action pattern library table

[0055]

[0056]

[0057] For autism children hand action recognition, based on hand action video data, a recurrent neural network (LSTM network) is used to consider the timing information for autism children hand action recognition. The hand action recognition algorithm includes the following steps:

[0058] (1) Data preprocessing

[0059] First, the video image frame is obtained and preprocessed. The preprocessing is to normalize the video image frame to ensure the standardization of the input data, so as to improve the generalization ability of the model. The mathematical process of data preprocessing is represented as follows:

[0060]

[0061] Where B is the original image, and B' is the normalized image.

[0062] (2) Hand key point detection

[0063] The open source hand key point detection algorithm MediaPipe Hands model is used to detect the hand key points in each image frame. The MediaPipe Hands model is based on convolutional neural network, which can detect 21 hand key points, including the fingertips of each finger, joints, etc. The key point coordinates output by the model are represented as (x i ,y i ,z i ), where i represents the index of the key point, and i ranges from 1 to 21.

[0064] (3) Action sequence generation

[0065] Extract the hand key points in each image frame and generate a fixed length action sequence. The fixed sequence length is represented as N frames, and L t represents the list of hand key points in the t-th frame. An action sequence C can be represented as:

[0066] C=[L1,L2,L3,…,L N]

[0067] wherein L t contains all 21 hand key point coordinates, L t refers to L1, L2, L3,..., L N any one of them, is expressed as:

[0068] L t = [(x1, y1, z1), (x2, y2, z2),..., (x 21 , y 21 , z 21 )]

[0069] After flattening the coordinates of each hand key point, the feature vector v t is obtained:

[0070] v t = [x1, y1, z1, x2, y2, z2,..., x 21 , y 21 , z 21 ]

[0071] The action sequence C can be expressed as a sequence of feature vectors:

[0072] C = [v1, v2, v3,..., v N ]

[0073] (4) Action recognition

[0074] The generated action sequence is classified using an LSTM network, which captures the time-dependent relationship in the sequence data, and performs hand action recognition classification on the image sequence, i.e. video. The feature vector sequence C is input into the LSTM network, and the output is the class y of the action. The LSTM long short-term memory neural network is mathematically represented, and the input gate in the LSTM network is represented as:

[0075] i t = σ(W i · [h t-1 , S] + b i )

[0076] The forget gate is represented as:

[0077] f t = σ(W f · [h t-1 , S] + b f )

[0078] The output gate is represented as:

[0079] o t = σ(W o · [h t-1, S]+b o )

[0080] Cell state is represented as:

[0081]

[0082]

[0083] Hidden state is represented as:

[0084] h t = o t *tanh(C t ))

[0085] where σ represents sigmoid function, tanh represents tanh activation function, W i , W f , W o , W C represent weight matrix, b i , b f , b o , b C represent bias vector, h t represents hidden state.

[0086] The time series feature h t output by the LSTM network is classified by a fully connected layer, represented as:

[0087] y = softmax(W y · h t + b y )

[0088] where W y is the weight matrix of the fully connected layer, and b y is the bias vector.

[0089] (5) Model training

[0090] For the training of the LSTM model, the cross-entropy Loss function is used, mathematically represented as:

[0091]

[0092] where K represents the number of categories, yk is the real label, is the predicted probability.

[0093] The gesture action category label data includes correct actions such as patting, tapping, lifting, holding, holding, holding, turning pages, picking up, opening, closing, adjusting brightness, and the like, and incorrect actions include actions such as slapping, hitting, pushing over, knocking over, lifting, knocking over, tearing pages, throwing away, pounding, disassembling, and the like. The gesture actions of the autistic children for the training content are recognized through the time sequence network LSTM, so as to obtain the reward or punishment of the autistic children under the current training content.

[0094] II. Training content recommendation method based on DQN reinforcement learning

[0095] The training content recommendation problem is modeled as a Markov decision process (MDP), denoted as (S, A, P, R, γ), where S represents the state of the autistic child, i.e. the feedback result for the historical training content, A represents the action, i.e. the training content item recommended by the recommendation agent, P represents the state transition probability, P(s t+1 |s t , a t ) represents the probability of transitioning to the next state s t after performing action a t in the current state s t+1 ; R represents the reward or punishment of the autistic child in this training, R(s t , a t ) represents the reward or punishment obtained by performing action a t in the current state s t , i.e. finding and obtaining the reward or punishment of the autistic child under the current training content in the hand action pattern library; γ is the discount factor, representing the discount degree of future rewards or punishments.

[0096] Using the Q-learning value function-based reinforcement learning algorithm, the optimal policy is approximated by iteratively updating the state-action value function Q(s, a), and the update formula is:

[0097] Q(s t , a t )←Q(s t , a t )+α(r t +γmax a′ Q(s t+1 , a′)-Q(s t , a t ))

[0098] where α is the learning rate, DQN uses a neural network Q(s, a|θ) to parameterize the Q function, and θ represents the parameters of the neural network. To improve training stability, a target network Q'(s, a|θ- ), parameter theta - is a delayed copy of theta, synchronized every certain number of steps, the experience replay buffer stores past experiences (s t , a t , r t , s t+1 , d t ), where d t indicates whether it is a terminal state, small batches of experiences are randomly sampled from the buffer, stabilizing training. For each small batch of samples (s i , a i , r i , s i+1 , d i ), the target value is computed:

[0099] y i = r i + gamma * (1-d i ) * max a′ Q'(s i+1 , a'| theta - )

[0100] The loss function is:

[0101]

[0102] The loss function is minimized using gradient descent to update the policy network parameters theta.

[0103] In summary, the hand action pattern training method for autism rehabilitation training based on reinforcement learning can be applied in combination with a naked-eye 3D display (see Figure 2 ), based on a gesture recognition algorithm to recognize the hand actions of autistic children, match the action patterns with the action pattern library to determine whether the action patterns are correct or not, train the training content recommendation agent based on the DQN reinforcement learning algorithm, and feed back the recognition action result as a reward, and train a recommendation agent that can guide children to perform correct action patterns.

[0104] Although the embodiments of the present application have been shown and described above, it should be understood that the above-described embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. An autism child rehabilitation training hand action pattern training method, characterized in that, The method comprises the following steps: S1: based on the naked eye three-dimensional display interaction mode, the current training content is displayed in front of the screen, and the autistic children interact with the displayed training content in a pseudo-real way; S2: a hand action mode library is constructed, a hand action recognition algorithm is used to recognize the action mode of the autistic children under the current training content, and the reward or punishment of the autistic children under the current training content is obtained according to the action mode in the hand action mode library; S3: a DQN reinforcement learning algorithm model is constructed, the training content is recommended based on the DQN reinforcement learning algorithm model according to the reward or punishment of the autistic children under the current training content and the current training content, a next training content is generated, and the autistic children are guided to perform hand action rehabilitation training; In the step S2, the processing process of the hand action recognition algorithm is as follows: S21: data preprocessing The video image frame is obtained and normalized; S22: hand key point detection The open-source hand key point detection algorithm MediaPipe Hands model is used to detect the hand key points in each image frame, and the output hand key point coordinates are represented as (x i ,y i ,z i ), wherein i represents the index of the key point, and i ranges from 1 to 21; S23: action sequence generation The hand key points in each image frame are extracted, a fixed-length action sequence C is generated, and the action sequence C is converted into a feature vector sequence; S24: action recognition The feature vector sequence is classified by using an LSTM network, the feature vector sequence is input into the LSTM network, and the output is the category y of the action; In the step S3, the DQN reinforcement learning algorithm model models the training content recommendation problem as a Markov decision process, denoted as (S, A, P, R, γ), where S represents the state of the autistic child, i.e. the feedback result of the historical training content, which refers to the action sequence made by the autistic child for the historical training content; A represents the action, i.e. the training content item recommended by the recommendation agent; P represents the state transition probability, P(s t+1 |s t ,a t ) represents the probability of transitioning to the next state s t after performing the action a t in the current state s t+1 ; R represents the reward or punishment of the autistic child in this training, R(s t ,a t ) represents the reward or punishment obtained by performing the action a t in the current state s t , i.e. finding and obtaining the reward or punishment of the autistic child under the current training content in the hand action mode library; γ is the discount factor, which represents the discount degree of future rewards or punishments; In the step S3, the specific process of recommending the training content by using the DQN reinforcement learning algorithm model is as follows: S31: state representation In the process of training content recommendation based on the DQN reinforcement learning algorithm model, the state of the autistic child is comprehensively represented by the completion of the current training content, the hand action recognition result of the autistic child, and the training time, and the state s t is represented as a multidimensional vector, s t = [x1, x2, x3…, x n ], wherein each x i is a feature describing the state of the autistic child; S32: action selection Action selection is performed using an e-greedy policy, action a t is a training content item recommended by the recommendation agent; S33: execute action and observe reward The recommended agent performs action a t After that, the autistic child interacts according to the recommended training content item, records the performance of the autistic child, and calculates the reward or punishment r of the autistic child according to the hand action mode library t At the same time, the recommended agent transfers to the next state s according to the performance of the autistic child t+1 ; S34: Q value update The agent is recommended to use experience replay and Q-value update to train the model, store past experiences (s t , t , t , t+1 ) in an experience replay pool, and then randomly draw samples from the experience replay pool to update the Q network. S35: experience replay The past experience is stored by using an experience replay buffer (s t ,a t ,r t ,s t+1 ,d t ), data is randomly extracted from the experience replay pool for training every set batch time period, so as to prevent the model from falling into a local optimal solution and enhance the generalization ability of the model, wherein d t represents whether it is a terminal state.

2. The hand movement pattern training method for autistic children rehabilitation training according to claim 1, characterized in that, In the step S2, the hand action mode library is constructed by defining multiple groups of actions and their corresponding rewards and punishments.

3. The hand movement pattern training method for autistic children rehabilitation training according to claim 1, characterized in that, In the step S23, the action sequence C is represented as follows: C = [L1, L2, L3,..., L N ] wherein N represents the total number of image frames, L1 to L N Each of the 21 hand key point coordinates is included in the hand key point coordinate set.

4. The hand movement pattern training method for autistic children rehabilitation training according to claim 1, characterized in that, In the step S24, before using the LSTM network, the LSTM network needs to be trained, and a cross-entropy Loss function is used during training, which is represented as follows: where K denotes the number of classes, y k is the true label, is the predicted probability.

5. The hand movement pattern training method for autistic children rehabilitation training according to claim 3, characterized in that, In the step S2, after obtaining the action mode, that is, the category y of the action, the reward or punishment of the autistic children under the current training content is obtained in the hand action mode library.

6. The hand movement pattern training method for autistic child rehabilitation training according to claim 1, characterized in that, In the step S32, a random action is selected with a probability of ε, and an action with the maximum Q value is selected with a probability of 1-ε, and the action selection formula is as follows: where Q(s t is the expected reward value resulting from performing action a in the current state s t is the parameter of the Q-network.

7. The hand movement pattern training method for autistic child rehabilitation training according to claim 1, characterized in that, In the step S34, the Q value update formula is as follows: Q(s t ,a t )←Q(s t ,a t )+α(r t +γmax a′ Q(s t+1 ,a′)-Q(s t ,a t )) Wherein, a is the learning rate, the DQN reinforcement learning algorithm model uses a neural network Q(s, a|θ) to parameterize the Q function, θ represents the parameters of the neural network, a target network Q'(s, a|θ - ) is introduced, and the parameters θ - are the delayed copies of θ, and synchronization is performed once every set number of steps.

Images