Hand action mode training method for rehabilitation training of autistic children

Abstract

The invention discloses a hand action mode training method for rehabilitation training of autism children, and belongs to the technical field of reinforcement learning. According to the invention, real-time analysis is carried out according to the recognition result of the hand action mode in rehabilitation training of the child autism patient by using the reinforcement learning model, and different training contents are recommended to train the child autism patient, so that the effect of guiding and optimizing the hand action mode of the child autism patient is achieved; according to the method, manual experience is replaced by experience of a reinforcement learning model, hand action rules of autism children are found, and the children are guided to efficiently master training content and make correct action modes through an optimal training content recommendation strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of reinforcement learning, and in particular to a hand movement pattern training method for rehabilitation training of children with autism. Background Art

[0002] Children with autism are mainly characterized by social disorders, communication disorders, and repetitive and limited interest behaviors. They are usually given rehabilitation education based on applied behavior analysis, including demonstration, behavior packages, comprehensive behavior therapy, natural teaching, etc. The behavior of children with autism is evaluated, interpreted, intervened, and guided to improve children's behavior and enable them to master basic behavioral abilities. With the development of graphic display and image processing technology, equipment and systems for rehabilitation training of children with autism have been developed. In terms of display, it is no longer limited to the display of two-dimensional images and videos. The use of VR, naked-eye three-dimensional displays, etc. can more realistically simulate the interaction between people and objects in the real world, reduce the obstacles to children with autism's cognition of the real world, and improve the rehabilitation training effect of children with autism; in terms of image processing, with the development of deep learning technology, the behavior of autistic children is identified and analyzed through images, videos and other data, and children's behavior is perceived and identified without contact.

[0003] How to achieve efficient guidance and optimization of hand movement rehabilitation training for autistic children is an urgent problem to be solved. To this end, a hand movement pattern training method for rehabilitation training of autistic children is proposed. Summary of the invention

[0004] The technical problem to be solved by the present invention is: how to achieve efficient guidance and optimization of hand movement rehabilitation training for autistic children. A hand movement pattern training method for rehabilitation training of autistic children is provided, and a reinforcement learning algorithm model for hand movement patterns of children with autism rehabilitation training is constructed. The movement pattern of the child under the current training content is identified through the hand movement recognition algorithm, and according to the child's movement pattern and the current training content, the next training content is generated based on the reinforcement learning algorithm model, so as to efficiently guide the child to perform hand movement rehabilitation training.

[0005] The present invention solves the above technical problems through the following technical solutions, and the present invention comprises the following steps:

[0006] S1: The current training content is displayed on the screen based on the naked-eye 3D display interaction method, and the autistic children interact with the displayed training content in a simulated reality;

[0007] S2: Construct a hand action pattern library, use the hand action recognition algorithm to identify the action pattern of autistic children under the current training content, and search the hand action pattern library to obtain the reward or punishment of autistic children under the current training content according to the action pattern;

[0008] S3: Construct a DQN reinforcement learning algorithm model, recommend training content based on the DQN reinforcement learning algorithm model according to the rewards or punishments of autistic children under the current training content and the current training content, generate the next training content, and then guide autistic children to perform hand movement rehabilitation training.

[0009] Furthermore, in step S2, the hand motion pattern library is constructed by defining multiple groups of actions and their corresponding rewards and penalties.

[0010] Furthermore, in step S2, the processing process of the hand motion recognition algorithm is as follows:

[0011] S21: Data Preprocessing

[0012] Obtain video image frames and normalize them;

[0013] S22: Hand key point detection

[0014] The MediaPipe Hands model, an open source hand key point detection algorithm, is used to detect the hand key points in each image frame. The output hand key point coordinates are expressed as (x i ,y i ,z i ), where i represents the index of the key point, and the range of i is 1 to 21;

[0015] S23: Action sequence generation

[0016] Extract the hand key points in each image frame, generate a fixed-length action sequence C and convert it into a feature vector sequence;

[0017] S24: Motion Recognition

[0018] Use the LSTM network to classify the feature vector sequence, input the feature vector sequence into the LSTM network, and output the action category y.

[0019] Furthermore, in step S23, the action sequence C is expressed as follows:

[0020] C=[L 1 ,L 2 ,L 3 ,…,L N ]

[0021] Where N represents the total number of image frames. 1 To L N In both, 21 hand key point coordinates are included.

[0022] Furthermore, in step S24, before using the LSTM network, it needs to be trained, and the cross entropy Loss function is used during training, which is expressed as follows:

[0023]

[0024] Among them, K represents the number of categories, y k is the true label, is the predicted probability.

[0025] Furthermore, in step S2, after obtaining the action pattern, that is, the action category y, the reward or punishment for the autistic child under the current training content is searched in the hand action pattern library.

[0026] Furthermore, in step S3, the DQN reinforcement learning algorithm model models the training content recommendation problem as a Markov decision process, expressed as (S, A, P, R, γ), where S represents the state of the autistic child, that is, the feedback result of the historical training content, which refers to the action sequence made by the autistic child for the historical training content; A represents the action, that is, the training content item recommended by the recommendation agent; P represents the state transition probability, P(s t+1 |s t ,a t ) indicates that in the current state s t Next, perform action a t Then transfer to the next state s t+1 The probability of autistic children in this training; R represents the reward or punishment of autistic children in this training, R(s t ,a t ) indicates that in the current state s t Next, perform action a t The reward or punishment obtained is to search the hand movement pattern library to obtain the reward or punishment for autistic children under the current training content; γ is the discount factor, which indicates the degree of discount of future rewards or punishments.

[0027] Furthermore, in step S3, the specific process of using the DQN reinforcement learning algorithm model to recommend training content is as follows:

[0028] S31: Status indication

[0029] In the process of training content recommendation based on the DQN reinforcement learning algorithm model, the status of autistic children is comprehensively represented by the completion of the current training content, the hand movement recognition results of autistic children, and the training time.t Represented as a multidimensional vector, s t =[x 1 , x 2 , x 3 …,x n ], where each x i It is a characteristic that describes the state of autistic children;

[0030] S32: Action Selection

[0031] Use the ε-greedy strategy for action selection, action a t is the training content item recommended by the recommendation agent;

[0032] S33: Perform actions and observe rewards

[0033] Recommend the agent to perform action a t After that, the autistic children interact according to the recommended training content items, record the performance of the autistic children, and calculate the reward or punishment r of the autistic children based on the hand movement pattern library. t , and the recommendation agent transfers to the next state s according to the performance of the autistic child t+1 ;

[0034] S34: Q value update

[0035] The recommendation agent uses experience replay and Q-value updates to train the model, using the experience replay pool to store past experiences (s t ,a t ,r t ,s t+1 ), and then randomly draw samples from the experience replay pool to update the Q network;

[0036] S35: Experience Replay

[0037] Use the experience replay buffer to store past experiences (s t ,a t ,r t ,s t+1 ,d t ), randomly extract data from the experience replay pool for training at every set batch time period to prevent the model from falling into the local optimal solution and enhance the generalization ability of the model. t Indicates whether it is in the termination state.

[0038] Furthermore, in step S32, a random action is selected with probability ε, and an action that maximizes the Q value is selected with probability 1-ε. The action selection formula is as follows:

[0039]

[0040] Among them, Q(st ,a;θ) is the current state s t is the expected reward value generated by executing action a, and θ is the parameter of the Q network.

[0041] Furthermore, in step S34, the Q value update formula is as follows:

[0042] Q(s t ,a t )←Q(s t ,a t )+α(r t +γmax a′ Q(s t+1 ,a′)-Q(s t ,a t ))

[0043] Among them, α is the learning rate. The DQN reinforcement learning algorithm model uses the neural network Q(s,a|θ) to parameterize the Q function. θ represents the parameters of the neural network. The target network Q′(s,a|θ - ), parameter θ - is a delayed copy of θ, synchronized every set number of steps.

[0044] Compared with the prior art, the present invention has the following advantages: the hand movement pattern training method for rehabilitation training of autistic children uses a reinforcement learning model to perform real-time analysis based on the hand movement pattern recognition results in the rehabilitation training of autistic children, and recommends different training contents to train autistic children, so as to achieve the effect of guiding and optimizing the hand movement patterns of autistic children. The experience of the reinforcement learning model is used to replace artificial experience to discover the hand movement rules of autistic children, and the optimal training content recommendation strategy is used to guide children to efficiently master the training content and make correct movement patterns. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of a hand movement pattern training method for rehabilitation training of children with autism based on reinforcement learning in an embodiment of the present invention;

[0046] Figure 2 It is a schematic diagram of the structure of a training device based on a naked-eye 3D display in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following embodiment.

[0048] like Figure 1As shown, the reinforcement learning-based hand movement pattern training method for rehabilitation training of children with autism proposed in the present invention provides an interactive recommendation agent between autistic children and training content. The recommendation agent is an intelligent agent in reinforcement learning. The recommendation agent and the autistic child perform multiple rounds of interaction. In each interaction process, the recommendation agent recommends training content to the autistic child according to the current recommendation strategy, and then the autistic child gives feedback on the recommended training content. For example, the recommended training content is a kitten, and the autistic child interacts with the kitten. The camera recognizes the hand movements of the autistic child based on the gesture recognition algorithm, including stroking, light touching or patting, and heavy hitting. Then the feedback is input into the recommendation agent. The recommendation agent calculates rewards based on user feedback, that is, rewards in reinforcement learning. The recommendation agent updates the training content recommendation strategy based on the previous round of interaction, and records the changes in the interaction records of the autistic child, that is, the state transfer in reinforcement learning, and recommends the next round of training content based on the changed interaction records of the autistic child and the updated recommendation strategy.

[0049] The process of training content recommendation problem is described mathematically. The set of autistic children is represented as U (|U|=M), the set of training content is represented as I (|I|=N), and time is t. The recommendation agent observes the target autistic child u t The state is established by the interaction history of the autistic child, and then based on the state s of the autistic child t and recommendation strategy π will train content i t ∈I is recommended to the target autistic child u t , targeting children with autism t According to their own behavioral abilities, they give feedback on the recommended training content, and the gesture recognition algorithm identifies the target autistic children. t The gesture action is judged and the correctness of the gesture action is fed back to the recommendation agent, which calculates the reward r based on the feedback. t , and observe the target autistic child u t Interaction records of target autistic children u t The state of , and then the recommendation agent recommends the next round of training content. The goal of this training content recommendation problem is to learn a recommendation strategy π:S→I to maximize the expected cumulative reward of the entire recommendation process:

[0050]

[0051] Among them, π * refers to the optimal training content recommendation strategy, Π represents the set of all possible recommendation strategies, T represents the time length of the training content recommendation process, t represents the time in T, E represents the expectation, and r t refers to the reward or penalty received by the recommending agent at time t.

[0052] 1. Reward and Penalty Mechanism Based on Gesture Recognition Algorithm

[0053] In the present invention, a hand motion pattern library is constructed by defining a set of actions and their corresponding rewards and penalties. An example of the hand motion pattern library is shown in the following table.

[0054] Table 1 Hand motion pattern library

[0055]

[0056]

[0057] For hand motion recognition of autistic children, a recurrent neural network (LSTM network) is used based on hand motion video data to consider time sequence information for hand motion recognition of autistic children. The hand motion recognition algorithm includes the following steps:

[0058] (1) Data preprocessing

[0059] First, obtain the video image frames and preprocess them. Preprocessing is to normalize the video image frames to ensure the standardization of the input data, thereby improving the generalization ability of the model. The mathematical process of data preprocessing is expressed as follows:

[0060]

[0061] Among them, B is the original image and B′ is the normalized image.

[0062] (2) Hand key point detection

[0063] The MediaPipe Hands model, an open source hand key point detection algorithm, is used to detect the hand key points in each image frame. The MediaPipe Hands model is based on a convolutional neural network and can detect 21 hand key points, including the fingertips and joints of each finger. The key point coordinates output by the model are expressed as (x i ,y i ,z i ), where i represents the index of the key point, and the range of i is 1 to 21.

[0064] (3) Action sequence generation

[0065] Extract the key points of the hand in each image frame and generate a fixed-length action sequence. The fixed sequence length is expressed as N frames, L t represents the list of hand key points in the tth frame, then an action sequence C can be expressed as:

[0066] C=[L 1 , L2 , L 3 , …, L N ]

[0067] Among them, L t Contains all 21 hand key point coordinates, L t It means L 1 , L 2 , L 3 , ..., L N Any one of them is expressed as:

[0068] L t =[(x 1 ,y 1 , z 1 ), (x 2 ,y 2 , z 2 ),…,(x 21 ,y 21 , z 21 )]

[0069] Flatten the coordinates of each hand key point to get the feature vector v t :

[0070] v t =[x 1 ,y 1 , z 1 , x 2 ,y 2 , z 2 , …, x 21 ,y 21 , z 21 ]

[0071] The action sequence C can be expressed as a sequence of feature vectors:

[0072] C=[v 1 , v 2 , v 3 , …, v N ]

[0073] (4) Action Recognition

[0074] The generated action sequence is classified using the LSTM network. The LSTM network captures the temporal dependencies in the sequence data, performs hand action recognition and classification of the image sequence, i.e., the video, and inputs the feature vector sequence C into the LSTM network, and outputs the action category y. The LSTM long short-term memory neural network is mathematically represented, and the input gate in the LSTM network is represented as:

[0075] i t =σ(W i ·[ht-1 , S]+b i )

[0076] The forget gate is expressed as:

[0077] f t =σ(W f ·[h t-1 , S]+b f )

[0078] The output gate is represented as:

[0079] o t =σ(W o ·[h t-1 , S]+b o )

[0080] The cell state is represented as:

[0081]

[0082]

[0083] The hidden state is represented as:

[0084] h t =o t *tanh(C t ))

[0085] Among them, σ represents the sigmoid function, tanh represents the tanh activation function, and W i , W f , W o , W C represents the weight matrix, b i , b f , b o , b C represents the bias vector, h t Indicates the hidden state.

[0086] The time series features h output by the LSTM network t , gesture classification is performed through the fully connected layer, expressed as:

[0087] y=softmax(W y ·h t +b y )

[0088] Among them, W y is the weight matrix of the fully connected layer, b y is the bias vector.

[0089] (5) Model training

[0090] For the training of LSTM model, the cross entropy Loss function is used, which is mathematically expressed as:

[0091]

[0092] Among them, K represents the number of categories, yk is the true label, is the predicted probability.

[0093] The gesture action category label data includes correct actions such as stroking, touching, lifting, holding up, holding up, turning pages, picking up, opening, closing, adjusting brightness, etc., and incorrect actions include slapping, hitting, overturning, knocking over, lifting, knocking over, tearing pages, throwing away, beating, disassembling, etc. The gesture actions of autistic children for the training content are identified through the time series network LSTM, so as to obtain the rewards or punishments of autistic children under the current training content.

[0094] 2. Training content recommendation method based on DQN reinforcement learning

[0095] The training content recommendation problem is modeled as a Markov decision process (MDP), expressed as (S, A, P, R, γ), where S represents the state of the autistic child, that is, the feedback result of the historical training content, which refers to the sequence of action patterns (action sequence) made by the autistic child in response to the historical training content; A represents the action, that is, the training content item recommended by the recommendation agent; P represents the state transition probability, P(s t+1 |s t , a t ) indicates that in the current state s t Next, perform action a t Then transfer to the next state s t+1 The probability of autistic children in this training; R represents the reward or punishment of autistic children in this training, R(s t , a t ) indicates that in the current state s t Next, perform action a t The reward or punishment obtained is to search the hand movement pattern library to obtain the reward or punishment for autistic children under the current training content; γ is the discount factor, which indicates the degree of discount of future rewards or punishments.

[0096] Use Q-learning to approach the optimal strategy by iteratively updating the state-action value function Q(s, a). The update formula is:

[0097] Q(s t , a t )←Q(s t , a t )+α(r t+γmax a′ Q(s t+1 , a′)-Q(s t , a t ))

[0098] Among them, α is the learning rate. DQN uses the neural network Q(s, a|θ) to parameterize the Q function. θ represents the parameters of the neural network. In order to improve the training stability, the target network Q′(s, a|θ - ), parameter θ - is a delayed copy of θ, synchronized every certain number of steps, and the experience replay buffer stores past experience (s t , a t , r t ,s t+1 , d t ), where d t Indicates whether it is a termination state, randomly sampling small batches of experience from the buffer to stabilize training. For each small batch sample (s i , a i , r i ,s i+1 , d i ), calculate the target value:

[0099] y i =r i +γ(1-d i )max a′ Q′(s i+1 , a′|θ - )

[0100] The loss function is:

[0101]

[0102] Use gradient descent to minimize the loss function and update the policy network parameters θ.

[0103] In summary, the hand movement pattern training method for children with autism rehabilitation training based on reinforcement learning can be combined with naked-eye 3D display (see Figure 2 ) is applied to identify the hand movements of autistic children based on the gesture recognition algorithm, and match them with the action pattern library to determine whether the action pattern is correct. The training content recommendation agent is trained based on the DQN reinforcement learning algorithm, and the feedback of the recognized action results is used as a reward to train a recommendation agent that can guide children to perform the correct action pattern.

[0104] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A hand movement pattern training method for rehabilitation training of autistic children, characterized in that: The following steps are involved: S1: The current training content is displayed on the screen based on the naked-eye 3D display interaction method, and the autistic children interact with the displayed training content in a simulated reality; S2: Construct a hand action pattern library, use the hand action recognition algorithm to identify the action pattern of autistic children under the current training content, and search the hand action pattern library to obtain the reward or punishment of autistic children under the current training content according to the action pattern; S3: Construct a DQN reinforcement learning algorithm model, recommend training content based on the DQN reinforcement learning algorithm model according to the rewards or punishments of autistic children under the current training content and the current training content, generate the next training content, and then guide autistic children to perform hand movement rehabilitation training.

2. A hand movement pattern training method for rehabilitation training of autistic children according to claim 1, characterized in that: In step S2, a hand motion pattern library is constructed by defining multiple groups of motions and their corresponding rewards and penalties.

3. The hand movement pattern training method for rehabilitation training of autistic children according to claim 1, characterized in that: In step S2, the processing process of the hand movement recognition algorithm is as follows: S21: Data Preprocessing Obtain video image frames and normalize them; S22: Hand key point detection The MediaPipe Hands model, an open source hand key point detection algorithm, is used to detect the hand key points in each image frame. The output hand key point coordinates are expressed as (x i ,y i ,z i ), where i represents the index of the key point, and the range of i is 1 to 21; S23: Action sequence generation Extract the hand key points in each image frame, generate a fixed-length action sequence C and convert it into a feature vector sequence; S24: Motion Recognition Use the LSTM network to classify the feature vector sequence, input the feature vector sequence into the LSTM network, and output the action category y.

4. A hand movement pattern training method for rehabilitation training of autistic children according to claim 3, characterized in that: In step S23, the action sequence C is expressed as follows: <h2 style=";text-align:left;direction:ltr">C=[L1,L2,L3,…,L<h2 style=";text-align:left;direction:ltr"> N <h2 style=";text-align:left;direction:ltr"> ] Where N represents the total number of image frames, from L1 to L N In both, 21 hand key point coordinates are included.

5. The hand movement pattern training method for rehabilitation training of autistic children according to claim 3, characterized in that: In step S24, before using the LSTM network, it needs to be trained, and the cross entropy Loss function is used during training, which is expressed as follows: Among them, K represents the number of categories, y k is the true label, is the predicted probability.

6. A hand movement pattern training method for rehabilitation training of autistic children according to claim 4, characterized in that: In step S2, after obtaining the action pattern, that is, the action category y, the reward or punishment for the autistic child under the current training content is searched in the hand action pattern library.

7. A hand movement pattern training method for rehabilitation training of autistic children according to claim 4, characterized in that: In step S3, the DQN reinforcement learning algorithm model models the training content recommendation problem as a Markov decision process, expressed as (S, A, P, R, γ), where S represents the state of the autistic child, that is, the feedback result of the historical training content, which refers to the action sequence made by the autistic child for the historical training content; A represents the action, that is, the training content item recommended by the recommendation agent; P represents the state transition probability, P(s t+1 |s t ,a t ) indicates that in the current state s t Next, perform action a t Then transfer to the next state s t+1 The probability of autistic children in this training; R represents the reward or punishment of autistic children in this training, R(s t ,a t ) indicates that in the current state s t Next, perform action a t The reward or punishment obtained is to search the hand movement pattern library to obtain the reward or punishment for autistic children under the current training content; γ is the discount factor, which indicates the degree of discount of future rewards or punishments.

8. A hand movement pattern training method for rehabilitation training of autistic children according to claim 7, characterized in that: In step S3, the specific process of using the DQN reinforcement learning algorithm model to recommend training content is as follows: S31: Status indication In the process of training content recommendation based on the DQN reinforcement learning algorithm model, the status of autistic children is comprehensively represented by the completion of the current training content, the hand movement recognition results of autistic children, and the training time. t Represented as a multidimensional vector, s t =[x1, x2, x3…, x n ], where each x i It is a characteristic that describes the state of autistic children; S32: Action Selection Use the ε-greedy strategy for action selection, action a t is the training content item recommended by the recommendation agent; S33: Perform actions and observe rewards Recommend the agent to perform action a t After that, the autistic children interact according to the recommended training content items, record the performance of the autistic children, and calculate the reward or punishment r of the autistic children based on the hand movement pattern library. t , and the recommendation agent transfers to the next state s according to the performance of the autistic child t+1 ; S34: Q value update The recommendation agent uses experience replay and Q-value updates to train the model, using the experience replay pool to store past experiences (s t ,a t ,r t ,s t+1 ), and then randomly draw samples from the experience replay pool to update the Q network; S35: Experience Replay Use the experience replay buffer to store past experiences (s t ,a t ,r t ,s t+1 ,d t ), randomly extract data from the experience replay pool for training at every set batch time period to prevent the model from falling into the local optimal solution and enhance the generalization ability of the model. t Indicates whether it is in the termination state.

9. A hand movement pattern training method for rehabilitation training of autistic children according to claim 8, characterized in that: In step S32, a random action is selected with probability ε, and an action that maximizes the Q value is selected with probability 1-ε. The action selection formula is as follows: Among them, Q(s t ,a;θ) is the current state s t is the expected reward value generated by executing action a, and θ is the parameter of the Q network.

10. A hand movement pattern training method for rehabilitation training of autistic children according to claim 8, characterized in that: In step S34, the Q value update formula is as follows: Q(s t ,a t )←Q(s t ,a t )+α(r t +γmax a′ Q(s t+1 ,a′)-Q(s t ,a t )) Among them, α is the learning rate. The DQN reinforcement learning algorithm model uses the neural network Q(s,a|θ) to parameterize the Q function. θ represents the parameters of the neural network. The target network Q′(s,a|θ - ), parameter θ - is a delayed copy of θ, synchronized every set number of steps.

Images