Gesture recognition method and system for rehabilitation of aphasia patients

The method improves gesture recognition accuracy for aphasia patients by extracting key points and analyzing gesture change vectors, segmenting dynamic gestures, and using clustering techniques to ensure precise classification of gesture meanings.

CN118898872BActive Publication Date: 2024-12-06西安国际医学中心有限公司
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411389844.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-10-08
Publication Date
2024-12-06
Estimated Expiration
2044-10-08

AI Technical Summary

Technical Problem

Traditional gesture recognition methods for aphasia patients suffer from reduced accuracy due to variations in gesture execution speed influenced by emotional states, leading to errors in matching gesture templates.

Method used

A method involving key point extraction, gesture change vector analysis, and clustering of action frame image sequences to improve accuracy and efficiency in gesture recognition, utilizing techniques like semantic segmentation and neural networks to identify gesture meanings.

Benefits of technology

Enhances the accuracy and efficiency of gesture recognition by simplifying gesture analysis through key point extraction and segmenting dynamic changes, ensuring precise classification of gesture meanings.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118898872B_ABST
    Figure CN118898872B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of motion recognition technology, and in particular to a gesture recognition method and system for rehabilitation of aphasia patients. The method first extracts key points in each action frame image; obtains the gesture change vector of the action frame image according to the difference between the spatial distribution of key points corresponding to each action frame image and the action frame images adjacent in time sequence; divides the action frame image sequence to obtain all local frame image sequences of the action frame image sequence; obtains the distance measurement value of each two action frame image sequences according to the difference between the gesture change vectors of the local frame image sequences corresponding to each two action frame image sequences; performs action recognition on the action frame image sequence to be processed according to the distance measurement value between the action frame image sequence to be processed and the action category. The present invention ensures the accuracy and efficiency of gesture recognition of aphasia patients and improves the gesture recognition effect by conducting an in-depth analysis of the features of gesture recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of motion recognition, and in particular to a gesture recognition method and system for rehabilitation of aphasia patients. Background Art

[0002] Aphasia is a language disorder caused by brain damage, that is, aphasic patients have language communication disorders when they are conscious, have normal consciousness, and have no pronunciation and articulation disorders. In order to promote the recovery of language function of aphasic patients, rehabilitation training is required for aphasic patients. The more advanced rehabilitation method currently available is to collect the patient's gestures through human-computer interaction, and then judge the relevant meanings expressed, and then generate voice and text for auxiliary treatment.

[0003] When aphasic patients are unable to express a phrase in words, they will use complete gestures to express the phrase. Traditional gesture recognition methods usually identify specific gestures that match gesture templates and use the phrase meaning corresponding to the gesture template as the gesture meaning. However, when patients perform gestures with the same phrase meaning, the gesture execution speed will be different due to the influence of their emotional state, which will cause errors in the gesture recognition system when matching the gesture template, thereby reducing the accuracy of gesture recognition and making it difficult to ensure the effect of gesture recognition. Summary of the invention

[0004] In order to solve the technical problem that the prior art is difficult to ensure the accuracy of gesture recognition for aphasia patients, the purpose of the present invention is to provide a gesture recognition method and system for rehabilitation of aphasia patients. The technical solution adopted is as follows:

[0005] A gesture recognition method for rehabilitation of aphasia patients, the method comprising:

[0006] Acquire each action frame image sequence and the action frame image sequence to be processed in the historical database; the action frame image sequence includes each action frame image;

[0007] Extract key points from each of the action frame images; obtain the gesture change vector of the action frame image according to the spatial distribution difference between each of the action frame images and the key points corresponding to the action frame images adjacent in time sequence; divide the action frame image sequence to obtain all local frame image sequences of the action frame image sequence; obtain the distance measurement value of each of the action frame image sequences according to the difference between the gesture change vectors of the local frame image sequences corresponding to each two action frame image sequences and the difference between the local frame image sequences and the number of action frame images corresponding to each two action frame image sequences;

[0008] According to the distance measurement value between every two action frame image sequences, all action frame image sequences in the historical database are clustered to obtain various action categories; according to the distance measurement value between the action frame image sequence to be processed and the action category, action recognition is performed on the action frame image sequence to be processed.

[0009] Furthermore, the method for acquiring the gesture change vector includes:

[0010] Matching key points of each action frame image with its next action frame image, and obtaining each matching key point pair and its corresponding ratio test value;

[0011] Acquire a local vector of each of the matching key point pairs according to the spatial positions of the two key points in each of the matching key point pairs;

[0012] Fusing the local vector of each matching key point pair with the ratio test value to obtain an overall vector of the action frame image;

[0013] According to the difference in the number of key points corresponding to each action frame image and the next action frame image, obtaining a quantity difference index of the action frame images;

[0014] A gesture change vector of the action frame image is acquired according to the quantity difference index and the overall vector of the action frame image.

[0015] Furthermore, the method for acquiring the local frame image sequence includes:

[0016] Obtaining a motion change difference value of each of the motion frame images according to an angle between each of the motion frame images and the gesture change vector corresponding to the next of the motion frame images;

[0017] In the action frame image sequence, taking the action frame images whose action change difference values ​​are greater than the preset amplitude value as the segmentation images;

[0018] The action frame image sequence is divided by using each of the divided images to obtain all local frame image sequences of the action frame image sequence.

[0019] Furthermore, the method for obtaining the distance metric value includes:

[0020] Sorting all the partial frame image sequences in the action frame image sequence in a time sequence order to obtain sequence numbers of all the partial frame image sequences in the action frame image sequence;

[0021] In each of two action frame image sequences, according to the difference in the number of action frame images of the local frame image sequences under the same sequence number and the difference in the gesture change vector, the action speed difference index of each of the two action frame image sequences is obtained;

[0022] In every two action frame image sequences, according to the action change difference value and the gesture change vector of the corresponding divided image of the local frame image sequence under the same sequence number, the action angle difference index of every two action frame image sequences is obtained;

[0023] According to the difference in the number of the local frame image sequences included in every two action frame image sequences, obtaining a quantity difference index;

[0024] According to the action speed difference index, the action angle difference index and the quantity difference index, a distance measurement value between every two action frame image sequences is obtained.

[0025] Furthermore, the method of the motion speed difference index includes:

[0026] In every two action frame image sequences, each local frame image sequence in the action frame image sequence containing the largest number of the action frame images is used as each target local sequence, and each local frame image sequence in the other action frame image sequence is used as each reference local sequence;

[0027] In every two action frame image sequences, the ratio of the number of the action frame images of the target local sequence and the reference local sequence under the same sequence number is calculated to obtain a frame image quantity difference index of the sequence number;

[0028] Calculating the mean value of the modulus lengths corresponding to the gesture change vectors of all the action frame images in the local frame image sequence to obtain the local sequence vector modulus length of the local frame image sequence;

[0029] In every two action frame image sequences, the sum of the local sequence vector modulus length and the denominator adjustment value of the reference local sequence is calculated, and the ratio of the local sequence vector modulus length and the sum of the local sequence vector modulus length of the target local sequence under the same sequence number is calculated to obtain the modulus length difference index of the sequence number;

[0030] Calculate the ratio of the frame image quantity difference index to the module length difference index to obtain the speed scaling amplitude of the sequence number;

[0031] The variance of the speed scaling amplitudes corresponding to all sequence numbers is calculated to obtain the motion speed difference index of two motion frame image sequences.

[0032] Furthermore, the method for obtaining the action angle difference index includes:

[0033] According to the directions of all the gesture change vectors of the local frame image sequence, obtaining an overall direction value of the local frame image sequence;

[0034] Obtaining a local sequence direction value of the local frame image sequence according to the motion change difference value and the overall direction value of the divided image of the local frame image sequence;

[0035] In every two action frame image sequences, according to the difference of the local sequence direction values ​​of the local frame image sequences under each same sequence number, the sequence number angle difference value of the sequence number is obtained;

[0036] According to the overall distribution of the sequence angle difference values ​​of all sequence numbers, the action angle difference index of every two action frame image sequences is obtained.

[0037] Furthermore, the method for obtaining the local sequence direction value includes:

[0038] The product of the motion change difference value of the divided image of the local frame image sequence and the overall direction value is calculated to obtain the local sequence direction value of the local frame image sequence.

[0039] Furthermore, the method for performing action recognition on the action frame image sequence to be processed includes:

[0040] Using semantic neural network, obtain action phrases corresponding to action categories;

[0041] Calculate the average of the distance measurement values ​​between the action frame image sequence to be processed and all action frame image sequences in each action category to obtain the category distance value between the action frame image sequence to be processed and each action category;

[0042] The action phrase corresponding to the action category with the smallest category distance value is used as the action phrase of the action frame image sequence to be processed.

[0043] Furthermore, the method for obtaining the key points includes:

[0044] The semantic segmentation algorithm is used to identify the hand region in each of the action frame images, and key points of the hand region are extracted to obtain the key points in each of the action frame images.

[0045] The present invention proposes a gesture recognition system for the rehabilitation of aphasia patients, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the system implements any one of the steps of the gesture recognition method for the rehabilitation of aphasia patients.

[0046] The present invention has the following beneficial effects:

[0047] The main purpose of the present invention is to ensure the efficiency and accuracy of gesture recognition, thereby improving the effect of gesture recognition. In order to improve the efficiency of gesture recognition, the gestures of the action frame images can be simplified by extracting the key points in each action frame image. Considering that when aphasic patients use gestures to express the meaning of a phrase, the gestures will change to a certain extent, and the gesture change vector of the action frame image is used to reflect the change of the gesture, thereby simplifying the dynamic change characteristics of the gestures of the action frame image. In order to ensure the accuracy of gesture recognition, considering that when expressing the meaning of a phrase, the gestures have a large change, the gesture process is segmented and studied by the large change of the gesture, that is, each local frame image sequence of the action frame image is obtained. In order to analyze the possibility of expressing the same phrase meaning by different action frame image sequences, the distance measurement value of each two action frame image sequences is obtained by the difference of the gesture change vectors of the local frame image sequences contained in each two action frame image sequences. The larger the distance measurement value, the greater the difference in the gesture change corresponding to the two action frame image sequences, and the less likely it is that the phrase meanings corresponding to the two action frame image sequences are the same. In order to more accurately identify the phrase meaning corresponding to the action frame image sequence to be processed, the present invention clusters all action frame image sequences in a historical database to obtain various action categories; the action category can reflect various action frame image sequences corresponding to the same phrase meaning; according to the distance measurement value between the action frame image sequence to be processed and the action category, the action frame image sequence to be processed is more accurately classified into the action category to which the corresponding phrase meaning belongs, thereby improving the effect of action meaning recognition of the action frame image sequence to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A flowchart of a gesture recognition method for rehabilitation of aphasia patients provided by one embodiment of the present invention;

[0050] Figure 2 A flow chart of a method for acquiring a gesture change vector provided by an embodiment of the present invention;

[0051] Figure 3 A flow chart of a method for obtaining a distance metric value provided by an embodiment of the present invention;

[0052] Figure 4A flow chart of a method for obtaining an action angle difference index provided by an embodiment of the present invention;

[0053] Figure 5 A structural diagram of a gesture recognition system for rehabilitation of aphasia patients provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of a gesture recognition method and system for rehabilitation of aphasia patients proposed by the present invention, its specific implementation, structure, features and effects, in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0055] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0056] The following is a detailed description of a method and system for hand gesture recognition for rehabilitation of aphasia patients provided by the present invention in conjunction with the accompanying drawings.

[0057] The embodiment of the present invention provides a gesture recognition method and system for rehabilitation of aphasia patients, please refer to Figure 1 , which shows a flowchart of a gesture recognition method for rehabilitation of aphasia patients provided by an embodiment of the present invention, the method comprising the following steps:

[0058] Step S1: Acquire each action frame image sequence in the historical database and the action frame image sequence to be processed; the action frame image sequence includes each action frame image.

[0059] First, it is necessary to obtain the action frame image sequence to provide data support for subsequent gesture recognition.

[0060] Considering that rehabilitation training is through human-computer interaction, by repeatedly guiding aphasic patients to use gestures to express phrases, so as to realize the recognition of the corresponding meanings of the gestures of aphasic patients. Obtain each action frame image sequence and the action frame image sequence to be processed from the rehabilitation training system. Specifically, in an embodiment of the present invention, considering that in the rehabilitation training process, by repeatedly guiding aphasic patients to use gestures to express phrases. In each training process of aphasic patients using gestures to express phrases, the gestures of aphasic patients are collected by a camera according to a preset sampling frequency to obtain the original frame image. In order to ensure the image quality of the subsequent image processing process, it is also necessary to perform image preprocessing operations on each original frame image, and then obtain each action frame image, and in a chronological order, the action frame images of aphasic patients in the process of using gestures to express phrases are counted to obtain the action frame image sequence. Among them, the action frame image sequence corresponding to the process of aphasic patients using gestures to express phrases is used as the action frame image sequence to be processed. The action frame image sequence corresponding to the process of aphasic patients using gestures to express the meaning of phrases each time in the historical rehabilitation training process is used as each action frame image sequence in the historical database. It should be noted that the number of action frame image sequences in the historical database of the present invention is as large as possible, so as to ensure that the meaning of the phrases expressed by the action frame image sequences to be processed is the same as the meaning of a corresponding phrase in the historical database.

[0061] The specific image preprocessing operation is a technical means well known to those skilled in the art and is not limited here. In the embodiment of the present invention, the image preprocessing operation includes graying, noise reduction, and contrast enhancement. The embodiment of the present invention uses a weighted average method for graying, uses a Gaussian filter for noise reduction, and uses a histogram equalization for contrast enhancement. The implementer can set it according to the actual situation. In one embodiment of the present invention, sampling is performed at a preset frequency, and each sampling is used as a sampling moment. The preset sampling frequency is 5 times / second. It should be noted that, in order to facilitate calculations, all indicator data involved in the calculations in the embodiment of the present invention are subjected to data preprocessing to eliminate the dimension effect. The specific means of removing the dimension effect are technical means well known to those skilled in the art and are not limited here.

[0062] Step S2: extract key points from each action frame image; obtain the gesture change vector of the action frame image based on the spatial distribution difference between each action frame image and the corresponding key points of the temporally adjacent action frame images; divide the action frame image sequence to obtain all local frame image sequences of the action frame image sequence; obtain the distance measurement value of each two action frame image sequences based on the difference in gesture change vectors of the local frame image sequences corresponding to each two action frame image sequences and the difference in the number of local frame image sequences corresponding to each two action frame image sequences and their action frame images.

[0063] The main purpose of the present invention is to ensure the efficiency and accuracy of gesture recognition, thereby improving the effect of gesture recognition. In order to improve the efficiency of gesture recognition, the gestures of the action frame images can be simplified by extracting the key points in each action frame image. Considering that when aphasic patients use gestures to express the meaning of a phrase, the gestures will change to a certain extent, and the gesture change vector of the action frame image is used to reflect the change of the gesture, thereby simplifying the dynamic change characteristics of the gestures of the action frame image. In order to ensure the accuracy of gesture recognition, considering that when expressing the meaning of a phrase, the gestures have a large change, the gesture process is segmented and studied by the large change of the gesture, that is, each local frame image sequence of the action frame image is obtained. In order to analyze the possibility of expressing the same phrase meaning by different action frame image sequences, the distance measurement value of each two action frame image sequences is obtained by the difference of the gesture change vectors of the local frame image sequences contained in each two action frame image sequences. The larger the distance measurement value, the greater the difference in the gesture change corresponding to the two action frame image sequences, and the less likely it is that the phrase meanings corresponding to the two action frame image sequences are the same.

[0064] Preferably, in order to simplify the complexity of gesture analysis and improve gesture recognition efficiency, the gesture action in the action frame image is reflected by extracting key points in each action frame image. In one embodiment of the present invention, the method for obtaining key points includes:

[0065] The semantic segmentation algorithm is used to identify the hand area in each action frame image, and the SIFT algorithm (Scale invariant feature transform) is used to extract the key points of the hand area in each action frame image.

[0066] It should be noted that the semantic segmentation algorithm and the SIFT algorithm are prior arts well known to those skilled in the art, and are only briefly described here. Since the semantic segmentation algorithm can segment an image into multiple regions with specific semantic categories, the hand region in each action frame image can be identified using the semantic segmentation algorithm. Since the SIFT algorithm is an algorithm for extracting and describing local features of an image, and the algorithm detects key points in different scale spaces, and calculates the main direction of the gradient of the neighborhood of the key point as the directional feature of the point, the algorithm has scale invariance and rotation invariance, and has certain stability and robustness to illumination changes, affine transformations, etc., the embodiment of the present invention can use the SIFT algorithm to extract key points in each hand region respectively, and can also select other algorithms with scale invariance and rotation invariance to extract key points in the hand region, which is not limited here.

[0067] Preferably, considering that the process of aphasia patients using gestures to express the meaning of a phrase is a dynamic process, the gestures will change dynamically during this process, that is, the key points reflecting the gesture features in the action frame images in the action frame image sequence will change dynamically, and the gesture change features can be quantified by constructing the gesture change vector of the action frame image. Figure 2 , which shows a flowchart of a gesture change vector method provided by an embodiment of the present invention. In an embodiment of the present invention, the gesture change vector acquisition method includes:

[0068] Step S201: Match key points of each action frame image with its next action frame image, and obtain each matching key point pair and its corresponding ratio test value.

[0069] The matching key point pairs reflect the part of the hand that the action frame image and the next action frame image in the time dimension correspond to. The ratio test value reflects the unreliability of the match. The smaller the ratio test value, the greater the matching credibility, and the more likely the matching key point pairs correspond to the same part of the hand.

[0070] In one embodiment of the present invention, the SIFT algorithm is used to match the key points of each action frame image with the next action frame image in the time dimension, and each matching key point pair and its corresponding ratio test value are obtained. It should be noted that the use of the SIFT algorithm to match the key points of two images is a prior art well known to those skilled in the art. Here, only a brief description is given of the process of using the SIFT algorithm to match the key points of each action frame image with the next action frame image in the time dimension, and obtaining each matching key point pair and its corresponding ratio test value:

[0071] The current action frame image is used as a template image, and its next action frame image in the time dimension is used as the image to be matched. For each key point in the template image, search for the two key points with the closest and second closest Euclidean distance to it in the image to be matched, obtain the closest distance and the second closest distance of each key point in the template image, and then calculate the ratio of the closest distance to the second closest distance to obtain the ratio test value of each key point in the template image. If the ratio test value of the key point in the template image is less than the preset threshold, it is considered that the key point and the closest key point in the image to be matched are a pair of matching key points, that is, a matching key point pair. At the same time, the ratio test value of the key point in the template image in the matching key point pair is used as the ratio test value of the matching key point pair. In an example of the present invention, the preset threshold value is 0.75. The implementer can set it according to the implementation scenario. Finally, the algorithm will output all matching key point pairs that meet the conditions and their corresponding ratio test values.

[0072] Step S202: Obtain the local vector of each matching key point pair according to the spatial positions of the two key points in each matching key point pair.

[0073] The changing characteristics of the same hand parts are reflected through local vectors.

[0074] In one embodiment of the present invention, in the matching key point pair, the local vector of each matching key point pair is determined according to the spatial positions of the two key points of the matching key point pair, and the starting point of the local vector of each matching key point pair is the spatial position corresponding to the key point of the action frame image; the end point of the local vector of each matching key point pair is the spatial position corresponding to the key point of the next action frame image in the temporal dimension. For example, based on the image coordinate system, in the matching key point pair, the coordinates of the key point of the action frame image are (x1, y1), and the coordinates of the key point of the next action frame image in the temporal dimension are (x2, y2). The local vector of the matching key point pair It should be noted that the image coordinate system is a prior art well known to those skilled in the art and will not be described in detail here.

[0075] Step S203: Fusing the local vector and the ratio test value of each matching key point pair to obtain the overall vector of the action frame image.

[0076] In order to preliminarily reflect the changes in gestures in action frame images, the smaller the ratio test value, the greater the matching credibility. The more likely the matching key point pair corresponds to the same part of the hand, that is, the greater the credibility of the local vector, the overall vector of the action frame image is obtained. The overall vector can preliminarily and reliably reflect the changing characteristics of the gesture.

[0077] The sum of the ratio test value of the matching key point pair and the denominator adjustment factor is calculated, and the ratio of the local vector to the sum is calculated to obtain the feature vector of the matching key point pair; the mean of the feature vectors of all matching key point pairs is calculated to obtain the overall vector of the action frame image. In one embodiment of the present invention, the denominator adjustment factor is used to ensure that the denominator is not 0. The denominator adjustment factor is a number greater than 0. In this embodiment, it is set to 0.001, and the implementer can set it according to the implementation scenario.

[0078] Step S204: obtaining a quantity difference index of the action frame images according to the quantity difference of the key points corresponding to each action frame image and the next action frame image.

[0079] Considering that the number of key points of two adjacent action frame images will change with the patient's gestures, the number difference index of the action frame images is obtained. The larger the number difference index, the smaller the difference in the number of key points.

[0080] In one embodiment of the present invention, the absolute value of the difference in the number of key points between each action frame image and the next action frame image in the time dimension is calculated to obtain a quantity difference index of the action frame images, and the quantity difference situation is reflected by the absolute value of the quantity difference.

[0081] Step S205: acquiring a gesture change vector of the action frame image according to the quantity difference index and the overall vector of the action frame image.

[0082] Considering that the smaller the number of key points between two adjacent action frame images, the smaller the change, the higher the credibility of the overall vector in reflecting the change characteristics of the gesture, and the gesture change vector of the action frame image is obtained. The gesture change vector more reliably reflects the change characteristics of the gesture, the direction of the gesture change vector can quantify the gesture change angle, and the size of the gesture change vector can quantify the degree of gesture change.

[0083] In one embodiment of the present invention, the sum of the quantity difference index and the denominator adjustment parameter is calculated, and the reciprocal of the sum is calculated to obtain the quantity similarity index; the product of the quantity similarity index and the overall vector is calculated to obtain the gesture change vector of the action frame image. In one embodiment of the present invention, the denominator adjustment parameter is used to ensure that the denominator is not 0. The denominator adjustment parameter is a number greater than 0. In this embodiment, it is set to 0.01, and the implementer can set it according to the implementation scenario.

[0084] Preferably, considering that the gestures vary greatly when expressing the meaning of a phrase, the gesture process is segmented and studied according to the variations of the gestures, that is, each local frame image sequence of the action frame image is obtained. In one embodiment of the present invention, the method for obtaining the local frame image sequence includes:

[0085] Obtaining a motion change difference value of each motion frame image according to the angle between each motion frame image and the gesture change vector corresponding to the next motion frame image in the time dimension;

[0086] In the action frame image sequence, the action frame images whose action change difference value is greater than the preset amplitude value are used as the segmentation images;

[0087] The action frame image sequence is divided by using each divided image to obtain all local frame image sequences of the action frame image sequence.

[0088] For the above steps, specifically, it should be noted that, given two vectors, obtaining the angle between the two vectors is a technical means well known to those skilled in the art, and the value of the angle between each action frame image and the gesture change vector corresponding to the next action frame image in the time dimension is normalized to obtain the action change difference value of the action frame image. In one embodiment of the present invention, the normalization method is: normalization is performed using the norm normalization function, and the value range is limited to between 0 and 1. In the action frame image sequence, the action frame image with an action change difference value greater than a preset amplitude value is used as a divided image. In one embodiment of the present invention, the preset amplitude value is 0.1, and the implementer can set it according to the implementation scenario. Using each divided image, the action frame image sequence is divided to obtain all local frame image sequences of the action frame image sequence. It should be noted that each divided image is located between the corresponding adjacent local frame image sequences, and each divided image is divided into the next local frame image sequence, which is not limited here.

[0089] Considering that the speed and duration of gestures may vary due to various factors such as health status and emotional state, this difference may lead to inaccurate distance metrics for analyzing different action frame image sequences. In order to reduce the influence of gesture speed,

[0090] In order to analyze the consistency of the meaning of the corresponding phrases of the two action frame image sequences in the subsequent analysis, it is necessary to construct a distance measurement value between the two action frame image sequences. Taking into account the similarities in the change characteristics of gestures corresponding to the same phrase meaning and the large-scale gesture changes, the distance measurement value is constructed to reflect the degree of difference in the gesture changes corresponding to the two action frame image sequences.

[0091] Preferably, see Figure 3 , which shows a flow chart of a distance metric value method provided by an embodiment of the present invention. In an embodiment of the present invention, the method for obtaining the distance metric value includes:

[0092] Step S211: sorting all partial frame image sequences included in the action frame image sequence in chronological order to obtain sequence numbers of all partial frame image sequences in the action frame image sequence.

[0093] Taking into account that actions expressing the same phrase meaning are expressed in a fixed order, in order to facilitate the subsequent comparison of all local frame image sequences in two action frame image sequences, all local frame image sequences contained in the action frame image sequence are sorted in chronological order to obtain the serial numbers of all local frame image sequences in the action frame image sequence.

[0094] Step S212: in every two action frame image sequences, according to the difference in the number of action frame images of the local frame image sequences with the same sequence number and the difference in the gesture change vectors, obtain the action speed difference index between every two action frame image sequences.

[0095] In order to further ensure the accuracy of gesture recognition, taking into account the influence of emotional state, the speed of gesture actions corresponding to the same phrase meaning is different, resulting in differences in the changing characteristics of gesture actions corresponding to the same phrase meaning. The action speed difference index is used to reflect the degree of difference in the gesture action speeds corresponding to the two action frame image sequences, thereby reducing the influence of the gesture action speed on the distance measurement value, so that the distance measurement value can more accurately reflect the possibility of the corresponding phrase meanings of the two action frame image sequences, thereby ensuring the accuracy of gesture recognition.

[0096] Considering that when the gesture action corresponding to the same phrase meaning moves a large distance in a short time, that is, the faster the gesture action rate, the number of action frame images in the local frame image sequence will decrease, resulting in an increase in the module length of the local frame image sequence, the action speed difference index is used to measure the overall speed difference of the two action frame image sequences.

[0097] Preferably, in one embodiment of the present invention, the method of calculating the motion speed difference index includes:

[0098] In every two action frame image sequences, each local frame image sequence in the action frame image sequence containing the largest number of action frame images is used as each target local sequence, and each local frame image sequence in the other action frame image sequence is used as each reference local sequence;

[0099] In every two action frame image sequences, the ratio of the number of action frame images of the target local sequence and the reference local sequence under the same sequence number is calculated to obtain the frame image quantity difference index of the sequence number;

[0100] Calculate the mean value of the modulus lengths corresponding to the gesture change vectors of all action frame images in the local frame image sequence to obtain the local sequence vector modulus length of the local frame image sequence;

[0101] In every two action frame image sequences, the sum of the local sequence vector modulus length and the denominator adjustment value of the reference local sequence is calculated, and the ratio of the local sequence vector modulus length and the sum of the local sequence vector modulus length of the target local sequence under the same sequence number is calculated to obtain the modulus length difference index of the sequence number;

[0102] Calculate the ratio of the module length difference index to the frame image number difference index to obtain the speed scaling amplitude of the sequence number;

[0103] The variance of the speed scaling amplitudes corresponding to all sequence numbers is calculated to obtain the action speed difference index of the two action frame image sequences. In one embodiment of the present invention, the denominator adjustment value is used to ensure that the denominator is not 0, and the denominator adjustment value is a number greater than 0. This embodiment sets it to 0.01, and the implementer can set it according to the implementation scenario.

[0104] For the above steps, in order to compare the difference between the action speeds corresponding to the two action frame image sequences, first, the local frame image sequences in the two action frame image sequences are named separately, and are named as the target local sequence and the reference local sequence respectively. The frame image quantity difference index reflects the difference in the number of action frame images contained in the target local sequence compared with the reference local sequence under the same sequence number. The larger the value of the frame image quantity difference index, the more action frame images contained in the target local sequence than in the reference local sequence under the same sequence number; the local sequence vector modulus reflects the overall modulus of the gesture change vectors of all action frame images contained in the local frame image sequence. The larger the local sequence vector modulus, the larger the overall modulus corresponding to the local frame image sequence. Obtain the modulus length difference index. The larger the modulus length difference index, the longer the local sequence vector modulus corresponding to the target local sequence under the same sequence number is compared with the reference local sequence.

[0105] The speed scaling reflects that as the number of action frame images contained in the target local sequence under the same sequence number is greater than that in the reference local sequence, the corresponding local sequence vector modulus is smaller, which means that the local gesture speed change is more likely to occur. Considering that when the same phrase meaning corresponds to two action frame image sequences, when the overall speed of one action frame image sequence is larger and the overall speed of the other action frame image sequence is smaller, the speed scaling consistency is higher. The action speed difference index reflects the volatility of the change of local gesture speed. The smaller the action speed difference index, the higher the consistency of the speed scaling of the two action frame image sequences, and the higher the possibility that the two action frame image sequences represent the same phrase meaning.

[0106] Step S213: in every two action frame image sequences, according to the action change difference values ​​and gesture change vectors of the corresponding divided images of the local frame image sequences under the same sequence number, obtain the action angle difference index of every two action frame image sequences.

[0107] Taking into account the similarity of gesture action change characteristics and large gesture changes corresponding to the same phrase meaning, that is, the gesture action change characteristics and large gesture changes in the action frame image sequence corresponding to the same phrase meaning are similar, the action angle difference index of each two action frame image sequences is obtained. The action angle difference index reflects the gesture action change characteristics corresponding to the two action frame image sequences. The larger the action angle difference index, the larger the gesture action change characteristics, and the smaller the possibility that the two action frame image sequences represent the same phrase meaning.

[0108] Preferably, see Figure 4 , which shows a flow chart of a method for obtaining an action angle difference index provided by an embodiment of the present invention. The method for obtaining the action angle difference index includes:

[0109] Step S221: acquiring the overall direction value of the local frame image sequence according to the directions of all gesture change vectors of the local frame image sequence.

[0110] The overall direction value is used to quantify the angle change characteristics of gesture actions.

[0111] In one embodiment of the present invention, based on the image coordinate system, the value of the angle between the gesture change vector and the X-axis is normalized to obtain the angle value of the gesture change vector; the mean of all angle values ​​in the local frame image sequence is calculated to obtain the overall direction value of the local frame image sequence. In one embodiment of the present invention, the normalization method used is: normalization is performed using the norm normalization function to limit the value range to between 0 and 1. It should be noted that obtaining the angle between the vector and the X-axis is a prior art well known to those skilled in the art and will not be elaborated here.

[0112] Step S222: obtaining a local sequence direction value of the local frame image sequence according to the motion change difference value and the overall direction value of the divided images of the local frame image sequence.

[0113] The local sequence direction values ​​are used to quantify the angle change characteristics of gesture movements and the large-scale change characteristics of gestures.

[0114] In one embodiment of the present invention, the product of the motion change difference value of the divided images of the local frame image sequence and the overall direction value is calculated to obtain the local sequence direction value of the local frame image sequence. It should be noted that each divided image is located between the corresponding adjacent local frame image sequences, and each divided image is divided into the next local frame image sequence, so the first local frame image sequence in the action frame image sequence does not have a corresponding motion change difference value of the divided image, and the local sequence direction value of the first local frame image sequence is set to the overall direction value.

[0115] Step S223: in two action frame image sequences, according to the difference of the local sequence direction values ​​of the local frame image sequences under each same sequence number, the sequence number angle difference value of the sequence number is obtained.

[0116] The serial number angle difference value reflects the difference between the corresponding gesture change features of the local frame image sequences of two action frame image sequences under the same serial number. The larger the serial number angle difference value, the greater the local gesture difference between the two action frame image sequences.

[0117] In one embodiment of the present invention, the absolute value of the difference between the local sequence direction values ​​of the local frame image sequences corresponding to the two action frame image sequences under each same sequence number is calculated to obtain the sequence number angle difference value.

[0118] Step S224: obtaining the action angle difference index of every two action frame image sequences according to the overall distribution of the sequence angle difference values ​​of all sequence numbers.

[0119] The action angle difference index reflects the overall gesture difference between two action frame image sequences. The larger the action angle difference index is, the greater the overall gesture difference between the two action frame image sequences is, and the smaller the possibility that the two action frame image sequences correspond to phrases with the same meaning is.

[0120] In one embodiment of the present invention, the average of the sequence angle difference values ​​of all sequence numbers is calculated to obtain the action angle difference index of two action frame image sequences.

[0121] Step S214: obtaining a quantity difference index according to the quantity difference of the local frame image sequences included in every two action frame image sequences.

[0122] The larger the quantity difference index is, the greater the difference in the large-scale gesture changes between the two action frame image sequences is, and the smaller the possibility that the two action frame image sequences correspond to phrases with the same meaning is.

[0123] In one embodiment of the present invention, the absolute value of the difference between the number of local frame image sequences included in two action frame image sequences is calculated to obtain a quantity difference index.

[0124] Step S215: obtaining the distance measurement value between every two action frame image sequences according to the action speed difference index, the action angle difference index and the quantity difference index.

[0125] The larger the distance metric value is, the greater the difference in the gesture changes corresponding to the two action frame image sequences is, and the less likely it is that the meanings of the corresponding phrases of the two action frame image sequences are the same.

[0126] In one embodiment of the present invention, the product of the action speed difference index, the action angle difference index and the quantity difference index corresponding to two action frame image sequences is calculated to obtain the distance measurement value of the two action frame image sequences.

[0127] At this point, the distance measurement value of the two action frame image sequences is obtained.

[0128] Step S3: clustering all action frame image sequences in the historical database according to the distance measurement value between every two action frame image sequences to obtain various action categories; performing action recognition on the action frame image sequence to be processed according to the distance measurement value between the action frame image sequence to be processed and the action category.

[0129] In order to more accurately identify the phrase meaning corresponding to the action frame image sequence to be processed, the present invention clusters all action frame image sequences in a historical database to obtain various action categories; the action category can reflect various action frame image sequences corresponding to the same phrase meaning; according to the distance measurement value between the action frame image sequence to be processed and the action category, the action frame image sequence to be processed is more accurately classified into the action category to which the corresponding phrase meaning belongs, thereby improving the effect of action meaning recognition of the action frame image sequence to be processed.

[0130] In one embodiment of the present invention, all action frame image sequences in a historical database are clustered to obtain various action categories. The method for obtaining various action categories includes:

[0131] Using the k-means clustering algorithm, all action frame image sequences in the historical database are clustered according to the distance measurement value between every two action frame image sequences to obtain various action categories.

[0132] It should be noted that the k-means clustering algorithm is a technical means well known to those skilled in the art, and will not be described in detail here. Only a brief description is given of a brief process of clustering all action frame image sequences in a historical database according to the distance metric value between every two action frame image sequences using the k-means clustering algorithm in one embodiment of the present invention to obtain various action categories:

[0133] Considering that the larger the distance metric value between two action frame image sequences, the greater the difference in gesture movements, the optimal k value is obtained through the elbow method, and the distance metric value between every two action frame image sequences is substituted into the clustering process. All action frame image sequences in the historical database are clustered to obtain clustering results and various action categories. The differences in action frame image sequences in different action categories are made as large as possible, and the differences in action frame image sequences in the same action category are made as small as possible. The similarity of gesture behaviors in action frame image sequences in each action category is made as large as possible. In other embodiments of the present invention, the optimal k value of the k-means clustering algorithm can be determined by other methods such as the silhouette coefficient method, which is not limited here.

[0134] In order to more accurately classify the action frame image sequence to be processed into the action category to which the corresponding phrase meaning belongs, in one embodiment of the present invention, the method for performing action recognition on the action frame image sequence to be processed includes:

[0135] Using semantic neural network, obtain action phrases corresponding to action categories;

[0136] Calculate the average of the distance measurement values ​​between the action frame image sequence to be processed and all action frame image sequences in each action category to obtain the category distance value between the action frame image sequence to be processed and each action category;

[0137] The action phrase corresponding to the smallest category distance value is used as the action phrase of the action frame image sequence to be processed.

[0138] It should be noted that semantic neural network is a technical means well known to those skilled in the art, and will not be described in detail here. Only a brief description is given of a brief process of using semantic neural network to obtain action phrases corresponding to action categories in one embodiment of the present invention:

[0139] Combined with the context and meaning corresponding to each action frame image sequence in the action category, the action phrase corresponding to each action category is obtained through a semantic neural network, and the action phrase reflects the actual meaning of the gesture action. In other embodiments of the present invention, each action frame image sequence and its corresponding gesture meaning are stored in a historical database, and the gesture meaning corresponding to the action frame image sequence in the action category is used as the action phrase of the action category.

[0140] The present invention also proposes a gesture recognition system for rehabilitation of aphasia patients, see Figure 5 , which shows a structure diagram of a gesture recognition system for rehabilitation of aphasia patients provided by an embodiment of the present invention. The system includes: an image acquisition module 101, a distance measurement module 102, and a motion recognition module 103.

[0141] The image acquisition module 101 is used to acquire each action frame image sequence in the historical database and the action frame image sequence to be processed; the action frame image sequence includes each action frame image.

[0142] The distance measurement module 102 is used to extract key points in each action frame image; obtain the gesture change vector of the action frame image based on the difference between the spatial distribution of the key points corresponding to each action frame image and the action frame images adjacent in time sequence; divide the action frame image sequence to obtain all local frame image sequences of the action frame image sequence; and obtain the distance measurement value of each two action frame image sequences based on the difference between the gesture change vectors of the local frame image sequences corresponding to each two action frame image sequences.

[0143] The action recognition module 103 is used to cluster all action frame image sequences in the historical database according to the distance measurement value between every two action frame image sequences to obtain various action categories; and perform action recognition on the action frame image sequence to be processed according to the distance measurement value between the action frame image sequence to be processed and the action category.

[0144] It should be noted that: the system provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the gesture recognition system for rehabilitation of aphasia patients provided in the above embodiment and the gesture recognition method embodiment for rehabilitation of aphasia patients belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0145] In summary, the embodiment of the present invention provides a gesture recognition method and system for rehabilitation of aphasia patients, first extracting key points in each action frame image; obtaining the gesture change vector of the action frame image according to the difference between the spatial distribution of key points corresponding to each action frame image and the action frame images adjacent in time sequence; dividing the action frame image sequence to obtain all local frame image sequences of the action frame image sequence; obtaining the distance measurement value of each two action frame image sequences according to the difference between the gesture change vectors of the local frame image sequences corresponding to each two action frame image sequences; performing action recognition on the action frame image sequence to be processed according to the distance measurement value between the action frame image sequence to be processed and the action category. In the embodiment of the present invention, the features of gesture recognition are deeply analyzed to ensure the accuracy and efficiency of gesture recognition of aphasia patients and improve the gesture recognition effect.

[0146] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0147] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A gesture recognition method for rehabilitation of aphasia patients, characterized in that: The method comprises: Acquire each action frame image sequence and the action frame image sequence to be processed in the historical database; the action frame image sequence includes each action frame image; Extract key points from each of the action frame images; obtain the gesture change vector of the action frame image according to the spatial distribution difference between each of the action frame images and the key points corresponding to the action frame images adjacent in time sequence; divide the action frame image sequence to obtain all local frame image sequences of the action frame image sequence; obtain the distance measurement value of each of the action frame image sequences according to the difference between the gesture change vectors of the local frame image sequences corresponding to each two action frame image sequences and the difference between the local frame image sequences and the number of action frame images corresponding to each two action frame image sequences; Clustering all action frame image sequences in the historical database according to the distance measurement value between each two action frame image sequences to obtain various action categories; performing action recognition on the action frame image sequence to be processed according to the distance measurement value between the action frame image sequence to be processed and the action category; The method for acquiring the gesture change vector includes: Matching key points of each action frame image with its next action frame image, and obtaining each matching key point pair and its corresponding ratio test value; Acquire a local vector of each matching key point pair according to the spatial positions of the two key points in each matching key point pair; Fusing the local vector of each matching key point pair with the ratio test value to obtain an overall vector of the action frame image; According to the difference in the number of key points corresponding to each action frame image and the next action frame image, obtaining a quantity difference index of the action frame images; Acquire a gesture change vector of the action frame image according to the quantity difference index and the overall vector of the action frame image; The method for acquiring the local frame image sequence comprises: Obtaining a motion change difference value of each of the motion frame images according to an angle between each of the motion frame images and the gesture change vector corresponding to the next of the motion frame images; In the action frame image sequence, taking the action frame images whose action change difference values ​​are greater than the preset amplitude value as the segmentation images; Using each of the divided images, the action frame image sequence is divided to obtain all local frame image sequences of the action frame image sequence; The method for obtaining the distance metric value includes: Sorting all the partial frame image sequences in the action frame image sequence in a time sequence order to obtain sequence numbers of all the partial frame image sequences in the action frame image sequence; In each of two action frame image sequences, according to the difference in the number of action frame images of the local frame image sequences under the same sequence number and the difference in the gesture change vector, the action speed difference index of each of the two action frame image sequences is obtained; In every two action frame image sequences, according to the action change difference value and the gesture change vector of the corresponding divided image of the local frame image sequence under the same sequence number, the action angle difference index of every two action frame image sequences is obtained; According to the difference in the number of the local frame image sequences included in every two action frame image sequences, obtaining a quantity difference index; According to the action speed difference index, the action angle difference index and the quantity difference index, a distance measurement value between every two action frame image sequences is obtained.

2. A gesture recognition method for rehabilitation of aphasia patients according to claim 1, characterized in that: The method of the motion speed difference index includes: In every two action frame image sequences, each local frame image sequence in the action frame image sequence containing the largest number of the action frame images is used as each target local sequence, and each local frame image sequence in the other action frame image sequence is used as each reference local sequence; In every two action frame image sequences, the ratio of the number of the action frame images of the target local sequence and the reference local sequence under the same sequence number is calculated to obtain a frame image quantity difference index of the sequence number; Calculating the mean value of the modulus lengths corresponding to the gesture change vectors of all the action frame images in the local frame image sequence to obtain the local sequence vector modulus length of the local frame image sequence; In every two action frame image sequences, the sum of the local sequence vector modulus length and the denominator adjustment value of the reference local sequence is calculated, and the ratio of the local sequence vector modulus length and the sum of the local sequence vector modulus length of the target local sequence under the same sequence number is calculated to obtain the modulus length difference index of the sequence number; Calculate the ratio of the frame image quantity difference index to the module length difference index to obtain the speed scaling amplitude of the sequence number; The variance of the speed scaling amplitudes corresponding to all sequence numbers is calculated to obtain the motion speed difference index of two motion frame image sequences.

3. A gesture recognition method for rehabilitation of aphasia patients according to claim 1, characterized in that: The method for obtaining the action angle difference index includes: According to the directions of all the gesture change vectors of the local frame image sequence, obtaining an overall direction value of the local frame image sequence; Obtaining a local sequence direction value of the local frame image sequence according to the motion change difference value and the overall direction value of the divided image of the local frame image sequence; In every two action frame image sequences, according to the difference of the local sequence direction values ​​of the local frame image sequences under each same sequence number, the sequence number angle difference value of the sequence number is obtained; According to the overall distribution of the sequence angle difference values ​​of all sequence numbers, the action angle difference index of every two action frame image sequences is obtained.

4. A gesture recognition method for rehabilitation of aphasia patients according to claim 3, characterized in that: The method for obtaining the local sequence direction value includes: The product of the motion change difference value of the divided image of the local frame image sequence and the overall direction value is calculated to obtain the local sequence direction value of the local frame image sequence.

5. The gesture recognition method for rehabilitation of aphasia patients according to claim 1, characterized in that: The method for performing action recognition on the action frame image sequence to be processed comprises: Using semantic neural network, obtain action phrases corresponding to action categories; Calculate the average of the distance measurement values ​​between the action frame image sequence to be processed and all action frame image sequences in each action category to obtain the category distance value between the action frame image sequence to be processed and each action category; The action phrase corresponding to the action category with the smallest category distance value is used as the action phrase of the action frame image sequence to be processed.

6. The gesture recognition method for rehabilitation of aphasia patients according to claim 1, characterized in that: The method for obtaining the key points includes: The semantic segmentation algorithm is used to identify the hand region in each of the action frame images, and key points of the hand region are extracted to obtain the key points in each of the action frame images.

7. A gesture recognition system for rehabilitation of aphasia patients, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a gesture recognition method for rehabilitation of aphasia patients as described in any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Authentication and authorization method, processor, equipment and mobile terminal

    CN102509053A

  • Gesture recognition method and related product

    CN115661917A