A gesture recognition and classification method and system based on fuzzy neural network

Abstract

This invention proposes a gesture recognition and classification method and system based on a fuzzy neural network. The method involves acquiring a gesture video to be recognized and preprocessing it to obtain a gesture image. A gesture recognition and classification model is constructed, comprising a fuzzy processing layer, a neural network layer, and a fusion layer. The gesture image is input into the fuzzy logic layer and the neural network layer for processing, respectively, to obtain a first feature output and a second feature output. The first and second feature outputs are weightedly fused through the fusion layer to output the gesture recognition result. This invention combines fuzzy systems with neural network learning to form a network structure with adaptive capabilities, experience accumulation, and fuzzy information processing capabilities.

Description

Technical Field

[0001] The present invention belongs to the technical field related to gesture recognition and classification, and in particular relates to a gesture recognition and classification method and system based on fuzzy neural network. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Gesture recognition is widely used across the industry, particularly in sign language interpretation for the deaf and mute and contactless hygiene solutions. Gesture recognition algorithms based on the K-nearest neighbor (KNN) method first segment the hand region, extracting the target region's contour and Hu moment features. Before classification, a template library suitable for the data must be established. This technology involves multiple steps, including segmentation, extraction and screening, and recognition and classification. Consequently, this technology suffers from poor recognition efficiency and a limited range. Using bioelectrical signals to analyze gesture information is also a popular technique. This technique first acquires muscle signals and then uses muscle changes to determine muscle electrical signals under different conditions. However, this technique requires wearable devices and carries the risk of injury to the human body. Sensor-based gesture recognition captures hand motion and spatial position to identify gestures. This technology delivers real-time data without segmentation, but recognition accuracy can be affected if the user's gestures are not standard. Image-based gesture recognition typically uses physical cameras to collect information, pre-processing and segmenting it before analysis. This technique offers high accuracy but requires high background sensitivity.

[0004] The actions in gesture recognition are becoming more and more diverse, and with them comes an increase in the amount of data. The current application of various neural network algorithms has greatly solved this problem, but neural networks can only describe the complex functional relationships between large amounts of data, but cannot express human empirical knowledge. Summary of the Invention

[0005] To overcome the shortcomings of the above-mentioned existing technologies, the present invention provides a gesture recognition and classification method and system based on a fuzzy neural network. This method utilizes a dual-stream structure to process the same information in different ways. First, a fuzzy processing layer is used to obtain fuzzy information, and then a neural network is used to obtain abstract information. The weights are dynamically adjusted according to the membership function value to achieve the fusion of the two types of information, thereby improving the classification accuracy.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: a gesture recognition and classification method based on a fuzzy neural network, comprising the following steps:

[0007] Obtain the gesture video to be recognized and preprocess it to obtain a gesture image;

[0008] Constructing a gesture recognition classification model, wherein the gesture recognition classification model includes a fuzzy processing layer, a neural network layer, and a fusion layer;

[0009] The gesture image is input into the fuzzy processing layer and the neural network layer for processing respectively to obtain the first feature output and the second feature output;

[0010] The first feature output and the second feature output are weighted fused through the fusion layer to output the gesture recognition result.

[0011] A second aspect of the present invention provides a gesture recognition and classification system based on a fuzzy neural network, comprising:

[0012] Preprocessing module: obtains the gesture video to be recognized and preprocesses it to obtain the gesture image;

[0013] Classification model building module: building a gesture recognition classification model, the gesture recognition classification model includes a fuzzy processing layer, a neural network layer and a fusion layer;

[0014] Data processing module: inputs the gesture image into the fuzzy processing layer and the neural network layer for processing, and obtains the first feature output and the second feature output;

[0015] Classification and recognition module: The first feature output and the second feature output are weighted fused through the fusion layer to output the gesture recognition result.

[0016] A third aspect of the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the above method.

[0017] A fourth aspect of the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein the computer instructions complete the steps of the above method when executed by the processor.

[0018] One or more of the above technical solutions have the following beneficial effects:

[0019] The present invention utilizes a dual-stream structure to process the same input information in different ways. First, a fuzzy processing layer is used to obtain fuzzy information, and then a neural network is used to obtain abstract information. The weights are dynamically adjusted according to the membership function value to achieve the fusion of the two types of information and improve the classification accuracy.

[0020] The present invention uses a grayscale separation method to preprocess gesture videos. The grayscale separation method can reduce interference from external factors and retain only the most valuable grayscale information. This grayscale information has low noise and high recognizability, and can also reduce the time cost of network operation. The overall classification effect is much better than the original image.

[0021] The present invention combines neural networks and fuzzy systems for gesture classification, fully considering the complementarity of the two, integrating logical reasoning, language calculation, and nonlinear dynamics, and has functions such as learning, association, recognition, adaptation, and fuzzy information processing capabilities.

[0022] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0024] Figure 1 This is a schematic diagram of image recognition in the first embodiment of the present invention;

[0025] Figure 2 It is a structural diagram of the classification model in the first embodiment of the present invention. DETAILED DESCRIPTION

[0026] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0027] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present invention.

[0028] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0029] Example 1

[0030] like Figure 1-Figure 2 As shown, this embodiment discloses a gesture recognition and classification method based on a fuzzy neural network, comprising the following steps:

[0031] Step 1: Obtain the gesture video to be recognized and preprocess it to obtain a gesture image;

[0032] Step 2: Constructing a gesture recognition classification model, which includes a fuzzy processing layer, a neural network layer, and a fusion layer;

[0033] Step 3: Input the gesture image into the fuzzy processing layer and the neural network layer for processing to obtain the first feature output and the second feature output;

[0034] Step 4: Perform weighted fusion on the first feature output and the second feature output through the fusion layer to output the gesture recognition result.

[0035] In this embodiment, the training data is based on the EgoHands dataset and the GTEA dataset. The EgoHands dataset is a dataset of human interactions, consisting of 48 video clips recorded using Google Class. Each video clip depicts two participants engaging in hand-to-hand interactions while playing puzzles, cards, Jenga, or chess. These videos were captured in three different environments: an office, a courtyard, and a bedroom. The dataset contains over 15,000 hand instances, each of which includes 100 manually annotated hand mask images, for a total of 4,800 annotated hand mask images.

[0036] The GTEA dataset collects videos of seven daily activities. The videos were collected under uniform environmental conditions, without any human interaction, and under static lighting conditions. The segmented dataset covers the hand and arm regions and contains 663 manually annotated images.

[0037] In step 1 of this embodiment, based on the idea of ​​the optical flow method, targeted adjustments and improvements are made according to the actual situation, and a "grayscale separation method" is proposed.

[0038] like Figure 1 As shown in the figure, the grayscale separation method first divides the gesture video into frames and collects action images at a frame rate of 0.5s. Because different backgrounds and faces appear in the video at the same time, the hand and face segmentation method is used to separate the hands and face from the background to obtain a picture of only the hands and face; finally, the image of each frame of the video with hand and face segmentation is combined into a grayscale image according to the principle of decreasing transparency based on the multiple images extracted according to the frame rate. In this way, each sign language video is converted into a picture, that is, 3D video recognition is converted into 2D image recognition.

[0039] For gesture segmentation, this embodiment adopts a hand segmentation method based on convolutional neural network, which mainly consists of two main branches: flow-guided feature propagation and lightweight occlusion-aware detail enhancement.

[0040] The flow-guided feature propagation branch runs the image segmentation network only on sparse frames and warps intermediate features to other frames based on the cross-frame flow field. This achieves significant speedup compared to per-frame inference, but incurs a significant accuracy drop due to distortion and occlusion issues during the warping. By introducing a lightweight occlusion-aware detail enhancement branch that leverages spatial attention over occluded regions to extract low-level detail features from each frame, the method becomes more robust to distortion and occlusion issues and achieves a better accuracy-latency trade-off.

[0041] This embodiment compares the classification accuracy of the original image and the image obtained by grayscale separation, and finds that the classification accuracy of the latter is higher than that of the original image, which shows that the grayscale separation method can play a role in noise reduction.

[0042] Fuzzy systems have advantages such as knowledge storage, fuzzy thinking, strong reasoning ability and expert knowledge, but their adaptability is poor. Neural networks have advantages such as strong learning ability, suitable for processing batch data, and high fault tolerance, but they cannot utilize system information and expert knowledge. Figure 2 As shown, this embodiment combines a neural network with a fuzzy system to form a dynamically integrated fuzzy neural network architecture as a classification model for gesture recognition. It has the capabilities of self-adaptation, experience accumulation, and fuzzy information processing. This structure mainly forms an architecture from fuzzy to clear by inputting fuzzy information into the model.

[0043] In this embodiment, the fuzzy processing layer adds membership functions and fuzzy rules to a neural network, creating another form of neural network. Based on these membership functions and fuzzy rules, the fuzzy processing layer fuzzifies and normalizes the input signal to obtain an output variable. The fuzzy processing layer consists of two layers: the first layer calculates the membership, and the second layer obtains the fuzzified output variable, normalizes it, and clarifies the variable.

[0044] After the input layer, we first use the membership function Calculate the membership of the input variables. Assume that the input variable x corresponding to the gesture image is x=[x1,x2,x3,…,x i ], according to the membership function Get x j The corresponding fuzzy output variable is:

[0045]

[0046] in, Represents the membership function, that is, the center of the Gaussian function, represents the width of the membership function, n is the number of fuzzy rules, and Me is the output variable after fuzzification.

[0047] By normalizing the output variables of the fuzzy processing layer, Mz is obtained, and then the weight variable ω is used to calculate the k The weight represents the confidence of each rule. Before being input into the fusion layer, the fuzzy output variable is clarified to form the accurate value Ms of the output variable:

[0048]

[0049] Among them, Ms is the first feature output that is finally input to the fusion layer.

[0050] For the neural network layer, the input variables are abstracted by the neural network layer, and any input variable of the input layer will enter each neuron. The variable M is obtained through the neural network layer. t .

[0051] In step 3 of this embodiment, the features extracted from a single image are generally unable to fully obtain the deep information of the data. The input data is fuzzy processed in the fuzzy processing layer to reduce uncertainty, and the depth and breadth of the input data are increased in the neural network layer. Therefore, after processing through the fuzzy processing layer and the neural network layer, a dynamic fusion layer is established to combine the fuzzy processed variables with the variables processed by the neural network layer to form parameter variables with two network characteristics, ultimately achieving the classification of gesture activities.

[0052] In the fusion layer of this embodiment, the outputs of the neural network layer and the fuzzy processing layer are regarded as extracted feature vectors. The two types of outputs are connected to weights S and T respectively. The two weights are dynamically adjusted according to the output values ​​and the fused information value is calculated based on the weights. The calculation formula is:

[0053]

[0054] Among them, S and T are based on M s and M t The value is adjusted dynamically.

[0055] In order to prove the effectiveness of the fused information, the present invention classifies the information extracted by the fuzzy processing layer and the neural network layer respectively. The accuracy of direct classification by the fuzzy processing layer is 0.82, while the accuracy of direct classification by the neural network layer is only 0.71. Therefore, compared with the classification accuracy after fusion, this fusion method is effective.

[0056] This embodiment uses the above classification model to recognize and classify gestures. Before training, the database is divided into 13 common sign language categories. The final result is the membership of a certain action in each category. The category with the highest membership is the predicted category.

[0057] In this paper, we use accuracy, precision and membership to evaluate the method.

[0058] The accuracy is calculated as follows:

[0059]

[0060] Among them, TP represents the number of samples correctly classified into the positive category, FP represents the number of samples incorrectly classified into the negative category, TN represents the number of samples correctly classified into the negative category, and FN represents the number of samples incorrectly classified into the positive category.

[0061] The precision is calculated as follows:

[0062]

[0063] Among them, TP represents the number of samples correctly classified into the positive category, and FP represents the number of samples incorrectly classified into the negative category.

[0064] In this embodiment, 500 original images are used for identification and classification. Table 1 shows the results of this classification.

[0065] Table 1 Classification results of the present invention

[0066]

[0067]

[0068] As shown in Table 1, the original data is subdivided into 13 categories, where categories with the same capital letter have the same label. For example, in the present invention, A represents a greeting gesture, and B represents a sports gesture. The results show that the average classification accuracy of the present invention is 0.85, and the accuracy reaches 0.87. It is worth noting that the test data used in the present invention are original images, some of which are noisy. Of course, if the aforementioned processing method is used for further testing, the accuracy can reach 0.95. Therefore, the classification results of the present invention are quite advantageous.

[0069] Example 2

[0070] The purpose of this embodiment is to provide a gesture recognition and classification system based on a fuzzy neural network, including:

[0071] Preprocessing module: obtains the gesture video to be recognized and preprocesses it to obtain the gesture image;

[0072] Classification model building module: building a gesture recognition classification model, the gesture recognition classification model includes a fuzzy processing layer, a neural network layer and a fusion layer;

[0073] Data processing module: inputs the gesture image into the fuzzy processing layer and the neural network layer for processing, and obtains the first feature output and the second feature output;

[0074] Classification and recognition module: The first feature output and the second feature output are weighted fused through the fusion layer to output the gesture recognition result.

[0075] Example 3

[0076] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0077] Example 4

[0078] The purpose of this embodiment is to provide a computer-readable storage medium.

[0079] A computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the above method.

[0080] The steps involved in the apparatuses of Examples 2, 3, and 4 above correspond to those of Method Example 1. For detailed implementations, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any method of the present invention.

[0081] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0082] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A gesture recognition and classification method based on fuzzy neural network, characterized in that: The following steps are involved: Obtain the gesture video to be recognized and preprocess it to obtain a gesture image; Construct a gesture recognition classification model, which includes a fuzzy processing layer, a neural network layer, and a fusion layer; the fuzzy processing layer uses fuzzy rules to fuzzify gesture images, specifically: The membership function is used to calculate the membership of the input variable corresponding to the gesture image, and the fuzzy output variable is obtained through fuzzy rules; Based on the fuzzy output variables belonging to different variables, the fuzzy output variables are normalized and weighted connected to obtain the first feature output; The gesture image is input into the fuzzy processing layer and the neural network layer for processing respectively to obtain the first feature output and the second feature output; The first feature output and the second feature output are weightedly fused through the fusion layer to output the gesture recognition result; wherein, the first feature output and the second feature output are weightedly fused through the fusion layer, specifically: the first feature output and the second feature output are connected to weights S and T respectively, the weights S and T are dynamically adjusted according to the values ​​of the first feature output and the second feature output, and the fused value is obtained according to the weight calculation.

2. The gesture recognition and classification method based on fuzzy neural network according to claim 1, characterized in that: The specific method for preprocessing the acquired gesture video is as follows: A hand gesture video is framed and segmented to separate the hands and face from the background, resulting in a picture of only the hands and face. The images of each frame of the video separated by the hands and face are synthesized into a grayscale image in a manner of decreasing transparency.

3. The gesture recognition and classification method based on fuzzy neural network according to claim 1, characterized in that: The membership calculation formula is: in, c k j Represents the membership function, that is, the center of the Gaussian function, b k j represents the width of the membership function, n is the number of fuzzy rules, Me is the output variable after fuzzification.

4. A gesture recognition and classification system based on fuzzy neural network, characterized in that: include: Preprocessing module: obtains the gesture video to be recognized and preprocesses it to obtain the gesture image; Classification model construction module: Constructs a gesture recognition classification model, which includes a fuzzy processing layer, a neural network layer, and a fusion layer. The fuzzy processing layer uses fuzzy rules to fuzzify gesture images, specifically: The membership function is used to calculate the membership of the input variable corresponding to the gesture image, and the fuzzy output variable is obtained through fuzzy rules; Based on the fuzzy output variables belonging to different variables, the fuzzy output variables are normalized and weighted connected to obtain the first feature output; Data processing module: inputs the gesture image into the fuzzy processing layer and the neural network layer for processing, and obtains the first feature output and the second feature output; Classification and recognition module: The first feature output and the second feature output are weightedly fused through the fusion layer to output the gesture recognition result; wherein, the first feature output and the second feature output are weightedly fused through the fusion layer, specifically: the first feature output and the second feature output are connected to weights S and T respectively, the weights S and T are dynamically adjusted according to the values ​​of the first feature output and the second feature output, and the fused value is obtained according to the weight calculation.

5. The gesture recognition and classification system based on fuzzy neural network according to claim 4, characterized in that: The pre-processing module includes: performing frame processing on a gesture video, separating the hands and face from the background using a segmentation method, and obtaining a picture of only the hands and face; The images of each frame of the video separated by the hands and face are synthesized into a grayscale image in a manner of decreasing transparency.

6. The gesture recognition and classification system based on fuzzy neural network according to claim 4, characterized in that: The data processing module includes: using a membership function to calculate the membership corresponding to each input component to obtain a fuzzy output variable; Based on the fact that the fuzzy output variables belong to different variables, the fuzzy output variables are matched with fuzzy rules one by one, normalized according to the obtained matching degree, and weight connection and variable clarification are performed to obtain the first feature output.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the gesture recognition and classification method based on a fuzzy neural network as claimed in any one of claims 1 to 3 are implemented.

8. A processing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the gesture recognition and classification method based on fuzzy neural network as described in any one of claims 1 to 3 are implemented.

Images