Deaf-mute gesture recognition method based on DCGAN-Yolov5 algorithm

Abstract

The invention belongs to the technical field of computer vision and artificial intelligence, and particularly relates to a deaf-mute gesture recognition method based on a DCGAN-Yolov5 algorithm, and the method comprises the following steps: 1, firstly, making a deaf-mute gesture original data set which comprises different gesture sample pictures of a deaf-mute; 2, expanding the different gesture sample pictures in the step 1 by using DCGAN to obtain a mixed data set; and step 3, performing label labeling operation on the mixed data set obtained in the step 2 to obtain a label corresponding to the mixed data set, and putting the label into a PostgreSQL database. According to the method, an original data set is expanded by adopting a DCGAN (deep convolutional generative adversarial network) model, so that the problem that an identification model is easy to over-fit under the condition of insufficient data volume is effectively solved. By means of the data enhancement technology, the recognition model can learn feature representation with richer samples, and therefore better generalization ability is shown in practical application.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and artificial intelligence, and in particular to a method for hand gesture recognition for deaf-mute people based on a DCGAN-Yolov5 algorithm. Background Art

[0002] In the field of computer vision and artificial intelligence, gesture recognition technology has become an important research direction. Especially for the deaf-mute group, gesture recognition technology can greatly improve their ability to communicate with society. Traditional gesture recognition methods often rely on complex sensor equipment and algorithms with high computational costs, which limits their popularity in practical applications. With the development of deep learning technology, especially the application of convolutional neural networks (CNN) and generative adversarial networks (GAN), the accuracy and real-time performance of gesture recognition have been significantly improved.

[0003] Deep Convolutional Generative Adversarial Network (DCGAN) is a model that combines deep learning with adversarial training ideas. It generates high-quality image data through the adversarial process of the generator and the discriminator. The goal of the generator is to produce realistic images, while the discriminator tries to distinguish between real images and images produced by the generator. In the field of gesture recognition, DCGAN can be used to generate training data and enhance the generalization ability of the model, especially when data is scarce.

[0004] YOLO (You Only Look Once) algorithm is a popular target detection algorithm. It predicts the target location and category in an image simultaneously through a unified neural network. It has the characteristics of fast recognition speed and high accuracy, and is particularly suitable for scenarios that require real-time processing, such as video surveillance and autonomous driving. In gesture recognition for the deaf, YOLOv5 can quickly and accurately detect sign language gestures, providing a basis for subsequent recognition and translation.

[0005] Domestic patents involving the application of Yolov5 method in gesture recognition include "Gesture Dataset Collection Method, Gesture Recognition Method and Device for YOLO Network" (CN201810280674.X), which solves the problem of poor suppression of complex background noise such as faces and skin-colored walls in pictures. The national invention patent "A gesture recognition method based on the yolo-v3 framework integrating horizontal and vertical information" (CN202111559476.5) improves the accuracy of gesture recognition by addressing the problem of inconsistent size scale of gestures in different scenes. Although the above two national invention patents have applied the Yolo series to solve the problem of gesture recognition accuracy from different angles and degrees in the field of gesture recognition, they both have insufficient sample size, uneven sample distribution, and lack of diversity in the data set, which leads to overfitting of the Yolo series model and weak generalization ability of the model. Therefore, we propose a gesture recognition method for the deaf-mute based on the DCGAN-Yolov5 algorithm to solve the above problems. Summary of the invention

[0006] 1. Technical issues to be solved

[0007] In view of the deficiencies in the prior art, the present invention provides a hand gesture recognition method for the deaf-mute based on the DCGAN-Yolov5 algorithm, which solves the problems raised in the above background technology.

[0008] (II) Technical solution

[0009] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0010] The hand gesture recognition method for the deaf-mute based on the DCGAN-Yolov5 algorithm includes the following steps:

[0011] Step 1: First, create an original data set of hand gestures for the deaf-mute. The original data set includes different hand gesture sample images of the deaf-mute.

[0012] Step 2: Use (DCGAN) to expand the different gesture sample images in step 1 to obtain a mixed data set;

[0013] Step 3: Label the mixed data set obtained in step 2 to obtain the labels corresponding to the mixed data set, and put the labels into the PostgreSQL database;

[0014] Step 4: Use the mixed data set and corresponding labels obtained in step 3 as the input of the Yolov5 target detection model and perform multiple rounds of training to complete the pre-training operation of the Yolov5 target detection and recognition model;

[0015] Step 5: Use a high-definition camera to collect different gesture pictures of the deaf-mute, and then use DCGAN to expand the collected gesture picture samples to obtain a new data set, perform labeling operations, and then put the corresponding labels into the PostgreSQL database;

[0016] Step 6: Use the newly added data set and its corresponding labels to train Yolov5 again, obtain the updated model and save it;

[0017] Step 7: Perform the final deaf-mute gesture recognition operation through the trained Yolov5 target detection and recognition model to detect the information and meaning represented by the gesture, and the whole process ends here.

[0018] Furthermore, the training steps of the DCGAN network model in step 2 are:

[0019] Step 2.1: Initialize the network, the generator G is used to generate new data samples, and the discriminator D is used to distinguish between real data and fake data generated by the generator;

[0020] Step 2.2: Input noise, the generator accepts a random noise that follows a Gaussian distribution as input;

[0021] Step 2.3: The generator generates a fake data sample through its network structure;

[0022] Step 2.4: Train the discriminator. The discriminator receives real data and fake data as input, uses the Neural Characteristic Functions Loss (NCFL) to calculate the discriminator's loss, and performs backpropagation to update the discriminator's weights.

[0023] Step 2.5: Train the generator and input the generated fake data into the discriminator. The discriminator will output a probability value, indicating the probability that the input data is real data. The neural feature loss function uses this probability value to calculate the loss of the generator and back-propagate to update the weight of the generator.

[0024] Step 2.6: Alternately perform the above two training steps until the image quality generated by the model meets the requirements or the predetermined number of training rounds is completed; the loss function of the entire adversarial process is expressed as:

[0025]

[0026] Among them, X represents the real data, Y represents the set of total label categories corresponding to each sample in the real data set X, Represents generated data, Representing and Generating Data The corresponding generated label, y represents a specific label category in Y; and Represent the empirical characteristic functions (ECFs) of the real and generated joint distributions, t iy is the projection function parameter sampled from a certain distribution, and k is the number of samples.

[0027] Furthermore, the training steps of the Yolov5 target detection and recognition model in step 4 are:

[0028] Step 4.1: Input the expanded gesture images and their corresponding labels into the Yolov5 model;

[0029] Step 4.2: The input is forward propagated through the backbone, neck and head layers in sequence; the backbone layer is responsible for extracting features from the input image; the neck layer further processes and fuses the features extracted by the backbone to enhance the expressiveness of the features; the head layer predicts the category, location and confidence of the target based on the fused feature map;

[0030] Step 4.3: Compare the output of the model with the true label and calculate the loss function, which includes three loss functions: classification loss, localization loss and confidence loss. The calculation formula of the classification loss function is as follows:

[0031]

[0032] Among them, N represents the total number of categories, yn is the probability predicted by the model, is the true label (0 or 1);

[0033] Positioning loss function:

[0034] L local =1-CIOU(B,B gt )

[0035] Among them, B is the prediction box, B gt For the ground-truth box, the CIOU calculation takes into account the overlapping area, center point distance, and aspect ratio;

[0036] Confidence loss function:

[0037]

[0038] Among them, p nis the confidence of the model prediction, is the true confidence label (0 or 1), usually for the predicted box that matches the target, otherwise

[0039] L=λ obj ·L obj +λ class ·L class +λ local ·L local

[0040] Among them, λ obj , class and λ local They are the weight coefficients of the corresponding loss items, which are used to balance the impact of different loss items on the total loss;

[0041] Step 4.4: Repeat the above steps, according to the loss function, through the back propagation algorithm, use the optimizer Adam to adjust the learning rate and update the model parameters until the loss function converges and the model training is completed.

[0042] Furthermore, the specific content of step 6 is: using the newly expanded additional data set and the corresponding labels to perform transfer learning training on the original trained Yolov5 recognition model, wherein the training process is the same as steps 4.1-4.4, so that the YOLOv5 recognition model has better generalization ability, so that it can accurately recognize gestures in different environments and conditions.

[0043] (III) Beneficial effects

[0044] Compared with the prior art, the present invention provides a deaf-mute gesture recognition method based on the DCGAN-Yolov5 algorithm, which has the following beneficial effects:

[0045] 1. This invention uses the DCGAN (Deep Convolutional Generative Adversarial Network) model to expand the original data set, effectively solving the problem that the recognition model is prone to overfitting when the amount of data is insufficient. Through this data enhancement technology, the recognition model can learn richer feature representations of samples, thereby showing better generalization ability in practical applications.

[0046] 2. The present invention effectively enhances the recognition ability of gesture image samples by using the Yolov5 model. Based on the extended image samples generated by the improved DCGAN, the model can accurately capture the distribution characteristics between different gestures, which not only reduces the overfitting problem, but also realizes the fast and accurate recognition of different gestures.

[0047] 3. The present invention utilizes the advantage of PostgreSQL database in storing unstructured data and can store image tags, thereby improving the efficiency of Yolov5 recognition model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the process of the present invention;

[0049] Figure 2 Schematic diagram of the training structure of the DCGAN model used in the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Example

[0052] like Figure 1-2 As shown, a method for hand gesture recognition for the deaf-mute based on the DCGAN-Yolov5 algorithm proposed in one embodiment of the present invention includes the following steps:

[0053] Step 1: First, create an original data set of hand gestures for the deaf-mute. The original data set includes different hand gesture sample images of the deaf-mute.

[0054] Step 2: Use (DCGAN) to expand the different gesture sample images in step 1 to obtain a mixed data set;

[0055] The training steps of the DCGAN network model are:

[0056] Step 2.1: Initialize the network, the generator G is used to generate new data samples, and the discriminator D is used to distinguish between real data and fake data generated by the generator;

[0057] Step 2.2: Input noise, the generator accepts a random noise that follows a Gaussian distribution as input;

[0058] Step 2.3: The generator generates a fake data sample through its network structure;

[0059] Step 2.4: Train the discriminator. The discriminator receives real data and fake data as input, uses the Neural Characteristic Functions Loss (NCFL) to calculate the discriminator's loss, and performs backpropagation to update the discriminator's weights.

[0060] Step 2.5: Train the generator and input the generated fake data into the discriminator. The discriminator will output a probability value, indicating the probability that the input data is real data. The neural feature loss function uses this probability value to calculate the loss of the generator and back-propagate to update the weight of the generator.

[0061] Step 2.6: Alternately perform the above two training steps until the image quality generated by the model meets the requirements or the predetermined number of training rounds is completed; the loss function of the entire adversarial process is expressed as:

[0062]

[0063] Among them, X represents the real data, Y represents the set of total label categories corresponding to each sample in the real data set X, Represents generated data, Representing and Generating Data The corresponding generated label, y represents a specific label category in Y; and Represent the empirical characteristic functions (ECFs) of the real and generated joint distributions, t iy is the projection function parameter sampled from a certain distribution, and k is the number of samples.

[0064] Step 3: Label the mixed data set obtained in step 2 to obtain the labels corresponding to the mixed data set, and put the labels into the PostgreSQL database;

[0065] Step 4: Use the mixed data set and corresponding labels obtained in step 3 as the input of the Yolov5 target detection model and perform multiple rounds of training to complete the pre-training operation of the Yolov5 target detection and recognition model;

[0066] The training steps of the Yolov5 target detection and recognition model are:

[0067] Step 4.1: Input the expanded gesture images and their corresponding labels into the Yolov5 model;

[0068] Step 4.2: The input is forward propagated through the backbone, neck and head layers in sequence; the backbone layer is responsible for extracting features from the input image; the neck layer further processes and fuses the features extracted by the backbone to enhance the expressiveness of the features; the head layer predicts the category, location and confidence of the target based on the fused feature map;

[0069] Step 4.3: Compare the output of the model with the true label and calculate the loss function, which includes three loss functions: classification loss, localization loss and confidence loss. The calculation formula of the classification loss function is as follows:

[0070]

[0071] Among them, N represents the total number of categories, y n is the probability predicted by the model, is the true label (0 or 1);

[0072] Positioning loss function:

[0073] L local =1-CIOU(B,B gt )

[0074] Among them, B is the prediction box, B gt For the ground-truth box, the CIOU calculation takes into account the overlapping area, center point distance, and aspect ratio;

[0075] Confidence loss function:

[0076]

[0077] Among them, p n is the confidence of the model prediction, is the true confidence label (0 or 1), usually for the predicted box that matches the target, otherwise

[0078] L=λ obj ·L obj +λ class ·L class +λ local ·L local

[0079] Among them, λ obj , class and λ local They are the weight coefficients of the corresponding loss items, which are used to balance the impact of different loss items on the total loss;

[0080] Step 4.4: Repeat the above steps, according to the loss function, through the back propagation algorithm, use the optimizer Adam to adjust the learning rate and update the model parameters until the loss function converges and the model training is completed.

[0081] Step 5: Use a high-definition camera to collect different gesture pictures of the deaf-mute, and then use DCGAN to expand the collected gesture picture samples to obtain a new data set, perform labeling operations, and then put the corresponding labels into the PostgreSQL database;

[0082] Step 6: Use the newly added data set and its corresponding labels to train Yolov5 again, obtain the updated model and save it;

[0083] The specific content is: the newly expanded data set and the corresponding labels are used to perform transfer learning training on the original trained Yolov5 recognition model. The training process is the same as steps 4.1-4.4, so that the YOLOv5 recognition model has better generalization ability, enabling it to accurately recognize gestures in different environments and conditions.

[0084] Step 7: Perform the final deaf-mute gesture recognition operation through the trained Yolov5 target detection and recognition model to detect the information and meaning represented by the gesture, and the whole process ends here.

[0085] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A deaf-mute gesture recognition method based on the DCGAN-Yolov5 algorithm, characterized by: The following steps are involved: Step 1: First, create an original data set of hand gestures for the deaf-mute. The original data set includes different hand gesture sample images of the deaf-mute. Step 2: Use (DCGAN) to expand the different gesture sample images in step 1 to obtain a mixed data set; Step 3: Label the mixed data set obtained in step 2 to obtain the labels corresponding to the mixed data set, and put the labels into the PostgreSQL database; Step 4: Use the mixed data set and corresponding labels obtained in step 3 as the input of the Yolov5 target detection model and perform multiple rounds of training to complete the pre-training operation of the Yolov5 target detection and recognition model; Step 5: Use a high-definition camera to collect different gesture pictures of the deaf-mute, and then use DCGAN to expand the collected gesture picture samples to obtain a new data set, perform labeling operations, and then put the corresponding labels into the PostgreSQL database; Step 6: Use the newly added data set and its corresponding labels to train Yolov5 again, obtain the updated model and save it; Step 7: Perform the final deaf-mute gesture recognition operation through the trained Yolov5 target detection and recognition model to detect the information and meaning represented by the gesture, and the whole process ends here.

2. The method for hand gesture recognition for the deaf-mute based on the DCGAN-Yolov5 algorithm according to claim 1, characterized in that: The training steps of the DCGAN network model in step 2 are: Step 2.1: Initialize the network, the generator G is used to generate new data samples, and the discriminator D is used to distinguish between real data and fake data generated by the generator; Step 2.2: Input noise, the generator accepts a random noise that follows a Gaussian distribution as input; Step 2.3: The generator generates a fake data sample through its network structure; Step 2.4: Train the discriminator. The discriminator receives real data and fake data as input, uses the Neural Characteristic Functions Loss (NCFL) to calculate the discriminator's loss, and performs backpropagation to update the discriminator's weights. Step 2.5: Train the generator and input the generated fake data into the discriminator. The discriminator will output a probability value, indicating the probability that the input data is real data. The neural feature loss function uses this probability value to calculate the loss of the generator and back-propagate to update the weight of the generator. Step 2.6: Alternately perform the above two training steps until the image quality generated by the model meets the requirements or the predetermined number of training rounds is completed; the loss function of the entire adversarial process is expressed as: Among them, X represents the real data, Y represents the set of total label categories corresponding to each sample in the real data set X, Represents generated data, Representing and Generating Data The corresponding generated label, y represents a specific label category in Y; and Represent the empirical characteristic functions (ECFs) of the real and generated joint distributions, t iy is the projection function parameter sampled from a certain distribution, and k is the number of samples.

3. The method for hand gesture recognition for the deaf-mute based on the DCGAN-Yolov5 algorithm according to claim 1, characterized in that: The training steps of the Yolov5 target detection and recognition model in step 4 are: Step 4.1: Input the expanded gesture images and their corresponding labels into the Yolov5 model; Step 4.2: The input is forward propagated through the backbone, neck and head layers in sequence; the backbone layer is responsible for extracting features from the input image; the neck layer further processes and fuses the features extracted by the backbone to enhance the expressiveness of the features; the head layer predicts the category, location and confidence of the target based on the fused feature map; Step 4.3: Compare the output of the model with the true label and calculate the loss function, which includes three loss functions: classification loss, localization loss and confidence loss. The calculation formula of the classification loss function is as follows: Where N represents the total number of categories, y n is the probability predicted by the model, is the true label (0 or 1); Positioning loss function: L local =1-CIOU(B,B gt ) Among them, B is the prediction box, B gt For the ground-truth box, the CIOU calculation takes into account the overlapping area, center point distance, and aspect ratio; Confidence loss function: Among them, p n is the confidence of the model prediction, is the true confidence label (0 or 1), usually for the predicted box that matches the target, otherwise L=λ obj ·L obj +λ class ·L class +λ local ·L local Among them, λ obj , class and λ local They are the weight coefficients of the corresponding loss items, which are used to balance the impact of different loss items on the total loss; Step 4.4: Repeat the above steps, according to the loss function, through the back propagation algorithm, use the optimizer Adam to adjust the learning rate and update the model parameters until the loss function converges and the model training is completed.

4. The method for hand gesture recognition for the deaf-mute based on the DCGAN-Yolov5 algorithm according to claim 3, characterized in that: The specific content of step 6 is: using the newly expanded additional data set and the corresponding labels to perform transfer learning training on the original trained Yolov5 recognition model, wherein the training process is the same as steps 4.1-4.4.

Images