A sign language recognition method and system based on a lightweight GLnet network

Abstract

The application discloses a sign language recognition method and system based on a light-weight GLnet network, and the method comprises the following steps: taking N*M sign language videos of labeled sign language words as a training sample set; preprocessing the sign language videos in the training sample set to obtain L frame sign language continuous frame images corresponding to each sign language video; constructing a sign language recognition model based on a light-weight GLnet network, wherein the GLnet network is a CNN-LSTM network with an improved Ghost module; training the sign language recognition model by using the training sample set; and inputting a sign language video to be recognized into the trained sign language recognition model after preprocessing, and outputting a recognition result. The improved Ghost module is introduced on the basis of the CNN-LSTM network to construct the light-weight GLnet network, so that the volume and calculation amount of the network model are greatly reduced, and the sign language recognition with high recognition rate is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sign language recognition, and in particular to a sign language recognition method and system based on a light-weight GLnet network. BACKGROUND

[0002] Sign language provides a way for deaf-mutes to communicate with the society, and helps deaf-mutes to realize their social value. Deaf-mutes convey information to healthy people or hearing-impaired people through body language such as hand movements and facial expressions. However, it is not an easy thing for healthy people with good hearing and expression ability to master sign language, so there is a huge communication barrier between deaf-mutes and healthy people. Sign language interpreters can help deaf-mutes communicate with non-deaf-mutes, but the number of people engaged in this work in the society is limited and this method has many limitations in time and space. Therefore, using visual technology to automatically translate sign language and deploying the sign language translation technology to mobile handheld devices can not only reduce the cost of sign language learning, but also reduce the communication barrier between deaf-mutes and non-deaf-mutes in time and space.

[0003] Currently, sign language translation technology has been developed on the basis of deep neural networks, mainly divided into traditional machine learning methods and deep learning methods. Traditional sign language recognition methods rely on artificial design features, require repeated verification, and the preprocessing process is tedious and inefficient; the development of deep learning technology has greatly promoted the development of sign language recognition technology. The sign language recognition technology represented by convolutional neural networks has the characteristics of high recognition rate and wide application scenarios. For example, Elboushaki A et al. proposed a multi-dimensional feature learning method for RGB-D sign language videos, which uses a 3D residual network and a ConvLSTM network to capture and learn the spatiotemporal features of RGB and depth sequences, which can effectively combine appearance and motion features to suppress background and other changes. Interference; Jiang S et al. proposed a multi-modal skeleton-aware sign language recognition model, which uses SSTCN to model two-dimensional coordinate information, and uses multiple 3DCNN networks to merge RGB images, optical flow, depth flow and HHA features for complementation, to obtain higher recognition rate; Li X et al. proposed an improved CNN-LSTM network, which uses a Swith function to improve the residual network and uses label smoothing technology for training, enhancing the network generalization ability.

[0004] However, conventional deep neural network models are large in size and generate a large number of parameters in the convolution process, resulting in the above-mentioned deep neural network-based methods having high requirements for hardware computing power, and existing mobile devices are usually difficult to meet their requirements, making it difficult to apply this method to mobile devices. Therefore, a light-weight deep neural network is needed that can be deployed to mobile handheld devices to achieve sign language recognition with lower parameter quantity and computational quantity. SUMMARY

[0005] To solve the above problems of the prior art, the application provides a sign language recognition method and system based on a lightweight GLnet network, an improved Ghost module is introduced on the basis of a CNN-LSTM network to construct a lightweight GLnet network, which greatly reduces the volume and calculation amount of the network model, and based on the lightweight network, high recognition rate of sign language recognition is realized under the condition of low parameter amount and calculation amount.

[0006] In a first aspect, the disclosure provides a sign language recognition method based on a lightweight GLnet network.

[0007] A sign language recognition method based on a lightweight GLnet network comprises the following steps:

[0008] M sign language videos corresponding to N sign language vocabulary respectively are obtained, and N*M sign language videos with labeled sign language vocabulary are taken as a training sample set;

[0009] The sign language videos in the training sample set are preprocessed to obtain L frames of sign language continuous frame images corresponding to each sign language video;

[0010] A sign language recognition model based on a lightweight GLnet network is constructed; the GLnet network is a CNN-LSTM network with an improved Ghost module;

[0011] The sign language continuous frame images in the training sample set are input into the sign language recognition model, and the sign language recognition model is trained using the training sample set;

[0012] After preprocessing, the sign language video to be recognized is input into the trained sign language recognition model, and the recognition result is output.

[0013] Further technical solutions, the preprocessing includes frame extraction, cropping, scaling, and specifically includes:

[0014] An equal time interval uniform sampling method is used to extract video information as multiple frames of continuous color images by frame extraction processing on each sign language video;

[0015] The obtained multiple frames of continuous color images are respectively cropped to the same size;

[0016] The cropped images are scaled to the same size.

[0017] Further technical solutions, further comprising:

[0018] The preprocessed images are scaled again to reduce the size of the sign language continuous frame images to Z*Z, and then standardized with a standard deviation of S and a mean of M.

[0019] Further technical solutions, in the GLnet network, first, a plurality of Gnet networks are used to extract the spatial features of the input multi-frame sign language continuous frame image respectively, then the LSTM network is used to learn the time features from the extracted spatial feature sequence, and finally, a linear fully connected layer is used as a classifier to output the final recognition result.

[0020] Further technical solutions, the Gnet network is a CNN network based on an improved G module, the Gnet network contains six layers in total, the first layer is a 3x3 convolution layer, followed by a BN layer, a ReLu function and a 3x3 maximum pooling layer, the middle four layers are four G modules with a 3x3 convolution kernel respectively, and the last layer is a 1x1 average pooling layer, wherein each G module corresponds to a feature extraction layer.

[0021] Further technical solutions, the G module is an improved Ghost module, the G module is composed of a Ghost module, a BN layer and a pooling layer connected in sequence, the Ghost module is used as a feature extraction layer, followed by a BN layer and a ReLu function as an activation function, and finally connected to a maximum pooling layer.

[0022] In the Ghost module, first, a regular convolution operation is performed on the input image to obtain a fixed feature map, then the obtained feature map is subjected to linear transformation respectively to obtain a plurality of linear transformation maps corresponding to each feature map, and finally the fixed feature map and its corresponding linear transformation map are spliced to obtain the output feature map.

[0023] Further technical solutions, label smoothing is introduced for the loss function during the training process, and the calculation formula of the label after introducing the label smoothing is:

[0024]

[0025] Wherein, K represents the total number of classes, θ is a small hyperparameter, y represents the true class, p' represents the predicted class, and y represents the label of the i-th sample. i

[0026] Secondly, the present disclosure provides a sign language recognition system based on a lightweight GLnet network.

[0027] A sign language recognition system based on a lightweight GLnet network, comprising:

[0028] A training sample set construction module is configured to obtain N sign language videos corresponding to M sign language vocabularies respectively, and use the NxM annotated sign language video as a training sample set.

[0029] ​A preprocessing module is configured to preprocess the sign language videos in the training sample set to obtain L frame sign language continuous frame images corresponding to each sign language video.

[0030] A sign language recognition model construction module is configured to construct a sign language recognition model based on a light-weight GLnet network.

[0031] A sign language recognition model training module is configured to input the sign language continuous frame images in the training sample set into the sign language recognition model and train the sign language recognition model by using the training sample set.

[0032] A sign language recognition module is configured to input the sign language video to be recognized into the trained sign language recognition model after preprocessing and output a recognition result.

[0033] In a third aspect, the present disclosure further provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the steps of the method of the first aspect are completed.

[0034] In a fourth aspect, the present disclosure further provides a computer readable storage medium for storing computer instructions, when the computer instructions are executed by a processor, the steps of the method of the first aspect are completed.

[0035] The above one or more technical solutions have the following beneficial effects:

[0036] 1. The present application provides a sign language recognition method and system based on a light-weight GLnet network, which introduces an improved Ghost module based on a CNN-LSTM network, constructs a light-weight GLnet network, greatly reduces the size and calculation amount of the network model, and realizes high-accuracy sign language recognition based on the light-weight network with low parameter amount and calculation amount.

[0037] 2. The present application introduces an improved Ghost module based on a CNN-LSTM network, combines the Ghost module with a max-pooling layer to improve the Ghost module, greatly reduces the parameters and calculation amount of the model, uses the improved Ghost module to cooperate with a conventional convolution layer to build a CNN network for spatial feature extraction, then uses an LSTM network to model the spatial sequence features, finally uses a linear layer as a classifier to complete the construction of a light-weight GLnet network, and realizes high-accuracy and low-computation sign language recognition based on the light-weight GLnet network. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0039] Figure 1 This is an overall flowchart of the sign language recognition method based on the lightweight GLnet network described in Embodiment 1 of the present invention;

[0040] Figure 2 This is a schematic diagram of the Ghost module in Embodiment 1 of the present invention;

[0041] Figure 3 This is a schematic diagram of the structure of module G in Embodiment 1 of the present invention;

[0042] Figure 4 This is a schematic diagram of the GLnet network structure in Embodiment 1 of the present invention;

[0043] Figure 5 This is a schematic diagram of the LSTM network structure in Embodiment 1 of the present invention;

[0044] Figure 6 The figures show the accuracy variation curves of different network structures of the recognition network on the test set in Embodiment 1 of the present invention. Detailed Implementation

[0045] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0046] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0047] Example 1

[0048] In order to solve the problem that the deep neural network model is not suitable for mobile devices due to its large size, large number of parameters and large amount of calculation in the existing hand sign recognition scheme based on deep neural network, the embodiment provides a hand sign recognition method based on a light-weight GLnet network. Considering that the Ghost module can reduce the calculation cost while ensuring the performance by replacing the convolution layer with convolution and linear transformation to generate ghost features, the improved Ghost module is introduced into the CNN-LSTM network in the scheme, the Ghost module is improved by combining the Ghost module with the maximum pooling layer, the number of parameters and the amount of calculation of the model are greatly reduced, the improved Ghost module is used to build the CNN network with the conventional convolution layer to extract spatial features, then the LSTM network is used to model the spatial sequence features, and finally the linear layer is used as a classifier to complete the construction of the light-weight GLnet network. Based on the light-weight GLnet network, high-accuracy and low-computation hand sign recognition is realized.

[0049] The embodiment provides a hand sign recognition method based on a light-weight GLnet network, as shown in Figure 1 The method comprises the following steps:

[0050] In step S1, M hand sign words are selected, and for each hand sign word, multiple hand sign interpreters repeatedly translate the hand sign word, and a Kinetic camera is used to shoot the translated video as a hand sign video. The shot video includes RGB color video, depth video and 3D skeleton joint information, etc. In the embodiment, the RGB color video is selected as the hand sign video, N hand sign videos are selected from the hand sign videos, and the N*M hand sign videos with the corresponding hand sign words are used as a training sample set.

[0051] In step S2, the hand sign videos in the training sample set are preprocessed to obtain L frame hand sign continuous frame images corresponding to each hand sign video.

[0052] In step S3, a hand sign recognition model based on a light-weight GLnet network is constructed. The GLnet network is a CNN-LSTM network with an improved Ghost module.

[0053] In step S4, the hand sign continuous frame images in the training sample set are input into the hand sign recognition model, and the hand sign recognition model is trained using the training sample set.

[0054] In step S5, the hand sign video to be recognized is preprocessed and input into the trained hand sign recognition model, and the recognition result is output.

[0055] In the above step S1, M Chinese hand sign words are selected, and for each hand sign word, multiple hand sign interpreters repeatedly translate the hand sign word, and a Kinetic camera is used to shoot the translated video as a hand sign video. The shot video includes RGB color video, depth video and 3D skeleton joint information, etc. In the embodiment, the RGB color video is selected as the hand sign video, N hand sign videos are selected from the hand sign videos, and the N*M hand sign videos with the corresponding hand sign words are used as a training sample set.

[0056] Preferably, for M sign language vocabularies, a corresponding sign language video is selected for each sign language vocabulary in a ratio of 10:1 as a training set and a verification set respectively, the training set is used to train the constructed sign language recognition model, and the verification set is used to verify the accuracy, calculation amount and calculation efficiency of the constructed sign language recognition model.

[0057] In the above step S2, the sign language video in the training sample set is preprocessed, including frame extraction, cropping, scaling, etc., so as to obtain L frames of sign language continuous frame images corresponding to each sign language video. Specifically, first, the equal time interval uniform sampling method is used to perform frame extraction processing on each sign language video, and the video information is extracted into multiple continuous color images; second, considering that the image contains a large amount of redundant information, therefore, the multiple continuous color images obtained are respectively cropped to retain key information, and the cropped images are unified to the same size, in this embodiment, the images are cropped to the same size of 600*600; finally, the cropped images are scaled to the same size, in this embodiment, the images are uniformly scaled to the size of 300*300.

[0058] Preferably, in order to ensure that the L frames of sign language continuous frame images input into the sign language recognition model can be more adaptive to the network model, in this embodiment, the preprocessed images are scaled again, the size of the sign language continuous frame images is reduced to Z*Z, and then standardized with a standard deviation of S and a mean of M.

[0059] After the above image acquisition and image processing operations are completed, the above step S3 is performed, that is, the sign language recognition model based on the lightweight GLnet network is constructed. In order to realize the lightweight of the recognition network, the improved Ghost module is introduced on the basis of the CNN-LSTM network in this embodiment, so as to construct the sign language recognition model based on the lightweight GLnet network.

[0060] First, the Ghost network is obtained by improving the conventional convolutional network, and the conventional convolutional operation can be described by the following formula (1):

[0061]

[0062] Wherein, f∈R c×k×k×n represents a filter, * represents a convolution operation, X∈R c×h×w represents input data, b represents a bias function, Y∈R h′×w′×n represents data output.

[0063] At present, the mainstream convolutional neural network uses a large number of convolution operations in the calculation process to generate a large number of feature maps, some of which are very similar or even overlapping. A sufficient number of feature maps ensures that the network can deeply understand the data, but also occupies a large amount of memory. Instead of reducing the amount of calculation by reducing the redundancy of feature maps, a low-computing-cost way is used to generate a sufficient number of feature maps, so the Ghost module emerges as the times require. The Ghost module divides the conventional convolution layer into two parts, namely the ordinary convolution and the linear function, and generates a sufficient number of redundant feature maps without changing the output size of the feature map, thereby reducing the computing cost while ensuring that the network can deeply understand the data. This is described by the following formulas (2)-(5):

[0064] y = f * X 2

[0065] g ij =h ij ( i ) 3

[0066] g=[g 11 ,g 12 ,…g ms ] 4

[0067] Y=[y,g] 5

[0068] Wherein, X∈R c×h×w represents the input data, f∈R c×k×k×m represents the filter, m≤n, y∈R h′×w′×m represents the feature map after convolution operation, h ij represents the linear operation, wherein g ij represents the result of linear operation on a convolution feature map, g represents the Ghost feature map, Y∈R h′×w′×n represents the data output.

[0069] As shown in Figure 2 , in the Ghost module, the input image is first subjected to a conventional convolution operation to obtain a fixed feature map, then the obtained feature map is subjected to linear transformation to obtain a plurality of linear transformation maps corresponding to each feature map, and finally the fixed feature map and its corresponding linear transformation map are spliced to obtain a sufficient number of output feature maps.

[0070] Further, considering that a single Ghost module has the problem of generating a large number of FLOPS and memory occupation in the stacking process, in view of this problem, the above-mentioned Ghost module is improved to construct a G module. As Figure 3As shown, the G module is composed of a Ghost module, a BN layer and a pooling layer connected in sequence, with the Ghost module as the feature extraction layer, followed by the BN layer and the ReLU function as the activation function, and finally connected to the max pooling layer to reduce the dimension and abstract the features, reduce the memory occupation and the FLOPS generation.

[0071] This embodiment introduces an improved Ghost module (i.e. G module) on the basis of the CNN-LSTM network, improves the CNN network based on the above G module, and names it as Gnet. The network structure of Gnet is shown in Table 1 below, which contains six layers in total. The first layer is a 3x3 regular convolution layer, followed by a BN layer, a ReLu function and a 3x3 max pooling layer, and the step of each layer is 2. The middle four layers are four G modules with a 3x3 convolution kernel, respectively. The last layer is a 1x1 average pooling layer. Each G module corresponds to a feature extraction layer, and the max pooling layer has a convolution kernel step of 2.

[0072] Table 1 Network structure of Gnet

[0073]

[0074] Based on the above-mentioned Gnet network, a lightweight GLnet network is constructed by combining LSTM, and a sign language recognition model based on the GLnet network is further constructed. In this embodiment, the GLnet network structure is as shown in Figure 4 Firstly, multiple Gnets are used to extract the spatial features of multiple frames of input continuous frame images of sign language, then the LSTM network is used to learn the temporal features from the extracted spatial feature sequence, and finally the linear fully connected layer is used as the classifier to output the final recognition result.

[0075] As shown in Figure 5 , the LSTM network uses input gate I t , output gate O t , forget gate F t , candidate memory unit memory unit C t and hidden unit H t to complete the control and learning of information. Specifically, under the condition of batch size n and input number d, the memory unit, input and output gate, and forget gate parameter update formula are shown in the following formulas (6)-(11), where t represents the current time, h represents the number of hidden units, W represents the weight parameter, and b represents the bias parameter, where O t ,F t , C t ,H t ∈R nxh;W xi , W xf , W xo , W xc ∈R dxh ; b i , b f , b o , b c ∈R 1xh .

[0076] I t = σ(X t W xi + H t-1 W hi + b i ) (6)

[0077] F t = σ(X t W xf + H t-1 W hf + b f ) (7)

[0078] O t = σ(X t W xo + H t-1 W ho + b o ) (8)

[0079]

[0080]

[0081]

[0082] In the above step S4, the continuous frame images of sign language in the training sample set are input into the sign language recognition model, and the sign language recognition model is trained by using the training sample set. Further, in order to reduce overfitting, enhance the generalization ability and anti-interference ability of the network, label smoothing is introduced for the loss function in the training process. Label smoothing is a regularization technique that reduces prediction accuracy by setting an error rate for the target variable, artificially introducing noise interference, thereby reducing overfitting, enhancing the generalization ability and anti-interference ability of the network. After introducing label smoothing, the label calculation formula is as shown in the following formula (12):

[0083]

[0084] Wherein, K represents the total number of categories, θ is a small hyperparameter, y represents the true category, and p' i represents the label of the i-th sample.

[0085] After the training of the above model is completed, step S5 is performed, that is, the sign language video to be recognized is preprocessed and input into the trained sign language recognition model, and the recognition result is output.

[0086] To verify the feasibility of the network model constructed by the above scheme of the embodiment and the accuracy of sign language recognition, the following experiments are performed for verification.

[0087] Specifically, the SLR500 dataset is a vocabulary-level dataset, which contains 500 commonly used Chinese sign language vocabularies. For each vocabulary, 50 sign language translation volunteers repeat 5 times, and a Kinetic camera is used for shooting. The sign language videos obtained by shooting include RGD videos, depth videos and 3D skeleton joint information. Ten vocabularies of RGB

[0088] video information are selected from the SLR500 dataset, a total of 1100 videos are selected, and 100 sign language videos are selected for each vocabulary as a training set and 10 sign language videos are selected as a validation set.

[0089] Then, the dataset is preprocessed, including frame extraction, cropping, scaling, etc. to obtain L frames of 300*300 sign language continuous frame images. Then, the input image is scaled again to reduce the size of the sign language continuous frame image to N*N, and then standardized with a standard deviation of S and a mean of M. The obtained image is input into the recognition network. Wherein, L = 16, N = 128, S = 0.5, M = 0.5.

[0090] In addition, during the training process, Nvidia RTX3090 GPU with 24G video memory is used, Adam optimizer is selected, the running environment is Ubuntu18.04, python3.8, pytorch1.7.0, the learning rate is 0.001, and the weight decay is 1e-4.

[0091] The shufflenet series network, the Mobilenet series network, SAnet, CAnet, GhostNet combined with LSTM network, GLnet network and GLnet(label smooth) network (i.e. the network described above in the embodiment) are selected for sign language recognition, and the recognition accuracy is tested. The experimental results are shown in Table 2 as follows.

[0092] Table 2 Performance comparison of each network structure

[0093]

[0094]

[0095] Wherein, Parameters and Flops are calculated by the ptflops library of python under the condition that the input is [3, 16, 224, 224], the parameter quantity of GLnet is 2.96M, and the FLOPS is 2.56G.

[0096] As can be seen from Table 2, for FLOPS, Mobile_V3_small is the smallest, which is 984.81M, and Mobile_Next is the largest, which is 5.2G; the parameter quantity of Shuffle_V2_2.0x network is the largest, which is 10.65M, and the parameter quantity of GLnet network is the smallest, which is 2.96M. In summary, the GLnet network exhibits the best comprehensive performance, achieves the highest accuracy, the smallest parameter quantity, and lower FLOPS, and has a recognition accuracy of 91% on the test set; after using label smoothing, the recognition accuracy of GLnet is improved by 1%, reaching 92%.

[0097] In addition, specific training images are as shown in Figure 6 The upper indexes in the figure correspond to SAnet, CAnet, Shuffle, Next, V3_Large and Gnet, respectively, and the image is the accuracy curve of SAnet-LSTM, CAnet-LSTM, Shuffle_V2_1.5x-LSTM, Mobile_Next-LSTM (label smooth), Mobile_V3_large-LSTM and GLnet (label smooth) network on the test set, respectively. The results show that GLnet has stronger stability and higher recognition rate.

[0098] The hand sign recognition method based on the lightweight GLnet network provided in the embodiment introduces an improved Ghost module on the basis of the CNN-LSTM network, constructs a lightweight GLnet network, greatly reduces the volume and calculation amount of the network model, and based on the lightweight network, realizes high recognition rate of hand sign recognition under the condition of low parameter quantity and calculation amount.

[0099] Embodiment two

[0100] The embodiment provides a hand sign recognition system based on a lightweight GLnet network, which comprises:

[0101] A training sample set construction module is configured to obtain N hand sign videos corresponding to M hand sign vocabularies, and use the N*M hand sign videos with labeled hand sign vocabularies as a training sample set;

[0102] A preprocessing module is configured to preprocess the hand sign videos in the training sample set, and obtain L frames of continuous hand sign frame images corresponding to each hand sign video.

[0103] The sign language recognition model construction module is configured to construct a sign language recognition model based on a lightweight GLnet network, wherein the GLnet network is a CNN-LSTM network with an improved Ghost module introduced.

[0104] The sign language recognition model training module is configured to input the sign language continuous frame images in the training sample set into the sign language recognition model, and train the sign language recognition model by using the training sample set.

[0105] The sign language recognition module is configured to input the sign language video to be recognized into the trained sign language recognition model after preprocessing, and output the recognition result.

[0106] Embodiment three

[0107] The embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the steps in the sign language recognition method based on the lightweight GLnet network are completed.

[0108] Embodiment four

[0109] The embodiment also provides a computer-readable storage medium for storing computer instructions, when the computer instructions are executed by the processor, the steps in the sign language recognition method based on the lightweight GLnet network are completed.

[0110] The steps and methods in the above embodiments two to four correspond to the method embodiment one, and the specific embodiments can refer to the related description part of the embodiment one. The term "computer-readable storage medium" should be understood as including a single medium or multiple media of one or more instruction sets; it should also be understood as including any medium capable of storing, encoding or carrying instruction sets for execution by a processor and causing the processor to perform any method in the present application.

[0111] Those skilled in the art should understand that each module or step of the present application described above can be realized by a general computer device, alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be respectively manufactured into each integrated circuit module, or a plurality of modules or steps among them can be manufactured into a single integrated circuit module to realize. The present application is not limited to any specific combination of hardware and software.

[0112] The above description is only the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0113] The above describes the specific embodiments of the present application in conjunction with the drawings, but is not a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the scope of protection of the present application.

Claims

1. A sign language recognition method based on a lightweight GLnet network, characterized by, The method comprises the following steps: Obtaining N sign language videos corresponding to M sign language vocabularies respectively, and taking N*M sign language videos with labeled sign language vocabularies as a training sample set; Preprocessing the sign language videos in the training sample set to obtain L frames of sign language continuous frame images corresponding to each sign language video; Constructing a sign language recognition model based on a light-weight GLnet network; the GLnet network is a CNN-LSTM network with an improved Ghost module; The improved Ghost module is a G module, and the Gnet network is a CNN network improved based on the G module; the Gnet network comprises six layers in total; the first layer is a convolutional layer with a kernel size of 3x3, followed by a BN layer, a ReLu function and a max-pooling layer with a kernel size of 3x3; the middle four layers are four G modules with a kernel size of 3x3 respectively; and the last layer is an average-pooling layer with a kernel size of 1x1; each G module corresponds to a feature extraction layer; Inputting the sign language continuous frame images in the training sample set into the sign language recognition model, and training the sign language recognition model by using the training sample set; Introducing label smoothing into the loss function during the training process; and the calculation formula of the label after introducing the label smoothing is: wherein, K denotes the total number of classes, θ is a small hyperparameter, y denotes the true class, denotes the label of the i th sample; Inputting a sign language video to be recognized into the trained sign language recognition model after preprocessing, and outputting a recognition result. 2.The sign language recognition method based on the lightened GLnet network according to claim 1, characterized in that, The preprocessing comprises frame extraction, cropping and scaling, and specifically comprises the following steps: Uniformly sampling the video information into multiple frames of continuous color images by using an equal time interval sampling method; Cropping the obtained multiple frames of continuous color images respectively, and unifying the cropped images to the same size; Scaling the cropped images to the same size.

3. The sign language recognition method based on the lightened GLnet network according to claim 2, further comprising: The method comprises the following steps: Rescaling the preprocessed images, reducing the size of the sign language continuous frame images to Z*Z, and then performing standardization with a standard deviation of S and a mean value of M. 4.The sign language recognition method based on the lightened GLnet network of claim 1, wherein, In the GLnet network, multiple Gnet networks are used to extract spatial features of the input multiple frames of sign language continuous frame images, then an LSTM network is used to learn time features from the extracted spatial feature sequence, and finally a linear fully connected layer is used as a classifier to output the final recognition result. 5.The sign language recognition method based on the lightened GLnet network according to claim 1, characterized in that, The G module is an improved Ghost module, and the G module comprises a Ghost module, a BN layer and a pooling layer connected in sequence; the Ghost module is used as a feature extraction layer, followed by a BN layer and a ReLu function as an activation function, and finally connected to a max-pooling layer; In the Ghost module, a regular convolution operation is first performed on the input image to obtain a fixed feature map, then linear transformation is performed on the obtained feature map to obtain multiple linear transformation maps corresponding to each feature map, and finally the fixed feature map and its corresponding linear transformation maps are spliced to obtain the output feature map.

6. A sign language recognition system based on a lightweight GLnet network, characterized in that, The method comprises the following steps: A training sample set construction module is used to obtain N sign language videos corresponding to M sign language vocabularies respectively, and take N*M sign language videos with labeled sign language vocabularies as a training sample set. A preprocessing module is configured to preprocess the sign language videos in the training sample set to obtain L frame sign language continuous frame images corresponding to each sign language video; The sign language recognition model construction module is configured to construct a sign language recognition model based on a light-weight GLnet network, wherein the GLnet network is a CNN-LSTM network with an improved Ghost module introduced therein; The sign language recognition model training module is configured to input the sign language continuous frame images in the training sample set into the sign language recognition model, and train the sign language recognition model by using the training sample set. The sign language recognition module is configured to input the sign language video to be recognized after preprocessing into the trained sign language recognition model, and output a recognition result.

7. An electronic device, characterized by comprising: The computer instructions are executed by the processor to complete the steps of the sign language recognition method based on the light-weight GLnet network according to any one of claims 1-5.

8. A computer readable storage medium characterized by, The computer instructions are executed by the processor to complete the steps of the sign language recognition method based on the light-weight GLnet network according to any one of claims 1-5.

Images