Method, system, device and medium for sign language vocabulary recognition based on deep learning

Abstract

The application discloses a sign language vocabulary recognition method, system, device and medium based on deep learning, and the method comprises the following steps: acquiring a sign language video; inputting the sign language video into a trained human body posture estimation network model to perform first feature extraction and obtain a heatmap graph in the sign language video; performing second feature extraction through a time sequence light-based feature rapid screening model to obtain heatmap space features; performing space feature screening of human body key point information on the heatmap space features to obtain human body key point space features; performing feature learning through a bidirectional LSTM time sequence model with an attention mechanism to obtain a sign language video learning result; performing classification and coding through a full connection layer and a softmax layer to obtain a sign language video classification coding result; and querying a sign language vocabulary recognition result according to the sign language video classification coding result. The application can improve the accuracy of sign language recognition.

Description

Technical Field

[0001] This invention relates to the field of sign language recognition technology, and in particular to a method, system, device and medium for sign language vocabulary recognition based on deep learning. Background Technology

[0002] Current sign language vocabulary recognition models mainly use RGB modal information and skeletal key point coordinates for recognition. However, effectively extracting sign language action information from RGB information requires a deep neural network, which not only increases the computational load of the model but also makes it difficult to achieve real-time results.

[0003] Existing sign language vocabulary recognition networks based on sequence models such as LSTM and transformers mostly do not pre-screen redundant information from the temporal video before input. Instead, they directly allow the sequence network to learn effective features. In general, the differences between frames in a 30fps video sequence are very small. In order to reduce redundant information from the video, the sequence information is compressed quickly before being input into the sequence model. This technique reduces the model's learning ability, thereby reducing the accuracy of sign language recognition. Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention proposes a deep learning-based sign language vocabulary recognition method, system, device, and medium, which can improve the learning ability of the model and improve the accuracy of sign language recognition.

[0005] In a first aspect, embodiments of the present invention provide a deep learning-based sign language vocabulary recognition method, the deep learning-based sign language vocabulary recognition method comprising:

[0006] Get sign language videos;

[0007] The sign language video is input into a trained human pose estimation network model for first feature extraction, and the heatmap image of the sign language video output by the human pose estimation network model is obtained.

[0008] The heatmap is subjected to a second feature extraction using a time-based lightweight feature fast filtering model to obtain the heatmap spatial features.

[0009] The spatial features of the heatmap are filtered for spatial features of human key points to obtain spatial features of human key points.

[0010] The spatial features of the human body key points are learned through a bidirectional LSTM time series model with attention mechanism to obtain the sign language video learning results output by the bidirectional LSTM time series model with attention mechanism.

[0011] The learning results of the sign language video are classified and encoded through a fully connected layer and a softmax layer to obtain the sign language video classification and encoding results.

[0012] Based on the sign language video classification and coding results, the sign language vocabulary recognition results are obtained.

[0013] Compared with the prior art, the first aspect of the present invention has the following beneficial effects:

[0014] This method acquires sign language videos; inputs the sign language videos into a trained human pose estimation network model for first feature extraction, obtaining a heatmap image from the sign language videos output by the human pose estimation network model; then, a lightweight temporal feature filtering model is used to perform second feature extraction on the heatmap image, obtaining spatial features of the heatmap. This method improves the speed of sign language recognition and reduces computational cost by using a lightweight temporal feature filtering model to extract features from the heatmap image. The spatial features of the heatmap are then filtered for spatial features of human key points, obtaining spatial features of human key points; these spatial features are then used for feature learning through a bidirectional LSTM time series model with attention mechanism, obtaining the sign language video learning results output by the bidirectional LSTM time series model with attention mechanism; the sign language video learning results are then classified and encoded through fully connected layers and softmax layers, obtaining sign language video classification and encoding results; finally, sign language vocabulary recognition results are obtained based on the sign language video classification and encoding results. This method improves the model's learning ability through feature selection and learning. Furthermore, by extracting heatmap images from sign language videos and performing feature extraction and learning based on these heatmap images, rather than directly extracting the coordinate information of key points on the human body as feature information, the accuracy of motion embedding feature information is increased, thereby improving the accuracy of sign language recognition.

[0015] According to some embodiments of the present invention, before inputting the sign language video into a trained human pose estimation network model for first feature extraction, the deep learning-based sign language vocabulary recognition method further includes:

[0016] The ideal number of frames in the sign language video is preset;

[0017] If the number of frames in the sign language video is less than the ideal number of frames, blank frames are added to fill the ideal number of frames to obtain a processed sign language video. The number of frames in the processed sign language video is greater than or equal to the ideal number of frames.

[0018] According to some embodiments of the present invention, the selection of key points on the human body includes selecting multiple key points on the nose, eyes, ears, arms, and fingers of the human body.

[0019] According to some embodiments of the present invention, the step of extracting second features from the heatmap using a time-based lightweight feature fast filtering model to obtain heatmap spatial features includes:

[0020] The heatmap image is input through multiple channels corresponding to the video frame sequence;

[0021] The heatmap is processed by two-dimensional adaptive average pooling, a fully connected layer, and ReLU activation to obtain global heatmap temporal features;

[0022] The heatmap is subjected to a first group convolution, batch normalization, ReLU activation function activation, and a second group convolution in the temporal dimension to obtain local heatmap temporal features;

[0023] The global heatmap temporal features and the local heatmap temporal features are added together to obtain the summed heatmap temporal features;

[0024] The sigmoid activation function is used to activate the summed heatmap temporal features to generate the frame weights corresponding to the summed heatmap temporal features;

[0025] The frame weights are multiplied by the corresponding input heatmap to obtain the heatmap spatial features.

[0026] According to some embodiments of the present invention, the step of performing spatial feature filtering of human keypoint information on the heatmap spatial features to obtain human keypoint spatial features includes:

[0027] The temporal dimension T and the channel dimension C of the human key points in the heatmap spatial feature of size BxTxCxHxW are merged to obtain a heatmap spatial feature of size Bx(T*C)xHxW, where B represents the number of data entries calculated at one time during training or inference, H represents the height of the heatmap, and W represents the width of the heatmap.

[0028] The heatmap spatial features of size Bx(T*C)xHxW are grouped by the channel dimension C of the human key points and then subjected to a first group convolution to obtain heatmap spatial features of size BxCxHxW.

[0029] The heatmap spatial features of size BxCxHxW are subjected to batch normalization, ReLU activation, and a second group convolution with the channel dimension C of the human key points as the group to obtain local heatmap spatial features.

[0030] The heatmap spatial features of size Bx(T*C)xHxW are subjected to adaptive average pooling and convolution to obtain global heatmap spatial features;

[0031] The local heatmap spatial features and the global heatmap spatial features are numerically multiplied to obtain the multiplied heatmap spatial features;

[0032] The multiplied heatmap spatial features are then activated by a third group convolution and the Mish activation function, grouped by the channel dimension C of the human keypoints, to obtain the spatial features of the human keypoints.

[0033] According to some embodiments of the present invention, before performing feature learning on the spatial features of the human body key points using a bidirectional LSTM time series model with an attention mechanism, the deep learning-based sign language vocabulary recognition method further includes:

[0034] The spatial features of the human body key points are subjected to Dropout random deactivation and dimensionality reduction processing with fully connected layers.

[0035] According to some embodiments of the present invention, the step of querying the sign language vocabulary recognition result based on the sign language video classification and coding result includes:

[0036] Generate a sign language vocabulary encoding table based on the sign language text corresponding to the sign language video;

[0037] The sign language video classification and encoding results are queried in the sign language vocabulary encoding table to obtain the predicted sign language vocabulary recognition results.

[0038] Secondly, embodiments of the present invention also provide a deep learning-based sign language vocabulary recognition system, the deep learning-based sign language vocabulary recognition system comprising:

[0039] The data acquisition unit is used to acquire sign language videos;

[0040] The first feature extraction unit is used to input the sign language video into the trained human pose estimation network model to perform first feature extraction and obtain the heatmap image in the sign language video output by the human pose estimation network model.

[0041] The second feature extraction unit is used to extract second features from the heatmap using a time-based lightweight feature fast filtering model to obtain heatmap spatial features.

[0042] The feature filtering unit is used to filter the spatial features of the heatmap spatial features for human key point information to obtain human key point spatial features.

[0043] The feature learning unit is used to learn the spatial features of the human body key points through a bidirectional LSTM time series model with attention mechanism, and obtain the sign language video learning results output by the bidirectional LSTM time series model with attention mechanism.

[0044] The result acquisition unit is used to classify and encode the sign language video learning results through a fully connected layer and a softmax layer to obtain the sign language video classification and encoding results.

[0045] The vocabulary recognition unit is used to query the sign language vocabulary recognition results based on the sign language video classification and coding results.

[0046] Thirdly, embodiments of the present invention also provide a deep learning-based sign language vocabulary recognition device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform a deep learning-based sign language vocabulary recognition method as described above.

[0047] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to execute a deep learning-based sign language vocabulary recognition method as described above.

[0048] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description

[0049] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0050] Figure 1 This is a flowchart of a deep learning-based sign language vocabulary recognition method according to an embodiment of the present invention;

[0051] Figure 2 This is a flowchart of another embodiment of the sign language vocabulary recognition method based on deep learning according to the present invention;

[0052] Figure 3 This is a schematic diagram illustrating the definition of key human body points according to an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram illustrating the definition of key points between fingers according to an embodiment of the present invention;

[0054] Figure 5 This is a structural diagram of a deep learning-based sign language vocabulary recognition system according to an embodiment of the present invention. Detailed Implementation

[0055] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0056] In the description of this invention, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of technical features indicated, or implicitly indicating the order of the technical features indicated.

[0057] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the drawings and are only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0058] In the description of this invention, it should be noted that, unless otherwise explicitly defined, terms such as "setting," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0059] First, let's analyze some of the terms used in this application:

[0060] Sign language vocabulary recognition: Sign language recognition converts a sequence of video clips of a person performing sign language into corresponding textual vocabulary. It can be understood as a classification task in deep learning; for example, with 500 vocabulary words, the input to the model is the sign language video, and the output is the word category.

[0061] Existing sign language vocabulary recognition networks based on sequence models such as LSTM and transformers mostly do not pre-screen redundant information from the temporal video before input. Instead, they directly allow the sequence network to learn effective features. In general, the differences between frames in a 30fps video sequence are very small. In order to reduce redundant information from the video, the sequence information is compressed quickly before being input into the sequence model. This technique reduces the model's learning ability, thereby reducing the accuracy of sign language recognition.

[0062] To address the aforementioned issues, this invention acquires sign language videos; inputs these videos into a trained human pose estimation network model for first feature extraction, obtaining a heatmap image from the sign language video output by the network model; and then uses a lightweight temporal feature filtering model to perform second feature extraction on the heatmap image, obtaining spatial features of the heatmap. This invention improves the speed of sign language recognition and reduces computational load by using a lightweight temporal feature filtering model to extract features from the heatmap image. The spatial features of the heatmap are then filtered for spatial features of human key points, obtaining spatial features of human key points; these features are then used for feature learning through a bidirectional LSTM time-series model with an attention mechanism, obtaining the sign language video learning result output by the bidirectional LSTM time-series model with an attention mechanism; the sign language video learning result is then classified and encoded through fully connected layers and softmax layers, obtaining the sign language video classification and encoding result; and based on the sign language video classification and encoding result, the sign language vocabulary recognition result is retrieved. This invention improves the learning ability of the model through feature selection and learning. Furthermore, by extracting heatmap images from sign language videos and performing feature extraction and learning based on these heatmap images, rather than directly extracting the coordinate information of key points on the human body as feature information, the accuracy of motion embedding feature information is increased, thereby improving the accuracy of sign language recognition.

[0063] Reference Figures 1 to 2 This invention provides a deep learning-based sign language vocabulary recognition method, which includes:

[0064] Step S100: Obtain the sign language video.

[0065] Specifically, the ideal number of frames in the pre-set sign language video;

[0066] If the number of frames in the sign language video is less than the ideal number of frames, blank frames are added to fill the gaps, resulting in a processed sign language video with a number of frames greater than or equal to the ideal number of frames. For example:

[0067] Input a sign language video. The sign language video contains multiple frames. Since the number of frames corresponding to different sign language videos is different, this embodiment first performs corresponding sampling to ensure that the number of frames in each sign language video is a fixed N frames. Here, N is a fixed ideal number of frames. For sign language videos with less than N frames, blank frames are directly used to fill in until N frames are reached.

[0068] Step S200: Input the sign language video into the trained human pose estimation network model for first feature extraction, and obtain the heatmap image of the sign language video output by the human pose estimation network model.

[0069] Specifically, in this embodiment, a pre-trained human pose estimation network model is used to extract the first feature from the sign language video, obtaining the heatmap image in the sign language video output by the human pose estimation network model. In this embodiment, the human pose estimation network takes RGB mode sign language video as input. The human pose estimation network model will infer and calculate the corresponding heatmap image by performing operations such as convolution on the key point information of the human body. This heatmap image can be converted into 2D coordinates (x, y) of a certain key point position through calculation. However, this embodiment does not calculate the 2D coordinates, but only calculates the heatmap image.

[0070] This embodiment generates as many heatmaps as there are human body key points. The selection of human body key points includes choosing multiple key points on the nose, eyes, ears, arms, and fingers. Specifically:

[0071] Regarding the selection of key points, refer to... Figures 3 to 4 In this embodiment, the chosen method for the sign language task is... Figure 3 and Figure 4 The diagram shows 42 points on the nose, eyes, ears, arms, and fingers. Specifically, they are labeled as follows: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 17, 18, 94, 97, 98, 99, 100, 102, 103, 104, 106, 107, 108, 110, 111, 112, 114, 115, 118, 119, 120, 121, 123, 124, 125, 127, 128, 129, 131, 132, 133, 135. Regarding sampling between fingers, the upper joints of the fingers, for example, points 101 and 105, have relatively small deformation and were therefore not sampled.

[0072] It should be noted that the pre-training process of the pre-trained human pose estimation network model in this embodiment can be found at "https: / / github.com / leoxiaobin / deep-high-resolution-net.pytorch". This URL contains the HRNet network model (High-Resolution Net), which is designed for 2D human pose estimation (or keypoint detection) tasks. This network primarily focuses on pose evaluation for a single individual (i.e., the image input to the network should contain only one human target). Human pose estimation has numerous applications today, such as human action recognition, human-computer interaction (e.g., a person performing a certain action triggers the system to perform certain tasks), and animation production (e.g., generating corresponding cartoon character actions based on human keypoint information). This embodiment does not describe the specific training process of the pre-trained human pose estimation network model.

[0073] In this embodiment, instead of directly extracting the coordinate information of the key points of the human body as feature information, the heatmap image before the final output coordinate position of the key point detection network is used as feature information. This increases the accuracy of action embedding feature information and improves the acquisition of sign language, thereby improving the accuracy of sign language vocabulary recognition.

[0074] Step S300: Extract the second feature from the heatmap using a time-based lightweight feature fast filtering model to obtain the heatmap spatial features.

[0075] Specifically, a lightweight, time-series-based feature filtering model is used to extract the second feature from the heatmap image to obtain its spatial features.

[0076] The heatmap is generated by inputting multiple channels corresponding to the video frame sequence.

[0077] The heatmap is processed by two-dimensional adaptive average pooling, a fully connected layer, and ReLU activation to obtain global heatmap temporal features.

[0078] The heatmap is processed in the temporal dimension by first group convolution, batch normalization, ReLU activation function activation, and second group convolution to obtain local heatmap temporal features;

[0079] The global heatmap temporal features and the local heatmap temporal features are added together to obtain the summed heatmap temporal features;

[0080] The sigmoid activation function is used to activate the summed heatmap temporal features to generate frame weights corresponding to the summed heatmap temporal features;

[0081] The frame weights are multiplied by the corresponding input heatmap to obtain the heatmap spatial features.

[0082] In this embodiment, a lightweight temporal feature fast filtering model with pre-learnable parameters is used to add a corresponding frame weight to the sequence dimension of the temporal features of each frame's heatmap information before inputting it into a bidirectional LSTM time series model with an attention mechanism. This can more efficiently improve the model's learning ability, thereby improving the sign language recognition effect.

[0083] Step S400: Filter the spatial features of human body key points from the heatmap spatial features to obtain the spatial features of human body key points.

[0084] Specifically, the spatial features of the heatmap are filtered for human body keypoint information to obtain the spatial features of human body keypoints, as follows:

[0085] The temporal dimension T and the channel dimension C of human keypoints in the heatmap spatial features of size BxTxCxHxW are merged to obtain the heatmap spatial features of size Bx(T*C)xHxW, where B represents the number of data entries calculated at one time during training or inference, H represents the height of the heatmap, and W represents the width of the heatmap.

[0086] The heatmap spatial features of size Bx(T*C)xHxW are grouped by the channel dimension C of human keypoints and then convolved to obtain heatmap spatial features of size BxCxHxW.

[0087] Local heatmap spatial features are obtained by batch normalization, ReLU activation, and second group convolution with the channel dimension C of human keypoints as the grouping of heatmap spatial features of size BxCxHxW.

[0088] The heatmap spatial features of size Bx(T*C)xHxW are subjected to adaptive average pooling and convolution to obtain global heatmap spatial features;

[0089] The local heatmap spatial features and the global heatmap spatial features are numerically multiplied to obtain the multiplied heatmap spatial features.

[0090] The multiplied heatmap spatial features are then activated by a third group convolution with the channel dimension C of the human keypoints as the grouping and the Mish activation function to obtain the spatial features of the human keypoints.

[0091] Step S500: The spatial features of human body key points are learned through a bidirectional LSTM time series model with attention mechanism to obtain the sign language video learning results output by the bidirectional LSTM time series model with attention mechanism.

[0092] Specifically, the spatial features of human key points obtained in step S400 are randomly deactivated with a 33% probability using the Dropout function, and then dimensionality is reduced through a fully connected layer. The dimensionality-reduced spatial features of human key points are then used for feature learning through a bidirectional LSTM time series model with an attention mechanism to obtain the sign language video learning results output by the bidirectional LSTM time series model with an attention mechanism.

[0093] Step S600: Classify and encode the sign language video learning results through a fully connected layer and a softmax layer to obtain the sign language video classification and encoding results.

[0094] Specifically, the sign language video learning results obtained in step S500 are classified and encoded through a fully connected layer and a softmax layer to obtain the sign language video classification and encoding results, which include the one-hot encoding of the sign language video classification category.

[0095] Step S700: Based on the sign language video classification and coding results, query the sign language vocabulary recognition results.

[0096] Specifically, based on the sign language video classification and coding results, the sign language vocabulary recognition results are obtained, as follows:

[0097] Based on the sign language text corresponding to the sign language video, a sign language vocabulary encoding table is generated. This encoding table is a one-hot vocabulary encoding table.

[0098] By querying the one-hot code in the sign language vocabulary coding table based on the one-hot code in the sign language video classification coding result, the predicted sign language vocabulary recognition result can be obtained.

[0099] In this embodiment, a sign language video is acquired; the video is then input into a trained human pose estimation network model for first feature extraction, resulting in a heatmap of the sign language video output by the model; a lightweight temporal feature filtering model is used to extract second features from the heatmap, obtaining spatial features. This embodiment improves the speed of sign language recognition and reduces computation by using a lightweight temporal feature filtering model to extract features from the heatmap. The spatial features of the heatmap are then filtered for spatial features of human key points, resulting in spatial features of human key points. These key points are then used for feature learning through a bidirectional LSTM time series model with attention mechanisms, yielding the sign language video learning results output by the model. This embodiment uses spatial feature filtering based on human key point information, achieving near real-time key point data extraction. Compared to other RGB training methods, this embodiment offers faster inference speeds. The sign language video learning results are then classified and encoded using fully connected layers and softmax layers, yielding sign language video classification and encoding results. Based on these classification and encoding results, sign language vocabulary recognition results are retrieved. This embodiment improves the model's learning ability through feature selection and learning. Furthermore, by extracting heatmap images from sign language videos and performing feature extraction and learning based on these heatmap images, rather than directly extracting the coordinate information of key points on the human body as feature information, the accuracy of motion embedding feature information is increased, thereby improving the accuracy of sign language recognition.

[0100] Reference Figure 5 This invention also provides a deep learning-based sign language vocabulary recognition system. This deep learning-based sign language vocabulary recognition system includes a data acquisition unit 100, a first feature extraction unit 200, a second feature extraction unit 300, a feature filtering unit 400, a feature learning unit 500, a result acquisition unit 600, and a vocabulary recognition unit 700, wherein:

[0101] Data acquisition unit 100 is used to acquire sign language videos;

[0102] The first feature extraction unit 200 is used to input the sign language video into the trained human pose estimation network model to extract the first feature and obtain the heatmap image in the sign language video output by the human pose estimation network model.

[0103] The second feature extraction unit 300 is used to extract the second feature from the heatmap using a time-based lightweight feature fast screening model to obtain the spatial features of the heatmap.

[0104] The feature filtering unit 400 is used to filter the spatial features of the heatmap spatial features for human key point information to obtain the spatial features of human key points.

[0105] The feature learning unit 500 is used to learn the spatial features of human key points through a bidirectional LSTM time series model with attention mechanism, and obtain the sign language video learning results output by the bidirectional LSTM time series model with attention mechanism.

[0106] The result acquisition unit 600 is used to classify and encode the sign language video learning results through a fully connected layer and a softmax layer to obtain the sign language video classification and encoding results.

[0107] The vocabulary recognition unit 700 is used to query the sign language vocabulary recognition results based on the sign language video classification and coding results.

[0108] It should be noted that since the deep learning-based sign language vocabulary recognition system in this embodiment is based on the same inventive concept as the deep learning-based sign language vocabulary recognition method described above, the corresponding content in the method embodiment is also applicable to this system embodiment, and will not be described in detail here.

[0109] This invention also provides a deep learning-based sign language vocabulary recognition device, comprising: at least one control processor and a memory for communicatively connecting to the at least one control processor.

[0110] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0111] The non-transitory software program and instructions required to implement the deep learning-based sign language vocabulary recognition method of the above embodiments are stored in memory. When executed by a processor, the deep learning-based sign language vocabulary recognition method of the above embodiments is executed, for example, the method described above is executed. Figure 1 The method steps S100 to S700.

[0112] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] This invention also provides a computer-readable storage medium storing computer-executable instructions. These instructions are executed by one or more control processors, causing the processors to perform a deep learning-based sign language vocabulary recognition method as described in the above-described method embodiments. For example, they can execute the methods described above. Figure 1 The functions of steps S100 to S700 in the method.

[0114] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0115] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. A method for sign language vocabulary recognition based on deep learning, characterized in that, The sign language vocabulary recognition method based on deep learning comprises: acquiring a sign language video; inputting the sign language video into a trained human pose estimation network model to perform first feature extraction, and obtaining a heatmap graph in the sign language video output by the human pose estimation network model; performing second feature extraction on the heatmap graph by a time series lightweight feature rapid screening model to obtain heatmap spatial features; performing spatial feature screening of human key point information on the heatmap spatial features to obtain human key point spatial features; performing feature learning on the human key point spatial features by a bidirectional LSTM time series model with an attention mechanism to obtain a sign language video learning result output by the bidirectional LSTM time series model with the attention mechanism; performing classification and coding on the sign language video learning result by a full connection layer and a softmax layer to obtain a sign language video classification coding result; querying a sign language vocabulary recognition result according to the sign language video classification coding result. 2.The method of claim 1, wherein, Before the sign language video is inputted into the trained human pose estimation network model to perform the first feature extraction, the sign language vocabulary recognition method based on deep learning further comprises: presetting an ideal frame number in the sign language video; if the number of frames in the sign language video is less than the ideal frame number, performing blank frame padding on the ideal frame number to obtain a processed sign language video, and the number of frames in the processed sign language video is greater than or equal to the ideal frame number. 3.The method of claim 1, wherein, The human key points comprise: a plurality of key points selected on a nose, eyes, ears, arms and fingers of a human body. 4.The method of claim 1, wherein, The second feature extraction on the heatmap graph by the time series lightweight feature rapid screening model to obtain the heatmap spatial features comprises: inputting the heatmap graph by a plurality of channels corresponding to a video frame sequence; performing two-dimensional adaptive average pooling, a full connection layer and a Relu activation function activation on the heatmap graph to obtain global heatmap time series features; performing first grouping convolution, batch normalization, Relu activation function activation and second grouping convolution on the heatmap graph in a time series dimension to obtain local heatmap time series features; adding the global heatmap time series features and the local heatmap time series features to obtain added heatmap time series features; generating frame weights corresponding to the added heatmap time series features by adopting a Sigmoid activation function to activate the added heatmap time series features; multiplying the frame weights and the inputted heatmap graph to obtain heatmap spatial features. 5.The method of claim 4, wherein, The spatial feature screening of human key point information on the heatmap spatial features to obtain human key point spatial features comprises: merge the time sequence dimension T in the heatmap space feature with a size of BxTxCxHxW and the channel dimension C of the human body key point, to obtain a heatmap space feature with a size of Bx(T*C)xHxW, wherein B represents the number of data entries calculated at one time during training or inference data, H represents the height of the heatmap, and W represents the width of the heatmap; perform first grouped convolution on the heatmap space feature with a size of Bx(T*C)xHxW in groups of the channel dimension C of the human body key point, to obtain a heatmap space feature with a size of BxCxHxW; perform batch normalization, Relu activation function activation, and second grouped convolution in groups of the channel dimension C of the human body key point on the heatmap space feature with a size of BxCxHxW, to obtain a local heatmap space feature; perform adaptive average pooling and convolution on the heatmap space feature with a size of Bx(T*C)xHxW, to obtain a global heatmap space feature; numerically multiply the local heatmap space feature and the global heatmap space feature, to obtain a multiplied heatmap space feature; perform third grouped convolution in groups of the channel dimension C of the human body key point and Mish activation function activation on the multiplied heatmap space feature, to obtain a human body key point space feature. 6.The method of claim 1, wherein, Before the human body key point space feature is subjected to feature learning by the bidirectional LSTM time sequence model with an attention mechanism, the sign language vocabulary recognition method based on deep learning further includes: performing Dropout random inactivation and full connection layer dimension reduction processing on the human body key point space feature. 7.The method of claim 1, wherein, According to the sign language video classification coding result, the query obtains a sign language vocabulary recognition result, which includes: generating a sign language vocabulary coding table according to a sign language text corresponding to the sign language video; querying the sign language vocabulary coding table with the sign language video classification coding result, to obtain a predicted sign language vocabulary recognition result. 8.A sign language vocabulary recognition system based on deep learning, characterized in that, The sign language vocabulary recognition system based on deep learning includes: a data acquisition unit configured to acquire a sign language video; a first feature extraction unit configured to input the sign language video into a trained human pose estimation network model to perform first feature extraction, to obtain a heatmap graph in the sign language video output by the human pose estimation network model; a second feature extraction unit configured to perform second feature extraction on the heatmap graph by a time sequence light feature fast screening model, to obtain a heatmap space feature; a feature screening unit configured to perform spatial feature screening of human body key point information on the heatmap space feature, to obtain a human body key point space feature; and The feature learning unit is used to learn the spatial features of the human body key points through a bidirectional LSTM time series model with attention mechanism, and obtain the sign language video learning results output by the bidirectional LSTM time series model with attention mechanism. The result acquisition unit is used to classify and encode the sign language video learning results through a fully connected layer and a softmax layer to obtain the sign language video classification and encoding results. The vocabulary recognition unit is used to query the sign language vocabulary recognition results based on the sign language video classification and coding results. 9.A sign language vocabulary recognition device based on deep learning, characterized by, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform the deep learning-based sign language vocabulary recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the deep learning-based sign language vocabulary recognition method as described in any one of claims 1 to 7.

Images