Sign language recognition method, device and equipment and storage medium

Abstract

The application discloses a sign language recognition method and device, equipment and a storage medium, the method comprises obtaining joint pose data and skeleton pose data of a target sign language action, further expanding into joint motion data flow, skeleton pose data flow and skeleton motion data flow, and constructing a topological relationship corresponding to a human body pose structure, which is input into a sign language graph convolutional neural network for sign language recognition to obtain a sign language prediction vocabulary corresponding to the target sign language action. The application is based on a multi-flow fusion framework in which multiple pose data representation forms are used as decision inputs, joint pose data flow, joint motion data flow, skeleton pose data flow and skeleton motion data flow are used, a high-precision sign language recognition process is realized, and the technical problem that the recognition accuracy of a current sign language recognition method for sign language videos belonging to different categories but having extremely similar local structures is not high is solved.

Description

Technical Field

[0001] This invention relates to the field of graph convolutional network technology, and in particular to a sign language recognition method, apparatus, device, and storage medium. Background Technology

[0002] Sign language is a language used by deaf or speech-impaired individuals to communicate. It is conveyed through rapid and complex gestures, body postures, and even facial expressions, requiring significant effort to master. Sign language recognition aims to bridge the gap between sign language users and others by recognizing the semantic meaning of words demonstrated in videos. Research in sign language recognition began in the 1990s, with its main challenge stemming from considering the dynamic connections in natural language, such as the relationships between body parts and the feature representation of non-hand regions, rather than simply recognizing static gestures or hand positions. In recent years, Graph Convolutional Neural Networks (GCNs) have attracted considerable interest, demonstrating significant performance advantages in numerous sign language recognition tasks due to their powerful reasoning capabilities and high efficiency.

[0003] Existing Generative Convolutional Networks (GCNs) models for sign language recognition can be broadly categorized into two types based on their temporal information processing strategies. One type feeds a two-dimensional spatial topology graph into the GCN to capture spatial interaction information within video frames, then uses serialization models such as LSTM or BERT to analyze the temporal dependencies between frames. The other type constructs a three-dimensional spatiotemporal topology graph and feeds it into the GCN to directly capture the spatiotemporal representations in the video. However, both types of GCNs share a common drawback: for sign language videos belonging to different categories but with extremely similar local structures, features learned based on existing graph convolutional structures are insufficient for accurate judgment. Existing GCN methods have not yet focused on learning the specific characteristics of sign language recognition, resulting in bottlenecks in the extraction and analysis of key topological features, thus hindering further performance optimization. Therefore, improving the accuracy of sign language recognition is a pressing technical problem that needs to be solved.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a sign language recognition method, apparatus, device, and storage medium, aiming to solve the technical problem that current sign language recognition methods have low recognition accuracy for sign language videos that belong to different categories but have extremely similar local structures.

[0006] To achieve the above objectives, the present invention provides a sign language recognition method, the method comprising the following steps:

[0007] The joint pose data of the target sign language movement are obtained by the Kinect acquisition device or pose estimator, and the skeletal pose data is determined based on the joint pose data.

[0008] Based on the joint pose data and the bone pose data, new pose representation forms are obtained: joint pose data stream, joint motion data stream, bone pose data stream and bone motion data stream;

[0009] The joint pose data stream, joint motion data stream, skeletal pose data stream, and skeletal motion data stream are combined with topological relationships and input into a sign language graph convolutional neural network for sign language recognition to obtain the predicted sign language words corresponding to the target sign language action.

[0010] Optionally, the step of acquiring joint pose data of the target sign language movement based on the Kinect acquisition device or pose estimator specifically includes:

[0011] The coordinate data of several key joints are extracted when the sign language demonstrator demonstrates the target sign language action, and the key joints are divided into source joints and target joints.

[0012] Optionally, the joint pose data includes coordinate scalar values ​​of several source joints and their corresponding confidence levels, as well as coordinate scalar values ​​of several target joints and their corresponding confidence levels. The bone pose data includes the source joints corresponding to the target bone and the coordinate vector values ​​determined by the target joints and their corresponding confidence levels.

[0013] Optionally, the joint pose data stream and the bone pose data stream are joint pose data and bone pose data extracted from several consecutive image frames, and the joint motion data stream and the bone motion data stream are the difference between joint pose data and bone pose data extracted from several adjacent image frames.

[0014] Optionally, the step of combining the joint pose data stream, joint motion data stream, skeleton pose data stream, and skeleton motion data stream with topological relationships and inputting them together into a sign language graph convolutional neural network for sign language recognition to obtain the sign language prediction vocabulary corresponding to the target sign language action specifically includes:

[0015] Based on the joint pose data stream, joint motion data stream, skeletal pose data stream, and skeletal motion data stream, combined with the adjacency matrix of the sign language demonstrator's full joints and hand joints, the data is input into a sign language graph convolutional neural network for sign language recognition to obtain the predicted sign language vocabulary corresponding to the target sign language action; wherein:

[0016] Each row of the adjacency matrix represents the set of adjacent nodes of a joint, and each column of the adjacency matrix represents each joint.

[0017] Optionally, the sign language graph convolutional neural network performs feature embedding on the local structure of each node, and the expression for the feature embedding is:

[0018]

[0019] Where l represents the number of layers, K v w represents the number of distinct partitions. k For the k-th partition, the weight matrix from layer i=1 to layer l, and the embedding feature Z of layer 0. (0) =X, Let A be the specific adjacency matrix of the k-th partition, enhanced with global and local information.

[0020] Optionally, the expression for the graph convolutional layer structure in the sign language graph convolutional neural network is:

[0021]

[0022] Among them, A k B is the adjacency matrix of all joints used in the k-th partition. k For A k The initialized learnable adjacency matrix, H k The adjacency matrix of the hand joint topology used for the k-th partition, PH k Indicates H k An initialized learnable adjacency matrix.

[0023] Furthermore, to achieve the above objectives, the present invention also provides a sign language recognition device, the sign language recognition device comprising:

[0024] The determination module is used to acquire joint pose data of the target sign language movement based on the Kinect acquisition device or pose estimator, and determine the skeletal pose data based on the joint pose data.

[0025] The module is used to obtain new pose representations based on joint pose data and bone pose data: joint pose data stream, joint motion data stream, bone pose data stream, and bone motion data stream.

[0026] The recognition module is used to input the joint pose data stream, joint motion data stream, bone pose data stream, and bone motion data stream together with the topological relationship into the sign language graph convolutional neural network for sign language recognition, and obtain the predicted sign language words corresponding to the target sign language action.

[0027] In addition, to achieve the above objectives, the present invention also provides a sign language recognition device, the device comprising: a memory, a processor, and a sign language recognition method program stored in the memory and executable on the processor, wherein the sign language recognition method program, when executed by the processor, implements the steps of the sign language recognition method described above.

[0028] In addition, to achieve the above objectives, the present invention also provides a storage medium storing a sign language recognition method program, which, when executed by a processor, implements the steps of the sign language recognition method described above.

[0029] This invention proposes a sign language recognition method, apparatus, device, and storage medium. The method includes acquiring joint pose data of a target sign language movement using a Kinect acquisition device or a pose estimator; determining skeletal pose data based on the joint pose data; obtaining new pose representation forms based on the joint pose data and the skeletal pose data: a joint pose data stream, a joint motion data stream, a skeletal pose data stream, and a skeletal motion data stream; and inputting these data streams, combined with topological relationships, into a sign language graph convolutional neural network for sign language recognition to obtain predicted sign language vocabulary corresponding to the target sign language movement. This invention utilizes a multi-stream fusion framework based on multiple pose data representation forms as decision inputs, leveraging joint pose data streams, joint motion data streams, skeletal pose data streams, and skeletal motion data streams to achieve a highly accurate sign language recognition process. This solves the technical problem of low recognition accuracy in current sign language recognition methods for sign language videos belonging to different categories but with extremely similar local structures. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the structure of a sign language recognition device according to an embodiment of the present invention;

[0031] Figure 2 This is a flowchart illustrating a sign language recognition method according to an embodiment of the present invention;

[0032] Figure 3 This is a framework diagram of the multi-stream sign language recognition system in an embodiment of the present invention;

[0033] Figure 4 This is a framework diagram of the preprocessing module for pose data in an embodiment of the present invention;

[0034] Figure 5 This is a flowchart of the graph convolutional neural network algorithm for pose data in an embodiment of the present invention;

[0035] Figure 6This is a schematic diagram of the Gated Compositive HA-GCM in an embodiment of the present invention;

[0036] Figure 7 This is a structural diagram of the SLR-GCN Block in an embodiment of the present invention;

[0037] Figure 8 This is a diagram of the overall network structure of SLR-GCN in this embodiment of the invention;

[0038] Figure 9 This is a structural block diagram of a sign language recognition device according to an embodiment of the present invention.

[0039] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0040] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0041] Currently, in the relevant technical field, existing sign language recognition methods do not have high accuracy in recognizing sign language videos that belong to different categories but have extremely similar local structures.

[0042] To address this issue, various embodiments of the sign language recognition method of the present invention are proposed. The sign language recognition method provided by the present invention inputs multiple pose data representation forms, such as joint pose data stream, joint motion data stream, skeletal pose data stream, and skeletal motion data stream, into a multi-stream decision fusion framework to achieve a highly accurate sign language recognition process. This solves the technical problem of low recognition accuracy of current sign language recognition methods for sign language videos belonging to different categories but with extremely similar local structures.

[0043] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of the sign language recognition device involved in the embodiment of the present invention.

[0044] The device can be a user equipment (UE) such as a mobile phone, smartphone, laptop, digital broadcast receiver, personal digital assistant (PDA), tablet computer (PAD), handheld device, in-vehicle device, wearable device, computing device or other processing device connected to a wireless modem, mobile station (MS), etc. The device may be referred to as a user terminal, portable terminal, desktop terminal, etc.

[0045] Typically, the device includes: at least one processor 301, a memory 302, and a sign language recognition method program stored in the memory and executable on the processor, the sign language recognition method program being configured to implement the steps of the sign language recognition method as described above.

[0046] Processor 301 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 301 may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). Processor 301 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 301 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. Processor 301 may also include an AI (Artificial Intelligence) processor, which processes operations related to sign language recognition methods, enabling the sign language recognition method model to train and learn autonomously, improving efficiency and accuracy.

[0047] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 302 are used to store at least one instruction, which is executed by the processor 301 to implement the sign language recognition method provided in the method embodiments of this application.

[0048] In some embodiments, the terminal may also optionally include a communication interface 303 and at least one peripheral device. The processor 301, memory 302, and communication interface 303 can be connected via a bus or signal line. Each peripheral device can be connected to the communication interface 303 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 304, a display screen 305, and a power supply 306.

[0049] The communication interface 303 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 301 and the memory 302. The communication interface 303 is used to receive movement trajectories and other data uploaded by the user from multiple mobile terminals via the peripheral device. In some embodiments, the processor 301, memory 302, and communication interface 303 are integrated on the same chip or circuit board; in other embodiments, any one or two of the processor 301, memory 302, and communication interface 303 can be implemented on separate chips or circuit boards, and this embodiment is not limited to this.

[0050] The radio frequency (RF) circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 304 communicates with communication networks and other communication devices via electromagnetic signals, thereby acquiring the movement trajectories and other data of multiple mobile terminals. The RF circuit 304 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 304 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 304 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0051] Display screen 305 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 305 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 301 for processing. In this case, display screen 305 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, display screen 305 can be a single screen, the front panel of an electronic device; in other embodiments, display screen 305 can be at least two screens, respectively disposed on different surfaces of the electronic device or in a folded design; in still other embodiments, display screen 305 can be a flexible display screen, disposed on a curved or folded surface of the electronic device. Furthermore, display screen 305 can also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 305 can be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0052] Power supply 306 is used to supply power to various components in an electronic device. Power supply 306 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 306 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0053] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the sign language recognition device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0054] This invention provides a sign language recognition method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the sign language recognition method of the present invention.

[0055] In this embodiment, the sign language recognition method includes the following steps:

[0056] Step S100: Obtain joint pose data of the target sign language movement using the Kinect acquisition device or pose estimator, and determine skeletal pose data based on the joint pose data.

[0057] Specifically, when acquiring the pose information of the target sign language movement, the coordinate data of several joints are extracted when the sign language demonstrator demonstrates the target sign language movement, and these joints are divided into source joints and target joints.

[0058] In this example, when acquiring pose information, the pose information of the target sign language action can be obtained using a Kinect acquisition device or a pose estimation model. It should be noted that all methods for acquiring pose information of the target sign language action that can be used by those skilled in the art are within the scope of protection of this application, and this embodiment does not limit them.

[0059] As is easily understood, the pose estimation model can use HRNet or OpenPose. Several joints represent the 2D or 3D coordinate data of several joints in the upper body and hand region of the sign language demonstrator, i.e., joint pose data. This joint pose data includes the coordinate scalar values ​​of several source joints and their corresponding confidence scores, as well as the coordinate scalar values ​​of several target joints and their corresponding confidence scores.

[0060] Subsequently, based on the coordinate data corresponding to the source joint and the coordinate data corresponding to the target joint, the bone pose data can be determined. The bone pose data includes the source joint points corresponding to the target bone and the coordinate vector values ​​and corresponding confidence levels of the determined target joint points.

[0061] In practical applications, for joint pose data, in a given coordinate system at time t, the source joint is... and its target joint is (x, y, s) represent the x and y axis coordinates and the confidence level, respectively; for skeletal pose data, in a given coordinate system at frame t, its basic vector can be obtained by subtracting its source joint coordinates from its current joint coordinates, i.e., for all (i, j) ∈ H, H represents a naturally connected set of human bodies.

[0062] Step S200: Based on the joint pose data and the bone pose data, obtain new pose representation forms: joint pose data stream, joint motion data stream, bone pose data stream, and bone motion data stream.

[0063] Specifically, the joint pose data stream and the bone pose data stream are joint pose data and bone pose data extracted from several consecutive image frames, while the joint motion data stream and the bone motion data stream are the differences between joint pose data and bone pose data extracted from several adjacent image frames.

[0064] In practical applications, after obtaining joint pose data and bone pose data, the difference between the same joint or bone in two consecutive frames is used to represent the pose. For example, given a joint in the coordinate system at time t. The same joint in the coordinate system at frame t+1 Then v i,t and v i,t+1 The motion information between them can be represented as The resulting sign language video information is denoted as the joint motion data stream (JM-stream). Similarly, the skeletal motion data stream (BM-stream) is generated by... Calculated generation.

[0065] In step S300, the joint pose data stream, joint motion data stream, bone pose data stream, and bone motion data stream are combined with topological relationships and input into the sign language graph convolutional neural network for sign language recognition to obtain the predicted sign language words corresponding to the target sign language action.

[0066] Specifically, when performing sign language recognition based on joint pose data stream, joint motion data stream, skeleton pose data stream, and skeleton motion data stream, the spatiotemporal skeleton diagram of the sign language demonstrator when demonstrating the target sign language action is determined by combining the adjacency matrix of the sign language demonstrator's full joints and hand joints. Then, the pose data stream and the adjacency matrix are input into the sign language graph convolutional neural network for sign language recognition to obtain the predicted sign language words corresponding to the target sign language action.

[0067] In this context, each row of the adjacency matrix represents the set of adjacent nodes of a key point, and each column of the adjacency matrix represents each key point in the input graph. The input graph expression of the spatiotemporal skeleton graph is G = (V, E), where V is the set of skeleton key points, and E is the connection between all adjacent key points in a single frame and the inter-frame connection between corresponding key points in adjacent frames.

[0068] Furthermore, the sign language graph convolutional neural network performs feature embedding on the local structure of each node in the input graph. The expression for the feature embedding is:

[0069]

[0070] Where l represents the number of layers, K v w represents the number of distinct partitions. k For the k-th partition, the weight matrix from the (l-1)-th layer to the l-th layer, and the embedding feature Z of the 0th layer. (0) =X, Let A be the specific adjacency matrix of the k-th partition, enhanced with global and local information.

[0071] Specifically, the expression for the graph convolutional layer structure in the sign language graph convolutional neural network for sign language recognition is as follows:

[0072]

[0073] Among them, A k B is the adjacency matrix of all joints used in the k-th partition. k For A k The initialized learnable adjacency matrix, H kThe adjacency matrix of the hand joint topology used for the k-th partition, PH k Indicates H k An initialized learnable adjacency matrix.

[0074] To explain this application more clearly, specific examples of the sign language recognition method of this application are provided below.

[0075] The sign language recognition method presented in this specific example is based on a multi-stream fusion framework that uses multiple pose data representations as decision inputs, such as... Figure 3 As shown.

[0076] The joint data is divided into joint pose data stream (J-stream) and joint motion data stream (JM-stream), and the skeleton data is divided into skeleton pose data stream (B-stream) and skeleton motion data stream (BM-stream). Then, the softmax scores obtained by the SLR-GCN sign language recognition network designed in this example are weighted, summed and fused to obtain the action score and predict the sign language vocabulary label.

[0077] For pose topology information, this example uses J-stream and B-stream to represent the joint topology data stream and skeleton topology data stream generated based on pose coordinate information, respectively, as follows: Figure 3 As shown.

[0078] 1) Each joint topology is represented as a vector pointing from the source joint to the target joint along the natural connections of the human body. In this example, the joint closer to the skeletal centroid is defined as the source joint, and the joint farther from the centroid as the target joint. The nose node is used as the root joint. For example, in a given coordinate system at frame t, its source joint is... and its target joint is (x,y,s) represent the x and y coordinates and the confidence level, respectively.

[0079] 2) The basic vectors in the skeletal topology graph can be obtained by subtracting the source joint coordinates from the current joint coordinates, i.e., for all (i,j)∈H, Here, H represents the set of naturally connected human bodies. There are no circular connections in the skeletal topology graph, and each bone vector needs to be assigned a unique target joint. Since the number of joints is one more than the number of bone information points, and the root joint has not been assigned a suitable target joint, to simplify the network design, this example assigns an empty node with a value of 0 as the target joint to the root joint, thus facilitating the design of a skeletal topology graph and network with the same number of joints.

[0080] For pose motion information, this example uses the difference between the same joints or bones in two consecutive frames. For example, given a joint in the coordinate system at time t. The same joint in the coordinate system at frame t+1 Then v i,t and v i,t+1 The motion information between them can be represented as The resulting sign language video information is denoted as the joint motion data stream (JM-stream). Similarly, the skeletal motion data stream (BM-stream) is generated by... Calculated generation.

[0081] The process of implementing sign language recognition includes the following steps:

[0082] 1. Data preprocessing for pose topology graphs, such as... Figure 4 As shown;

[0083] 1.1 Use a Kinect acquisition device or an existing pose estimation model (such as HRNet or OpenPose) to capture / extract and select 2D or 3D coordinate data X of n joints distributed in the upper body and hand area of ​​the sign language demonstrator in the video sequence.

[0084] 1.2 Random Continuous Sampling. This involves randomly selecting the starting position of the video and extracting a segment of length T frames, limiting the sample space size fed into the network to C×T×n. When the video is less than T frames, the current video sequence is repeated to fill it into a fixed-length frame sequence. When the video length is greater than T frames, a continuous video frame of length T is randomly selected.

[0085] 1.3 Random Jitter. For the first-order information provided by the joint pose data stream, random jitter, i.e., displacement operation, is performed on all coordinates one by one; for the second-order vector information such as the skeleton pose data stream, joint motion data stream, and skeleton motion data stream, random jitter is performed only on the origin coordinate.

[0086] 1.4 Random Mirroring. When the generated random number is greater than the random threshold (0.5), the arrangement order of the human body's symmetrical joints is replaced; at the same time, for the first-order information provided by the joint pose data stream, the x and y coordinate positions of each coordinate are successively subtracted from the length and width of the video frame; for the second-order vector information such as the skeleton pose data stream, joint motion data stream, and skeleton motion data stream, the values ​​are set to their opposites.

[0087] 1.5 Normalization. For the first-order information provided by the joint pose data stream, the mean of the origin is subtracted from all coordinates one by one; for the second-order vector information such as the skeleton pose data stream, joint motion data stream, and skeleton motion data stream, only the coordinates of the origin are normalized; the coordinates of key points are normalized to [-1, 1].

[0088] 2. Sign language recognition using a graph convolutional neural network flowchart oriented towards pose data, such as... Figure 5 As shown;

[0089] 2.1 Based on the physical connections of the human body, construct an adjacency matrix A with dimensions n×n, representing the actual topological relationships of the n selected joints. Each row of matrix A represents the set of neighboring nodes of a node, and each column element of that row corresponds to each node in the input graph. If the element is 1, the node in the corresponding column belongs to the set of neighboring nodes of the node in that row; if the element in the column is 0, it does not belong to the set of neighboring nodes. Similarly, an adjacency matrix H can be constructed that only involves the topological relationships of hand joints, still with dimensions n×n, and all element values ​​related to non-hand joints are set to 0.

[0090] 2.2 Construct a spatiotemporal skeleton graph with human joints as vertices and bones as edges; connect corresponding nodes in adjacent frames in the form of time edges to construct a T-frame spatiotemporal topology input graph G = (V, E) with n joints, where the vertex set V is the set of skeleton key points; the edge set E includes all connections between adjacent nodes in a single frame and the inter-frame connections between corresponding joints in adjacent frames.

[0091] 2.3 The spatiotemporal skeleton graph constructed by combining pose data and adjacency matrix is ​​fed into the Sign Language Recognition Graph Convolution Network (SLR-GCN), and finally passed through a global average pooling layer, a fully connected layer and a softmax classifier to obtain the probability of the corresponding sign language words.

[0092] The novel sign language graph convolutional structure proposed in this example embeds Z features into the local structure of each node in the input graph;

[0093]

[0094] Where l represents the layer number, thus obtaining the output of the l-th layer of the model; K v w represents the number of distinct partitions. k For the k-th partition, the weight matrix from the (l-1)-th layer to the l-th layer, and the embedding feature Z of the 0th layer. (0) =X, Let A be the specific adjacency matrix of the k-th partition, enhanced with global and local information. For the sign language recognition task, this example innovatively designs a novel graph convolutional layer structure (Gated Compositive Hand-Aware Graph Convolution Module), such as... Figure 6 As shown:

[0095]

[0096] Among them, A k B is the adjacency matrix of all joints used in the k-th partition. k For A k An initialized learnable adjacency matrix is ​​used to enhance the model's flexibility. k An n×n adjacency matrix, involving only the topological relationships of hand joints, is constructed for the k-th partition to enhance the convolutional layer's focus on the hand region. PH k Indicates H k An initialized, n×n learnable adjacency matrix. PH k The elements will be parameterized and optimized along with other parameters during the training process.

[0097] It should be noted that the subgraph PH k Without constraints, it learns entirely from the training data, allowing for the creation of new edges and the deletion of old ones based on the initial settings. In a data-driven approach, this method can learn topological relationships specifically tailored to the sign language recognition task and personalize the different information contained in different hidden layers.

[0098] As is easily understood, this example uses a gating mechanism to adjust the importance of hand images in different layers. H... k It is multiplied by a parameterized coefficient α or β, which is unique for each layer and is dynamically updated during training based on the dataset’s dependence on the overall topological relationships of the hand.

[0099] In this example, each SLR-GCN Block contains the novel sign language graph convolutional structure (HA-GCM) proposed in this example, the STC attention mechanism, the temporal convolutional layer (TCN), the temporal drop graph module (DropT), the batch normalization (BN) layer, and the ReLU layer, as follows. Figure 7 As shown.

[0100] Among them, 1) HA-GCM applies the novel graph convolutional structure proposed above, improving the topological evolution capability of GCNs for sign language recognition tasks without increasing extraction costs. 2) The STC attention mechanism is composed of cascaded spatial attention, temporal attention, and channel attention modules, strengthening the weight ratio of effective feature vectors. 3) TCN, which performs K... l4) A 2D convolution of ×1 is used to handle temporal features. A DropGraph module is introduced in the time dimension to avoid overfitting and enhance the model's generalization ability, making it suitable for various differentiated data samples. 5) To stabilize training and slow down gradient propagation, a residual connection is added to each base block. In summary, the SLR-GCN Block constructed in this example can meet the basic performance requirements of sign language recognition graph convolutional networks.

[0101] It should be noted that the sign language graph convolutional neural network SLR-GCN used in this example is mainly composed of 10 SLR-GCNBlock basic blocks, such as... Figure 8 As shown, the first four basic blocks each have 64-dimensional output channels, the middle three basic blocks have 128-dimensional output channels, and the last three basic blocks have 256-dimensional output channels. Initially, a batch normalization (BN) layer is added to standardize the input data. Finally, a global average pooling (GAP) layer is performed in both spatial and temporal dimensions to pool the feature maps of different samples to the same size. The output is then fed into a softmax classifier to obtain the prediction result.

[0102] The following section provides a further description of the sign language recognition performance of this application, based on experimental data.

[0103] (1) Table 1 reports the Top-1 recognition rates of isolated sign language recognition networks using only skeletal pose data, based on the open-source datasets AUTSL (http: / / cvml.ankara.edu.tr / datasets / ) and INCLUDE (https: / / zenodo.org / record / 4010759) respectively.

[0104]

[0105]

[0106] Table 1. Comparison of Top-1 recognition rates of independent sign language recognition networks

[0107] The SLR-GCN proposed in this application enhances attention to hand topological details and effectively utilizes the self-learning capability of graph convolutional networks. It achieves recognition accuracies of 96.26% and 99.75% on the AUTSL and INCLUDE datasets, respectively, representing state-of-the-art results. Compared to other existing methods, SLR-GCN demonstrates overwhelming performance advantages, indicating its greater suitability for sign language recognition tasks.

[0108] (2) Table 2 reports the results of applying the proposed SLR-GCN in different topology data streams with the Top-1 recognition rate. Each data stream is one of the different representations of the pose topology graph. Among them, the joint flow performs best in the four topology data streams, while the skeletal flow is slightly inferior; the sign language recognition accuracy of the joint motion flow and skeletal motion flow is reduced to varying degrees, which indicates that the GCNs method is slightly less effective in capturing and analyzing pose motion information. As different representations of pose topology data, there is still complementarity between them. This application uses a weighted strategy to explore this, and the experimental results show that the integrated multi-stream pose topology data has a stable improvement over the single topology data stream.

[0109]

[0110] Table 2. Results of multi-flow topology graph recognition based on SLR-GCN

[0111] In this embodiment, a sign language recognition method is proposed based on a multi-stream fusion framework that uses multiple pose data representations as decision inputs. This method utilizes joint pose data streams, joint motion data streams, skeletal pose data streams, and skeletal motion data streams to achieve a high-precision sign language recognition process. This solves the technical problem that current sign language recognition methods have low recognition accuracy for sign language videos that belong to different categories but have extremely similar local structures.

[0112] Reference Figure 9 , Figure 9 This is a structural block diagram of an embodiment of the sign language recognition device of the present invention.

[0113] like Figure 9 As shown, the sign language recognition device proposed in this embodiment of the invention includes:

[0114] The determination module 10 is used to obtain joint pose data of the target sign language movement based on the Kinect acquisition device or pose estimator, and determine the skeletal pose data based on the joint pose data.

[0115] The module 20 is used to obtain new pose representations based on joint pose data and the bone pose data: joint pose data stream, joint motion data stream, bone pose data stream and bone motion data stream.

[0116] The recognition module 30 is used to input the joint pose data stream, joint motion data stream, bone pose data stream, and bone motion data stream together with the topological relationship into the sign language graph convolutional neural network for sign language recognition, and obtain the sign language predicted words corresponding to the target sign language action.

[0117] Other embodiments or specific implementations of the sign language recognition device of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0118] Furthermore, embodiments of the present invention also propose a storage medium storing a sign language recognition method program, which, when executed by a processor, implements the steps of the sign language recognition method as described above. Therefore, further details will not be repeated here. Additionally, the beneficial effects of employing the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application. As an example, program instructions may be deployed to execute on a single computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0119] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0120] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware, and of course, it can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memory, special components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for the present invention, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, portable hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

Claims

1. A method for sign language recognition, characterized in that, The method includes the following steps: The joint pose data of the target sign language movement are obtained by the Kinect acquisition device or pose estimator, and the skeletal pose data is determined based on the joint pose data. Based on the joint pose data and the bone pose data, new pose representation forms are obtained: joint pose data stream, joint motion data stream, bone pose data stream and bone motion data stream; The joint pose data stream and the bone pose data stream are joint pose data and bone pose data extracted from several consecutive image frames. The joint motion data stream and the bone motion data stream are the differences between joint pose data and bone pose data extracted from several adjacent image frames. Based on the joint pose data stream, joint motion data stream, skeletal pose data stream, and skeletal motion data stream, combined with the adjacency matrix of the sign language demonstrator's full joints and hand joints, the data is input into a sign language graph convolutional neural network for sign language recognition to obtain the predicted sign language vocabulary corresponding to the target sign language action; wherein: Each row of the adjacency matrix represents the set of adjacent nodes of a joint, and each column of the adjacency matrix represents each joint. The sign language graph convolutional neural network embeds features into the local structure of each node, and the expression for the feature embedding is: wherein, denotes the number of layers, is the number of different partitions, is the weight matrix of the k-th layer for the i-th partition, layer to the i-th layer of the k-th layer, layer of the embedding feature, , is the specific adjacency matrix A of the k-th partition reinforced by global and local information attention. The expression for the graph convolutional layer structure in the sign language graph convolutional neural network is: wherein, is an adjacency matrix of the full joint used in the kth partition, is an adjacency matrix of the hand joint topology relationship used in the kth partition, is a learnable adjacency matrix initialized with is an adjacency matrix of the full joint used in the kth partition, denotes a learnable adjacency matrix initialized with is a learnable adjacency matrix initialized with 2. The sign language recognition method of claim 1, wherein, The step of obtaining joint pose data of the target sign language movement based on the Kinect acquisition device or pose estimator specifically includes: Extract the coordinate data of several key joints when the sign language demonstrator demonstrates the target sign language action, and divide the key joints into source joints and target joints.

3. The sign language recognition method of claim 2, wherein, The joint pose data includes the coordinate scalar values ​​of several source joints and target joints and their corresponding confidence scores. The bone pose data includes the coordinate vector values ​​of the source joints and target joints corresponding to the target bone and their corresponding confidence scores.

4. A sign language recognition apparatus capable of performing a sign language recognition method according to claim 1, characterized by The sign language recognition device includes: The determination module is used to acquire joint pose data of the target sign language movement based on the Kinect acquisition device or pose estimator, and determine the skeletal pose data based on the joint pose data. The module is used to obtain new pose representations based on joint pose data and bone pose data: joint pose data stream, joint motion data stream, bone pose data stream, and bone motion data stream. The recognition module is used to input the joint pose data stream, joint motion data stream, skeletal pose data stream, and skeletal motion data stream, together with the adjacency matrix of the sign language demonstrator's full joints and hand joints, into the sign language graph convolutional neural network for sign language recognition, and obtain the predicted sign language words corresponding to the target sign language action.

5. A sign language recognition device, characterized by, The sign language recognition device includes: a memory, a processor, and a sign language recognition method program stored in the memory and executable on the processor. When the sign language recognition method program is executed by the processor, it implements the steps of the sign language recognition method as described in any one of claims 1 to 3.

6. A storage medium, characterized by The storage medium has stored thereon a sign language recognition method program, which, when executed by the processor, implements the steps of the sign language recognition method according to any one of claims 1 to 3.

Images