Real-time Sign Language Recognition and Translation System and Method Based on Edge Devices

Abstract

The present invention discloses a real-time sign language recognition and translation system and method based on edge devices, comprising: a video data acquisition module for acquiring sign language video data; a video preprocessing module comprising: a video frame size adjustment module, a data normalization module, and a video frame grayscale module; a sign language model module for obtaining a sign language model; a structural reparameterization module for optimizing the sign language model obtained by the sign language model module to obtain an optimized sign language model; and ensuring that the output of the optimized model remains unchanged. The present invention extracts key local region features and global frame features of sign language, and constructs these features into a regional perception spatiotemporal graph, uses a shallow graph convolutional network to extract sign language features, and trains a sign language model based on the extracted features. The model is optimized using structural reparameterization technology, and then the optimized model is used to recognize sign language videos as word sequences and translate them into natural language text.

Description

Technical Field

[0001] The present invention belongs to the technical field of sign language recognition and translation, and specifically relates to a real-time sign language recognition and translation system and method based on edge devices. Background Art

[0002] Sign language is the primary means of communication for people with hearing impairments. Current sign language research focuses on two areas: continuous sign language recognition (CSLR) and sign language translation (SLT). The goal of continuous sign language recognition is to identify a sign language video sequence as a corresponding sequence of sign language words (gloss). However, because sign language has different grammatical rules from natural language, the recognized sign language word sequences often do not conform to natural grammatical rules and are difficult for the public to understand. The goal of sign language translation is to translate a sign language sequence into natural language. Because the translation conforms to natural language grammatical rules, it is also easy for the public to understand.

[0003] Current sign language research (including continuous sign language recognition and sign language translation) focuses on extracting effective features from videos using 2D and 3D convolutions. However, existing work rarely addresses computational overhead. These methods require powerful graphics processing units (GPUs) for computation, making them difficult to run on edge devices with standard computing capabilities (i.e., without GPUs).

[0004] In light of this, the present invention proposes a technology that uses visual methods, does not require a high-performance graphics processing unit (GPU), and can achieve real-time sign language recognition and translation on edge devices. Furthermore, the present invention can also be applied to new human-computer interaction scenarios, analyzing user gestures to enable gesture-based control of smart devices. Summary of the Invention

[0005] To address the shortcomings of the aforementioned existing technologies, the present invention aims to provide a real-time sign language recognition and translation system and method based on edge devices. This system addresses the problem that existing sign language translation and recognition technologies require high-performance computing platforms and cannot achieve real-time translation. This invention can facilitate communication between the hearing-impaired and the general public, enabling the wider deployment and application of sign language recognition and translation models.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] The present invention provides a real-time sign language recognition and translation system based on edge devices, comprising: a video data acquisition module, a video preprocessing module, a sign language model module, and a structure reparameterization module; wherein,

[0008] Video data acquisition module, used to collect sign language video data;

[0009] Video preprocessing module, including: video frame size adjustment module, data normalization module and video frame grayscale module;

[0010] The video frame size adjustment module is used to scale the collected sign language video frames to a uniform size;

[0011] The data normalization module is used to normalize the pixel values of the resized video frame from 0-255 to 0-1;

[0012] The video frame grayscale module is used to convert the sign language video frame from an RGB three-channel image into a grayscale image;

[0013] The sign language model module is used to obtain a sign language model. The sign language model includes: a region representation sub-model, a graph convolution sub-model, and a text generation sub-model;

[0014] The region representation sub-model is used to obtain feature representations of key regions, including: a body network (BodyNet), a face network (FaceNet), a left hand network (LHandNet), and a right hand network (RHandNet);

[0015] The trunk network is used to extract skeleton points (nose, left wrist, and right wrist) from a sign language video frame, and extract the video frame into a 256-dimensional vector. The corresponding key areas (face area, left hand area, and right hand area) are cropped according to the skeleton points.

[0016] The facial network is used to extract the cropped facial region into a 256-dimensional vector;

[0017] The left-hand network is used to extract the cropped left-hand area into a 256-dimensional vector;

[0018] The right-hand network is used to extract the cropped right-hand area into a 256-dimensional vector;

[0019] The graph convolution sub-model is used to obtain feature representations of sign language videos, including: a region-aware temporal graph and a shallow graph convolutional network (S-GCN);

[0020] The region-aware spatiotemporal graph G is composed of nodes V and edges E, G = (V; E), where V is a node set, V = {v i,j ,i∈[1,N],j∈[1,4]}, where v i,j represents the jth node in the i-th frame, and N represents the number of input video frames. This region-aware spatiotemporal graph is used to provide the basic input for the subsequent graph network, representing the spatiotemporal relationship between the local and global features of sign language, and providing information transfer for the subsequent graph convolutional network to extract sign language features.

[0021] The shallow graph convolution network consists of four graph convolution modules and one fully connected layer. The graph convolution module is composed of a 1*1 convolution module, a residual graph convolution module (ResGraphConv), a 3*1 convolution module, and a residual graph convolution module (ResGraphConv). The shallow graph convolution module extracts gesture-related feature vectors based on the extracted hand features, facial features, global frame features, and region-aware spatiotemporal graph, and outputs feature vectors with a dimension of 1024.

[0022] The text generation sub-model is used to identify the feature representation of the sign language video as a sign language word sequence and translate the sign language word sequence into natural language text, including: a connectionist temporal classification decoder and a translation model;

[0023] The connectionist temporal classification decoder is used to decode the output features of the shallow graph convolution module into a corresponding sign language word sequence;

[0024] The translation model is used to translate the decoded sign language word sequence into natural language text;

[0025] The structure reparameterization module is used to optimize the sign language model obtained by the sign language model module to obtain an optimized sign language model; and ensure that the output of the optimized model remains unchanged.

[0026] Furthermore, the video frame size adjustment module includes: center cropping and resizing to a uniform size, and the video size adjustment adopts the size adjustment (reshape) function of the open source computer vision library (opencv) to adjust the image to a uniform size.

[0027] Furthermore, the data normalization module normalizes the pixel values of all video frames to 0-1 by dividing by 255.

[0028] Furthermore, the video frame grayscale module converts the RGB image into a grayscale image using the color conversion (cvtColor) function of the open source computer vision library (opencv).

[0029] Furthermore, the region representation submodel adopts a modified RepVGG model to construct a trunk network, extracts skeleton points (three key points of nose, left wrist and right wrist) from a video frame, and extracts the video frame into a 256-dimensional vector. At the same time, the corresponding key areas (face area, left hand area and right hand area) are cropped according to the above skeleton points, and then the face network, left hand network and right hand network are constructed to extract the face area, left hand area and right hand area into 256-dimensional vectors respectively; specifically: the modified RepVGG model adopts the residual network (resnet18) as the basic architecture, replaces the residual structure with the RepBlock module, and adds the skeleton extraction branch network to the third stage (Stage 3) and the fourth stage (Stage 4) of the network to obtain the trunk network; the face network, left hand network and right hand network adopt the same structure as the trunk network, but do not include the skeleton extraction branch network in the trunk network; wherein,

[0030] RepBlock: It uses three parallel branches. The first branch is an n*n convolution and batch normalization layer. The formula is as follows:

[0031] y=BN(w n (x)+b n )

[0032] Among them, x is the input vector, y is the output vector, and w n is the weight coefficient of the n*n convolution kernel, b n is the bias value, BN is the batch normalization layer, and the formula is as follows:

[0033]

[0034] Among them, γ and β are learnable parameters. are the mean and variance of the input x respectively;

[0035] The second branch is a 1*1 convolution and batch normalization layer, which is expressed as follows:

[0036] y=BN(w 1 (x)+b 1 )

[0037] The third branch is a single batch normalization layer;

[0038] Skeleton extraction branch network: It consists of two parallel deconvolutions, and the formula is as follows:

[0039] y=P1(D1(x))+P2(D2(x))

[0040] Among them, x is the input vector, y is the output vector, D1 and D2 are deconvolution layers, and P1 and P2 are pointwise convolutions; the skeleton extraction branch converts the x input vector into the y output vector to obtain the corresponding skeleton heat map, and then extracts the coordinates of the skeleton points from the skeleton heat map. Then, with the extracted skeleton points as the center, the key areas are cropped to obtain the key images of the left hand, right hand, and facial areas.

[0041] Among them, the network structures of the face network, left-hand network, and right-hand network are similar to the main structure of the torso network. They are all modified RepVGG models, using the residual network (resnet18) as the basic architecture, replacing the residual structure with the RepBlock module, but excluding the skeleton extraction branch network in the torso network.

[0042] Furthermore, the graph convolution sub-model includes a region-aware temporal graph and a shallow graph convolutional network (S-GCN); wherein the region-aware temporal graph G is composed of nodes V and edges E, G = (V; E), and for the i-th frame, four nodes are constructed, which are respectively composed of feature vectors corresponding to the global frame, the facial region, the left hand region, and the right hand region; the edge set E is composed of the intra-frame edge set E a and the intra-frame edge set E e The edge set E in the frame a ={v ip v iq |i∈[1,N],(p,q)∈S},v ip v iq Represents the edges of different key regions in the same frame, S = {(1,2), (1,3), (1,4), (2,1), (3,1), (4,1)}, where nodes 1-4 represent the global frame, facial region, left hand region, and right hand region features respectively; the inter-frame edge set E e Indicated as E a ={v ip v jp |i,j∈[1,N],|ij|=1,p∈[1,4]},v ip v jp The edges represent the corresponding nodes in adjacent frames.

[0043] Among them, the shallow graph convolution network consists of 4 graph convolution modules and 1 fully connected layer. The 1*1 convolution module of the shallow graph convolution module consists of a RepBlock module, where the first branch of RepBlock is replaced by a 1*1 convolution kernel instead of an n*n convolution kernel; the 3*1 convolution module consists of a RepBlock module, where the first branch of RepBlock is replaced by a 3*1 convolution kernel instead of an n*n convolution kernel; the residual graph convolution module consists of a graph convolution plus a residual connection, and the formula is expressed as follows:

[0044]

[0045] Among them, x ij is the feature vector of the jth node in the i-th frame of the graph, x i ′ j is the updated node feature vector, GCN(*) is the graph convolution function, θ1 and θ2 are graph convolution parameters, and N(ij) represents the neighbor nodes of node ij.

[0046] Furthermore, the connectionist temporal classification decoder is implemented using the open-source ctcdecode library, which takes the output of the graph convolutional sub-model as input, finds the path with the maximum probability sum based on the input as the final decoding path, and obtains the decoded sign language word sequence;

[0047] The translation model takes the sign language word sequence decoded by the connectionist temporal classification decoder as input and translates it into the corresponding natural language sequence. The model includes a character-level encoder, a word-level encoder, and a Transformer decoder. The character-level encoder and word-level encoder map the sign language words into high-dimensional vectors, and the Transformer decoder decodes the high-dimensional vectors as input to obtain natural language text.

[0048] Specifically, for a sign language word Indicates the jth character, n t Indicates the total number of characters, characters

[0049] The encoder will Mapped to the initial embedding vector Then use 1D convolution CONV c And maximum pooling MP[*] to obtain a fixed-length character-level word embedding vector e t , the formula is expressed as follows:

[0050]

[0051] The word-level encoder converts the sign language word gt Get the original word vector r through word embedding method t , then use 1D convolution CONV w Fusion character-level word embedding vector e t Get the final word vector The formula is as follows:

[0052]

[0053] Among them, [·] represents the vector concatenation operation;

[0054] The Transformer decoder consists of a 3-layer self-attention model (transformer). The decoder transforms the above word vector As input, the character [SOS] is used as the starting mark to gradually predict the final natural language sequence, and the character [EOS] is used as the end symbol.

[0055] Furthermore, the structure reparameterization module optimizes the sign language model obtained by the sign language model module to obtain an optimized sign language model for recognizing and translating sign language data. Specifically, the structure reparameterization module reduces the branch structure of the sign language model obtained by the sign language model module so that a multi-branch network is merged into a single branch. For the first type of branch, which is a combination of all n*m convolutional layers and batch normalization (BN) layers in the network, the structure reparameterization module merges the two layers into a new single n*m convolutional layer. The specific formula is as follows:

[0056]

[0057] Among them, W n,m is the weight coefficient of the n*m convolution kernel, b is the bias value, BN is the batch normalization layer, CONV is the convolution layer, γ and β are learnable parameters, are the mean and variance of the input x respectively;

[0058] Through the above merging, the original convolution layer and the batch normalization layer are merged into a new convolution layer. The convolution kernel of the new convolution layer is The new convolutional layer bias value is

[0059] For the second type of branch, it is the first branch and the second branch of the RepBlock module in the network, where the first branch is a combination of an n*n convolutional layer and a batch normalization layer. The convolutional layer and the batch normalization layer are merged into a new n*n convolutional layer through the above formula (1). The second branch is a combination of a 1*1 convolutional layer and a batch normalization layer. The convolutional layers are merged into a new 1*1 convolutional layer through the above formula (1), and the convolution kernel is padded with 0 to obtain an n*n convolutional layer. For the third type of branch, it is all single batch normalization layers in the network, which is equivalent to a combination of a 1*1 convolutional layer with a value of 1 and a batch normalization layer, and then merged and padded into a new n*n convolutional layer in the above way. The three branch structures in the RepBlock module are merged into one convolutional layer. The specific formula is as follows:

[0060] y=BN(W n (x)+b n )+BN(W 1 (x)+b 1 )+BN(x)

[0061] =W n′ (x)+W 1′ (x)+W 0′ (x)+b n′ +b 1′ +b 0′

[0062] =(W n′ +W 1′ +W 0′ )(x)+(b n′ +b 1′ +b 0′ )

[0063] Among them, W n′ 、W 1′ 、W 0′ is the convolutional layer parameter after merging the batch normalization layer, b n′ 、b 1′ 、b 0′ To merge the bias value parameters after batch normalization, the new convolution parameter after merging the branches is W n′ +W 1′ +W 0′ , the combined bias value is b n′ +b 1′ +b 0′ ;

[0064] The residual graph convolution modules are merged into a graph convolution layer through structural reparameterization. The formula is as follows:

[0065]

[0066] Where I is the identity matrix, x i is the eigenvector of the i-th node in the graph, x i ′ is the updated node feature vector, GCN(*) is the graph convolution function, θ1 and θ2 are graph convolution parameters; GCN * (*) is the new graph convolutional layer after merging, and N(i) represents the neighbor nodes of node i.

[0067] The present invention also provides a real-time sign language recognition and translation method based on an edge device. Based on the above system, the steps are as follows:

[0068] 1) Obtain sign language video data;

[0069] 2) Perform center cropping on each frame of the sign language video data and adjust the video frame size to a uniform size;

[0070] 3) Normalize each frame of the sign language video with a uniform size;

[0071] 4) Convert the normalized video frame from RGB three-channel image to grayscale image;

[0072] 5) A series of video frames are passed through the torso network to extract a 256-dimensional feature vector and the coordinates of the skeleton points (nose, left wrist, and right wrist);

[0073] 6) Crop the corresponding key areas (face area, left hand area, right hand area) based on the above skeleton points;

[0074] 7) Extracting the cropped facial region into a 256-dimensional vector through the facial network; extracting the cropped left-hand region into a 256-dimensional vector through the left-hand network; and extracting the cropped right-hand region into a 256-dimensional vector through the right-hand network;

[0075] 8) The obtained global frame vector and key area vector are constructed into a graph, and the sign language video features are obtained through graph convolution;

[0076] 9) Obtaining a sign language model based on the above sign language video feature training;

[0077] 10) Reducing the branch structure of the trained sign language model, merging all multi-branch structures in the model structure into a single branch, and obtaining an optimized sign language model;

[0078] 11) Based on the optimized sign language model, sign language recognition is performed, and translation is performed according to the recognized sign language sequence to obtain natural language text.

[0079] Beneficial effects of the present invention:

[0080] The present invention extracts key area features and global frame features of sign language, constructs these features into a region-aware spatiotemporal graph, uses a shallow graph convolutional network to extract sign language features, trains a sign language model based on the extracted features, optimizes the model using structural reparameterization technology, and then uses the optimized model to recognize sign language videos as word sequences and translate them into natural language text.

[0081] While maintaining the performance of sign language recognition and translation, this invention significantly reduces the computing power required by the model. This allows for a sign language model that is independent of high-performance graphics processing units (GPUs) and can be deployed on low-computing edge devices equipped only with a central processing unit (CPU). This enables real-time local translation of sign language, facilitating daily communication for the hearing-impaired and providing users with a more natural experience communicating with smart devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 This is a system structure diagram of the present invention.

[0083] Figure 2 Schematic diagram of the application scenario of the present invention.

[0084] Figure 3 Schematic diagram of the structure of the torso network, facial network, left-hand network and right-hand network in the present invention.

[0085] Figure 4 This is a schematic diagram of the structure of the regional shallow graph convolutional network of the present invention.

[0086] Figure 5 Schematic diagram of the structural reparameterization module of the present invention. DETAILED DESCRIPTION

[0087] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and drawings. The contents mentioned in the embodiments are not intended to limit the present invention.

[0088] Reference Figure 1-Figure 5 As shown, the present invention provides a real-time sign language recognition and translation system based on edge devices, including: a video data acquisition module, a video preprocessing module, a sign language model module and a structure reparameterization module; wherein,

[0089] Video data acquisition module, used to collect sign language video data;

[0090] Video preprocessing module, including: video frame size adjustment module, data normalization module and video frame grayscale module;

[0091] The video frame resizing module is used to scale the collected sign language video frames to a uniform size. The video frame resizing module includes: center cropping and resizing to a uniform size. The video resizing uses the reshape function of the open source computer vision library (opencv) to adjust the image to a uniform size.

[0092] The data normalization module is used to normalize the pixel values of the resized video frames from 0-255 to the range of 0-1; the data normalization module normalizes the pixel values of all video frames to 0-1 by dividing by 255.

[0093] The video frame grayscale module is used to convert the sign language video frame from an RGB three-channel image into a grayscale image; the video frame grayscale module uses the color conversion (cvtColor) function of the open source computer vision library (opencv) to convert the RGB image into a grayscale image.

[0094] The sign language model module is used to obtain a sign language model. The sign language model includes: a region representation sub-model, a graph convolution sub-model, and a text generation sub-model;

[0095] The region representation sub-model is used to obtain feature representations of key regions, including: a body network (BodyNet), a face network (FaceNet), a left hand network (LHandNet), and a right hand network (RHandNet);

[0096] The trunk network is used to extract skeleton points (nose, left wrist, and right wrist) from a sign language video frame, and extract the video frame into a 256-dimensional vector. The corresponding key areas (face area, left hand area, and right hand area) are cropped according to the skeleton points.

[0097] The facial network is used to extract the cropped facial region into a 256-dimensional vector;

[0098] The left-hand network is used to extract the cropped left-hand area into a 256-dimensional vector;

[0099] The right-hand network is used to extract the cropped right-hand area into a 256-dimensional vector;

[0100] Specifically, the region representation submodel adopts a modified RepVGG model to construct a trunk network, extracts skeleton points (three key points of nose, left wrist and right wrist) from a video frame, and extracts the video frame into a 256-dimensional vector. At the same time, the corresponding key areas (face area, left hand area and right hand area) are cropped according to the above skeleton points, and then the face network, left hand network and right hand network are constructed to extract the face area, left hand area and right hand area into 256-dimensional vectors respectively; specifically: the modified RepVGG model adopts the residual network (resnet18) as the basic architecture, replaces the residual structure with the RepBlock module, and adds the skeleton extraction branch network to the third stage (Stage 3) and the fourth stage (Stage 4) of the network to obtain the trunk network; the face network, left hand network and right hand network adopt the same structure as the trunk network, but do not contain the skeleton extraction branch network in the trunk network; wherein,

[0101] RepBlock: It uses three parallel branches. The first branch is an n*n convolution and batch normalization layer. The formula is as follows:

[0102] y=BN(w n (x)+b n )

[0103] Among them, x is the input vector, y is the output vector, and w n is the weight coefficient of the n*n convolution kernel, b n is the bias value, BN is the batch normalization layer, and the formula is as follows:

[0104]

[0105] Among them, γ and β are learnable parameters. are the mean and variance of the input x respectively;

[0106] The second branch is a 1*1 convolution and batch normalization layer, which is expressed as follows:

[0107] y=BN(w 1 (x)+b 1 )

[0108] The third branch is a single batch normalization layer;

[0109] Skeleton extraction branch network: It consists of two parallel deconvolutions, and the formula is as follows:

[0110] y=P1(D1(x))+P2(D2(x))

[0111] Among them, x is the input vector, y is the output vector, D1 and D2 are deconvolution layers, and P1 and P2 are pointwise convolutions; the skeleton extraction branch converts the x input vector into the y output vector, which is the corresponding skeleton heat map, and then extracts the coordinates of the skeleton points from the skeleton heat map. Then, with the extracted skeleton points as the center, the key areas are cropped to obtain the key images of the left hand, right hand, and facial areas.

[0112] Among them, the network structures of the face network, left-hand network, and right-hand network are similar to the main structure of the torso network. They are all modified RepVGG models, using the residual network (resnet18) as the basic architecture, replacing the residual structure with the RepBlock module, but excluding the skeleton extraction branch network.

[0113] The graph convolution sub-model is used to obtain feature representations of sign language videos, including: a region-aware temporal graph and a shallow graph convolutional network (S-GCN);

[0114] The region-aware spatiotemporal graph G is composed of nodes V and edges E, G = (V; E), where V is a node set, V = {v i,j ,i∈[1,N],j∈[1,4]}, where v i,j represents the jth node in the i-th frame, and N represents the number of input video frames. This region-aware spatiotemporal graph is used to provide the basic input for the subsequent graph network, representing the spatiotemporal relationship between the local and global features of sign language, and providing information transfer for the subsequent graph convolutional network to extract sign language features.

[0115] The shallow graph convolution network consists of four graph convolution modules and one fully connected layer. The graph convolution module is composed of a 1*1 convolution module, a residual graph convolution module (ResGraphConv), a 3*1 convolution module, and a residual graph convolution module (ResGraphConv). The shallow graph convolution module extracts gesture-related feature vectors based on the extracted hand features, facial features, global frame features, and regional perception spatiotemporal graph, and outputs feature vectors with a dimension of 1024.

[0116] Specifically, the graph convolution submodel includes a region-aware temporal graph and a shallow graph convolutional network (S-GCN); wherein the region-aware temporal graph G is composed of nodes V and edges E, that is, G = (V; E). For the i-th frame, four nodes are constructed, which are respectively composed of feature vectors corresponding to the global frame, facial region, left hand region and right hand region; the edge set E is composed of the intra-frame edge (inter-frame) set E aand the intra-frame edge set E e The edge set E in the frame a ={v ip v iq |i∈[1,N],(p,q)∈S},v ip v iq Represents the edges of different key regions in the same frame, S = {(1,2), (1,3), (1,4), (2,1), (3,1), (4,1)}, where nodes 1-4 represent the global frame, facial region, left hand region, and right hand region features respectively; the inter-frame edge set E e Indicated as E a ={v ip v jp |i,j∈[1,N],|ij|=1,p∈[1,4]},v ip v jp The edges represent the corresponding nodes in adjacent frames.

[0117] Among them, the shallow graph convolution network consists of 4 graph convolution modules and 1 fully connected layer. The 1*1 convolution module of the shallow graph convolution module consists of a RepBlock module, where the first branch of RepBlock is replaced by a 1*1 convolution kernel instead of an n*n convolution kernel; the 3*1 convolution module consists of a RepBlock module, where the first branch of RepBlock is replaced by a 3*1 convolution kernel instead of an n*n convolution kernel; the residual graph convolution module consists of a graph convolution plus a residual connection, and the formula is expressed as follows:

[0118]

[0119] Among them, x ij is the feature vector of the jth node in the i-th frame of the graph, x i ′ j is the updated node feature vector, GCN(*) is the graph convolution function, θ1 and θ2 are graph convolution parameters, and N(ij) represents the neighbor nodes of node ij.

[0120] The text generation sub-model is used to identify the feature representation of the sign language video as a sign language word sequence and translate the sign language word sequence into natural language text, including: a connectionist temporal classification decoder and a translation model;

[0121] The connectionist temporal classification decoder is used to decode the output features of the graph convolutional sub-model into corresponding sign language word sequences;

[0122] The translation model is used to translate the decoded sign language word sequence into natural language text;

[0123] Specifically, the connectionist temporal classification decoder is implemented using the open-source ctcdecode library. It takes the output of the graph convolutional sub-model as input, finds the path with the maximum probability sum based on the input, and uses it as the final decoding path to obtain the decoded sign language word sequence.

[0124] The translation model takes the sign language word sequence decoded by the connectionist temporal classification decoder as input and translates it into the corresponding natural language sequence. The model includes a character-level encoder, a word-level encoder, and a Transformer decoder. The character-level encoder and word-level encoder map the sign language words into high-dimensional vectors, and the Transformer decoder decodes the high-dimensional vectors as input to obtain natural language text.

[0125] Specifically, for a sign language word Indicates the jth character, n t Indicates the total number of characters, the character-level encoder will Mapped to the initial embedding vector j∈[1,n t ], then use 1D convolution CONV c And maximum pooling MP[*] to obtain a fixed-length character-level word embedding vector e t , the formula is expressed as follows:

[0126]

[0127] The word-level encoder converts the sign language word g t Get the original word vector r through word embedding method t , then use 1D convolution CONV w Fusion character-level word embedding vector e t Get the final word vector The formula is as follows:

[0128]

[0129] Among them, [·] represents the vector concatenation operation;

[0130] The Transformer decoder consists of a 3-layer self-attention model (transformer). The decoder transforms the above word vector As input, the character [SOS] is used as the starting mark to gradually predict the final natural language sequence, and the character [EOS] is used as the end symbol.

[0131] The structure reparameterization module is used to optimize the sign language model obtained by the sign language model module to obtain an optimized sign language model; and ensure that the output of the optimized model remains unchanged;

[0132] Specifically, the structure reparameterization module optimizes the sign language model obtained by the sign language model module to obtain an optimized sign language model for recognizing and translating sign language data; specifically, the structure reparameterization module reduces the branch structure of the sign language model obtained by the sign language model module so that the multi-branch network is merged into a single branch; for the first type of branch, which is a combination of all n*m convolutional layers and batch normalization (BN) layers in the network, the structure reparameterization module merges the two layers into a new single n*m convolutional layer. The specific formula is as follows:

[0133]

[0134] Among them, W n,m is the weight coefficient of the n*m convolution kernel, b is the bias value, BN is the batch normalization layer, CONV is the convolution layer, γ and β are learnable parameters, are the mean and variance of the input x respectively;

[0135] Through the above merging, the original convolution layer and the batch normalization layer are merged into a new convolution layer. The convolution kernel of the new convolution layer is The new convolutional layer bias value is

[0136] For the second type of branch, it is the first branch and the second branch of the RepBlock module in the network, where the first branch is a combination of an n*n convolutional layer and a batch normalization layer. The convolutional layer and the batch normalization layer are merged into a new n*n convolutional layer through the above formula (1). The second branch is a combination of a 1*1 convolutional layer and a batch normalization layer. The convolutional layers are merged into a new 1*1 convolutional layer through the above formula (1), and the convolution kernel is padded with 0 to obtain an n*n convolutional layer. For the third type of branch, it is all single batch normalization layers in the network, which is equivalent to a combination of a 1*1 convolutional layer with a value of 1 and a batch normalization layer, and then merged and padded into a new n*n convolutional layer in the above way. The three branch structures in the RepBlock module are merged into one convolutional layer. The specific formula is as follows:

[0137] y=BN(W n (x)+b n )+BN(W 1 (x)+b 1 )+BN(x)

[0138] =W n′ (x)+W 1′ (x)+W0′ (x)+b n′ +b 1′ +b 0′

[0139] =(W n′ +W 1′ +W 0′ )(x)+(b n′ +b 1′ +b 0′ )

[0140] Among them, W n′ 、W 1′ 、W 0′ is the convolutional layer parameter after merging the batch normalization layer, b n′ 、b 1′ 、b 0′ To merge the bias value parameters after batch normalization, the new convolution parameter after merging the branches is W n′ +W 1′ +W 0′ , the combined bias value is b n′ +b 1′ +b 0′ ;

[0141] The residual graph convolution modules are merged into a graph convolution layer through structural reparameterization. The formula is as follows:

[0142]

[0143] Where I is the identity matrix, x i is the eigenvector of the i-th node in the graph, x i ′ is the updated node feature vector, GCN(*) is the graph convolution function, θ1 and θ2 are graph convolution parameters; GCN * (*) is the new graph convolutional layer after merging, and N(i) represents the neighbor nodes of node i.

[0144] The present invention also provides a real-time sign language recognition and translation method based on an edge device. Based on the above system, the steps are as follows:

[0145] 1) Obtain sign language video data;

[0146] 2) Perform center cropping on each frame of the sign language video data and adjust the video frame size to a uniform size;

[0147] 3) Normalize each frame of the sign language video with a uniform size;

[0148] 4) Convert the normalized video frame from RGB three-channel image to grayscale image;

[0149] 5) A series of video frames are passed through the torso network to extract a 256-dimensional feature vector and the coordinates of the skeleton points (nose, left wrist, and right wrist);

[0150] 6) Crop the corresponding key areas (face area, left hand area, right hand area) based on the above skeleton points;

[0151] 7) Extracting the cropped facial region into a 256-dimensional vector through the facial network; extracting the cropped left-hand region into a 256-dimensional vector through the left-hand network; and extracting the cropped right-hand region into a 256-dimensional vector through the right-hand network;

[0152] 8) The obtained global frame vector and key area vector are constructed into a graph, and the sign language video features are obtained through graph convolution;

[0153] 9) Obtaining a sign language model based on the above sign language video feature training;

[0154] 10) Reducing the branch structure of the trained sign language model, merging all multi-branch structures in the model structure into a single branch, and obtaining an optimized sign language model;

[0155] 11) Based on the optimized sign language model, sign language recognition is performed, and translation is performed according to the recognized sign language sequence to obtain natural language text.

[0156] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principles of the present invention. These improvements should also be considered as the scope of protection of the present invention.

Claims

1. A real-time sign language recognition and translation system based on edge devices, characterized in that: include: Video data acquisition module, video preprocessing module, sign language model module and structure reparameterization module; Video data acquisition module, used to collect sign language video data; Video preprocessing module, including: video frame size adjustment module, data normalization module and video frame grayscale module; The video frame size adjustment module is used to scale the collected sign language video frames to a uniform size; The data normalization module is used to normalize the pixel values of the resized video frame from 0-255 to 0-1; The video frame grayscale module is used to convert the sign language video frame from an RGB three-channel image into a grayscale image; The sign language model module is used to obtain a sign language model. The sign language model includes: a region representation sub-model, a graph convolution sub-model, and a text generation sub-model; The region representation sub-model is used to obtain feature representations of key regions, including: a torso network, a face network, a left hand network, and a right hand network; The trunk network is used to extract skeleton points from a sign language video frame, extract the video frame into a 256-dimensional vector, and crop the corresponding key areas based on the skeleton points. The facial network is used to extract the cropped facial region into a 256-dimensional vector; The left-hand network is used to extract the cropped left-hand area into a 256-dimensional vector; The right-hand network is used to extract the cropped right-hand area into a 256-dimensional vector; The graph convolution sub-model is used to obtain feature representations of sign language videos, including: a region-aware spatiotemporal graph and a shallow graph convolutional network; The region-aware spatiotemporal graph G is composed of nodes V and edges E, G = (V; E), V is a node set, V = {v i,j ,i∈[1,N],j∈[1,4]}, where v i,j represents the jth node in the i-th frame, and N represents the number of input video frames. This region-aware spatiotemporal graph is used to provide the basic input for the subsequent graph network, representing the spatiotemporal relationship between the local and global features of sign language. The shallow graph convolution network consists of four graph convolution modules and one fully connected layer. The graph convolution module is composed of a 1*1 convolution module, a residual graph convolution module, a 3*1 convolution module, and a residual graph convolution module. The shallow graph convolution module extracts gesture-related feature vectors based on the extracted hand features, facial features, global frame features, and regional perception spatiotemporal graph, and outputs feature vectors with a dimension of 1024. The text generation sub-model is used to identify the feature representation of the sign language video as a sign language word sequence and translate the sign language word sequence into natural language text, including: a connectionist temporal classification decoder and a translation model; The connectionist temporal classification decoder is used to decode the output features of the shallow graph convolution module into a corresponding sign language word sequence; The translation model is used to translate the decoded sign language word sequence into natural language text; The structure reparameterization module is used to optimize the sign language model obtained by the sign language model module to obtain an optimized sign language model; and ensure that the output of the optimized model remains unchanged.

2. The real-time sign language recognition and translation system based on edge devices according to claim 1 is characterized in that: The video frame size adjustment module includes: center cropping and resizing to a uniform size. The video size adjustment uses the size adjustment function of the open source computer vision library to adjust the image to a uniform size.

3. The real-time sign language recognition and translation system based on edge devices according to claim 1 is characterized in that: The data normalization module normalizes the pixel values of all video frames to 0-1 by dividing by 255.

4. The real-time sign language recognition and translation system based on edge devices according to claim 1, characterized in that: The video frame grayscale module converts the RGB image into a grayscale image using the color conversion function of the open source computer vision library.

5. The real-time sign language recognition and translation system based on edge devices according to claim 1 is characterized in that: The region representation sub-model adopts the modified RepVGG model to construct the trunk network, extracts the skeleton points from a video frame, and extracts the video frame into a 256-dimensional vector. At the same time, the corresponding key areas are cropped according to the above skeleton points, and then the facial network, left-hand network, and right-hand network are constructed to extract the facial area, left-hand area, and right-hand area into 256-dimensional vectors respectively; specifically: the modified RepVGG model adopts the residual network as the basic architecture, replaces the residual structure with the RepBlock module, and adds the skeleton extraction branch network to the third and fourth stages of the network to obtain the trunk network; the facial network, left-hand network, and right-hand network adopt the same structure as the trunk network, but do not include the skeleton extraction branch network in the trunk network; wherein, RepBlock: It uses three parallel branches. The first branch is an n*n convolution and batch normalization layer. The formula is as follows: y=BN(w n (x)+b n ) Among them, x is the input vector, y is the output vector, and w n is the weight coefficient of the n*n convolution kernel, b n is the bias value, BN is the batch normalization layer, and the formula is as follows: Among them, γ and β are learnable parameters. are the mean and variance of the input x respectively; The second branch is a 1*1 convolution and batch normalization layer, which is expressed as follows: y=BN(w 1 (x)+b 1 ) The third branch is a single batch normalization layer; Skeleton extraction branch network: It consists of two parallel deconvolutions, and the formula is as follows: y=P1(D1(x))+P2(D2(x)) Among them, x is the input vector, y is the output vector, D1 and D2 are deconvolution layers, and P1 and P2 are point-by-point convolutions; the skeleton extraction branch converts the x input vector into the y output vector to obtain the corresponding skeleton heat map, and then extracts the coordinates of the skeleton points from the skeleton heat map. Then, with the extracted skeleton points as the center, the key areas are cropped to obtain the key images of the left hand, right hand, and facial areas.

6. The real-time sign language recognition and translation system based on edge devices according to claim 1, characterized in that: The graph convolution submodel includes a region-aware spatiotemporal graph and a shallow graph convolution network; the region-aware spatiotemporal graph G is composed of nodes V and edges E, G = (V; E), for the i-th frame, four nodes are constructed, which are respectively composed of feature vectors corresponding to the global frame, facial region, left hand region and right hand region; the edge set E is composed of the intra-frame edge set E a and the inter-frame edge set E e The edge set E in the frame a ={v ip v iq |i∈[1,N],(p,q)∈S},v ip v iq Represents the edges of different key regions in the same frame, S = {(1,2), (1,3), (1,4), (2,1), (3,1), (4,1)}, where nodes 1-4 represent the global frame, facial region, left hand region, and right hand region features respectively; the inter-frame edge set E e Indicated as E a ={v ip v jp |i,j∈[1,N],|ij|=1,p∈[1,4]},v ip v jp The edges represent the corresponding nodes in adjacent frames.

7. The real-time sign language recognition and translation system based on edge devices according to claim 6, characterized in that: The shallow graph convolution network consists of 4 graph convolution modules and 1 fully connected layer. The 1*1 convolution module of the shallow graph convolution module consists of a RepBlock module, where the first branch of RepBlock is replaced by a 1*1 convolution kernel instead of an n*n convolution kernel; the 3*1 convolution module consists of a RepBlock module; the first branch of RepBlock is replaced by a 3*1 convolution kernel instead of an n*n convolution kernel; the residual graph convolution module consists of a graph convolution plus a residual connection, and the formula is expressed as follows: Among them, x ij is the feature vector of the jth node in the i-th frame of the graph, x i ′ j is the updated node feature vector, GCN(*) is the graph convolution function, θ1 and θ2 are graph convolution parameters, and N(ij) represents the neighbor nodes of node ij.

8. The real-time sign language recognition and translation system based on edge devices according to claim 1, characterized in that: The connectionist temporal classification decoder is implemented using the open-source ctcdecode library. It takes the output of the graph convolutional sub-model as input, finds the path with the maximum probability sum based on the input, and uses it as the final decoding path to obtain the decoded sign language word sequence. The translation model takes the sign language word sequence decoded by the connectionist temporal classification decoder as input and translates it into the corresponding natural language sequence. It includes a character-level encoder, a word-level encoder, and a Transformer decoder. The character-level encoder and the word-level encoder map the sign language words into high-dimensional vectors, and the Transformer decoder decodes the high-dimensional vectors as input to obtain natural language text. Specifically, for a sign language word Indicates the jth character, n t Indicates the total number of characters, the character-level encoder will Mapped to the initial embedding vector Then use 1D convolution CONV c And maximum pooling MP[*] to obtain a fixed-length character-level word embedding vector e t , the formula is expressed as follows: The word-level encoder converts the sign language word g t Get the original word vector r through word embedding method t , then use 1D convolution CONV w Fusion character-level word embedding vector e t Get the final word vector The formula is as follows: Among them, [·] represents the vector concatenation operation; The Transformer decoder consists of a three-layer self-attention model. The decoder transforms the above word vectors As input, the character [SOS] is used as the starting mark to gradually predict the final natural language sequence, and the character [EOS] is used as the end symbol.

9. The real-time sign language recognition and translation system based on edge devices according to claim 1, characterized in that: The structure reparameterization module optimizes the sign language model obtained by the sign language model module to obtain an optimized sign language model for recognizing and translating sign language data. Specifically, the structure reparameterization module reduces the branch structure of the sign language model obtained by the sign language model module so that a multi-branch network is merged into a single branch. For the first type of branch, which is a combination of all n*m convolutional layers and a batch normalization layer in the network, the structure reparameterization module merges the two layers into a new single n*m convolutional layer. The specific formula is as follows: Among them, W n,m is the weight coefficient of the n*m convolution kernel, b is the bias value, BN is the batch normalization layer, CONV is the convolution layer, γ and β are learnable parameters, are the mean and variance of the input x respectively; Through the above merging, the original convolution layer and the batch normalization layer are merged into a new convolution layer. The convolution kernel of the new convolution layer is The new convolutional layer bias value is For the second type of branch, it is the first branch and the second branch of the RepBlock module in the network, where the first branch is a combination of an n*n convolutional layer and a batch normalization layer. The convolutional layer and the batch normalization layer are merged into a new n*n convolutional layer through the above formula (1). The second branch is a combination of a 1*1 convolutional layer and a batch normalization layer. The convolutional layers are merged into a new 1*1 convolutional layer through the above formula (1), and the convolution kernel is padded with 0 to obtain an n*n convolutional layer. For the third type of branch, it is all single batch normalization layers in the network, which is equivalent to a combination of a 1*1 convolutional layer with a value of 1 and a batch normalization layer, and then merged and padded into a new n*n convolutional layer in the above way. The three branch structures in the RepBlock module are merged into one convolutional layer. The specific formula is as follows: y=BN(W n (x)+b n )+BN(W 1 (x)+b 1 )+BN(x) =W n′ (x)+W 1′ (x)+W 0′ (x)+b n′ +b 1′ +b 0′ =(W n′ +W 1′ +W 0′ )(x)+(b n′ +b 1′ +b 0′ ) Among them, W n′ 、W 1′ 、W 0′ is the convolutional layer parameter after merging the batch normalization layer, b n′ 、b 1′ 、b 0′ To merge the bias value parameters after batch normalization, the new convolution parameter after merging the branches is W n′ +W 1′ +W 0′ , the combined bias value is b n′ +b 1′ +b 0′ ; The residual graph convolution modules are merged into a graph convolution layer through structural reparameterization. The formula is as follows: Where I is the identity matrix, x i is the eigenvector of the i-th node in the graph, x i ′ is the updated node feature vector, GCN(*) is the graph convolution function, θ1 and θ2 are graph convolution parameters; GCN * (*) is the new graph convolutional layer after merging, and N(i) represents the neighbor nodes of node i.

10. A real-time sign language recognition and translation method based on an edge device, based on the system according to any one of claims 1 to 9, characterized in that: Here are the steps: 1) Obtain sign language video data; 2) Perform center cropping on each frame of the sign language video data and adjust the video frame size to a uniform size; 3) Normalize each frame of the sign language video with a uniform size; 4) Convert the normalized video frame from RGB three-channel image to grayscale image; 5) A series of video frames are passed through the trunk network to extract a 256-dimensional feature vector and the coordinates of the skeleton points; 6) Cut out the corresponding key areas according to the above skeleton points; 7) Extracting the cropped facial region into a 256-dimensional vector through the facial network; extracting the cropped left-hand region into a 256-dimensional vector through the left-hand network; and extracting the cropped right-hand region into a 256-dimensional vector through the right-hand network; 8) The obtained global frame vector and key area vector are constructed into a graph, and the sign language video features are obtained through graph convolution; 9) Obtaining a sign language model based on the above sign language video feature training; 10) Reducing the branch structure of the trained sign language model, merging all multi-branch structures in the model structure into a single branch, and obtaining an optimized sign language model; 11) Based on the optimized sign language model, sign language recognition is performed, and translation is performed according to the recognized sign language sequence to obtain natural language text.

Images