Sign language real-time translation system based on multi-dimensional data fusion

Abstract

The invention relates to the field of sign language translation. The invention discloses a sign language real-time translation system based on multi-dimensional data fusion. The sign language real-time translation system comprises an image information acquisition module, a data processing module and a result display module, the image information acquisition module captures sign language actions of a user in real time through a camera to generate sign language videos, and operates four core task architectures including an HTTP server task, a camera acquisition task, a streaming media transmission task and a video downloading task on the basis of FreeRTOS, the HTTP server task receives a client request, and the video downloading task receives the client request. Comprising an MJPEG video stream, a JPEG single frame capture and a video downloading access endpoint, a built-in CORS cross-domain access support, a camera acquisition task continuously captures JPEG frames with a fixed frame rate, each task is connected with a queue transfer client through semaphore protection frame data to realize efficient collaboration, multi-task parallel processing is realized, the transmission frame rate is accurately controlled to meet the algorithm requirement, and the algorithm requirement is met. And ESP32 dual-core resources are utilized to the maximum extent.

Description

Technical Field

[0001] This invention relates to the field of sign language translation. Background Technology

[0002] Sign language translation is a technology / service that enables communication by converting spoken or written language into sign language gestures, or vice versa. Current sign language translation technology faces multiple bottlenecks in data acquisition, algorithms, and practical application. In terms of data acquisition, single cameras are prone to losing crucial information due to limited field of view. At the algorithm level, traditional keyframe sampling misses frames with abrupt changes in motion, and the inability to perform multi-dimensional data fusion leads to low recognition accuracy. In terms of practical application, existing solutions suffer from high latency, high cost, or low recognition rates, and lack multi-dimensional standardized datasets, making it difficult to meet the needs of the hearing impaired and users in various gesture-based interaction scenarios. Summary of the Invention

[0003] This invention aims to provide a sign language translation system based on multidimensional data fusion, which solves the problem of low accuracy in existing sign language translation by innovatively integrating three modal features: RGB video, depth information, and skeleton data.

[0004] The technical problem to be solved by this invention is: a real-time sign language translation system based on multi-dimensional data fusion, including an image information acquisition module, a data processing module, and a result display module; the image information acquisition module captures user sign language movements in real time through a camera to generate sign language videos, and runs a four-core task architecture based on FreeRTOS: an HTTP server task, a camera acquisition task, a streaming media transmission task, and a video download task. The HTTP server task receives client requests, including MJPEG video streams, JPEG single-frame capture, and video download access endpoints, and has built-in CORS cross-domain access support. The camera acquisition task continuously captures JPEG frames at a fixed frame rate. The streaming media transmission task pushes MJPEG video streams in real time. The video download task encapsulates the video into an AVI format file for client download. The AVI format contains a complete file header, stream header, and index block structure. In terms of real-time transmission, each task protects frame data with semaphores and transmits client connection data through queues. The system achieves high efficiency through multi-task parallel processing, precisely controlling the transmission frame rate to meet algorithm requirements and maximizing the utilization of the ESP32 dual-core resources. The data processing module includes a multi-dimensional sign language recognition model, DWSTST-MSD-Net, used to perform sign language recognition on video images acquired by the image acquisition module. The result display module converts the sign language recognition results into readable text and displays them. The built-in sign language recognition database stores database entries that map sign language gestures to translated text. When the multi-dimensional sign language recognition model DWSTST-MSD-Net outputs the recognition result, it first performs similarity matching between the sign language gesture corresponding to the recognition result and the database entry, generating the corresponding sign language vocabulary sequence and encapsulating it. Then, it transmits the data encrypted through the wireless communication module. After receiving the data, the display module first parses and extracts the sign language vocabulary sequence, performs text normalization processing, applies Named Entity Recognition (NER) technology to generate the target language text, and finally renders the target language text onto the display screen.

[0005] The image information acquisition module includes at least a color camera, a depth camera, and a human skeleton detection camera. The acquired data includes RGB video acquired by the color camera, depth video acquired by the depth camera, and 3D skeleton coordinate time-series data acquired by the human skeleton detection camera.

[0006] The multidimensional sign language recognition model DWSTST-MSD-Net includes a data preprocessing module, a multi-scale sign language recognition network MSD-Net, a dynamically weighted spatiotemporal graph convolutional network DWTST-GCN, and a Softmax classifier.

[0007] In the data preprocessing module, histogram equalization technology is used to enhance the image of the acquired RGB video to improve contrast. The processing can be represented by the following formula:

[0008]

[0009] In the formula, H(i,j) is the pixel value at the pixel coordinate (i,j) after histogram equalization, where i is the x-coordinate of the pixel coordinate, j is the y-coordinate of the pixel coordinate, and g is the y-coordinate of the pixel coordinate. i,j This describes the pixel value at pixel coordinates (i,j) in the input image, where I represents the total number of gray levels, typically ranging from 0 to 255 for an 8-bit image (I = 256), and n is an integer from 0 to I-1. n This represents the probability of a pixel with grayscale value n appearing in the input image. For the RGB video processed by histogram equalization, a keyframe selection strategy is used to remove redundant information. An initial index S is randomly selected from the RGB video, S = rand(0, N-16), and 16 consecutive frames are extracted as keyframes, where N is the total number of frames in the sign language video. While preserving the integrity of the actions, the input length is standardized. In the data preprocessing module, when processing the acquired depth video, since the depth video and RGB video are time-synchronized, the 16 keyframe indices of the RGB video are directly reused to extract the corresponding depth frames, ensuring that the multi-dimensional data are strictly aligned in the "16-frame sequence" dimension, thus achieving depth video preprocessing. In the data preprocessing module, when preprocessing the acquired 3D skeleton coordinate time-series data, a method combining filtering and time-series normalization is used. First, noise is removed through Gaussian filtering; the specific formula is as follows.

[0010]

[0011] Where, x t Let G(x) be the spatial coordinates of the skeleton keypoints at time t, k be the filtering window that determines the neighborhood range involved in the calculation, τ be the time offset, σ be the pixel standard deviation, and G(x) be the coordinates of the skeleton keypoints at time t. t The data is Gaussian filtered, and the filtered coordinate sequence is then normalized to a length of 16 frames using linear interpolation. The specific process can be performed using the following formula.

[0012]

[0013] Where s is the sampling interval of the original sequence, ia is the index of the new sequence, ja is the adjacent index of the original sequence, and x new,ia x represents the normalized keypoint coordinates of frame ia. old,ja This represents the coordinates of the keypoint corresponding to the ja-th frame in the original sequence.

[0014] The multi-scale sign language recognition network MSD-Net includes a spatial transformation network STN modified with ResNet50, a dynamic multi-scale attention module D-EMA, and a Transformer network. The spatial transformation network STN comprises a localization network, a mesh generator, and a sampler. The localization network calculates the affine transformation parameters of the input image. The input feature map U is processed by a convolutional neural network f. loc The formula for learning the transformation parameter θ is as follows:

[0015] θ=f loc (U)

[0016] Where θ is a 2×3 affine transformation matrix: And each of the parameters was initially...

[0017] The parameters are obtained through random initialization (such as Xavier initialization), and the optimal parameters are learned through subsequent training. Where θ 11 θ 12 θ 21 θ 22 Control rotation, scaling, and shearing, θ 13 θ 23 Control the translation.

[0018] Mesh generator: Its function is to generate a coordinate grid of the target feature map and calculate the mapping relationship between these coordinates and the original image coordinates, that is, to calculate the input image coordinates through the affine transformation matrix θ. The formula is as follows:

[0019]

[0020] in, These are the coordinates of a pixel in the source network (input). These are the coordinates of the corresponding pixel in the target network (output). Since the affine transformation matrix is ​​a 2×3 matrix, therefore... Transform it into a 3×1 matrix while maintaining the correct format.

[0021] Sampler: Its function is to use bilinear interpolation to sample pixel values ​​from the original image based on the transformed coordinates, generating a target feature map, that is, for each output pixel V i Using bilinear interpolation, the formula is as follows:

[0022]

[0023] Among them, (χ n ,y n ) represents the pixel coordinates of the original image U. V represents the sampling coordinates calculated from the affine transformation. iU represents the value of the i-th pixel in the output feature map. n This represents the value of the nth pixel in the input feature map;

[0024] ResNet50 employs two Bottleneck structures: Bottleneck1 and Bottleneck2. The core difference lies in the adjustment of the number of channels and the presence or absence of downsampling operations, adapting to the differentiated feature extraction needs of each layer of the network. ResNet50 consists of five convolutional blocks and one initial convolutional layer. The number and specific structure of the Bottleneck residual modules included in different stages vary. Detailed configurations are as follows: Stage-1: Output size [16, 256, 32, 32], composed of 1 Bottleneck1 module and 2 Bottleneck2 modules, initially extracting basic features through different module combinations; Stage-2: Output size [16, 512, 16, 16], containing 1 Bottleneck1 module and 3 Bottleneck2 modules. In feature extraction, the spatial dimension is gradually compressed and the channel dimension is expanded. Stage-3: The output size is [16, 1024, 8, 8], configured with 1 Bottleneck1 module and 5 Bottleneck2 modules to further deepen the feature expression and capture more complex patterns. Stage-4: The output size is [16, 2048, 4, 4], composed of 1 Bottleneck1 module and 2 Bottleneck2 modules, to complete the final construction of high-dimensional features. The standard Bottleneck2 module is used for residual blocks other than the first module of each stage. The backbone network of this module consists of 1×1 dimensionality reduction convolution, 3×3 main feature extraction convolution and 1×1 dimensionality increase convolution. The 1×1 dimensionality reduction convolution is used to reduce the computation of the network, and the 1×1 dimensionality increase convolution is used to restore the number of channels, so as to fully extract feature information and improve the computational efficiency of the network. Residual connections directly pass the input to the output and perform element-wise addition along the channel dimension to keep the size of the input data constant, making it convenient for the network to be superimposed for the next step of deep learning;

[0025] The Dynamic Multi-Scale Attention Module (D-EMA) is used to address the problems of multi-scale information aggregation and cross-space learning, for the input feature map X∈R. C×H×W Where C is the number of channels, H is the feature map height, and W is the feature map width, the D-EMA module first segments the feature map X along the channel dimension, dividing it into G sub-feature groups to learn the feature information of different channels. The grouped feature map is represented as X = [X0, X1, ..., X...]. G-1, where each sub - feature group sets the value of G to G << C, that is, the input feature map is divided into G sub - feature groups in the channel direction, and the number of channels in each group is much less than the total number of channels, ensuring that each sub - feature group still has multiple channels, so as to assist the D - EMA module to learn different features in different sub - feature groups, thereby enhancing the feature expression ability;

[0026] In the determination of G, a strategy for dynamically adjusting the number of groups according to the number of training epochs is defined to enhance the adaptability of the D - EMA module to multi - scale features in different training stages. Compared with the method of fixed number of groups, since the model's focus on features varies in different training stages, dynamically adjusting the number of groups can better adapt to the learning requirements of global and local information in the training process; G changes dynamically according to the current number of training epochs epoch: in the early stage of training, a smaller value of G is adopted to capture a larger range of global information; in the later stage of training, G is gradually increased to focus on more in - depth local features. Its adjustment rule can be expressed by the following formula: Among them, by using different values of G to correspond to the number of groups required in specific training stages respectively, in the early stage of training, a smaller number of groups is used for learning global information; in the later stage of training, a larger number of groups is used for capturing in - depth local features in the later stage of training. This dynamic adjustment strategy effectively helps the D - EMA module to adaptively adjust in different training stages, contributing to enhancing the feature expression ability of the model. Relying on the characteristic of the neural network that "different local receptive fields can enable neurons to capture multi - scale features", D - EMA uses three parallel paths to extract the grouped feature matrix, specifically: two branches are in the 1×1 branch, and the remaining one branch is in the 3×3 branch. In the 1×1 branch, the channels are encoded bidirectionally; in the 3×3 branch, 3×3 convolutional kernels are stacked to capture multi - scale feature representations;

[0027] After completing pooling, the D - EMA module applies the Softmax function to the pooling output to optimize the calculation efficiency, and then through a dot - product operation, multiplies the outputs of parallel processing to generate the first spatial attention map that aggregates multi - scale spatial information. Similarly, 2D global average pooling is used for the large - scale feature information of the 3×3 branch, and the 1×1 branch is adjusted to the corresponding dimension shape. Through a dot - product operation, the outputs of the above - mentioned parallel processing are multiplied to obtain the second spatial attention map. Finally, the two groups of feature maps are fused, and through the Sigmoid function, multi - scale feature information aggregation is completed;

[0028] The Dynamically Weighted Spatiotemporal Graph Convolutional Network (DWTST-GCN) is used to process skeleton information, specifically, to process pre-processed 3D skeleton coordinate temporal data. DWTST-GCN includes a Dynamic Weighting Module (DWM) and a Temporally Aware Module (DCSpatialDropout layer). The DC temporally aware module consists of two different convolutional layers and a sigmoid function. The dilated convolutions in the network expand the model's receptive field while maintaining computational efficiency, acquiring more comprehensive temporal information. Batch normalization is performed after each convolutional layer to accelerate training, and the structure of convolutions and ReLU activation functions enhances the model's non-linear expressive power. In addition to combining dilated and causal convolutions into a dilated causal convolution, the network also uses a dilated non-causal convolution in the second convolutional layer. Conv allows this layer to be unconstrained by causal relationships, ensuring that the model can rely on past and future inputs, capture the global contextual information of the complete input as much as possible, and ensure the extraction of complete sign language features. In the DC time-aware module, a SpatialDropout layer is used instead of a traditional Dropout layer. Compared to a regular Dropout layer, SpatialDropout can randomly drop features across each channel of the feature matrix, making it more flexible than dropping entire neurons. This effectively maintains the complete spatial structure of the feature matrix, preventing the destruction of correlations between spatial features and preserving feature continuity. SpatialDropout is more suitable for processing continuous graph data. The Sigmoid function calculates attention weights based on the output feature matrix of the network and performs a weighted summation operation on the input features, ultimately yielding... The output is enhanced with attention weights; the DC time-aware module ensures that causal relationships in the time series of sign language recognition are not lost during network training, and also retains some global contextual information, which can effectively establish distance dependencies in sign language recognition features; the DWM module can introduce temporal weight parameters in the generation of the adjacency matrix. For skeleton keypoints, not only their spatial distance is considered, but also temporal weight parameters are introduced to form a spatiotemporally dynamic weighted adjacency matrix with both spatial and temporal representations. At the same time, nodes with different hop counts are classified and assigned different weights. Together with the temporal weights, the adjacency matrix is ​​dynamically adjusted to effectively capture the spatiotemporal features in the skeleton information and further improve the model's performance. Finally, the adjacency matrices with different hop counts are stacked together to form a multi-scale adjacency matrix, which is input into the network built in this invention for graph convolution operations. The specific generation strategy of the adjacency matrix is ​​as follows:

[0029] For each valid hop count (hop), generate three adjacency matrices: a root: Represents the adjacency relationship from the central node to other nodes, mainly capturing the direct connection between a node and its central node; a close : Represents the adjacency relationship close to the central node, capturing the relationships between nodes that are relatively close to the central node. further : Represents the adjacency relationships far from the central node, capturing the relationships between nodes that are relatively far from the central node. The formula for generating the initial adjacency matrix is ​​as follows:

[0030]

[0031] Among them, A ij This represents the adjacency matrix elements between node i and node j, `normalize_adjacency` represents the normalized adjacency matrix values ​​between node i and node j, and `hop_dis`... ji This represents the hop distance between node j and node i, where hop represents the current effective hop count. Then, based on the time weight parameter hop_dis... ji Adjust the adjacency matrix:

[0032]

[0033] Here, `time_weight` is the time weight parameter, used to balance the weights of different types of connections. Finally, these adjacency matrices generated with different hop counts are stacked into the final adjacency matrix, which is used for graph convolution operations.

[0034] After fusing multidimensional information, it is input into the Softmax classifier. Depth video, skeleton keypoint information, and RGB video information complement each other: depth information focuses on spatial distance relationships and target contours, while skeleton information emphasizes the motion trajectories and connectivity of keypoints in three-dimensional space. Both provide recognition features that are difficult to represent in RGB video. The multidimensional fusion algorithm is based on dynamic adaptive weighting of feature layers. By integrating complementary information from RGB, depth, and skeleton modalities, and utilizing convolution and pooling to optimize feature representation, it improves the accuracy of multidimensional sign language recognition. First, a separate training strategy is used, training the MSD-Net network separately. Features F1 and F2 were obtained from RGB video data and depth data, with a size of [16, 256]. Simultaneously, the skeleton data was trained using the DWTST-GCN network to obtain skeleton feature F3, also with a size of [16, 256]. These networks focus on extracting spatiotemporal features from different modalities, ensuring that features for each data modality are learned independently and effectively. After training, the softmax layer of each network was removed, and the output was adjusted to the specified feature size. Then, corresponding weight parameters W1, W2, and W3 were assigned to each feature F1, F2, and F3 to obtain the weighted features.

[0035] F′1=W1·F1 F′2=W2·F2 F′3=W3·F3

[0036] After weighting, the weighted features are concatenated along the channel dimension to obtain a tensor of size [16, 768].

[0037] F concat =concat(F′1,F′2,F′3)

[0038] Tensors are processed through convolutional layers to extract higher-level features, followed by overlapping max pooling for downsampling. This pooling window overlaps with each other, preserving more local information and reducing information loss. ReLU activation and batch normalization are then applied to obtain the final fused features, which are finally input into a Softmax classifier to complete the sign language recognition task.

[0039] The D-EMA module establishes interdependencies between channels and spatial locations to enrich the effect of feature aggregation. For 1×1 branches, 2D global average pooling is used to encode the output, capturing global spatial information, as shown in the following formula:

[0040]

[0041] For a 3×3 branch, its output is directly adjusted to the corresponding dimension shape, and then processed accordingly to facilitate subsequent feature union; where H represents the height of the input feature map, i.e., the number of pixels or units in the vertical direction of the feature map, and this invention uses H=128; W represents the width of the input feature map, i.e., the number of pixels or units in the horizontal direction of the feature map, and this invention uses W=128; x C (i, j) represents the feature value of the c-th channel in the input feature map at pixel coordinates (i, j), Z C This represents the output result after 2D global averaging.

[0042] The beneficial effects of this invention are: this invention realizes fully automated processing from recognition results to visualization output, ensuring the accuracy, real-time performance and ease of use of the translation results. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the real-time streaming screen connection for the image information acquisition module.

[0044] Figure 2 This is a schematic diagram illustrating the AVI file download workflow of the image information acquisition module.

[0045] Figure 3 Here is a diagram of the MSD-Net sign language recognition network structure.

[0046] Figure 4 This is a schematic diagram illustrating the process of generating the adjacency matrix.

[0047] Figure 5 The structure is DWTST-GCN.

[0048] Figure 6 The model structure is DWTST-MSD-Net;

[0049] Figure 7 This is a flowchart illustrating the workflow of the results display module. Detailed Implementation

[0050] like Figure 1-7 As shown, this invention proposes a sign language translation system based on multi-dimensional data fusion. The system adopts a three-level processing architecture design, namely an image information acquisition module, a data processing module, and a result display module (the modules communicate with each other through a data bus).

[0051] The image information acquisition module captures user sign language gestures in real time using a camera installed in the mobile device. First, the system checks the pin connections, clock frequency, and pixel format parameters for correctness and initializes the system camera to ensure acquisition quality. Then, a stable WiFi connection is established (currently only WiFi is supported), and the system is set to STA mode to access the designated network, providing a network foundation for subsequent services. The built-in high-performance OV2640 camera captures user sign language video input in real time and runs a multi-task architecture based on FreeRTOS (an existing, mature, small real-time operating system kernel designed for embedded systems and widely used in microcontrollers (MCUs) and Internet of Things (IoT) devices). This architecture consists of four core tasks: an HTTP server task, a camera acquisition task, a streaming media transmission task, and a video download task. The HTTP server task receives client requests, including various access endpoints such as MJPEG video streams, JPEG single-frame capture, and video download. Built-in CORS cross-domain access support (the core mechanism for browsers to handle cross-domain requests, a standardized Web technology) greatly improves the system's usability and compatibility. The camera capture task continuously captures JPEG frames at a fixed frame rate, while the streaming media transmission task pushes MJPEG video streams in real time. The video download task encapsulates the video stream into an AVI file and transmits it, or directly extracts frames from the video stream on the server side using FFmpeg (an open-source, cross-platform multimedia processing framework for recording, converting, and streaming audio and video content, allowing for custom transmission durations and offering greater compatibility) to download MP4 files. For real-time transmission, each task achieves efficient collaboration through semaphore protection of frame data (one of the core synchronization mechanisms in operating systems and embedded systems; in FreeRTOS, semaphores are widely used to solve resource contention and task coordination problems in multi-tasking environments) and queue-based client connection passing. Through multi-task parallel processing, the transmission frame rate is precisely controlled to meet algorithm requirements, maximizing the utilization of the ESP32 dual-core resources. A PSRAM-first dynamic memory allocation strategy is adopted. To meet different application scenarios, the system innovatively supports encapsulating the captured video stream into a standard AVI format for client download. The AVI format includes a complete file header, stream header, and index block structure. In terms of energy efficiency management, the system implements an intelligent task suspension mechanism (tasks are only awakened when a client request is detected, and suspended when there is no request, reducing CPU idling and saving power) and dynamic memory adjustment (PSRAM priority), which significantly reduces power consumption. The entire system achieves efficient resource utilization and stable long-term operation while ensuring real-time performance.

[0052] The data processing module is used to translate the video recognition video acquired by the image information acquisition module, specifically covering two major sections: model training and real-time data processing. This part is also the core technology of this patent. The first section, model construction and training, is carried out according to the following steps:

[0053] Step 1: Dataset Creation and Preprocessing. This invention constructs a multidimensional sign language dataset. RGB video was recorded using industrial cameras at 45° angles on both sides, and data was collected using a Microsoft Kinect V2 depth camera (with built-in color camera, depth camera, and human skeleton detection function) deployed in the front. The data collection process involved 20 deaf volunteers with varying individual differences. Under the guidance of a professional sign language teacher, following standardized movements, and in a strictly controlled environment with a green wall background, 60 isolated sign language words and 20 sentences were demonstrated five times each. This resulted in a dataset containing RGB video, depth video, and 3D skeleton coordinate time-series data, providing multidimensional data support for subsequent model training.

[0054] In the dataset preprocessing stage, for RGB video image data, this invention uses histogram equalization technology to enhance image contrast, laying the foundation for the accuracy of subsequent model training. The processing can be represented by the following formula:

[0055]

[0056] Where i and j represent the position coordinates of pixels in the image, i represents the row index of the pixel, j represents the column index of the pixel, and g i,j H(i,j) represents the pixel value at position (i,j) in the input image, and H(i,j) represents the pixel value at position (i,j) in the image after histogram equalization; round() represents the rounding function; I represents the number of intensity values, usually 256; g refers to the original input image; P nThe intensity value n represents the probability of a pixel with an intensity value of n in the image, i.e., the normalized histogram value; n takes values ​​from 0, 1...I-1. After enhancement, a fixed-length keyframe extraction strategy is adopted: an initial index S is randomly selected from the video, S = rand(0, N-16), where N is the total number of frames in the sign language video, and 16 consecutive frames are extracted as keyframes. This method preserves the integrity of the action while standardizing the input length to meet the model training requirements. For depth video image data, this invention performs adaptive histogram equalization on the depth video through synchronous enhancement and keyframe alignment to optimize the contrast of local depth details. Since the depth video and RGB video are time-synchronized, the 16 keyframe indices of the RGB video are directly reused to extract the depth frames at the corresponding time, ensuring that the multi-dimensional data are strictly aligned in the "16-frame sequence" dimension, avoiding feature fusion errors caused by spatiotemporal misalignment. For 3D skeleton coordinate temporal data, this invention adopts a method combining filtering and temporal normalization. First, Gaussian filtering is used to remove noise, and the processing can be expressed by the following formula:

[0057]

[0058] Where, x t Let be the spatial coordinates of the skeleton keypoints at time t, k be the filtering window that determines the neighborhood range involved in the calculation, τ be the time offset, and σ be the pixel standard deviation. The filtered coordinate sequence is then normalized to a length of 16 frames using linear interpolation. The specific process can be expressed by the following formula:

[0059]

[0060] Where s is the sampling interval of the original sequence, ia is the index of the new sequence, ja is the adjacent index of the original sequence, and x new,ia x represents the normalized keypoint coordinates of frame ia. old,ja This represents the coordinates of the keypoint corresponding to the ja-th frame in the original sequence; ensuring that the skeletal data matches the length of the video keyframes. The advantage of dataset preprocessing lies in achieving strong alignment of multi-dimensional data in temporal length through the "16-frame keyframe" strategy. Combined with enhancement, filtering, and other operations, it provides high-quality, standardized input for model training, adapting to the dual requirements of sign language recognition for both action integrity and computational efficiency.

[0061] Step two involves constructing a Multi-Scale Dynamic Sign Language Recognition Network (MSD-Net) and processing the RGB and depth videos. The MSD-Net network comprises a Spatial Transformer Networks (STN) modified ResNet50 network, a Dynamic Efficient Multi-Scale Attention Module (D-EMA), and a Transformer network.

[0062] First, the STN (Spatial Transformation Network) is used to address the shortcomings of traditional convolutional neural networks in spatial transformation. STN not only achieves translation, scaling, and rotation invariance but can also be embedded within convolutional neural networks to transform and align data, thereby improving classification accuracy. The STN structure consists of three parts: a localization network, a mesh generator, and a sampler. The localization network calculates the affine transformation parameters of the input image. Specifically, the input feature map U is processed by a convolutional neural network f. loc The formula for learning the transformation parameter θ is as follows:

[0063] θ=f loc (U)

[0064] Where θ is a 2×3 affine transformation matrix: The parameters are initially randomly initialized (e.g., Xavier initialization), and the optimal parameters are learned through subsequent training. Among them, θ... 11 θ 12 θ 21 θ 22 Control rotation, scaling, and shearing, θ 13 θ 23 Control the translation.

[0065] Mesh generator: Its function is to generate a coordinate grid of the target feature map and calculate the mapping relationship between these coordinates and the original image coordinates, that is, to calculate the input image coordinates through the affine transformation matrix θ. The formula is as follows:

[0066]

[0067] in, These are the coordinates of a pixel in the source network (input). These are the coordinates of the corresponding pixel in the target network (output). Since the affine transformation matrix is ​​a 2×3 matrix, therefore... Transform it into a 3×1 matrix while maintaining the correct format.

[0068] Sampler: Its function is to use bilinear interpolation to sample pixel values ​​from the original image based on the transformed coordinates, generating a target feature map, that is, for each output pixel V i Using bilinear interpolation, the formula is as follows:

[0069]

[0070] Among them, (χ n ,y n ) represents the pixel coordinates of the original image U. V represents the sampling coordinates calculated from the affine transformation. i U represents the value of the i-th pixel in the output feature map. n This represents the value of the nth pixel in the input feature map.

[0071] Secondly, ResNet, as an important branch of CNNs, inherits the excellent image processing capabilities of CNNs while also possessing the ability to suppress gradient vanishing in deep networks. Building upon this, the activation function of ResNet50 is replaced with Swish, resulting in smoother gradient characteristics, preventing neuron "death," and enhancing non-linear expressive power, thus improving its adaptability to deep networks. Swish can maintain small gradients in negative regions, improving training stability. Its differentiability and adaptive β parameter make it more flexible in fitting complex data distributions. Experiments demonstrate that in tasks such as image classification, Swish often outperforms ReLU.

[0072] Furthermore, by leveraging the self-attention mechanism of Transformer, parallel processing can be achieved, long-distance dependencies can be directly modeled, and multi-head attention can be used to flexibly capture global information, significantly improving training efficiency and model expressive power, making it particularly suitable for large-scale data and pre-training scenarios.

[0073] ResNet50 employs two Bottleneck structures (Bottleneck1 and Bottleneck2) (this is existing content, not innovative; the explanation is repeated to better illustrate the network's data processing capabilities). The core difference lies in the adjustment of the number of channels and the presence or absence of downsampling operations, thus adapting to the differentiated feature extraction needs of each layer of the network. From an overall architecture perspective, ResNet50 consists of five convolutional blocks and one initial convolutional layer. The number and specific structure of the Bottleneck residual modules included in different stages (network stages) vary, as detailed below: Stage-1: Output size [16, 256, 32, 32], composed of 1 Bottleneck1 module and 2 Bottleneck2 modules, initially extracting basic features through different module combinations; Stage-2: Output size [16, 512, 16, 16], containing 1 Bottleneck1 module and 3 Bottleneck2 modules, gradually compressing the spatial dimension and expanding the channel dimension during feature extraction. Stage-3: Output size [16, 1024, 8, 8], configured with 1 Bottleneck1 module and 5 Bottleneck2 modules, further deepening feature representation and capturing more complex patterns. Stage-4: Output size [16, 2048, 4, 4], composed of 1 Bottleneck1 module and 2 Bottleneck2 modules, completing the final construction of high-dimensional features. The standard Bottleneck2 module is used for residual blocks except for the first module of each stage. The backbone network of this module consists of 1×1 dimensionality reduction convolutions, 3×3 main feature extraction convolutions, and 1×1 dimensionality increase convolutions. The 1×1 dimensionality reduction convolutions are used to reduce the computational cost of the network, and the 1×1 dimensionality increase convolutions are used to restore the number of channels, so as to fully extract feature information and improve the computational efficiency of the network. The residual connections directly pass the input to the output and perform element-wise addition in the channel dimension to keep the size of the input data unchanged, so as to facilitate the network to stack and carry out the next deep learning step.

[0074] In the feature extraction stage, we propose a Dynamic Multi-Scale Attention Module (D-EMA). The D-EMA module primarily addresses the problems of multi-scale information aggregation and cross-space learning. For the input feature map X∈R... C×H×W (Where C is the number of channels, H is the feature map height, and W is the feature map width), the D-EMA module first segments the feature map X along the channel dimension, dividing it into G sub-feature groups to learn the feature information of different channels. The grouped feature map can be represented as X = [X0, X1, ..., X...]. G-1, where each sub - feature group generally sets the value of G to G << C, that is, the input feature map is divided into G sub - feature groups along the channel direction, and the number of channels in each group is much smaller than the total number of channels, ensuring that each sub - feature group still has multiple channels, thus assisting the D - EMA module to learn different features in different sub - feature groups, and then enhancing the feature expression ability.

[0075] In determining the number of groups G, the present invention defines a strategy for dynamically adjusting the number of groups according to the number of training epochs, so as to enhance the adaptability of the D - EMA module to multi - scale features in different training stages. Compared with the method of fixed number of groups, since the model's focus on features varies in different training stages, dynamically adjusting the number of groups can better adapt to the learning requirements of global and local information during training. Specifically, it is stipulated that the number of groups G changes dynamically according to the current number of training epochs (epoch): in the early stage of training, a smaller value of G is used to capture a larger range of global information; in the later stage of training, G is gradually increased to focus on more in - depth detailed local features. The adjustment rule can be expressed by the following formula:

[0076]

[0077] Among them, G represents the number of groups in the current training stage, and epoch represents the current number of training rounds. By using different values of G to correspond to the number of groups required in specific training stages respectively. In the early stage of training, a smaller number of groups is used for learning global information; in the later stage of training, a larger number of groups is used for capturing in - depth detailed features in the later stage of training. This dynamic adjustment strategy can effectively help the D - EMA module to adaptively adjust in different training stages, and contribute to enhancing the feature expression ability of the model. By virtue of the characteristic of the neural network that "different local receptive fields can enable neurons to capture multi - scale features", D - EMA adopts three parallel paths to extract the grouped feature matrix, specifically: two branches are located in the 1×1 branch, and the remaining one branch is located in the 3×3 branch. In the 1×1 branch, the channels are encoded bidirectionally; in the 3×3 branch, 3×3 convolutional kernels are stacked to capture multi - scale feature representations.

[0078] In addition, the D - EMA module can establish mutual dependence between channels and spatial positions to enrich the effect of feature aggregation. For the 1×1 branch, 2D global average pooling is used to encode its output and capture global spatial information. The formula is as follows:

[0079]

[0080] For the 3×3 branch, its output is directly adjusted to the corresponding dimension shape, and then corresponding processing is carried out to facilitate subsequent feature combination.

[0081] Where H represents the height of the input feature map, i.e., the number of pixels or units in the feature map in the vertical direction; in this invention, H = 128. W represents the width of the input feature map, i.e., the number of pixels or units in the feature map in the horizontal direction; in this invention, W = 128. c (i, j) represents the feature value of the c-th channel in the input feature map at spatial location (i, j). C This represents the output result after 2D global averaging.

[0082] After pooling, the D-EMA module applies the Softmax function to the pooling output to optimize computational efficiency. Then, a dot product operation is performed to multiply the parallel processing outputs, generating the first spatial attention map that aggregates multi-scale spatial information. Similarly, 2D global average pooling is used for the large-scale feature information of the 3×3 branch, and the 1×1 branch is adjusted to its corresponding dimensional shape. A dot product operation is then performed to multiply the outputs of the parallel processing, yielding the second spatial attention map. Finally, the two feature maps are fused, and the multi-scale feature information aggregation is completed using the Sigmoid function.

[0083] In the data processing section, skeleton data is not processed by this network, while RGB data needs to be enhanced through histogram equalization. Depth video does not need to be enhanced. Both are then processed by the same MSD-Net network to extract feature information for the next step of processing.

[0084] For skeleton information, we model the skeleton information, construct a Dynamic Weighted Spatiotemporal Graph Convolutional Network (DWTST-GCN), and process the skeleton information.

[0085] First, the collected skeleton information is modeled. Since skeleton data has the advantage of being less dependent on lighting changes, background complexity, and appearance features, feature extraction methods based on skeleton data have attracted considerable attention. Skeleton data typically consists of joints and their inter-joint topological structures (edges), representing human motion in a time-series format.

[0086] In the skeleton data used in this paper, the sequence of human motions consists of joint coordinates from consecutive frames. The skeleton data for each frame can be represented as:

[0087] X∈R T×N×C

[0088] Where T represents the number of frames selected (16 frames in this invention), N represents the key points (25 in this invention), and C represents the three-dimensional spatial information of the key points. (R is the set of real numbers.)

[0089] Furthermore, we use an adjacency matrix (A) to describe the topological relationships in the skeleton structure, defined as follows:

[0090] A∈R N×N

[0091] This matrix encodes the topological structure of joints in the skeleton, representing the natural connections between human joints (elbow to shoulder, spine to neck, etc.). That is, A ij This indicates that joints i and j are directly connected; otherwise, A ij =0.

[0092] The DWTST-GCN network consists of a Dynamic Weights Module (DWM), a Dilated Causal Temporal Perception Module (DC), and a Spatial Dropout layer.

[0093] The DC time-aware module consists of two different types of convolutional layers and a sigmoid function. Dilated convolutions in the network can expand the model's receptive field while maintaining computational efficiency, thereby acquiring more comprehensive temporal information. Batch normalization is performed after each convolutional layer to accelerate training, and the structure of convolution and ReLU activation functions enhances the model's nonlinear expressive power.

[0094] In addition to combining dilated convolution and causal convolution into dilated causal convolution, the network also uses dilated noncausal convolution in the second convolutional layer. This allows the layer to be unconstrained by causal relationships, ensuring that the model can rely on past and future inputs, capture the global contextual information of the complete input as much as possible, and ensure that complete sign language features can be extracted.

[0095] Furthermore, the DC time-aware module uses a SpatialDropout layer instead of a traditional Dropout layer. Compared to a regular Dropout layer, SpatialDropout can randomly drop features in each channel of the feature matrix. This is more flexible than dropping the entire neuron, effectively maintaining the complete spatial structure of the feature matrix, preventing the correlation between spatial features from being destroyed, and preserving the continuity of features. SpatialDropout is more suitable for processing continuous graph data.

[0096] The Sigmoid function calculates attention weights based on the feature matrix output from the network and performs a weighted summation operation on the input features, ultimately obtaining a weighted, attention-enhanced output. The DC time-aware module ensures that causal relationships in the time series of sign language recognition are not lost during network training and also retains some global contextual information, effectively establishing distance dependencies in sign language recognition features.

[0097] The DWM module introduces temporal weight parameters during the adjacency matrix generation process. For skeleton keypoints, it considers not only their spatial distance but also temporal weight parameters, forming a spatiotemporally dynamic weighted adjacency matrix with both spatial and temporal representations. Simultaneously, nodes with different hop counts are categorized and assigned different weights, which, together with the temporal weights, dynamically adjust the adjacency matrix, effectively capturing the spatiotemporal features of the skeleton information and further improving model performance. Finally, adjacency matrices with different hop counts are stacked together to form a multi-scale adjacency matrix, which is then input into the network constructed in this invention for graph convolution operations. The specific generation strategy for the adjacency matrix is ​​as follows:

[0098] For each valid hop, we generate three adjacency matrices:

[0099] a root : Indicates the adjacency relationship from the central node to other nodes, mainly capturing the direct connection between a node and its central node;

[0100] a close : Represents the adjacency relationship close to the central node, capturing the relationship between nodes that are relatively close to the central node.

[0101] a furth This represents the adjacency relationships between nodes far from the central node, capturing the relationships between nodes that are relatively far from the central node. The formula for generating the initial adjacency matrix is ​​as follows:

[0102]

[0103] Among them, A ij `i` represents the adjacency matrix elements between nodes `i` and `j`. `normalize_adjacency` represents the normalized adjacency matrix values ​​between nodes `i` and `j`. `hop_dis` ji This represents the hop distance between node j and node i. `hop` represents the current effective hop count. Then, based on the time weight parameter `hop_dis`... ji Adjust the adjacency matrix:

[0104]

[0105] Here, `time_weight` is the time weight parameter, used to balance the weights of different types of connections. Finally, these adjacency matrices generated with different hop counts are stacked into the final adjacency matrix, which is then used for graph convolution operations. In this way, the model can simultaneously capture the spatiotemporal dynamics between nodes, thereby significantly improving the model's performance on spatiotemporal graph data.

[0106] Using the methods described above, a complete DWM module can be constructed, thereby enabling spatiotemporal feature modeling of the skeleton sequence. Based on this module, the overall DWTST-GCN structure is as follows: Figure 4 As shown.

[0107] Step 3: The multi-dimensional information is fused and input into the Softmax network. Multi-dimensional information fusion is performed and input into the Softmax network. In sign language translation tasks, depth video, skeleton keypoint information, and RGB video information are complementary: depth information focuses on spatial distance relationships and target contours, while skeleton information emphasizes the motion trajectory and connectivity of key points in three-dimensional space. Both can provide recognition features that are difficult to represent in RGB video. This invention employs a multi-dimensional data fusion sign language recognition algorithm (Dynamic Weighted Spatial-Temporal Multi-Scale Dynamic Net, DWTST-MSD-Net), which includes the MSD-Net, DWTST-GCN, multi-dimensional feature fusion algorithm, and Softmax classifier mentioned in the previous steps. The multi-dimensional fusion algorithm is based on dynamic adaptive weighting of feature layers, integrating complementary information from RGB, depth, and skeleton modalities, and utilizing convolution and pooling to optimize feature representation, thereby improving the accuracy of multi-dimensional sign language recognition. First, a separate training strategy is used. Features F1 and F2 are obtained by training the MSD-Net network separately on RGB video data and depth data, with feature sizes [16, 256]. Simultaneously, the DWTST-GCN network is used to train the skeleton data, obtaining skeleton feature F3, also with a size of [16, 256]. These networks focus on extracting spatiotemporal features of different data modalities, ensuring that features for each data modality are learned independently and effectively. After training, the softmax layer of each network is removed, and the output is adjusted to the specified feature size. Next, each feature F1, F2, and F3 is assigned corresponding weight parameters W1, W2, and W3 to obtain the weighted features:

[0108] F′1=W1·F1 F′2=W2·F2 F′3=W3·F3

[0109] After weighting, the weighted features are concatenated along the channel dimension to obtain a tensor of size [16, 768].

[0110] F concat=concat(F′1,F′2,F′3)

[0111] Tensors are processed through convolutional layers to extract higher-level features, followed by overlapping max pooling for downsampling. This pooling window overlaps with each other, preserving more local information and reducing information loss. ReLU activation and batch normalization are then applied to obtain the final fused features, which are finally input into a Softmax classifier to complete the sign language recognition task.

[0112] Step four: Train the DWTST-MSD-Net network using a multidimensional sign language dataset. This invention employs a separate training strategy. First, the three feature extraction modules—RGB video information, depth video information, and skeleton keypoint information—are trained independently. After training, the resulting feature matrices are concatenated and fused, then input into the proposed three-modal weighted feature fusion module for final classification prediction.

[0113] A multidimensional sign language recognition model, DWSTST-MSD-Net, is constructed. This model first extracts spatiotemporal features from RGB and depth video data and skeleton data, respectively. The dimensions of the RGB and depth video data are [16, 3, 16, 128, 128]. MSD-Net is used to extract the spatiotemporal features of the RGB and depth data, with a feature size of [16, 256]. For the skeleton data, DWTST-GCN is used to extract the spatiotemporal features of this modality, with a feature size of [16, 256]. After obtaining the spatiotemporal features of these three modalities, the multidimensional feature fusion algorithm proposed in this invention can fuse them to obtain fused features. Finally, a Softmax classifier is used to classify the fused features to achieve sign language recognition.

[0114] The results display module converts sign language recognition results into readable text and displays the output. Its workflow is as follows: The system's built-in sign language recognition database stores the mapping relationship between sign language gestures and translated text. After the sign language recognition model outputs the recognition result, the system first performs similarity matching between the result and database entries, generating and encapsulating the corresponding sign language vocabulary sequence. Then, it transmits the data encrypted via Bluetooth / WiFi wireless communication, employing an optimized transmission protocol to ensure secure and reliable data delivery and minimizing end-to-end latency. After receiving the data, the display module first parses and extracts the sign language vocabulary sequence, performs text normalization processing, and then applies Named Entity Recognition (NER) technology to generate the target language text. Finally, the translation result is rendered onto the display screen. As an optional feature, the system can also use an irregular freeform prism to refract the displayed text to the optimal viewing position, significantly improving the accessibility experience. This module achieves fully automated processing from recognition results to visual output, ensuring the accuracy, real-time performance, and ease of use of the translation results.

Claims

1. A real-time sign language translation system based on multi-dimensional data fusion, characterized in that: It includes an image information acquisition module, a data processing module, and a result display module; The image information acquisition module captures user sign language gestures in real time using a camera to generate sign language videos. Based on FreeRTOS, it runs a four-core task architecture: an HTTP server task, a camera acquisition task, a streaming media transmission task, and a video download task. The HTTP server task receives client requests, including MJPEG video streams, JPEG single-frame capture, and video download access endpoints, with built-in CORS cross-domain access support. The camera acquisition task continuously captures JPEG frames at a fixed frame rate. The streaming media transmission task pushes MJPEG video streams in real time. The video download task encapsulates the video into AVI format files for client download. The AVI format includes a complete file header, stream header, and index block structure. Alternatively, it can download MP4 format videos using FFmpeg built on the server side. For real-time transmission, each task achieves efficient collaboration through semaphore-protected frame data and queue-based client connection transmission. Through multi-task parallel processing, it achieves precise... The transmission frame rate is precisely controlled to meet algorithm requirements and maximize the utilization of the ESP32 dual-core resources. The data processing module includes a multidimensional sign language recognition model, DWSTST-MSD-Net, used to perform sign language recognition on the video recognition video acquired by the image information acquisition module. The result display module converts the sign language recognition results into readable text and displays them. The built-in sign language recognition database stores database entries that map the mapping relationship between sign language gestures and translated text. When the multidimensional sign language recognition model DWSTST-MSD-Net outputs the recognition result, it first performs similarity matching between the sign language gesture corresponding to the recognition result and the database entry, generates the corresponding sign language vocabulary sequence, and encapsulates it. Then, it transmits the data encrypted through the wireless communication module. After receiving the data, the display module first parses and extracts the sign language vocabulary sequence, performs text normalization processing, applies Named Entity Recognition (NER) technology to generate the target language text, and finally renders the target language text onto the display screen.

2. The real-time sign language translation system based on multi-dimensional data fusion according to claim 1, characterized in that: The image information acquisition module includes at least a color camera, a depth camera, and a human skeleton detection camera. The acquired data includes RGB video acquired by the color camera, depth video acquired by the depth camera, and 3D skeleton coordinate time-series data acquired by the human skeleton detection camera.

3. The real-time sign language translation system based on multi-dimensional data fusion according to claim 2, characterized in that: The multidimensional sign language recognition model DWSTST-MSD-Net includes a data preprocessing module, a multi-scale sign language recognition network MSD-Net, a dynamically weighted spatiotemporal graph convolutional network DWTST-GCN, and a Softmax classifier. In the data preprocessing module, histogram equalization technology is used to enhance the contrast of the acquired RGB video images. The processing can be represented by the following formula: In the formula, H(i,j) is the pixel value at the pixel coordinate (i,j) after histogram equalization, where i is the x-coordinate of the pixel coordinate, j is the y-coordinate of the pixel coordinate, and g is the y-coordinate of the pixel coordinate. i,j This describes the pixel value at pixel coordinates (i,j) in the input image, where I represents the total number of gray levels, typically ranging from 0 to 255 for an 8-bit image (I = 256), and n is an integer from 0 to I-1. n This represents the probability of a pixel with gray value n appearing in the input image; For RGB videos processed using histogram equalization, a keyframe selection strategy is employed to remove redundant information. An initial index S is randomly selected from the RGB video, S = rand(0, N-16), and 16 consecutive frames are extracted as keyframes, where N is the total number of frames in the sign language video. This approach preserves the integrity of the actions while standardizing the input length. In the data preprocessing module, when processing the acquired depth video, since the depth video and RGB video are time-synchronized, the 16 keyframe indices of the RGB video are directly reused to extract the corresponding depth frames, ensuring strict alignment of multi-dimensional data along the "16-frame sequence" dimension, thus achieving depth video preprocessing. In the data preprocessing module, when preprocessing the acquired 3D skeleton coordinate time-series data, a combination of filtering and time-series normalization is used. First, Gaussian filtering is applied to remove noise; the specific formula is as follows. Where, x t Let G(x) be the spatial coordinates of the skeleton keypoints at time t, k be the filtering window that determines the neighborhood range involved in the calculation, τ be the time offset, σ be the pixel standard deviation, and G(x) be the coordinates of the skeleton keypoints at time t. t The data is Gaussian filtered, and the filtered coordinate sequence is then normalized to a length of 16 frames using linear interpolation. The specific process can be performed using the following formula. Where s is the sampling interval of the original sequence, ia is the index of the new sequence, ja is the adjacent index of the original sequence, and x new,ia x represents the normalized keypoint coordinates of frame ia. old,ja This represents the coordinates of the keypoint corresponding to the ja-th frame in the original sequence. The multi-scale sign language recognition network MSD-Net includes a spatial transformation network STN modified with ResNet50, a dynamic multi-scale attention module D-EMA, and a Transformer network. The spatial transformation network STN comprises a localization network, a mesh generator, and a sampler. The localization network calculates the affine transformation parameters of the input image. The input feature map U is processed by a convolutional neural network f. loc The formula for learning the transformation parameter θ is as follows: θ=f loc (U) Where θ is a 2×3 affine transformation matrix: Mesh generator: Its function is to generate a coordinate grid of the target feature map and calculate the mapping relationship between these coordinates and the original image coordinates, that is, to calculate the input image coordinates through the affine transformation matrix θ. The formula is as follows: in, These are the coordinates of a pixel in the source network. These are the coordinates of the corresponding pixels in the target network. Since the affine transformation matrix is ​​a 2×3 matrix, therefore... Transform it into a 3×1 matrix while maintaining the correct format; Sampler: Its function is to use bilinear interpolation to sample pixel values ​​from the original image based on the transformed coordinates, generating a target feature map, that is, for each output pixel V i Using bilinear interpolation, the formula is as follows: Among them, (χ n ,y n ) represents the pixel coordinates of the original image U. V represents the sampling coordinates calculated from the affine transformation. i U represents the value of the i-th pixel in the output feature map. n This represents the value of the nth pixel in the input feature map; ResNet50 employs two Bottleneck structures: Bottleneck1 and Bottleneck2. The core difference lies in the adjustment of the number of channels and the presence or absence of downsampling operations, adapting to the differentiated feature extraction needs of each layer of the network. ResNet50 consists of five convolutional blocks and one initial convolutional layer. The number and specific structure of the Bottleneck residual modules included in different stages vary. Detailed configurations are as follows: Stage-1: Output size [16, 256, 32, 32], composed of 1 Bottleneck1 module and 2 Bottleneck2 modules, initially extracting basic features through different module combinations; Stage-2: Output size [16, 512, 16, 16], containing 1 Bottleneck1 module and 3 Bottleneck2 modules. In feature extraction, the spatial dimension is gradually compressed and the channel dimension is expanded. Stage-3: The output size is [16, 1024, 8, 8], configured with 1 Bottleneck1 module and 5 Bottleneck2 modules to further deepen the feature expression and capture more complex patterns. Stage-4: The output size is [16, 2048, 4, 4], composed of 1 Bottleneck1 module and 2 Bottleneck2 modules, to complete the final construction of high-dimensional features. The standard Bottleneck2 module is used for residual blocks other than the first module of each stage. The backbone network of this module consists of 1×1 dimensionality reduction convolution, 3×3 main feature extraction convolution and 1×1 dimensionality increase convolution. The 1×1 dimensionality reduction convolution is used to reduce the computation of the network, and the 1×1 dimensionality increase convolution is used to restore the number of channels, so as to fully extract feature information and improve the computational efficiency of the network. Residual connections directly pass the input to the output and perform element-wise addition along the channel dimension to keep the size of the input data constant, making it convenient for the network to be superimposed for the next step of deep learning; The dynamic multi-scale attention module D-EMA is used to solve the problems of multi-scale information aggregation and cross-space learning. For the input feature map X ∈ R C×H×W , where C is the number of channels, H is the height of the feature map, and W is the width of the feature map. The D-EMA module first performs a slicing operation on the feature map X in the channel dimension and divides it into G sub-feature groups to achieve learning of different channel feature information. The grouped feature map is represented as X = [X0, X1,..., X G-1 , where the value of G for each sub-feature group is set to G << C, that is, the input feature map is divided into G sub-feature groups in the channel direction, and the number of channels contained in each group is much smaller than the total number of channels, ensuring that each sub-feature group still has multiple channels, thereby assisting the D-EMA module to learn differentiated features in different sub-feature groups and further enhancing the feature expression ability; Regarding the determination of G, a strategy is defined to dynamically adjust the number of groups based on the number of training epochs. This enhances the adaptability of the D-EMA module to multi-scale features at different training stages. Compared to a fixed number of groups, dynamically adjusting the number of groups better adapts to the learning needs of global and local information during training, since the model's focus on features differs at different training stages. G changes dynamically according to the current training epoch: in the early stages of training, a smaller G value is used to capture a wider range of global information; in the later stages of training, G is gradually increased to focus on more in-depth and detailed local features. The adjustment rule can be expressed as the following formula: By using different G values ​​to correspond to the group number requirements of specific training stages, a smaller number of groups is used in the early stages of training for learning global information, while a larger number of groups is used in the later stages for capturing deep and detailed features. This dynamic adjustment strategy effectively helps the D-EMA module adapt to different training stages, helping to improve the model's feature representation ability. Leveraging the characteristic of neural networks that "different local receptive fields enable neurons to capture multi-scale features," D-EMA uses three parallel paths to extract the group feature matrix: two branches are located in a 1×1 branch, and the remaining branch is located in a 3×3 branch. In the 1×1 branch, bidirectional channel encoding is performed; in the 3×3 branch, 3×3 convolutional kernels are stacked to capture multi-scale feature representations. After pooling, the D-EMA module applies the Softmax function to the pooling output to optimize computational efficiency. Then, a dot product operation is performed to multiply the parallel processing outputs, generating the first spatial attention map that aggregates multi-scale spatial information. Similarly, 2D global average pooling is used for the large-scale feature information of the 3×3 branch, and the 1×1 branch is adjusted to its corresponding dimensional shape. A dot product operation is then performed to multiply the outputs of the parallel processing, yielding the second spatial attention map. Finally, the two feature maps are fused, and the multi-scale feature information aggregation is completed using the Sigmoid function. The Dynamically Weighted Spatiotemporal Graph Convolutional Network (DWTST-GCN) is used to process skeleton information, specifically, to process pre-processed 3D skeleton coordinate temporal data. DWTST-GCN includes a Dynamic Weighting Module (DWM) and a Temporally Aware Module (DCSpatialDropout layer). The DC temporally aware module consists of two different convolutional layers and a sigmoid function. The dilated convolutions in the network expand the model's receptive field while maintaining computational efficiency, acquiring more comprehensive temporal information. Batch normalization is performed after each convolutional layer to accelerate training, and the structure of convolutions and ReLU activation functions enhances the model's non-linear expressive power. In addition to combining dilated and causal convolutions into a dilated causal convolution, the network also uses a dilated non-causal convolution in the second convolutional layer. Conv allows this layer to be unconstrained by causal relationships, ensuring that the model can rely on past and future inputs, capture the global contextual information of the complete input as much as possible, and ensure the extraction of complete sign language features. In the DC time-aware module, a SpatialDropout layer is used instead of a traditional Dropout layer. Compared to a regular Dropout layer, SpatialDropout can randomly drop features across each channel of the feature matrix, making it more flexible than dropping entire neurons. This effectively maintains the complete spatial structure of the feature matrix, preventing the destruction of correlations between spatial features and preserving feature continuity. SpatialDropout is more suitable for processing continuous graph data. The Sigmoid function calculates attention weights based on the output feature matrix of the network and performs a weighted summation operation on the input features, ultimately yielding... The output is enhanced with attention weights; the DC time-aware module ensures that causal relationships in the time series of sign language recognition are not lost during network training, and also retains some global contextual information, which can effectively establish distance dependencies in sign language recognition features; the DWM module can introduce temporal weight parameters in the generation of the adjacency matrix. For skeleton keypoints, not only their spatial distance is considered, but also temporal weight parameters are introduced to form a spatiotemporally dynamic weighted adjacency matrix with both spatial and temporal representations. At the same time, nodes with different hop counts are classified and assigned different weights. Together with the temporal weights, the adjacency matrix is ​​dynamically adjusted to effectively capture the spatiotemporal features in the skeleton information and further improve the model's performance. Finally, the adjacency matrices with different hop counts are stacked together to form a multi-scale adjacency matrix, which is input into the network built in this invention for graph convolution operations. The specific generation strategy of the adjacency matrix is ​​as follows: For each valid hop count (hop), generate three adjacency matrices: a root : Represents the adjacency relationship from the central node to other nodes, mainly capturing the direct connection between a node and its central node; a close : Represents the adjacency relationship close to the central node, capturing the relationships between nodes that are relatively close to the central node. further : Represents the adjacency relationships far from the central node, capturing the relationships between nodes that are relatively far from the central node. The formula for generating the initial adjacency matrix is ​​as follows: Among them, A ij This represents the adjacency matrix elements between node i and node j, `normalize_adjacency` represents the normalized adjacency matrix values ​​between node i and node j, and `hop_dis`... ji This represents the hop distance between node j and node i, where hop represents the current effective hop count. Then, based on the time weight parameter hop_dis... ji Adjust the adjacency matrix: Here, `time_weight` is the time weight parameter, used to balance the weights of different types of connections. Finally, these adjacency matrices generated with different hop counts are stacked into the final adjacency matrix, which is used for graph convolution operations. After fusing multidimensional information, it is input into the Softmax classifier. Depth video, skeleton keypoint information, and RGB video information complement each other: depth information focuses on spatial distance relationships and target contours, while skeleton information emphasizes the motion trajectories and connectivity of keypoints in three-dimensional space. Both provide recognition features that are difficult to represent in RGB video. The multidimensional fusion algorithm is based on dynamic adaptive weighting of feature layers. By integrating complementary information from RGB, depth, and skeleton modalities, and utilizing convolution and pooling to optimize feature representation, it improves the accuracy of multidimensional sign language recognition. First, a separate training strategy is used, training the MSD-Net network separately. Features F1 and F2 were obtained from RGB video data and depth data, with a size of [16, 256]. Simultaneously, the skeleton data was trained using the DWTST-GCN network to obtain skeleton feature F3, also with a size of [16, 256]. These networks focus on extracting spatiotemporal features from different modalities, ensuring that features for each data modality are learned independently and effectively. After training, the softmax layer of each network was removed, and the output was adjusted to the specified feature size. Then, corresponding weight parameters W1, W2, and W3 were assigned to each feature F1, F2, and F3 to obtain the weighted features. F′1=W1·F1 F′2=W2·F2 F′3=W3·F3 After weighting, the weighted features are concatenated along the channel dimension to obtain a tensor of size [16, 768]. F concat =concat(F′1,F′2,F′3) Tensors are processed through convolutional layers to extract higher-level features, followed by overlapping max pooling for downsampling. This pooling window overlaps with each other, preserving more local information and reducing information loss. ReLU activation and batch normalization are then applied to obtain the final fused features, which are finally input into a Softmax classifier to complete the sign language recognition task.

4. A real-time sign language translation system based on multi-dimensional data fusion according to claim 3, characterized in that: The D-EMA module establishes interdependencies between channels and spatial locations to enrich the effect of feature aggregation. For 1×1 branches, 2D global average pooling is used to encode the output, capturing global spatial information, as shown in the following formula: For a 3×3 branch, its output is directly adjusted to the corresponding dimension shape, and then processed accordingly to facilitate subsequent feature union; where H represents the height of the input feature map, i.e., the number of pixels or units in the vertical direction of the feature map, and this invention uses H=128; W represents the width of the input feature map, i.e., the number of pixels or units in the horizontal direction of the feature map, and this invention uses W=128; x C (i, j) represents the feature value of the c-th channel in the input feature map at pixel coordinates (i, j), Z C This represents the output result after 2D global averaging.

Images