Real-time voice-to-sign language conversion and synthesis method based on cooperative computing

Abstract

The invention discloses a real-time voice-to-sign language conversion and synthesis method based on cooperative computing, and the method comprises the steps: S10, voice collection and preprocessing: collecting a voice signal through a wearable device, and generating a compressed voice packet; an improved sparse coding algorithm is adopted to compress the data volume so as to reduce the transmission load; s20, performing cloud semantic coding, modeling long-range dependence of the voice signals by using a deep bidirectional transducer model, extracting high-order semantic vectors through a self-attention mechanism, generating compact semantic representation, and performing semantic enhancement in combination with the paranguage features; s30, edge node action mapping is carried out, a space-time convolutional network ST-CNN is deployed, and a double-branch structure of the space-time convolutional network ST-CNN analyzes a syntactic-level action trajectory and refines hand micro-motion features; action continuity is ensured through residual connectivity and historical gradient moving average filtering; a biomechanical constraint module is applied, a joint angle threshold value and a motion smooth item are included, and it is guaranteed that the motion conforms to the human kinematics law; s40, terminal skeleton rendering; and S50, reinforcement learning dynamic scheduling.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction and edge computing technology, specifically relating to a real-time speech-to-sign language conversion and synthesis method based on collaborative computing. Background Technology

[0002] Current real-time sign language synthesis systems based on wearable devices face severe technical bottlenecks. At the terminal hardware level, commonly used wearable devices are generally limited by power consumption and size constraints, making it difficult for their local computing power to support the generation of high-quality sign language animations. Complex sign language scenes require the real-time driving of more than 50 biometric joint parameters, causing traditional local rendering solutions to often experience sharp drops in frame rate (below 15fps) and interaction latency (exceeding 300 milliseconds), failing to meet the real-time requirements of natural dialogue for hearing-impaired users. Especially when processing long sentence synthesis, insufficient terminal computing power can cause motion interruptions or mechanical jitter, severely damaging the communication experience.

[0003] Low network transmission efficiency further restricts system performance. Existing cloud-edge collaborative architectures mostly adopt video streaming or full-volume action data transmission modes, which expose significant defects in 5G mobile environments: First, high-definition sign language video streams require continuous and stable high bandwidth support (>10Mbps), and the stuttering rate surges in weak network scenarios (such as subway tunnels and dense commercial areas); second, there are a large number of repetitive parameters in continuous gesture actions, but traditional solutions have not established a differentiated transmission mechanism, resulting in more than 60% redundant data transmission; more importantly, the cross-modal semantics of voice text and sign language actions are disconnected, and the lack of an end-to-end alignment mechanism causes a mismatch between paralinguistic information such as facial expressions and rhythm and the core semantics.

[0004] The quality defects in motion generation constitute the third obstacle. Mainstream technologies generally suffer from insufficient biological rationality: keyframe interpolation-based schemes ignore the continuity of joint movements, resulting in non-physiological trajectory abrupt changes that make gestures appear stiff and mechanical; at the same time, most systems do not integrate paralinguistic features such as speech rate and intonation, failing to accurately convey emotional semantics such as questions and emphasis; furthermore, the conversion of Chinese to sign language word order relies on a static rule base, resulting in an excessively high error rate for complex imperative and interrogative sentences, severely limiting its adaptability to practical scenarios.

[0005] Static resource scheduling mechanisms exacerbate system vulnerability. Traditional cloud-edge-device architectures employ fixed computation offloading strategies (such as cloud semantic parsing + terminal rendering), which cannot dynamically respond to network fluctuations and load changes: the lack of a degradation rendering mechanism when edge nodes are congested leads to animation interruptions or frame skipping; continuous high load operation of terminals shortens the battery life of wearable devices; and the preloading strategy, due to the lack of consideration for semantic relevance, has a consistently low cache hit rate, frequently triggering high-latency data reloading. Summary of the Invention

[0006] In view of the above-mentioned problems, the present invention provides a real-time speech-to-sign language conversion and synthesis method based on collaborative computing.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A real-time speech-to-sign language conversion and synthesis method based on collaborative computing includes the following steps:

[0009] S10, Voice Acquisition and Preprocessing: Acquires voice signals through wearable devices, performs noise suppression and feature extraction, and generates compressed voice packets; employs an improved sparse coding algorithm to compress data volume to reduce transmission load;

[0010] S20, cloud-based semantic coding, uses a deep bidirectional transformer model to model the long-range dependencies of speech signals, extracts high-order semantic vectors through a self-attention mechanism, generates compact semantic representations, and combines paralinguistic features for semantic enhancement.

[0011] S30, edge node action mapping, deploys a spatiotemporal convolutional network ST-CNN, whose dual-branch structure parses syntactic-level action trajectories and refines hand micro-movement features respectively; residual connectivity and historical gradient moving average filtering ensure action continuity; biomechanical constraint modules are applied, including joint angle thresholds and motion smoothing terms, to ensure that the action conforms to the laws of human kinematics.

[0012] S40, terminal skeletal rendering, organizes skeletal animation data based on the Global Semantic Tree (GST) caching architecture. The root node stores basic action primitives, the intermediate nodes encode composite semantic clusters, and the leaf nodes bind complete animation packages. It calculates relevant word cluster prefetch data by improving point mutual information (PMI*) and only transmits differences in action parameters between nodes. It adopts a lightweight skeleton topology structure to dynamically adjust the precision, and combines Bloom filter indexing to reduce computational complexity. It loads animation primitive nodes from the local machine and renders sign language output.

[0013] S50, reinforcement learning dynamic scheduling, constructs a system state vector, which integrates node computing and cloud load L. cloud Edge node load L edge Real-time uplink bandwidth B uplink Real-time downlink bandwidth B downlink and cache hit rate H cache Maximize the reward function through the Q-Learning strategy; adopt dynamic decision-making for activation state and data transmission granularity.

[0014] In one possible implementation, the cloud-based semantic encoding specifically includes:

[0015] S201, Text semantic encoding. Input the converted text S into the BERT model, and extract the last layer hidden state H = BERTπ(S); Take the token corresponding vector as the basic semantic vector E s ;

[0016] S202, Paralinguistic feature extraction. Extract the speech rate r, intonation p, and context c from the speech signal to form the paralinguistic feature vector P = [r, p, c];

[0017] S203, Semantic vector fusion. Generate an enhanced semantic vector through weighted fusion: E = E s +λP, where λ is the paralinguistic feature weight coefficient;

[0018] S204, Vector dimension compression. Input the enhanced semantic vector E into the adaptive quantization module and output the compressed vector (dimension D′, satisfying D′ < D), where D represents the initial dimension of the enhanced semantic vector E, and D′ refers to the dimension of the output vector after compression by the adaptive module of the vector.

[0019] In a possible implementation manner, the implementation of the spatio-temporal convolutional network ST-CNN includes:

[0020] S301, Dual-branch feature processing. The main branch analyzes the syntactic-level action trajectory F g , and the auxiliary branch receives the same input and refines the hand micro-motion feature F l ;

[0021] S302, Feature fusion and action generation. Weightedly fuse the global and local features into F f =αF g +(1 - α)F l (0.5 < α < 0.7), and output the action parameter matrix through residual connection

[0022] S303, Biomechanical constraints. The joint angle threshold constraint is The motion smoothness constraint is

[0023] In a possible implementation manner, the terminal bone rendering specifically includes:

[0024] S401, Construct a global semantic tree GST cache structure. The root node stores the basic action primitives, the intermediate nodes encode the composite semantic clusters, and the leaf nodes bind the complete context animation packages, and organize the bone animation data according to semantic relevance;

[0025] S402, perform improved point mutual information (PMI) prefetch calculation. When the target word is identified, the relevant word clusters are calculated through the PMI algorithm, and animation data is prefetched along the vertical direction of the tree structure, i.e. the parent-child node direction, and the horizontal direction, i.e. the sibling node direction.

[0026] S403 transmits differentiated action parameters, only transmitting action parameter data that differs between the current node and the prefetched node, reducing the amount of data transmitted;

[0027] S404 dynamically optimizes skeleton rendering by using a lightweight skeleton topology structure combined with Bloom filter indexing to update the skeleton chain. It dynamically adjusts motion parameters based on a pre-trained inverse kinematics model, reducing the rendering computational complexity from quadratic to logarithmic.

[0028] In one possible implementation, the reinforcement learning scheduling specifically includes:

[0029] S501 constructs a system state vector, and collects the edge node computing load, current network bandwidth and terminal cache hit rate in real time to form a state space;

[0030] S502 designs a multi-dimensional reward function that integrates end-to-end latency, system energy consumption, and semantic relevance strength to calculate instantaneous reward values ​​to drive policy optimization.

[0031] S503 executes a dynamic scheduling strategy: It selects the optimal action using the Q-Learning algorithm and adjusts the data transmission granularity and computation task migration based on network conditions. Specifically, this includes transmitting the complete semantic subtree to the terminal during high bandwidth usage; and pushing only basic action primitives during low bandwidth usage, with the terminal generating transition animations using lightweight LSTM interpolation to maintain visual continuity.

[0032] The present invention has the following beneficial effects:

[0033] (1) Through weighted fusion of paralinguistic features and BERT semantic vectors (E = E s +λP) to achieve cross-modal emotion expression enhancement;

[0034] (2) ST-CNN dual-branch network combined with historical gradient moving average filtering is used to ensure reasonable biomechanical motion generation;

[0035] (3) Design a semantically driven GST tree cache and PMI prefetching mechanism to significantly reduce data transmission volume;

[0036] (4) A reinforcement learning scheduling model with end-to-end latency, energy consumption, and semantic relevance as multi-dimensional rewards is constructed to adaptively balance the load of heterogeneous resources. This method provides hearing-impaired users with a low-latency, high-fidelity sign language interaction experience in a 5G mobile environment. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the steps of a real-time speech-to-sign language conversion and synthesis method based on collaborative computing, according to an embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] See Figure 1 The diagram shows a flowchart of a real-time speech-to-sign language conversion and synthesis method based on collaborative computing according to an embodiment of the present invention. It achieves efficient speech-to-sign language conversion through a three-tiered distributed architecture: semantic encoding is completed in the cloud, action parameters are generated at the edge nodes, lightweight rendering is performed on the terminal, and resource scheduling is dynamically optimized using reinforcement learning. Specifically, the method includes the following steps:

[0040] S10, Voice Acquisition and Preprocessing: Acquires voice signals through wearable devices, performs noise suppression and feature extraction, and generates compressed voice packets; employs an improved sparse coding algorithm to compress data volume to reduce transmission load;

[0041] S20, cloud-based semantic coding, uses a deep bidirectional transformer model to model the long-range dependencies of speech signals, extracts high-order semantic vectors through a self-attention mechanism, generates compact semantic representations, and combines paralinguistic features for semantic enhancement.

[0042] S30, edge node action mapping, deploys a spatiotemporal convolutional network ST-CNN, whose dual-branch structure parses syntactic-level action trajectories and refines hand micro-movement features respectively; residual connectivity and historical gradient moving average filtering ensure action continuity; biomechanical constraint modules are applied, including joint angle thresholds and motion smoothing terms, to ensure that the action conforms to the laws of human kinematics.

[0043] S40, terminal skeletal rendering, organizes skeletal animation data based on the Global Semantic Tree (GST) caching architecture. The root node stores basic action primitives, the intermediate nodes encode composite semantic clusters, and the leaf nodes bind complete animation packages. It calculates relevant word cluster prefetch data by improving point mutual information (PMI) and only transmits differences in action parameters between nodes. It adopts a lightweight skeleton topology structure to dynamically adjust the accuracy and combines Bloom filter indexing to reduce computational complexity. It loads animation primitive nodes from the local machine and renders sign language output.

[0044] S50, reinforcement learning dynamic scheduling, constructs a system state vector, which integrates node computing and cloud load L.cloud Edge node load L edge Real-time uplink bandwidth B uplink Real-time downlink bandwidth B downlink and cache hit rate H cache Maximize the reward function through the Q-Learning strategy; adopt dynamic decision-making for activation state and data transmission granularity.

[0045] The above-described real-time speech-to-sign language conversion and synthesis method based on collaborative computing involves a wearable device collecting speech signals, performing noise suppression and feature extraction, and uploading the data to the cloud after compressing the data volume using an improved sparse coding algorithm. The cloud uses a deep bidirectional transformer model to extract high-order semantic vectors from the speech text and integrates paralinguistic features such as speech rate, intonation, and context to generate enhanced semantic representations. Edge nodes receive the semantic vectors and use a two-branch spatiotemporal convolutional network (ST-CNN) to parse syntactic-level action trajectories and hand micro-movement features, optimizing action continuity through a biomechanical constraint module. The terminal device prefetches skeletal animation data based on a Global Semantic Tree (GST) architecture, loading only the differential action parameters between nodes, and using Bloom filter indexing to achieve efficient skeleton chain updates. Finally, a reinforcement learning mechanism monitors node computational load, network bandwidth, and cache hit rate in real time, dynamically adjusting data transmission granularity and computational task migration through a Q-Learning strategy. For example, under high bandwidth, the complete semantic subtree is transmitted, while under low bandwidth, only basic action primitives are pushed and the terminal interpolates to generate transition animations.

[0046] The local terminal is responsible for preprocessing the voice signal, reducing the amount of raw data through noise suppression and feature extraction, and generating lightweight semantic packets using an improved sparse coding algorithm. Edge nodes are deployed at 5G base stations or local servers near the user, possessing stronger computing power and more stable network connections than wearable terminals. They can quickly perform action mapping on semantic vectors sent from the cloud, shortening end-to-end latency. The cloud server uses a large-scale pre-trained bidirectional transformer and self-attention mechanism to extract high-order semantics and refine semantic expressions by combining paralinguistic features, ensuring the accuracy and diversity of the conversion. The above three components achieve private and stable data exchange in home or mobile environments through dedicated low-latency channels and encrypted communication, effectively alleviating the transmission delay problem caused by public Internet congestion and avoiding the loss of real-time performance due to insufficient computing power at the terminal and edge sides. Scheduling control dynamically allocates processing tasks and data granularity at each layer based on runtime computing load, network bandwidth, and cache hit rate, ensuring overall system throughput while achieving rapid adaptation to network jitter and resource fluctuations.

[0047] In another embodiment of the present invention, a real-time speech-to-sign language conversion and synthesis method based on collaborative computing is provided. In S10, a wearable device collects user speech through a built-in microphone, and the sampling rate is set to f.s The sampling bit depth is 16 bits, and the audio frames are stored in PCM format. Each frame has a length of T milliseconds, and the frame shift is T / 2 to ensure sufficient time resolution and continuity. In preprocessing, spectral subtraction and adaptive gain control (AGC) are used to denoise the original speech and normalize the energy, eliminating the influence of environmental noise and ensuring that the energy level of each frame signal is consistent, thereby improving the robustness of subsequent feature extraction. The time-frequency domain features of the speech are extracted by means of time-frequency convolution. After mapping each frame signal into a time-frequency spectrogram, a lightweight convolutional neural network is applied to extract multi-scale feature responses for capturing the short-time dynamic features of the speech.

[0048] For the extracted feature vectors An improved sparse coding algorithm is introduced to generate a sparse representation c, and the optimization objective is defined as follows:

[0049]

[0050] where is the pre-trained sparse dictionary, and λ>0 is the sparse regularization coefficient. Through this compression method, the amount of feature data can be significantly reduced. The generated semantic data packet c not only retains the main information but also achieves a high compression ratio.

[0051] For a real-time speech-to-sign language conversion and synthesis method based on collaborative computing according to another embodiment of the present invention, the cloud semantic encoding specifically includes:

[0052] S201, text semantic encoding. The converted text S is input into the BERT model, and the last layer hidden state H = BERT(S) is extracted; the vector corresponding to the token is taken as the basic semantic vector E s ;

[0053] S202, paralinguistic feature extraction. The speech rate r, intonation p, and context c are extracted from the speech signal to form a paralinguistic feature vector P = [r, p, c];

[0054] S203, semantic vector fusion. An enhanced semantic vector is generated through weighted fusion: E = E s + λP, where λ is the paralinguistic feature weight coefficient;

[0055] S204, vector dimension compression. The enhanced semantic vector E is input into the adaptive quantization module, and the compressed vector (with dimension D′, satisfying D′ < D), where D represents the initial dimension of the enhanced semantic vector E, and D′ refers to the dimension of the output vector after being compressed by the adaptive module.

[0056] For a real-time speech-to-sign language conversion and synthesis method based on collaborative computing according to another embodiment of the present invention, the implementation of the spatio-temporal convolutional network ST-CNN includes:

[0057] S301, dual-branch feature processing, main branch parses syntactic-level action trajectories F g The auxiliary branch receives the same input and refines the hand micro-movement features F. l ;

[0058] S302, Feature Fusion and Action Generation, Weighted Fusion of Global and Local Features as F f =αF g +(1-α)F l (0.5 < α < 0.7), output action parameter matrix via residual connection

[0059] S303, biomechanical constraint, joint angle threshold constraint is... Motion smoothing constraint is

[0060] Another embodiment of the present invention provides a real-time speech-to-sign language conversion and synthesis method based on collaborative computing, wherein terminal skeletal rendering specifically includes:

[0061] S401 constructs a global semantic tree (GST) cache structure, with the root node storing basic action primitives, the intermediate nodes encoding composite semantic clusters, and the leaf nodes binding complete context animation packages, organizing skeletal animation data according to semantic relevance.

[0062] S402, perform improved point mutual information (PMI) prefetch calculation. When the target word is identified, the relevant word clusters are calculated through the PMI algorithm, and animation data is prefetched along the vertical direction of the tree structure, i.e. the parent-child node direction, and the horizontal direction, i.e. the sibling node direction.

[0063] S403 transmits differentiated action parameters, only transmitting action parameter data that differs between the current node and the prefetched node, reducing the amount of data transmitted;

[0064] S404 dynamically optimizes skeleton rendering by using a lightweight skeleton topology structure combined with Bloom filter indexing to update the skeleton chain. It dynamically adjusts motion parameters based on a pre-trained inverse kinematics model, reducing the rendering computational complexity from quadratic to logarithmic.

[0065] Specifically, to improve loading efficiency, after identifying the target words, the edge nodes calculate the improved mutual information between node c and word cluster ω:

[0066]

[0067] Where ∈ is the smoothing constant. According to PMI * The values ​​are sorted, and highly relevant nodes are prefetched first along the three directions of GST (vertical, horizontal and cross-layer), and only the parameter differences between the nodes to be loaded are transmitted.

[0068] The terminal uses a Bloom filter to maintain the index of loaded nodes, reducing the complexity of updating the skeletal chain from O(n^2) to O(n^2). 2 The rendering latency is reduced to O(logn). A lightweight skeletal topology model is used to perform fast frame interpolation and pose decoding on candidate motion parameters, thereby reducing rendering latency.

[0069] During the terminal sign language loading and rendering process, when the edge node sends out the action parameter sequence A t Then, the terminal first quickly locates the corresponding action node c through the Global Semantic Tree (GST); if this node has mutual information (PMI) with the improvement points of the current input word cluster... * If (c,ω)≥τ, then the incremental animation package ΔA is prefetched from GST first to reduce subsequent network requests. Subsequently, a Bloom filter is used to check if the required skeleton chain already exists in the local cache; if it is already loaded, the cache is reused directly; otherwise, the complete skeleton chain data is loaded from the local cache or an edge node. After the skeleton chain is loaded, the module processes the motion primitive parameter A for each frame. t Applying exponential smoothing:

[0070]

[0071] A t These are the basic action primitive parameters for the current time step t, i.e., the basic action parameter vector for the current time step t. This is the interpolated output of the previous frame (time t-1); This is the interpolated output for the current frame (time t).

[0072] To eliminate abrupt changes and ensure motion continuity, a lightweight skeletal topology algorithm is used to calculate the pose for each frame and drive the rendering engine, updating the sign language animation on the display screen in real time. This invention achieves efficient rendering and natural, smooth sign language output in low-bandwidth environments through multiple judgment and caching strategies.

[0073] Another embodiment of the present invention provides a real-time speech-to-sign language conversion and synthesis method based on collaborative computing, wherein reinforcement learning scheduling specifically includes:

[0074] S501 constructs a system state vector, and collects the edge node computing load, current network bandwidth and terminal cache hit rate in real time to form a state space;

[0075] S502 designs a multi-dimensional reward function that integrates end-to-end latency, system energy consumption, and semantic relevance strength to calculate instantaneous reward values ​​to drive policy optimization.

[0076] S503 executes a dynamic scheduling strategy: the optimal action is selected through the Q-Learning algorithm, and the data transmission granularity and computation task migration are adjusted according to the network status; specifically, when the bandwidth is high, the complete semantic subtree is transmitted to the terminal; when the bandwidth is low, only basic action primitives are pushed, and the terminal generates transition animations through lightweight LSTM interpolation to maintain visual coherence.

[0077] Specifically, real-time collection of cloud load L cloud Edge load L edge Uplink bandwidth B uplink Downlink bandwidth B downlink and cache hit rate H cache Construct the state vector:

[0078] s t =[L cloud ,L edge B uplink H cache ] T

[0079] Calculate immediate rewards based on action execution results, incorporating end-to-end latency T. latency Energy consumption E energy and semantic relevance PMI * :

[0080]

[0081] Where ω1, ω2, and ω3 are adjustable weights.

[0082] High bandwidth scenarios (B downlink >τ high ): Transmit the complete semantic subtree (directly call the GST leaf node to bind the animation package ΔA);

[0083] Low bandwidth scenarios (B downlink <τ low ): Push basic action primitives (root node data), and the terminal generates transition animations through LSTM interpolation: Where 0 < β < 1, it is used to control the weight between the original motion parameters of the current frame and the smoothing result of the previous frame; A t These are the basic action primitive parameters for the current time step t, i.e., the basic action parameter vector for the current time step t. This is the interpolated output of the previous frame (time t-1); This is the interpolated output for the current frame (time t).

[0084] The above-described real-time speech-to-sign language conversion and synthesis method based on collaborative computing, through a cloud-edge-device collaborative architecture, effectively solves the technical problems faced by existing sign language synthesis systems on wearable devices, such as poor real-time performance, stiff movements, and strong network dependence. This system fully utilizes the powerful computing capabilities of the cloud, the low latency advantages of edge nodes, and the local rendering capabilities of terminal devices to achieve efficient, coherent, and real-time speech-to-sign language conversion. Furthermore, through a reinforcement learning dynamic scheduling mechanism, it can intelligently adapt to network fluctuations and load changes, optimizing resource allocation and data transmission strategies, further improving the system's performance and stability.

[0085] It should be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A real-time speech-to-sign language conversion and synthesis method based on collaborative computing, characterized in that, The following steps are involved: S10, Voice Acquisition and Preprocessing: Acquires voice signals through wearable devices, performs noise suppression and feature extraction, and generates compressed voice packets; An improved sparse coding algorithm is used to compress data volume to reduce transmission load; S20, cloud-based semantic coding, uses a deep bidirectional transformer model to model the long-range dependencies of speech signals, extracts high-order semantic vectors through a self-attention mechanism, generates compact semantic representations, and combines paralinguistic features for semantic enhancement. S30, edge node action mapping, deploys a spatiotemporal convolutional network ST-CNN, whose dual-branch structure parses syntactic-level action trajectories and refines hand micro-movement features respectively; residual connectivity and historical gradient moving average filtering ensure action continuity; biomechanical constraint modules are applied, including joint angle thresholds and motion smoothing terms, to ensure that the action conforms to the laws of human kinematics. S40, terminal skeletal rendering, organizes skeletal animation data based on the Global Semantic Tree (GST) caching architecture. The root node stores basic action primitives, the intermediate nodes encode composite semantic clusters, and the leaf nodes bind complete animation packages. It calculates relevant word cluster prefetch data by improving point mutual information (PMI) and only transmits differences in action parameters between nodes. It adopts a lightweight skeleton topology structure to dynamically adjust the accuracy and combines Bloom filter indexing to reduce computational complexity. It loads animation primitive nodes from the local machine and renders sign language output. S50, reinforcement learning dynamic scheduling, constructs a system state vector, which integrates node computing and cloud load L. cloud Edge node load L edge Real-time uplink bandwidth B uplink Real-time downlink bandwidth B downlink and cache hit rate H cache Maximize the reward function through the Q-Learning strategy; adopt dynamic decision-making for activation state and data transmission granularity.

2. The real-time speech-to-sign language conversion and synthesis method based on collaborative computing as described in claim 1, characterized in that, The cloud-based semantic encoding specifically includes: S201, text semantic encoding, inputs the transformed text S into the BERT model, extracts the last hidden state H = BERT(S); takes the corresponding vector of the label as the basic semantic vector E. s ; S202, Paralinguistic feature extraction, extracts speech rate r, intonation p, and context c from the speech signal to form a paralinguistic feature vector P = [r, p, c]; S203, Semantic Vector Fusion, generating enhanced semantic vectors through weighted fusion: E = E s +λP, where λ is the weight coefficient of the secondary language features; S204, Vector dimension compression. Input the enhanced semantic vector E into the adaptive quantization module to output a compressed vector (Dimension D′, where D′ < D), where D represents the initial dimension of the enhanced semantic vector E, and D′ refers to the dimension of the output vector after compression by the adaptive module .

3. The real-time speech-to-sign language conversion and synthesis method based on collaborative computing as described in claim 2, characterized in that, The spatiotemporal convolutional network ST-CNN includes: S301, dual-branch feature processing, main branch parses syntactic-level action trajectories F g The auxiliary branch receives the same input and refines the hand micro-movement features F. l ; S302, Feature Fusion and Action Generation, Weighted Fusion of Global and Local Features as F f =αF g +(1-α)F l (0.5 < α < 0.7), output action parameter matrix via residual connection S303, biomechanical constraint, joint angle threshold constraint is θ. min ≤θ t ≤θ max Motion smoothing constraint is 4. The real-time speech-to-sign language conversion and synthesis method based on collaborative computing as described in claim 1, characterized in that, The terminal skeleton rendering specifically includes: S401 constructs a global semantic tree (GST) cache structure, with the root node storing basic action primitives, the intermediate nodes encoding composite semantic clusters, and the leaf nodes binding complete context animation packages, organizing skeletal animation data according to semantic relevance. S402, perform improved point mutual information (PMI) prefetch calculation. When the target word is identified, the relevant word clusters are calculated through the PMI algorithm, and animation data is prefetched along the vertical direction of the tree structure, i.e. the parent-child node direction, and the horizontal direction, i.e. the sibling node direction. S403 transmits differentiated action parameters, only transmitting action parameter data that differs between the current node and the prefetched node, reducing the amount of data transmitted; S404 dynamically optimizes skeleton rendering by using a lightweight skeleton topology structure combined with Bloom filter indexing to update the skeleton chain. It dynamically adjusts motion parameters based on a pre-trained inverse kinematics model, reducing the rendering computational complexity from quadratic to logarithmic.

5. The real-time speech-to-sign language conversion and synthesis method based on collaborative computing as described in claim 1, characterized in that, The reinforcement learning scheduling specifically includes: S501 constructs a system state vector, and collects the edge node computing load, current network bandwidth and terminal cache hit rate in real time to form a state space; S502 designs a multi-dimensional reward function that integrates end-to-end latency, system energy consumption, and semantic relevance strength to calculate instantaneous reward values ​​to drive policy optimization. S503 executes a dynamic scheduling strategy: the optimal action is selected through the Q-Learning algorithm, and the data transmission granularity and computation task migration are adjusted according to the network status; specifically, when the bandwidth is high, the complete semantic subtree is transmitted to the terminal; when the bandwidth is low, only basic action primitives are pushed, and the terminal generates transition animations through lightweight LSTM interpolation to maintain visual coherence.

Images