Sign Language Recognition Method Based on Multi-Level State Feature Optimization

Abstract

This invention relates to the fields of computer vision and artificial intelligence, and discloses a sign language recognition method based on multi-level state feature optimization. The method includes acquiring a sign language video sequence and extracting original visual features; constructing a semantic potential energy field, calculating the potential energy value of the current frame features and their gradient vector relative to the feature space; constructing a local orthogonal basis based on the gradient vector, and orthogonally decoupling the features into normal transition components along the gradient direction and tangential steady-state components along the equipotential surface direction; performing differential evolution processing on the normal transition components based on multi-level dynamic states, using graph networks to repair transition features, and smoothing the tangential steady-state components; finally, reconstructing features and temporally decoding the processed components to obtain the sign language text. This invention utilizes the geometric properties of the potential energy field to achieve physical-level decoupling between core semantics and transitional actions, effectively solving the feature drift problem caused by action switching in continuous sign language, and significantly improving recognition accuracy.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically to a sign language recognition method based on multi-level state feature optimization. Background Technology

[0002] Continuous sign language recognition aims to convert continuous sign language video sequences into natural language text, a key technology for enabling barrier-free communication between the hearing-impaired and hearing-speaking communities. With the development of deep learning, architectures based on convolutional neural networks to extract spatiotemporal features and combined with sequence models for temporal modeling have become mainstream solutions, improving recognition performance to some extent. However, in practical applications, continuous sign language recognition technology still faces many challenges, the most critical being the ambiguity of semantic boundaries caused by the continuity of sign language movements.

[0003] Unlike speech signals where there are clear pauses between words, continuous sign language expression is a coherent fluid motion process. Between two core sign language words with clear semantics, there must be a transitional movement to change hand shape or position; this phenomenon is called "coarticulation" or "motion insertion" in linguistics. In current technological systems, deep neural network models typically treat each frame in a video as an equally important input unit, failing to effectively distinguish between stable frames carrying core semantics and transitional frames that only serve a connecting function. This leads to a large number of transitional frame features being mixed into the semantic space during the feature extraction stage. These features belong neither to the preceding nor following word, but rather present a semantically ambiguous mixed state.

[0004] More seriously, existing feature extraction methods often map all attributes of sign language actions into the same high-dimensional feature space for entangled representation. In this single feature space, rapid changes in action are coupled with the fine morphology of hand shapes. When sign language users make rapid inter-word transitions, the drastic movement changes often mask subtle hand shape features, causing feature vectors to drift in the manifold space. This feature drift makes it difficult for models to accurately capture the moment of semantic transition, easily misidentifying transitional actions as additional words, or missing key short words due to noise interference during the transition phase. Traditional methods attempt to alleviate this problem by introducing temporal attention mechanisms or boundary detection modules, but due to the lack of deep physical modeling of the feature manifold geometry, they cannot fundamentally separate "changing the trend of action" from "preserving the details of action," thus making it difficult to achieve high-precision semantic decoding in complex and ever-changing continuous sign language scenarios. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a sign language recognition method based on multi-level state feature optimization. This method solves the technical problem in existing continuous sign language recognition technologies where the core semantic actions and transitional actions become entangled in the feature space, leading to blurred semantic boundaries and feature drift, which in turn causes recognition errors.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a sign language recognition method based on multi-level state feature optimization, the method comprising the following steps:

[0007] A sign language video sequence is acquired, and the original visual features of each frame in the sign language video sequence are extracted using a deep neural network to form an original visual feature sequence.

[0008] A semantic potential field is constructed, and the potential value of the current frame feature in the original visual feature sequence is calculated using a potential energy mapping network. The gradient vector of the semantic potential field relative to the feature space is also calculated. The potential energy mapping network is configured to learn the stable distribution of the feature space manifold.

[0009] Based on the gradient vector, a local orthogonal basis is constructed, and the current frame features are orthogonally decoupled into normal transition components along the gradient direction and tangential steady-state components along the equipotential surface direction.

[0010] The normal transition component is subjected to differential evolution processing based on multi-level dynamic states, and the tangential steady-state component is smoothed.

[0011] The processed normal transition components and the processed tangential steady-state components are used to reconstruct features, and the reconstructed feature sequence is input into a temporal decoder to obtain the sign language text result.

[0012] Furthermore, in the step of constructing the semantic potential field, the calculation of the potential value is performed by using the original visual features. The input is a nonlinear mapping network consisting of a multilayer perceptron (MLP). The output is a scalar normalized to a preset interval. The potential energy value is used to quantify the uncertainty of the current frame features on the semantic manifold; a low potential energy value represents a stable state of the semantic core, while a high potential energy value represents a transitional unstable state. The gradient vector... It is obtained by calculating the first-order partial derivative of the potential energy value with respect to the original visual features, that is:

[0013] ;

[0014] The gradient vector is obtained using the backpropagation mechanism and indicates the direction in which semantic uncertainty increases the fastest in the feature space, thus providing geometric guidance for feature decoupling.

[0015] Furthermore, in the orthogonal decoupling step, the gradient vector is first normalized to obtain the unit normal vector. :

[0016] ;

[0017] in To prevent division by zero by small constants, the current frame features are then... Projecting onto the unit normal vector direction, the normal transition component is extracted. The components contain the dynamic trends of sign language actions switching between semantic categories:

[0018] ;

[0019] Simultaneously, the difference between the current frame feature and the normal transition component is calculated to obtain the tangential steady-state component. The components contain morphological details that preserve the current semantic state:

[0020] ;

[0021] This step, based on the geometric properties of the potential field, achieves strict geometric decoupling between the direction of semantic change and the direction of semantic preservation.

[0022] Furthermore, in the dynamic differential evolution processing step, a multi-level dynamic state including velocity and acceleration is constructed for the normal transition component rich in dynamic information. The velocity characteristics are obtained by calculating the first-order time difference of the normal transition component. The acceleration features are obtained by calculating the first-order time difference of the velocity features. :

[0023] ;

[0024] ;

[0025] The normal transition component, the velocity feature, and the acceleration feature are concatenated to form a multi-level dynamic state vector. .

[0026] Furthermore, the differential evolution processing also includes constructing a dynamic graph network containing time nodes, and using graph convolution operations to perform temporal evolution updates on the multi-level dynamic state vectors. During the update process, the potential energy difference between nodes in the dynamic graph network is calculated, and attention coefficients are calculated based on the potential energy difference. These attention coefficients are used to enhance the feature correction weights of low-potential nodes on high-potential nodes during the aggregation process, enabling frames in transitional unsteady states to utilize the dynamic inertia of adjacent frames in semantically stable states for trajectory repair. The corrected normal transition components... Extracted from the evolved node features.

[0027] Furthermore, in the feature reconstruction step, a learnable linear weighting matrix is ​​used. The processed normal transition components With the processed tangential steady-state component The features are then fused and mapped back to the original feature dimensions to obtain the reconstructed feature sequence. :

[0028] The temporal decoder uses the Connection Temporal Classification (CTC) algorithm to decode the reconstructed feature sequence and output the final predicted sequence.

[0029] Furthermore, the method of the present invention employs a joint loss function for end-to-end optimization during the model training phase. This joint loss function is configured to simultaneously guide semantic recognition accuracy, potential field distribution morphology, and the independence of feature decoupling. The joint loss function includes potential constraint loss, orthogonal independence loss, and connection-temporal classification loss.

[0030] The calculation of the potential energy constraint loss relies on pseudo-label supervision. Specifically, a pre-trained sign language recognition model is used to infer the input video and obtain the prediction confidence for each frame;

[0031] When the prediction confidence is higher than a preset threshold, a pseudo-label indicating low potential energy is generated.

[0032] When the prediction confidence is lower than a preset threshold, a pseudo-label indicating high potential energy is generated. The cross-entropy is calculated using the pseudo-label and the potential energy value output by the potential energy mapping network, thereby forcing the core semantic frame to converge towards the potential energy valley and the transition frame to diverge towards the potential energy peak.

[0033] The orthogonal independence loss is used to constrain the statistical independence of the decoupled features, preventing information leakage and re-coupling during the evolution process. The loss is calculated by taking the Frobenius norm of the cross-covariance matrix between the reconstructed normal transition component sequence and the tangential steady-state component sequence.

[0034] ;

[0035] in and These represent the normal and tangential characteristic matrices at the sequence level, respectively.

[0036] This invention provides a sign language recognition method based on multi-level state feature optimization. It has the following beneficial effects:

[0037] 1. This invention innovatively establishes a semantic stability metric on a feature manifold by constructing a semantic potential energy field and mapping high-dimensional visual features to scalar potential energy values. The mechanism utilizes the principle in physics that potential energy troughs correspond to steady states and potential energy peaks correspond to unsteady states, precisely quantifying the semantic uncertainty of each frame of the image. This enables the model to automatically distinguish between stable frames carrying core word meanings and transitional frames that only serve a connecting function, effectively suppressing the noise interference from transitional actions commonly found in continuous sign language, and fundamentally improving the purity of feature extraction.

[0038] 2. This invention utilizes the potential energy field gradient to construct a locally orthogonal basis and implements a feature orthogonal decoupling strategy based on geometric properties. By decomposing the original features into normal transition components along the gradient direction and tangential steady-state components along the tangential plane direction, the physical separation of "movement changing trend" and "movement retaining details" is successfully achieved. This decoupling process breaks the deadlock of the entanglement between dynamic motion information and static morphological information in traditional methods, ensuring that the subtle morphological features of the hand are not masked by large-scale motion signals when dramatic inter-word transitions occur, thereby significantly enhancing the recognition ability of complex sign language vocabulary.

[0039] 3. This invention designs a multi-level dynamic differential evolution mechanism for the normal transition component, combining first-order velocity and second-order acceleration features to perform temporal restoration of the motion trajectory. This scheme overcomes the deficiency of single features easily losing spatiotemporal context during high-speed motion, and utilizes the principle of dynamic inertia to fill semantic gaps. Even when video frames are severely distorted due to motion blur, the system can still infer a reasonable action transformation path based on the evolved dynamic state vector, ensuring the continuity and integrity of semantic understanding in the temporal dimension.

[0040] 4. This invention introduces an attention guidance mechanism based on potential energy difference into the graph neural network, establishing a directional information channel flowing from low-potential energy nodes to high-potential energy nodes. This design aligns with the logic of information entropy reduction, forcibly utilizing semantically clear core frames to guide and correct semantically ambiguous transition frames. This context-dependent repair strategy enables the model to effectively eliminate ambiguity in intermediate transition segments when processing long and complex sign language sentences, based on the deterministic information of the preceding and following context, significantly reducing the incidence of insertion and substitution errors.

[0041] 5. This invention introduces an orthogonal independence loss function to statistically force the two subspaces after constraint decomposition to remain uncorrelated. This not only guarantees the orthogonality of features in geometric projection but also prevents information leakage and re-coupling between dynamic and morphological features in data distribution. This deep constraint optimization enables the model to adaptively learn more robust and interpretable feature representations during end-to-end training, significantly improving the system's generalization performance under different sign language styles. Attached Figure Description

[0042] Figure 1 This is an overall flowchart of the sign language recognition method of the present invention;

[0043] Figure 2 This is a schematic diagram of the overall model architecture and data flow of the present invention;

[0044] Figure 3 This is a schematic diagram of the semantic potential field and orthogonal decoupling geometry principle of the present invention. Detailed Implementation

[0045] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Please see the appendix Figure 1 Appendix Figure 3 This invention provides a sign language recognition method based on multi-level state feature optimization. The method runs in an electronic device, which includes a processor and a memory. The memory stores computer program instructions, and the processor executes the computer program instructions to implement the method. The method aims to solve the problems of semantic boundary ambiguity and feature drift caused by transitional movements in continuous sign language recognition. It constructs a semantic potential energy field at the physical level and uses the gradient properties in field theory to perform geometric decoupling and dynamic repair of the feature manifold.

[0047] The sign language recognition system based on multi-level state feature optimization includes a feature extraction module, a potential energy field construction module, an orthogonal decoupling module, a dynamic evolution module, and a decoding module. These modules work collaboratively to achieve end-to-end processing from raw video input to semantic text output.

[0048] The sign language recognition method based on multi-level state feature optimization includes the following steps:

[0049] S1: Multimodal Basic Feature Extraction The feature extraction module receives a sign language video sequence as input and uses a deep neural network to extract the original visual features of each frame in the sign language video sequence to form an original visual feature sequence.

[0050] Specifically, for the input sign language video ,in Indicates the first Frame image, Given the total sequence length, the feature extraction module uses a pre-trained spatiotemporal convolutional neural network as the backbone encoder to map the high-dimensional image space to a low-dimensional feature space, outputting the original visual feature sequence. ,in For the first time step 3D feature vector. The feature vector contains spatial semantic information about hand shape, facial expression, and limb position.

[0051] S2: Semantic Potential Field Construction and Gradient Estimation. The potential field construction module builds a semantic potential field based on the original visual feature sequence. This module does not directly perform classification prediction, but instead uses a potential energy mapping network to calculate the features of the current frame. potential energy value Potential energy value It is a scalar used to quantify the uncertainty state of the current feature point on the semantic manifold. Define the potential energy mapping function. Then we have:

[0052]

[0053] Among them, potential energy value The distribution characteristics are configured such that when a feature is in a core stable state with a clear semantic meaning, It tends towards a minimum value; when the feature is in a semantically ambiguous transitional non-stationary state, It tends towards the maximum value.

[0054] Building upon this, the potential field construction module further calculates the gradient vector of the semantic potential field relative to the feature space. The gradient vector is obtained by calculating the first-order partial derivative of the potential value with respect to the original visual features:

[0055]

[0056] gradient vector Geometrically, it indicates the direction in which semantic uncertainty increases the fastest in the feature space; that is, the direction in which semantic transition occurs and the current semantic state is broken.

[0057] S3: Equipotential surface-based orthogonal decoupling; the orthogonal decoupling module receives the gradient vector. and original visual features Based on the gradient vector, a local orthogonal basis is constructed to orthogonally decouple the current frame features into two independent components: the normal transition component along the gradient direction and the tangential steady-state component along the equipotential surface direction. First, the gradient vector is normalized to obtain the unit normal vector. :

[0058]

[0059] in To prevent division by zero of small constants.

[0060] Then, the current frame features Projecting onto the unit normal vector direction yields the normal transition component. :

[0061]

[0062] Normal transition components This characterizes the dynamic trend of sign language actions switching between semantic categories, i.e., changing the feature components of the action. Simultaneously, the difference between the current frame features and the normal transition components is calculated to obtain the tangential steady-state components. :

[0063]

[0064] Tangential steady-state components This step represents the morphological details while maintaining the current semantic state, i.e., the feature components of "maintaining action". Through this step, the motion ambiguity information mixed in the original features is strictly separated from the static detail information in geometric space.

[0065] S4: Multi-level dynamic differential evolution processing. The dynamic evolution module performs targeted processing on the two components obtained from the decomposition. This is specifically for the normal transition component. Because it carries the main motion transformation information and is susceptible to transient noise, the module performs differential evolution processing on it based on multi-level dynamic states. Specifically, this includes calculating first-order velocity and second-order acceleration features, constructing a multi-level dynamic state vector containing position, velocity, and acceleration, and using a graph neural network combined with a potential energy difference-guided attention mechanism to perform temporal restoration of the feature trajectory, obtaining the corrected normal transition components. For the tangential steady-state component Since it mainly carries stable detailed information, the module performs smoothing and noise reduction processing on it to obtain the processed tangential steady-state component. The divide-and-conquer strategy allows the system to preserve subtle differences in sign language expression while suppressing interference from excessive actions.

[0066] S5: Manifold Reconstruction and Timing Decoding. The decoding module processes the normal transition components. With the processed tangential steady-state component Feature reconstruction is performed. The two components are fused and mapped back to the original feature dimensions using a learnable linear transformation, resulting in a reconstructed feature sequence. :

[0067]

[0068] in and The parameters are then used for reconstruction. Finally, the decoding module inputs the reconstructed feature sequence into the temporal decoder, calculates the conditional probability of each frame belonging to each sign language word, and outputs the final sign language text recognition result. This process, based on the reconstructed high signal-to-noise ratio features, can significantly improve the accuracy of recognizing blurred boundaries in continuous sign language.

[0069] In this embodiment, after obtaining the original visual feature sequence, to address the difficulty in distinguishing between core and transitional actions in sign language in existing technologies, a semantic potential field is first constructed to explicitly model the topological structure of the feature manifold. The process does not directly classify the features, but rather learns a scalar field function to measure the stability of feature points in the semantic space. Specifically, a potential energy mapping network is used to process the current frame features in the original visual feature sequence. The potential energy mapping network is configured as a multilayer perceptron structure consisting of fully connected layers, which can map high-dimensional feature vectors to one-dimensional potential energy scalars.

[0070] In this invention, the input current frame feature is assumed to be... The potential energy value output by the potential energy mapping network is The mathematical expression for the mapping process is shown below:

[0071]

[0072] in, and Here is the learnable weight matrix for the potential energy mapping network. and For bias vectors, ReLU activation function is chosen to enhance the nonlinear expressive power of the model. Using the Sigmoid activation function, the output potential energy value The constraint is within a closed interval between 0 and 1.

[0073] Under this definition, the potential energy value It has a clear dual meaning, both physical and semantic: low potential energy states with values ​​approaching 0 correspond to keyframes of sign language movements, that is, the core region where the semantics are clear and the movements are stable, equivalent to the "valley" in the potential energy field; high potential energy states with values ​​approaching 1 correspond to transition frames between sign language movements, that is, the non-steady-state region where the semantics are ambiguous and the movements undergo drastic changes, equivalent to the "peak" in the potential energy field. Through this mapping, a continuous sequence of sign language movements is transformed into a motion trajectory on the potential energy surface.

[0074] To further capture the directional information of action evolution, this invention calculates the gradient vector of the semantic potential field relative to the feature space. The gradient vector indicates the direction of the fastest increase in potential energy at the current feature point, and is geometrically perpendicular to the equipotential surface passing through the point. The calculation formula is as follows:

[0075]

[0076] in, This represents the gradient vector. During the calculation, the potential energy value is calculated using an automatic differentiation mechanism. Relative to input features The first-order partial derivative. Gradient vector. The direction is the direction in which semantic uncertainty grows most dramatically, which means that the direction is the "fastest path" for a feature to leave the current semantic category and leap to the next semantic category.

[0077] Based on the aforementioned gradient information, this embodiment further implements feature orthogonal decoupling based on equipotential surfaces. The step utilizes the geometric properties of the potential energy field to construct a locally orthogonal coordinate system, decomposing the original feature into two geometrically orthogonal and semantically functionally distinct components. First, the gradient vector... conduct Norm normalization yields the unit normal vector. :

[0078]

[0079] in, This is a very small constant used to prevent division operations from malfunctioning due to the gradient magnitude being zero. Unit normal vector. The normal direction of the local feature space is defined.

[0080] Then, the current frame features Projected onto unit normal vector The indicated direction is used to extract the normal transition component. The formula for calculating the components is:

[0081]

[0082] Normal transition components The component that changes along the potential energy gradient direction in the feature vector was captured. In the context of sign language, the component represents the switching and transition of action states, such as large-scale movement of hand position or fundamental change in hand shape. It is the main factor leading to the ambiguity of semantic boundaries, and also a key clue for recognizing action switching.

[0083] At the same time, calculate the features of the current frame. With normal transition components The vector difference between them yields the tangential steady-state component. The formula for calculating the components is:

[0084]

[0085] Tangential steady-state components It lies within the tangent plane of the equipotential surface perpendicular to the gradient direction. Since the potential energy remains constant when moving along the equipotential surface, the component represents subtle changes within the feature vector while maintaining the current semantic state, such as stylistic differences between different sign language speakers for the same sign language word, minute tremors of hand details, or redundancy of background information.

[0086] Through the above orthogonal decoupling operations, the present invention successfully decouples the original features. Decomposed into and The two satisfy the orthogonality condition, that is... This decoupling mechanism, which combines physics and geometry, allows subsequent processing modules to adopt customized optimization strategies for the two distinct dynamic characteristics of state change and state maintenance. This avoids the information confusion and interference caused by homogenizing all feature dimensions in traditional methods.

[0087] In this embodiment, differentiated evolution strategies are adopted for the normal transition component and the tangential steady-state component obtained through orthogonal decoupling to optimize feature representation. Considering that the normal transition component mainly carries the dynamic information of the sign language action switching between different semantic states, and that the component is highly susceptible to motion blur during the transition phase, resulting in trajectory breakage, a multi-level dynamic differential evolution mechanism is introduced to perform temporal repair.

[0088] Specifically, this invention first calculates the first-order and second-order differences of the normal transition components in the temporal dimension to explicitly model the velocity and acceleration properties of the action. Let... The normal transition component at time t is Then the velocity characteristics With acceleration characteristics The calculation formulas are as follows:

[0089]

[0090]

[0091] Based on this, the normal transition components, velocity characteristics, and acceleration characteristics are concatenated along the channel dimension to construct a multi-level dynamic state vector for the current moment. :

[0092]

[0093] State vector It not only includes the current location information, but also incorporates the inertial trend of action evolution, providing a physical basis for traversing semantically ambiguous areas.

[0094] To utilize contextual information to repair the feature trajectory in the high-potential transition region, this embodiment constructs a dynamic graph network with time steps as nodes. In the graph network, the node features are the aforementioned multi-level dynamic state vectors. To achieve physically guided feature aggregation, this invention designs an attention mechanism based on potential energy difference. The core logic of the mechanism is to use nodes with low potential energy to correct nodes with high potential energy, that is, to use the precise semantic information of the core frame to fill the information gaps in the transition frame.

[0095] Define nodes With nodes Attention coefficient between The coefficient is determined by the difference in potential energy between the two nodes. The calculation formula is as follows:

[0096]

[0097] in, Represents a node Temporal neighborhood, To control the hyperparameters of the decay rate, For the indicator function, ensure that only neighboring nodes The potential energy is lower than that of the current node. Significant aggregation weights are only assigned when the flow is at a steady state. This means that information mainly flows from a steady state to a non-steady state, which aligns with the principles of energy dissipation and steady-state attraction in physics.

[0098] Based on the aforementioned attention coefficients, graph convolution operations are used to evolve and update the multi-level dynamic state vector. The update formula is as follows:

[0099] in, To evolve the weight matrix of the graph, The activation function is derived from the evolved and updated state vector. In the middle, the corrected normal transition component is separated. Meanwhile, for the tangential steady-state component... Since it represents relatively stable morphological details, this embodiment uses residual smoothing connection for processing to obtain the processed tangential steady-state component. This is to filter out high-frequency random noise and maintain the coherence of detailed features.

[0100] After completing targeted component optimization, this invention performs manifold reconstruction and temporal decoding steps. Using a learnable linear fusion layer, the corrected normal transition components and the processed tangential steady-state components are mapped back to the original feature dimensions, achieving a reunification of geometric components. The feature sequence is then reconstructed. The calculation is as follows:

[0101]

[0102] in, To reconstruct the weight matrix, This is the bias term. The reconstruction process is not a simple addition, but rather an adaptive weighted fusion of two orthogonal subspaces achieved through parameter learning, ensuring that the synthesized features have both clear semantic boundaries and retain rich morphological details.

[0103] Finally, the feature sequence will be reconstructed. The input temporal decoder consists of a bidirectional long short-term memory network and a connection-based temporal classification layer. The BiLSTM network performs contextual temporal modeling on the feature sequence, outputting the probability distribution of each frame corresponding to a predefined sign language vocabulary. The CTC layer uses a dynamic programming algorithm to calculate path probabilities, decoding the sign language text sequence corresponding to the maximum posterior probability without requiring frame-level alignment labels. Through these multi-level state feature optimizations, the system significantly reduces the interference of transitional actions on the recognition results, improving the accuracy of continuous sign language recognition.

[0104] In this embodiment, to achieve end-to-end parameter optimization of the aforementioned neural network model, a joint loss function is designed. This function collaboratively guides the semantic recognition accuracy, potential field distribution, and the independence of feature decoupling. The construction of the potential field relies on effective supervision signals; however, sign language video data typically only contains sentence-level text labels, lacking frame-level potential field annotations. Therefore, this invention employs a confidence-based pseudo-label generation strategy. Specifically, a pre-trained sign language recognition model is used to infer the training set videos, obtaining the probability distribution of each frame belonging to each category, and the maximum probability value is used as the frame's prediction confidence.

[0105] Based on the predicted confidence level, binary pseudo-labels are generated for supervising the potential energy mapping network. High-confidence and low-confidence thresholds are set. When the predicted confidence level of a frame is higher than the high-confidence threshold, the frame is determined to be in a semantically clear core stable state, and its corresponding potential energy pseudo-label is set to 0. When the predicted confidence level of a frame is lower than the low-confidence threshold, the frame is determined to be in a semantically ambiguous transitional unstable state, and its corresponding potential energy pseudo-label is set to 1. Frames falling between these two thresholds are not included in the potential energy loss calculation. Based on this, a potential energy constraint loss is defined. The binary cross-entropy function is used to measure the difference between the predicted potential value and the pseudo-label:

[0106]

[0107] in, The set of valid frames that meet the threshold condition. Number of valid frames For the first The potential energy pseudo-label of the frame, This represents the predicted potential energy value output by the potential energy mapping network. By minimizing the loss, the network is forced to push core semantic frames to the potential energy trough and transition frames to the potential energy peak, thereby forming a semantic potential energy field with clear physical meaning.

[0108] In this invention, to ensure that the normal transition component and the tangential steady-state component separated by the orthogonal decoupling module remain statistically independent and to prevent information leakage or re-coupling of features during subsequent dynamic evolution, an orthogonal independence loss is introduced. The loss is achieved by minimizing the Frobenius norm of the cross-covariance matrix between the two component sequences. Let the reconstructed normal transition characteristic matrix be... The tangential steady-state characteristic matrix is The orthogonal independence loss is calculated as follows:

[0109]

[0110] in, and These are the mean vectors of the normal eigenma matrix and the tangential eigenma matrix, respectively. Let Frobenius norm be denoted. The constraint forces the feature distributions in the two subspaces to be not only geometrically orthogonal but also statistically close to zero, thus ensuring that the dynamic repair only acts on the dimension related to action changes and does not interfere with the static detail dimension.

[0111] Furthermore, this embodiment employs connection-time classification loss. The loss function, used as the primary task loss, maximizes the alignment probability between the reconstructed feature sequence and the actual sign language text labels. Ultimately, the model's overall optimization objective function... Defined as the weighted sum of the three losses mentioned above:

[0112]

[0113] in, and To balance the hyperparameters of the various loss weights, a backpropagation algorithm is used during training to calculate the gradient of the total loss function with respect to the network parameters. Stochastic gradient descent or the Adam optimizer is then used to update all learnable parameters of the feature extraction module, potential mapping network, dynamics evolution module, and decoding module. Through this joint optimization strategy, the model can adaptively learn how to separate and repair dynamic trajectories in complex sign language manifolds, ultimately achieving high-precision continuous sign language recognition.

[0114] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A sign language recognition method based on multi-level state feature optimization, characterized in that, Includes the following steps: Step S1: Obtain the sign language video sequence and use a deep neural network to extract the original visual features of each frame in the sign language video sequence to form an original visual feature sequence; Step S2: Construct a semantic potential energy field, use a potential energy mapping network to calculate the potential energy value of the current frame feature in the original visual feature sequence, and calculate the gradient vector of the semantic potential energy field relative to the feature space. Step S3: Construct a local orthogonal basis based on the gradient vector, and orthogonally decouple the current frame features into normal transition components along the gradient direction and tangential steady-state components along the equipotential surface direction; Step S4: Perform differential evolution processing based on multi-level dynamic states on the normal transition component, and smooth the tangential steady-state component; Step S5: Reconstruct the features of the processed normal transition components and the processed tangential steady-state components, and input the reconstructed feature sequence into the temporal decoder to obtain the sign language text result.

2. The method according to claim 1, characterized in that, In step S2, calculating the potential energy value of the current frame feature in the original visual feature sequence includes: The original visual features are input into a nonlinear mapping network, and the output is a scalar normalized to a preset interval as the potential energy value. The potential energy value is used to characterize the probability that the current frame features are in a semantic core stable state or a transitional non-stable state. The calculation of the gradient vector of the semantic potential field with respect to the feature space is to calculate the first-order partial derivative of the potential value with respect to the original visual feature.

3. The method according to claim 1, characterized in that, In step S3, the step of constructing a local orthogonal basis based on the gradient vector to orthogonally decouple the current frame features includes: The gradient vector is normalized to obtain the unit normal vector; The projection of the current frame feature onto the direction of the unit normal vector is calculated to obtain the normal transition component, which represents the dynamic trend of sign language actions switching between semantic categories. The difference between the current frame feature and the normal transition component is calculated to obtain the tangential steady-state component, which represents the morphological details that maintain the current semantic state.

4. The method according to claim 1, characterized in that, In step S4, the differential evolution processing of the normal transition component based on multi-level dynamic states includes: The velocity characteristics are obtained by calculating the first-order time difference of the normal transition component. The acceleration characteristics are obtained by calculating the first-order time difference of the velocity characteristics; The normal transition component, the velocity feature, and the acceleration feature are concatenated to form a multi-level dynamic state vector.

5. The method according to claim 4, characterized in that, The differential evolution processing of the normal transition component based on multi-level dynamic states further includes: Construct a dynamic graph network containing time nodes, and use the multi-level dynamic state vectors as node features; The node features are updated temporally using graph convolution operations, and the corrected normal transition components are extracted from the updated node features.

6. The method according to claim 5, characterized in that, The step of using graph convolution operations to perform temporal evolution updates on the node features includes: Calculate the potential energy difference between nodes in a dynamic graph network; The attention coefficients are calculated based on the potential energy difference. These attention coefficients are used to enhance the feature correction weights of low-potential nodes on high-potential nodes during graph convolution aggregation, so that frames in transitional unsteady states can use the dynamic inertia of adjacent frames in semantic core stable states for trajectory repair.

7. The method according to claim 1, characterized in that, In step S5, the feature reconstruction of the processed normal transition component and the processed tangential steady-state component includes: By using a learnable linear weighting matrix, the processed normal transition components and the processed tangential steady-state components are added together and mapped back to the original feature dimension to obtain the reconstructed feature sequence. The temporal decoder uses a connection-time classification algorithm to calculate the conditional probability of the sign language text sequence corresponding to the reconstructed feature sequence.

8. The method according to claim 7, characterized in that, It also includes a model training step, wherein the model training step uses a joint loss function to perform end-to-end optimization of the deep neural network, the potential energy mapping network and the temporal decoder, wherein the joint loss function includes potential energy constraint loss; The potential energy constraint loss is calculated as follows: a binary pseudo-label is generated based on the prediction confidence of the pre-trained model. When the prediction confidence is higher than a preset threshold, the pseudo-label indicates low potential energy; when the prediction confidence is lower than the preset threshold, the pseudo-label indicates high potential energy; and the generation of the potential energy value is supervised by the binary pseudo-label.

9. The method according to claim 8, characterized in that, The joint loss function also includes orthogonal independence loss; The orthogonality independence loss is calculated as follows: the cross-covariance matrix between the reconstructed normal transition component sequence and the tangential steady-state component sequence is calculated, and the Frobenius norm of the cross-covariance matrix is ​​calculated to minimize the correlation between the normal transition component and the tangential steady-state component in the feature space.

10. The method according to claim 8, characterized in that, The joint loss function also includes a connection temporal classification loss, which is used to maximize the alignment probability between the reconstructed feature sequence and the real sign language text labels.

Images