An intelligent sign language translation method and system based on multi-modal fusion

Abstract

The application discloses a kind of intelligent sign language translation method and system based on multi-modal fusion, it is related to the field of barrier-free human-computer interaction, the method comprises: S1, four modalities synchronous timing acquisition, S2, data pre-processing and occlusion confidence grading repair, S3, cross-modal feature extraction, S4, timing segmentation window length adaptive adjustment, S5, multi-modal text translation.The application frame-by-frame occlusion confidence grading matching interpolation repair, adaptive compensation hand, facial modal occlusion missing feature, effectively overcome the problem of key point loss under complex scenes such as object occlusion, mask face covering, simultaneously based on hand skeleton motion gradient dynamic adjustment timing segmentation window length, give consideration to real-time inference speed and the timing integrity of complete sign language morpheme, and perform morpheme reordering aggregation to fragmented slice feature, solve the morpheme fragmentation problem caused by fixed window cutting.

Description

Technical Field

[0001] This invention relates to the field of barrier-free human-computer interaction technology, specifically to an intelligent sign language translation method and system based on multimodal fusion. Background Technology

[0002] Sign language is the core medium for daily communication among the hearing impaired. Intelligent sign language translation devices are the core tools for achieving barrier-free information and eliminating communication barriers among the hearing impaired. At present, sign language recognition and translation technology has developed from early static gesture template matching and single image classification to video temporal recognition solutions based on deep learning. However, in real-world usage scenarios, there are still many interference conditions, such as indoor low light, outdoor backlight, hand crossing and occlusion, blurred images at a distance, local sign language variations, and inconsistent speech speed. Existing technologies struggle to balance recognition accuracy and real-time translation speed in complex scenarios. Translation results often suffer from problems such as missing words, disordered word order, and loss of emotional and semantic meaning, making it difficult to meet the needs of hearing impaired individuals in real-world scenarios such as daily conversations, government affairs, and medical communication. Therefore, research on intelligent sign language translation methods and systems based on multimodal fusion is necessary.

[0003] Existing technologies meet certain requirements for research on intelligent sign language translation methods based on multimodal fusion, but they also have some shortcomings. For example, existing multimodal sign language translation methods simply stitch together RGB images, skeletal key points, and facial image features, ignoring the bidirectional constraint interaction mechanism between modalities. Facial expressions and head postures in sign language can correct the semantics of hand gestures. Shallow fusion cannot uncover the complementary relationships between modalities, often resulting in correct gesture recognition but loss of emotional semantics such as joy, anger, sorrow, doubt, and negation, leading to a one-sided interpretation of the translated text.

[0004] On the other hand, existing technologies for temporal processing mostly use fixed duration and fixed frame intervals to cut sign language videos, which cannot adapt to changes in the speed of sign language. When users perform fast short sign language or slow long sign language, the fixed slices will split the complete sign language morphemes, resulting in cross-frame segmentation of morphemes and semantic gaps in the context. Traditional temporal models have weak long-distance dependency capture capabilities, and long continuous sign language translations are prone to word order reversal and keyword loss. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent sign language translation method and system based on multimodal fusion, which solves the problems existing in the background technology.

[0006] To solve the above-mentioned technical problems, the first aspect of the present invention provides an intelligent sign language translation method based on multimodal fusion. The method includes: S1, four-modal synchronous temporal acquisition: acquiring hand image data, 3D skeletal key points of the hand, facial micro-expression pixel sequence and head posture data of the sign language user in real-time video stream, adding timestamps to achieve temporal alignment, and obtaining the original four-modal data of the sign language user.

[0007] S2. Data Preprocessing and Occlusion Confidence Grading Repair: The original four-modal data of the sign language speaker is preprocessed, the occlusion confidence of each modality is calculated frame by frame, the occlusion confidence level is divided according to the calculation results, and the original four-modal data of the sign language speaker is interpolated and repaired frame by frame to obtain the repaired four-modal data of the sign language speaker.

[0008] S3. Cross-modal feature extraction: Construct a sign language modal causal constraint matrix, filter cross-modal feature interactions without semantic dependence in the four-modal data after sign language repair, and generate sign language intermediate fusion features carrying modal causal constraints.

[0009] S4. Adaptive adjustment of temporal segmentation window length: The length of the temporal segmentation window is dynamically adjusted according to the real-time extracted hand skeletal motion gradient of the signer. The intermediate fusion features of the signer are segmented according to the length of the temporal segmentation window, and the segmented intermediate fusion feature set of the signer is aggregated according to morpheme reordering.

[0010] S5. Multimodal text translation: Decode and translate the reordered and aggregated sign language intermediate fusion feature set to generate natural language.

[0011] A second aspect of the present invention provides an intelligent sign language translation system based on multimodal fusion, the system comprising: a four-modal data acquisition module, a data preprocessing module, an occlusion confidence level repair module, a cross-modal feature extraction module, a temporal window segmentation module, a morpheme reordering and aggregation module, and a text translation module.

[0012] The four-modal data acquisition module is used to acquire hand image data, 3D skeletal key points of the hand, facial micro-expression pixel sequence, and head posture data from the real-time video stream of the sign language user.

[0013] The data preprocessing module is used to calculate the occlusion confidence of each modality frame by frame from the original four-modal data of the sign language speaker, and to classify the occlusion confidence level according to the calculation results.

[0014] The occlusion confidence level repair module is used to perform interpolation repair on a frame-by-frame basis on the signer's original four-modal data based on the occlusion confidence level of each frame.

[0015] The cross-modal feature extraction module is used to extract features from the repaired four-modal data to obtain sign language intermediary fusion features carrying modal causal constraints.

[0016] The temporal window segmentation module is used to dynamically adjust the length of the temporal segmentation window based on the real-time extracted hand skeletal motion gradient of the sign language user, and to segment the intermediate fusion features of the sign language user according to the length of the temporal segmentation window.

[0017] The morpheme reordering and aggregation module is used to aggregate the segmented sign language inter-fusion feature set according to morpheme reordering to obtain the reordered and aggregated sign language inter-fusion feature set.

[0018] The text translation module is used to decode and translate the reordered and aggregated sign language inter-fusion feature set to generate natural language.

[0019] The advantages of this invention compared to existing technologies are: I. This invention uses four-modal synchronous acquisition with timestamps to unify the multi-dimensional information of sign language users’ hand images, skeletal key points, facial micro-expressions and head postures with a unified temporal reference, avoiding semantic conflicts caused by multimodal asynchronous misalignment, and providing complete raw data support for high-precision recognition.

[0020] Second, the present invention uses frame-by-frame occlusion confidence hierarchical matching causal interpolation repair to adaptively compensate for the missing features of hand and face modal occlusion, effectively overcoming the problem of key point loss in complex scenarios such as object occlusion, limb crossing, and face mask covering, and improving the robustness of the system.

[0021] Third, this invention dynamically adjusts the length of the temporal segment window based on the motion gradient of the hand skeleton, taking into account both real-time inference speed and the temporal integrity of complete sign language morphemes, and performs morpheme reordering and aggregation on fragmented slice features, solving the problems of morpheme fragmentation and disordered word order caused by fixed window cutting. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0024] Figure 2 This is a system connection diagram of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Reference Figure 1 As shown, the first aspect of the present invention provides an intelligent sign language translation method based on multimodal fusion. The method includes: S1, four-modal synchronous temporal acquisition: acquiring hand image data, 3D skeletal key points of the hand, facial micro-expression pixel sequence and head posture data of the sign language user in real-time video stream, adding timestamps to achieve temporal alignment, and obtaining the original four-modal data of the sign language user.

[0027] S2. Data Preprocessing and Occlusion Confidence Grading Repair: The original four-modal data of the sign language speaker is preprocessed, the occlusion confidence of each modality is calculated frame by frame, the occlusion confidence level is divided according to the calculation results, and the original four-modal data of the sign language speaker is interpolated and repaired frame by frame to obtain the repaired four-modal data of the sign language speaker.

[0028] S3. Cross-modal feature extraction: Construct a sign language modal causal constraint matrix, filter cross-modal feature interactions without semantic dependence in the four-modal data after sign language repair, and generate sign language intermediate fusion features carrying modal causal constraints.

[0029] S4. Adaptive adjustment of temporal segmentation window length: The length of the temporal segmentation window is dynamically adjusted according to the real-time extracted hand skeletal motion gradient of the signer. The intermediate fusion features of the signer are segmented according to the length of the temporal segmentation window, and the segmented intermediate fusion feature set of the signer is aggregated according to morpheme reordering.

[0030] S5. Multimodal text translation: Decode and translate the reordered and aggregated sign language intermediate fusion feature set to generate natural language.

[0031] In a specific embodiment of the present invention, the method for obtaining the original four-modal data of the sign language interpreter is as follows: The color images of the sign language interpreter's hands are captured frame by frame in the real-time video stream, and the effective pixel RGB value matrix is ​​recorded as the sign language interpreter's hand image data.

[0032] Using a depth camera and 3D recognition, the 3D coordinates of 21 key points of the sign language person's hand skeleton are obtained frame by frame and recorded as the 3D key points of the sign language person's hand skeleton. The 21 key points of the hand skeleton are 1 wrist, 5 palm base, 15 fingertips, and 15 first and second joints.

[0033] The sign language interpreter's face is captured frame by frame in the real-time video stream, and the effective pixel RGB value matrix is ​​recorded as the sign language interpreter's facial micro-expression pixel sequence.

[0034] Using a depth camera and 3D recognition, the standard facial feature pixel key points detected in the image are used as the observation output frame by frame. Combined with a general 3D facial reference model, the spatial coordinates of the sign language interpreter's head and three rotation angles, including the pitch angle, yaw angle and roll angle, are obtained through the perspective point algorithm. These six sets of values ​​are recorded as the sign language interpreter's head posture data.

[0035] The hand image data, 3D skeletal key points of the hand, facial micro-expression pixel sequence, and head posture data of the sign language interpreter were collected and time-stamped according to the processing of each frame, and then time-series aligned to obtain the original four-modal data of the sign language interpreter.

[0036] This invention uses four-modal synchronous acquisition with timestamps to unify the temporal reference of multi-dimensional information such as the sign language user's hand images, skeletal key points, facial micro-expressions, and head posture, avoiding semantic conflicts caused by multimodal asynchronous misalignment and providing complete raw data support for high-precision recognition.

[0037] Specifically, in this embodiment of the invention, the hand image data is uniformly scaled and the pixel value range is normalized. After visual features are extracted by a convolutional encoder, they are mapped by a learnable projection layer to generate a hand image feature vector of length d for the sign language speaker.

[0038] For example, the raw simplified RGB data of a hand image is denoted as... After normalization and projection layer mapping, a hand image feature vector of length d is obtained for the sign language speaker. .

[0039] The 21 hand skeleton 3D coordinates of the signer are processed by wrist centering and scale normalization, flattened into a 63-dimensional coordinate sequence, and compressed and mapped by one-dimensional convolution and projection layer to output the signer's hand skeleton feature vector of length d.

[0040] For example, the three-dimensional coordinates of the 21 hand bones of a sign language speaker are: , Wait, after normalization and flattening, a unified four-dimensional skeleton vector is obtained after projection layer mapping. .

[0041] Specifically, in an embodiment of the present invention, the sign language speaker's facial image is uniformly scaled and the pixel value range is normalized. After visual features are extracted by a convolutional encoder, they are mapped through a learnable projection layer to generate a facial micro-expression feature vector of length d for the sign language speaker.

[0042] The signer's 6-dimensional translational and rotational head posture values ​​are normalized to intervals and mapped to the signer's head posture feature vector of dimension d using a single-layer learnable fully connected projection.

[0043] For example, the numerical values ​​of the sign language interpreter's 6-dimensional head translation and rotation posture are: After interval-normalized language projection mapping, a unified four-dimensional head pose feature vector of the sign language user is obtained. .

[0044] In a specific embodiment of the present invention, the specific analysis method for preprocessing the original four-modal data of sign language speakers is as follows: for the hand image data of sign language speakers, the ratio of effective pixels to theoretical hand pixels is recorded as the hand occlusion confidence level.

[0045] Specifically, in an embodiment of the present invention, for example, if the theoretical number of complete hand pixels is 10,000 and the actual number of effective hand pixels collected is 5,500, then the confidence level of hand occlusion is calculated to be 0.55.

[0046] For the 3D skeleton key points of the sign language user's hand, the percentage of valid and unlost key points of 21 3D skeleton key points of the hand was counted and recorded as the skeleton occlusion confidence.

[0047] Specifically, in an embodiment of the present invention, for example, if the number of valid and unlost key points of the 3D skeleton of the hand actually collected is 9, then the calculated confidence level of skeleton occlusion is 0.428.

[0048] For the facial micro-expression pixel sequence of sign language users, the proportion of effective pixels to standard face pixels is recorded as the facial occlusion confidence level.

[0049] For sign language users' head pose data, the ratio of the effective standard facial feature pixel key points detected by head pose detection to 68 complete key points is recorded as the head occlusion confidence.

[0050] A threshold is set to classify the occlusion confidence of the four modalities of the signer, including occlusion level and no occlusion level. The classification results are simultaneously sent to the interpolation and repair module to perform corresponding missing feature compensation processing.

[0051] Specifically, in an embodiment of the present invention, for example, the occlusion threshold is set to 0.7. If the occlusion confidence level is greater than or equal to 0.7, the occlusion confidence level is determined to be no occlusion level. If the occlusion confidence level is less than 0.7, the occlusion confidence level is determined to be occlusion level.

[0052] In a specific embodiment of the present invention, the specific analysis method for performing the corresponding missing feature compensation processing is as follows: a sign language modality causal dependency mapping library is established in advance, with the key points of the 3D skeleton of the hand as the main causal modality, and the hand image data, facial micro-expression pixel sequence and head posture data as secondary constraint modalities.

[0053] For the main causal modality, when the sign language interpreter's skeleton occlusion confidence is occlusion level, and at least one of the sign language interpreter's hand modality, facial modality, and head modality is unoccluded, constraint interpolation repair is triggered, and the three-dimensional coordinate sequence of historical intact hand 3D skeleton key points of a preset length before the current missing frame is extracted.

[0054] Extract the data corresponding to the unoccluded level modality in the current missing frame as a causal constraint vector, construct a causal interpolation loss function, and perform interpolation by fusing the three-dimensional coordinate sequence of historical intact hand 3D skeleton key points with a preset length before the missing frame and the causal constraint vector, and output the coordinates of the 21 completed hand 3D skeleton key points.

[0055] If the skeleton occlusion confidence is occlusion level, and the sign language user's hand modality, facial modality, and head modality are all occlusion level, then only the average of the three-dimensional coordinate sequence of the historical intact hand 3D skeleton key points of the preset length before the current missing frame is used for filling, and the coordinates of the 21-point hand 3D skeleton key points are output.

[0056] For hand image data, facial micro-expression pixel sequences, and head pose data as secondary constraint modalities, if there is an occlusion confidence level of a secondary constraint modality, the mean of the historical no-occlusion modality dataset with a preset length before the current missing frame is used to fill the missing frame and complete the repair of the current missing frame of that modality.

[0057] Frame-by-frame interpolation was performed to repair the sign language speaker's four-modal occlusion confidence level, resulting in the repaired four-modal data.

[0058] This invention employs frame-by-frame occlusion confidence-based hierarchical matching and causal interpolation repair, adaptively compensating for missing features in hand and facial modal occlusion. This effectively overcomes the problem of key point loss in complex scenarios such as object occlusion, limb crossing, and face mask obscuring, thereby improving the robustness of the system.

[0059] In a specific embodiment of the present invention, the specific analysis method for constructing the sign language modal causal constraint matrix is ​​as follows: pre-divide four types of modal causal levels, set the key points of the 3D skeleton of the hand as the unique query backbone modality, and use the hand image data, facial micro-expression pixel sequence, and head posture data as the modal modification key value modality.

[0060] Construct a 4×4 static binary causal constraint matrix, where the matrix rows correspond to the query modality and the columns correspond to the key-value modality. Only the matrix cells whose query rows are the key points of the 3D skeleton of the hand and whose columns are the hand image data, facial micro-expression pixel sequence and head pose data, are assigned a value of 1, while all other cells are set to 0, forming a fixed causal mask.

[0061] The occlusion confidence of hand image data, facial micro-expression pixel sequence, and head pose data are extracted frame by frame. For the units with a value of 1 in the static matrix, the corresponding key modal occlusion confidence is multiplied to complete the weight scaling, and the sign language modal causal constraint matrix is ​​generated.

[0062] In a specific embodiment of the present invention, the specific analysis method for generating intermediate fusion features of sign language speakers carrying modal causal constraints is as follows: for the four modal data of sign language speakers after restoration, feature linear projection is performed with the fixed learnable three-class projection matrices loaded from the database to generate query matrix, key matrix and value matrix, and unconstrained cross-modal score matrix is ​​obtained by scaling dot product attention calculation.

[0063] The sign language modal causal constraint matrix and the unconstrained cross-modal score matrix are processed by Hadamard product to generate sign language intermediary fusion features carrying modal causal constraints.

[0064] In a specific embodiment of the present invention, the method for calculating the unconstrained cross-modal score matrix using scaled dot product attention is as follows: the four modal data of the sign language user after restoration are stacked in rows to generate the four modal stacked feature matrix of the sign language user.

[0065] The key matrix generated by linear projection of the four-modal stacked feature matrix is ​​transposed, and the query matrix is ​​then inner productd with the transposed key matrix to obtain the sign language speaker's original cross-modal score matrix.

[0066] Using the square root of the dimension of the four-modal stacked feature matrix as a scaling factor, each element in the sign language's original cross-modal score matrix is ​​divided and scaled to obtain the sign language's scaled cross-modal score matrix.

[0067] Normalize the scaled cross-modal score matrix of the sign language user to obtain the unconstrained cross-modal score matrix of the sign language user.

[0068] In a specific embodiment of the present invention, the method for dynamically adjusting the length of the time-series segmented window is as follows: extract the three-dimensional coordinates of all 21 key points of the hand 3D skeleton in the previous frame and the current frame, calculate the displacement distance between adjacent frames for each of the 21 key points of the hand 3D skeleton, calculate the arithmetic mean of the displacement distance of all key points, and record it as the hand skeleton motion gradient of the sign language interpreter in the current frame.

[0069] Specifically, in an embodiment of the present invention, the method for calculating the displacement distance between two adjacent frames for each of the 21 key points of the hand's 3D skeleton is as follows: for example, the three-dimensional coordinates of the thumb tip in the previous frame are... The three-dimensional coordinates of the thumb tip in the current frame are: Through formula The displacement distance between two adjacent frames of the thumb tip was calculated. The displacement distance between two adjacent frames of the 3D skeleton key points of the hand is calculated sequentially, and the arithmetic mean of the displacement distance of all key points is calculated to obtain the motion gradient of the hand skeleton of the signer in the current frame.

[0070] Based on the calculation method of the hand skeletal motion gradient of the signer in the current frame, the hand skeletal motion of the previous two frames is extracted, a gradient sliding buffer queue with a length of 3 frames is maintained, and the hand skeletal motion gradient of the signer within 3 frames is averaged. The result is recorded as the hand skeletal motion gradient of the signer.

[0071] It should be noted that the hand skeletal motion gradient of a sign language interpreter represents the overall spatial motion variation of the hand; the larger the value, the faster the hand movement and the greater the displacement.

[0072] Two sets of fixed motion gradient thresholds, one for low speed and one for high speed, are extracted from the database, along with the corresponding three time-series segmented frame numbers: low-speed long window, medium-speed standard window, and high-speed short window. The motion gradient of the sign language user's hand skeleton is compared with the two fixed motion gradient thresholds to classify the sign language user's hand skeleton motion into three levels: low-speed, medium-speed, and high-speed, and the corresponding time-series window length is matched.

[0073] Specifically, in embodiments of the present invention, for example, the low-speed threshold is set to 0.03, the high-speed threshold is set to 0.12, the number of frames in the low-speed long window timing segment is set to 15, the number of frames in the medium-speed standard window timing segment is set to 8, and the number of frames in the high-speed short window timing segment is set to 4.

[0074] Using the current frame as the end point of the temporal slice, sign language intermediate fusion features carrying modal causal constraints are extracted from the corresponding number of consecutive frames forward according to the length of the matched temporal window to generate sign language intermediate fusion feature fragments.

[0075] The motion gradient of the signer's hand skeleton in the current frame is cached. The motion gradient of the signer's hand skeleton is calculated frame by frame in a loop. The temporal window length is dynamically updated. The obtained multi-segment intermediate fusion feature fragments of the signer are integrated and recorded as the segmented intermediate fusion feature set of the signer, which is used as the input for the morpheme reordering aggregation operation.

[0076] In a specific embodiment of the present invention, the morpheme reordering aggregation is specifically analyzed as follows: For the segmented sign language inter-fusion feature set, the first sign language inter-fusion feature segment is analyzed, and the cosine similarity between the first frame and the last frame in the sign language inter-fusion feature segment is calculated. If the cosine similarity between the first frame and the last frame in the sign language inter-fusion feature segment is less than the preset fixed cosine similarity threshold extracted from the database, then the last frame in the sign language inter-fusion feature segment is recorded as the starting point.

[0077] The last frame of the sign language intermediate fusion feature segment is sequentially compared with each frame of the subsequent sign language intermediate fusion feature segments to calculate cosine similarity until the cosine similarity is greater than a preset fixed cosine similarity threshold. This frame is then recorded as the end node. The first sign language intermediate fusion feature segment and the intermediate fusion features corresponding to the end node are then reordered and aggregated to obtain the first reordered and aggregated sign language intermediate feature set.

[0078] If the cosine similarity between the first and last frames of the sign language intermediate fusion feature segment is greater than or equal to the preset fixed cosine similarity threshold extracted from the database, then the cosine similarity between the first and middle frames of the sign language intermediate fusion feature segment is recalculated. If the cosine similarity between the first and middle frames of the sign language intermediate fusion feature segment is less than the preset fixed cosine similarity threshold extracted from the database, starting from the middle frame of the sign language intermediate fusion feature segment, the cosine similarity between the sign language intermediate fusion feature segment and subsequent frames, as well as each frame in subsequent sign language intermediate fusion feature segments, is calculated sequentially until the cosine similarity is greater than the preset fixed cosine similarity threshold. This frame is then recorded as the end node, and the first reordered aggregated sign language intermediate feature set is obtained.

[0079] The first sign language inter-fusion feature segment is analyzed using a binary method until the first reordered and aggregated sign language inter-feature set is obtained. The next frame corresponding to the end node in the first reordered and aggregated sign language inter-feature set is taken as the starting point of the next reordered and aggregated feature set. The segmented sign language inter-fusion feature sets are reordered and aggregated sequentially to obtain the reordered and aggregated sign language inter-fusion feature set.

[0080] This invention dynamically adjusts the temporal segment window length based on the hand skeletal motion gradient, balancing real-time inference speed with the temporal integrity of complete sign language morphemes. It also performs morpheme reordering and aggregation on fragmented slice features, solving the problems of morpheme fragmentation and disordered word order caused by fixed window cutting.

[0081] Reference Figure 2 As shown, the second aspect of the present invention provides an intelligent sign language translation system based on multimodal fusion. The system includes a four-modal data acquisition module for acquiring hand image data, 3D skeletal key points of the hand, facial micro-expression pixel sequences, and head posture data from a real-time video stream of a sign language user.

[0082] The data preprocessing module is used to calculate the occlusion confidence of each modality frame by frame from the original four-modal data of the sign language speaker, and to classify the occlusion confidence level according to the calculation results.

[0083] The occlusion confidence level repair module is used to perform interpolation repair on a frame-by-frame basis on the sign language speaker's original four-modal data based on the occlusion confidence level of each frame.

[0084] The cross-modal feature extraction module is used to extract features from the repaired four-modal data to obtain sign language intermediary fusion features carrying modal causal constraints.

[0085] The temporal window segmentation module is used to dynamically adjust the length of the temporal segmentation window based on the real-time extracted hand skeletal motion gradient of the sign language user, and to segment the intermediate fusion features of the sign language user according to the length of the temporal segmentation window.

[0086] The morpheme reordering and aggregation module is used to aggregate the segmented sign language inter-fusion feature set according to morpheme reordering to obtain the reordered and aggregated sign language inter-fusion feature set.

[0087] The text translation module is used to decode and translate the reordered and aggregated sign language intermediary feature set to generate natural language.

[0088] The above content is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined by the present invention, and all such modifications and additions should fall within the protection scope of the present invention.

Claims

1. A smart sign language translation method based on multimodal fusion, characterized in that, The method includes: S1. Synchronous acquisition of four modalities: Acquire hand image data, 3D skeleton key points of the hand, facial micro-expression pixel sequence and head posture data of the sign language interpreter in real-time video stream, add timestamps to achieve temporal alignment, and obtain the original four modal data of the sign language interpreter. S2. Data preprocessing and occlusion confidence level repair: The original four-modal data of the sign language speaker is preprocessed, the occlusion confidence of each modality is calculated frame by frame, the occlusion confidence level is divided according to the calculation results, and the original four-modal data of the sign language speaker is repaired frame by frame by interpolation to obtain the repaired four-modal data of the sign language speaker. S3. Cross-modal feature extraction: Construct a sign language modal causal constraint matrix, filter cross-modal feature interactions without semantic dependence in the four-modal data after sign language repair, and generate sign language intermediate fusion features carrying modal causal constraints; S4. Adaptive adjustment of temporal segmentation window length: The length of the temporal segmentation window is dynamically adjusted according to the real-time extracted hand skeletal motion gradient of the signer. The intermediate fusion features of the signer are segmented according to the length of the temporal segmentation window. The segmented intermediate fusion feature set of the signer is then aggregated according to morpheme reordering. S5. Multimodal text translation: Decode and translate the reordered and aggregated sign language intermediate fusion feature set to generate natural language.

2. The intelligent sign language translation method based on multimodal fusion according to claim 1, characterized in that, The specific analysis method for obtaining the original four-modal data of the sign language interpreter is as follows: The color images of the sign language interpreter's hands are captured frame by frame in the real-time video stream, and the effective pixel RGB value matrix is ​​recorded as the sign language interpreter's hand image data. Using a depth camera and 3D recognition, the 3D coordinates of 21 key points of the hand skeleton of the sign language person are obtained frame by frame and recorded as the 3D key points of the hand skeleton of the sign language person. The 21 key points of the hand skeleton are 1 wrist, 5 palm base, 15 fingertips, and 15 first and second joints. The sign language interpreter's face is captured frame by frame in the real-time video stream, and the effective pixel RGB value matrix is ​​recorded as the sign language interpreter's facial micro-expression pixel sequence. Using a depth camera and 3D recognition, the standard facial features pixel key points detected in the image are used as the observation output frame by frame. Combined with a general facial 3D benchmark model, the spatial coordinates and three rotation angles of the sign language person's head are obtained through the perspective point algorithm, including the pitch angle, yaw angle and roll angle. These six sets of values ​​are recorded as the sign language person's head posture data. The hand image data, 3D skeletal key points of the hand, facial micro-expression pixel sequence, and head posture data of the sign language interpreter were collected and time-stamped according to the processing of each frame, and then time-series aligned to obtain the original four-modal data of the sign language interpreter.

3. The intelligent sign language translation method based on multimodal fusion according to claim 2, characterized in that, The specific analytical method for preprocessing the raw four-modal data of sign language speakers is as follows: For hand image data of sign language users, the ratio of effective pixels to theoretical hand pixels is denoted as the hand occlusion confidence level. For the 3D skeleton key points of the sign language user's hand, the percentage of valid and unlost key points of 21 3D skeleton key points of the hand was counted and recorded as the skeleton occlusion confidence. For the facial micro-expression pixel sequence of sign language users, the proportion of effective pixels to standard face pixels is recorded as the facial occlusion confidence level; For sign language users’ head pose data, the ratio of the effective standard facial feature pixel key points detected by head pose detection to 68 complete key points is recorded as the head occlusion confidence. A threshold is set to classify the occlusion confidence of the four modalities of the signer, including occlusion level and no occlusion level. The classification results are simultaneously sent to the interpolation and repair module to perform corresponding missing feature compensation processing.

4. The intelligent sign language translation method based on multimodal fusion according to claim 3, characterized in that, The specific analysis method for performing the corresponding missing feature compensation processing is as follows: A pre-established sign language modality causal dependency mapping library is used, with the key points of the 3D skeleton of the hand as the main causal modality, and the hand image data, facial micro-expression pixel sequence and head pose data as secondary constraint modalities. For the main causal modality, when the sign language interpreter’s skeleton occlusion confidence is occlusion level, and at least one of the sign language interpreter’s hand modality, facial modality and head modality is unoccluded level, constraint interpolation repair is triggered, and the three-dimensional coordinate sequence of historical intact hand 3D skeleton key points of a preset length before the current missing frame is extracted. Extract the data corresponding to the unoccluded level modality in the current missing frame as a causal constraint vector, construct a causal interpolation loss function, and perform interpolation by fusing the three-dimensional coordinate sequence of historical intact hand 3D skeleton key points with a preset length before the missing frame and the causal constraint vector, and output the coordinates of the 21 completed hand 3D skeleton key points; If the skeleton occlusion confidence is occlusion level, and the sign language user's hand modality, facial modality, and head modality are all occlusion level, then only the average of the three-dimensional coordinate sequence of the historical intact hand 3D skeleton key points with a preset length before the current missing frame is used for filling, and the coordinates of the 21-point hand 3D skeleton key points are output. For hand image data, facial micro-expression pixel sequences, and head pose data as secondary constraint modalities, if there is an occlusion confidence level of a secondary constraint modality, the mean of the historical no-occlusion modal dataset with a preset length before the current missing frame is used to fill the missing frame and complete the repair of the current missing frame of that modality. Frame-by-frame interpolation was performed to repair the sign language speaker's four-modal occlusion confidence level, resulting in the repaired four-modal data.

5. The intelligent sign language translation method based on multimodal fusion according to claim 4, characterized in that, The specific analytical method for constructing the sign language modality causal constraint matrix is ​​as follows: Four modal causal hierarchies are pre-defined, with the key points of the 3D skeleton of the hand set as the unique query backbone modality, and the hand image data, facial micro-expression pixel sequence, and head pose data used as modification key value modalities. Construct a 4×4 static binary causal constraint matrix, where the matrix rows correspond to the query modality and the columns correspond to the key-value modality. Only the matrix cells whose query rows are the key points of the 3D skeleton of the hand and whose columns are the hand image data, facial micro-expression pixel sequence and head pose data, are assigned a value of 1, and all other cells are set to 0, forming a fixed causal mask. The occlusion confidence of hand image data, facial micro-expression pixel sequence, and head pose data are extracted frame by frame. For the units with a value of 1 in the static matrix, the corresponding key modal occlusion confidence is multiplied to complete the weight scaling, and the sign language modal causal constraint matrix is ​​generated.

6. The intelligent sign language translation method based on multimodal fusion according to claim 5, characterized in that, The specific analytical method for generating inter-sign language fusion features carrying modal causal constraints is as follows: For the repaired four-modal data of sign language users, feature linear projection is performed on the fixed learnable three-class projection matrix loaded from the database to generate query matrix, key matrix and value matrix. The unconstrained cross-modal score matrix is ​​obtained by scaling dot product attention. The sign language modal causal constraint matrix and the unconstrained cross-modal score matrix are processed by Hadamard product to generate sign language intermediary fusion features carrying modal causal constraints.

7. The intelligent sign language translation method based on multimodal fusion according to claim 6, characterized in that, The specific analysis method for obtaining the unconstrained cross-modal score matrix using scaled dot product attention is as follows: The repaired four-modal data of the sign language speaker are stacked in rows to generate the four-modal stacked feature matrix of the sign language speaker; The key matrix generated by linear projection of the four-modal stacked feature matrix is ​​transposed, and the query matrix is ​​then inner productd with the transposed key matrix to obtain the sign language speaker’s original cross-modal score matrix. Using the square root of the dimension of the four-modal stacked feature matrix as a scaling factor, each element in the sign language's original cross-modal score matrix is ​​divided and scaled to obtain the sign language's scaled cross-modal score matrix. Normalize the scaled cross-modal score matrix of the sign language user to obtain the unconstrained cross-modal score matrix of the sign language user.

8. The intelligent sign language translation method based on multimodal fusion according to claim 1, characterized in that, The specific analysis method for dynamically adjusting the length of the time-series segmented window is as follows: Extract the 3D coordinates of all 21 key points of the hand 3D skeleton in the previous frame and the current frame, calculate the displacement distance between adjacent frames for each of the 21 key points of the hand 3D skeleton, calculate the arithmetic mean of the displacement distance of all key points, and record it as the hand skeleton motion gradient of the signer in the current frame. According to the calculation method of the hand skeletal motion gradient of the signer in the current frame, the hand skeletal motion of the previous two frames is extracted, a gradient sliding buffer queue with a length of 3 frames is maintained, the hand skeletal motion gradient of the signer within 3 frames is averaged, and the result is recorded as the hand skeletal motion gradient of the signer. Extract two sets of fixed motion gradient thresholds (low speed and high speed) from the database, along with the corresponding three time-series segmented frame numbers (low speed long window, medium speed standard window, and high speed short window). Compare the hand skeletal motion gradient of the sign language interpreter with the two fixed motion gradient thresholds to classify the sign language interpreter's hand skeletal motion into three levels of hand gestures: low speed, medium speed, and high speed, and match the corresponding time-series window length. Using the current frame as the end point of the temporal slice, the sign language intermediate fusion features carrying modal causal constraints are extracted from the corresponding number of consecutive frames forward according to the length of the matched temporal window, and a sign language intermediate fusion feature fragment is generated. The motion gradient of the signer's hand skeleton in the current frame is cached. The motion gradient of the signer's hand skeleton is calculated frame by frame in a loop. The temporal window length is dynamically updated. The obtained multi-segment intermediate fusion feature fragments of the signer are integrated and recorded as the segmented intermediate fusion feature set of the signer, which is used as the input for the morpheme reordering aggregation operation.

9. The intelligent sign language translation method based on multimodal fusion according to claim 8, characterized in that, The specific analysis method for the morpheme reordering aggregation is as follows: For the segmented sign language inter-fusion feature set, the first sign language inter-fusion feature segment is analyzed. The cosine similarity between the first frame and the last frame in the sign language inter-fusion feature segment is calculated. If the cosine similarity between the first frame and the last frame in the sign language inter-fusion feature segment is less than the preset fixed cosine similarity threshold extracted from the database, then the last frame in the sign language inter-fusion feature segment is recorded as the starting point. The last frame of the sign language intermediate fusion feature segment is sequentially compared with each frame of the subsequent sign language intermediate fusion feature segments to calculate the cosine similarity until the cosine similarity is greater than the preset fixed cosine similarity threshold. The frame is then recorded as the end node. The first sign language intermediate fusion feature segment and the intermediate fusion feature corresponding to the end node are reordered and aggregated to obtain the first reordered and aggregated sign language intermediate feature set. If the cosine similarity between the first and last frames in the sign language intermediate fusion feature segment is greater than or equal to the preset fixed cosine similarity threshold extracted from the database, then the cosine similarity between the first and middle frames in the sign language intermediate fusion feature segment is recalculated. If the cosine similarity between the first and middle frames in the sign language intermediate fusion feature segment is less than the preset fixed cosine similarity threshold extracted from the database, starting from the middle frame of the sign language intermediate fusion feature segment, the cosine similarity between the sign language intermediate fusion feature segment and subsequent frames, as well as each frame in subsequent sign language intermediate fusion feature segments, is calculated sequentially until the cosine similarity is greater than the preset fixed cosine similarity threshold. This frame is then recorded as the end node, and the first reordered aggregated sign language intermediate feature set is obtained. The first sign language inter-fusion feature segment is analyzed using a binary method until the first reordered and aggregated sign language inter-feature set is obtained. The next frame corresponding to the end node in the first reordered and aggregated sign language inter-feature set is taken as the starting point of the next reordered and aggregated feature set. The segmented sign language inter-fusion feature sets are reordered and aggregated sequentially to obtain the reordered and aggregated sign language inter-fusion feature set.

10. A system for executing the intelligent sign language translation method based on multimodal fusion as described in claims 1-9, characterized in that, The system includes: The four-modal data acquisition module is used to acquire hand image data, 3D skeletal key points of the hand, facial micro-expression pixel sequence and head posture data of the sign language user's real-time video stream; The data preprocessing module is used to calculate the occlusion confidence of each modality frame by frame from the original four-modal data of the sign language speaker, and to classify the occlusion confidence level according to the calculation results; The occlusion confidence level repair module is used to perform interpolation repair on each frame of the sign language's original four-modal data based on the occlusion confidence level of each frame. The cross-modal feature extraction module is used to extract features from the repaired four-modal data to obtain sign language intermediary fusion features carrying modal causal constraints; The temporal window segmentation module is used to dynamically adjust the length of the temporal segmentation window based on the real-time extracted hand skeletal motion gradient of the sign language user, and to segment the intermediate fusion features of the sign language user according to the length of the temporal segmentation window. The morpheme reordering and aggregation module is used to aggregate the segmented sign language inter-fusion feature set according to morpheme reordering to obtain the reordered and aggregated sign language inter-fusion feature set. The text translation module is used to decode and translate the reordered and aggregated sign language intermediary feature set to generate natural language.

Images