Sentence-level sign language video segmentation system and method based on boundary learning and optimization

Abstract

The invention discloses a sentence-level sign language video segmentation system and method based on boundary learning and optimization. The sentence-level sign language video segmentation system comprises a video data acquisition module, a video preprocessing module, a boundary learning module, a boundary optimization module and a frame-level label generation module. According to the method, combined modeling is carried out on the frame-level visual features and the text labeling information in the long sign language video, the time sequence context information under different receptive fields is extracted by using a two-way expansion convolution structure, and multi-modal feature fusion is realized through a gated cross attention mechanism, so that the accuracy of boundary positioning is improved, and the accuracy of long sign language video positioning is improved. Predicting a frame-level boundary confidence sequence; performing Gaussian filtering smoothing on the boundary confidence sequence to suppress local fluctuation, calculating a frame-by-frame uncertainty score in combination with the smoothed boundary confidence sequence, constructing a candidate set according to the difference between the predicted segmentation number and the annotated sentence number, and performing insertion or deletion operation, and finally generating a binary label sequence with the same length as the input frame sequence and completing sentence-level video segmentation.

Description

Technical Field

[0001] This invention belongs to the field of multimodal learning and sign language research, specifically involving a sentence-level sign language video segmentation system and method based on boundary learning and optimization. Background Technology

[0002] Sign language is the primary means of communication for the hearing impaired, possessing unique linguistic rules and expressive forms. Current sign language understanding research primarily focuses on tasks such as Continuous Sign Language Recognition (CSLR) and Sign Language Translation (SLT), which generally rely on sentence-level labeled samples—one text sentence per video. However, sentence-level sign language samples are scarce, limiting further development in sign language research. Therefore, obtaining sentence-level sign language samples is of paramount importance for sign language research and technological applications.

[0003] Traditional methods for acquiring sentence-level sign language data typically include two approaches: one is to invite sign language interpreters to record videos sentence by sentence, and the other is for experts to manually segment and annotate long videos. Both methods have significant problems: the former is inefficient and costly, while the latter requires domain experts to participate in annotation and is a heavy task, making large-scale adoption difficult. Therefore, effectively segmenting existing long sign language videos to obtain sentence-level sign language data is of great significance. However, long sign language videos often lack obvious inter-sentence transitions, and sentence boundaries change naturally and subtly, posing numerous challenges to automatic segmentation of long sign language videos.

[0004] In view of this, it is necessary to propose a sentence-level segmentation method suitable for long sign language videos. By introducing a boundary learning mechanism to detect boundary frames in the video and combining text modality to optimize the boundary detection results, the method can automatically identify sentence boundaries in the video and complete non-overlapping video segmentation, providing an efficient and practical technical solution for building large-scale labeled datasets. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a sentence-level sign language video segmentation system and method based on boundary learning and optimization to obtain sentence-level sign language samples, thereby solving the problems of low efficiency and high cost in the construction of existing sign language datasets. The present invention can automatically segment long sign language videos into sentence-level non-overlapping segments, significantly reducing the cost of sign language data annotation, providing support for the construction of large-scale, high-quality sign language corpora, and providing richer training samples for subsequent sign language recognition and translation models, thus promoting the development and application of sign language recognition and translation technologies.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] This invention discloses a sentence-level sign language video segmentation system based on boundary learning and optimization, comprising: a video data acquisition module, a video preprocessing module, a boundary learning module, a boundary optimization module, and a frame-level label generation module; wherein,

[0008] The video data acquisition module is used to collect long sign language video data and its corresponding annotated text information;

[0009] The video preprocessing module is used to extract frames and standardize the acquired long sign language video data.

[0010] The boundary learning module is used to extract features from the processed long sign language video frames and labeled text, and to perform boundary discrimination for each frame based on a deep neural network, outputting the corresponding frame-level boundary confidence sequence.

[0011] The boundary optimization module is used to smooth, estimate uncertainty, filter boundaries, and adjust the number of boundaries in the boundary confidence sequence output by the boundary learning module to obtain a set of boundary frames.

[0012] The frame-level label generation module is used to construct a frame-level segmentation label sequence based on the set of boundary frames obtained by the boundary optimization module, and to perform sentence-level segmentation on long sign language videos.

[0013] Furthermore, the video preprocessing module includes: a video frame size adjustment module, a random cropping module, an image flipping module, and a timing scaling module;

[0014] The video frame size adjustment module is used to uniformly scale the captured sign language video frames to a size of 256×256 pixels.

[0015] The random cropping module is used to randomly crop the resized sign language video frames to 224×224 pixels.

[0016] The image flipping module is used to horizontally flip the sign language video frame obtained after being processed by the random cropping module with a preset probability in the range of [0,1].

[0017] The timing scaling module is used to adjust the duration of sign language video frame sequences within a preset ratio range, and to expand or compress the number of frames by random frame interpolation or downsampling.

[0018] Furthermore, the boundary learning module includes: a feature extraction module, a dual-path dilated convolution module, a gated cross-attention module, and a boundary prediction module;

[0019] The feature extraction module includes: a visual feature extraction unit and a text feature extraction unit;

[0020] The visual feature extraction unit performs frame-level encoding on the sign language video output by the video preprocessing module. It uses the I3D model to extract the visual features of each frame, outputting a visual feature representation with a feature dimension of 1024. Where V is the number of channels and L is the number of frames;

[0021] The text feature extraction unit encodes the labeled text and uses the BERT model to extract semantic features, resulting in a 768-dimensional text feature representation. Where P is the number of channels and S is the number of tokens in the sentence;

[0022] The visual features and text features are respectively mapped to a unified intermediate feature dimension d through 1×1 convolution, thereby obtaining the visual embedding features. Text embedding features Where d is the dimension of the embedding space;

[0023] The dual-path dilated convolution module consists of two parallel one-dimensional convolutional paths used to extract multi-scale temporal context information. One convolutional path uses an increasing dilation factor sequence of 1, 2, 4, 8, ... to progressively expand the receptive field to capture long-term dependencies, while the other convolutional path uses a decreasing dilation factor sequence of 1024, 512, 256, 128, ... to preserve local fine-grained information. Each convolutional path consists of several dilated convolutional layers, a ReLU activation function, and a normalization layer. The features extracted by the two convolutional paths are concatenated to form a visual multi-scale representation X. DDA ;

[0024] The gated cross-attention module is used to achieve multimodal fusion between visual and textual features. It employs a gating mechanism to regulate the information flow from different modalities, enhancing semantic alignment capabilities. Its input is the visual multi-scale representation X output by the dual-path dilated convolution module. DDA With the text embedding feature T′, cross-modal attention CrossAttn(T′,X) is calculated. DDA This yields a multimodal representation; a learnable gating parameter α is introduced to weight and adjust the multimodal and visual multiscale representations, which are then fused with the visual multiscale representation. The formula is expressed as follows:

[0025] X fused =tanh(α)·CrossAttn(T′,X) DDA )+(1-tanh(α))·X DDA

[0026] Where tanh(α) represents the gating weight, used to control the degree to which the text guides the visual features, and the fusion result X fused As input for subsequent boundary prediction;

[0027] The boundary prediction module, consisting of a fully connected layer and a sigmoid activation function, maps the fused features to an original boundary confidence sequence C∈[0,1] with the same length as the input sequence. L , is used to represent the confidence level of each frame as a boundary frame.

[0028] Furthermore, the boundary optimization module includes: a confidence smoothing module, an uncertainty assessment module, a boundary merging module, and a boundary adjustment module;

[0029] The confidence smoothing module is used to perform temporal smoothing on the frame-level boundary confidence sequence output by the boundary learning module, suppressing local fluctuations and enhancing the global boundary trend. Specifically, it smooths the original boundary confidence sequence C = [C1, C2, ..., C...]. L After applying Gaussian filtering, the smoothed confidence sequence is: The calculation formula is as follows:

[0030]

[0031] Among them, C t+i This represents the boundary confidence of the (t+i)th frame in the original boundary confidence sequence; i is the time offset relative to the current frame t; Z is the normalization coefficient, used to ensure that the weighted sum of the Gaussian kernel is 1; r is the convolution kernel radius, and the formula for calculating the normalization coefficient Z is as follows:

[0032]

[0033] Where σ is the standard deviation hyperparameter;

[0034] The uncertainty assessment module calculates the boundary score U for each frame based on the smoothed confidence sequence. t This is used to measure the degree of uncertainty in boundary determination of a frame, and thus to evaluate whether each frame is a boundary frame. The calculation formula is as follows:

[0035]

[0036] Where ∈ represents the local peak weighting coefficient. This represents the confidence value of the t-th frame in the smoothed boundary confidence sequence. This represents the confidence value of the (t-1)th frame in the sequence. This represents the confidence value of the (t+1)th frame in the sequence;

[0037] The boundary merging module is used to merge consecutive high-confidence frames (i.e., multi-frame boundaries) in the original boundary confidence sequence into a single-frame boundary. Within each consecutive boundary segment, the frame with the highest boundary score is selected as the final boundary frame, resulting in a boundary set.

[0038] The boundary adjustment module is used to perform boundary insertion or deletion according to the difference between the current predicted segmentation number m and the number of labeled text sentences n, so that the final number of boundary frames approaches the reference number of sentences, as follows:

[0039] When m < n, boundary insertion is performed. A number of potential boundary frames are selected from the current non-boundary frames for supplementation. By selecting the set of frames with a boundary confidence less than 0.5 and a confidence greater than that of adjacent frames as the candidate insertion frame set, where the number of candidate boundaries in the candidate insertion frame set is k, and the min(k, n - m) frames with the highest boundary scores are selected and inserted into the boundary set;

[0040] Construct the candidate insertion frame set As follows:

[0041]

[0042] From the candidate insertion frame set Select the min(k, n - m) frames with the highest boundary scores as the final inserted boundary frame set As follows:

[0043]

[0044] When m > n, boundary deletion is performed. The m - n boundary frames with the lowest boundary scores are selected from the candidate deletion frame set for removal. By selecting the set of frames with a boundary confidence greater than 0.5 and a confidence greater than that of adjacent frames as the candidate deletion frame set, the candidate deletion frame set Is defined as:

[0045]

[0046] Select the m - n frames with the lowest boundary scores in the candidate deletion frame set as the final deleted boundary frame set As follows:

[0047]

[0048] The final boundary frame set Is expressed as:

[0049]

[0050] Where, \ is the set difference operation, which means removing the elements identical to the latter from the former.

[0051] Furthermore, the frame-level label generation module constructs a frame-level segmentation label sequence Y = [y1, y2,..., y , where the frame-level segmentation label value y L Of the t-th frame is defined as follows: t As follows:

[0052]

[0053] The output frame-level segmentation label sequence Y is directly used as the sentence-level segment boundary labeling result of the sign language video. Specifically, after binary classification, the output boundary sequence is used to obtain a binary label sequence with the same length as the input frame sequence, where 1 represents a boundary frame and 0 represents an inner frame. The video segments are automatically segmented according to the frame-level segmentation label sequence Y: starting from the first frame, scanning backward from the current position, the first frame with a label of 1 is the end frame of the current sentence segment, and the start point to this frame constitutes a complete sentence segment; the next frame is used as the start point of the next segment, and the above process continues until the entire frame sequence is processed, thereby realizing automatic sentence-level segmentation of long sign language videos.

[0054] The present invention provides a sentence-level sign language video segmentation method based on boundary learning and optimization. Based on the above system, the steps are as follows:

[0055] 1) Video data acquisition: Obtain long sign language videos containing multiple sentence-level sign language segments and their corresponding annotated text information from online platforms or public sign language corpora;

[0056] 2) Video preprocessing: Frame extraction and standardization are performed on the acquired video, including uniform size scaling, random cropping, horizontal flipping, and temporal scaling;

[0057] 3) Visual feature extraction: The I3D model is used to encode the preprocessed video frames frame by frame to obtain a frame-level visual feature sequence;

[0058] 4) Text semantic modeling: Input the labeled text corresponding to the video into the BERT model for semantic encoding to obtain text features;

[0059] 5) Boundary learning modeling: Visual features and text features are fused across modalities to extract multi-scale temporal context information, and boundary discrimination is performed on each frame under cross-modal guidance to obtain the corresponding frame-level boundary confidence sequence.

[0060] 6) Boundary smoothing and uncertainty analysis: Gaussian filtering is applied to the boundary confidence sequence to suppress local fluctuations, and frame-level boundary scores are calculated based on the smoothed sequence;

[0061] 7) Boundary quantity optimization: Merge consecutive high-confidence frames in the original boundary confidence sequence into single-frame boundaries to construct a boundary set. Perform boundary insertion or deletion based on the difference between the predicted number of segments and the number of labeled sentences. Construct a candidate insertion frame set or a candidate deletion frame set based on the smoothed boundary confidence. Then calculate the frame-by-frame boundary score to select the final inserted or deleted boundary frame and complete the boundary adjustment.

[0062] 8) Frame tag generation: Based on the final set of boundary frames obtained after optimization of the number of boundaries, construct a binary tag sequence with the same length as the input frame sequence, and scan the tag sequence in sequence to determine the start and end positions of sentence segments, thereby realizing automatic sentence-level segmentation of long sign language videos.

[0063] The beneficial effects of this invention are:

[0064] This invention jointly models frame-level visual features and text annotation information in long sign language videos, extracts temporal contextual information under different receptive fields using a dual-path dilated convolution structure, and achieves multimodal feature fusion through a gated cross-attention mechanism to improve the accuracy of boundary localization, thereby predicting frame-level boundary confidence sequences. The boundary confidence sequences are then smoothed using Gaussian filtering to suppress local fluctuations. Frame-by-frame uncertainty scores are calculated based on the smoothed boundary confidence sequences. A candidate set is constructed based on the difference between the predicted number of segments and the number of annotated sentences, and insertion or deletion operations are performed. Finally, a binary label sequence with the same length as the input frame sequence is generated, completing sentence-level video segmentation.

[0065] This invention can automatically segment long sign language videos into sentence-level non-overlapping sign language video segments without additional manual intervention. By combining cross-modal feature fusion and boundary optimization, it achieves effective boundary localization, significantly reducing manual annotation costs and improving corpus construction efficiency. At the same time, this invention balances accuracy and scalability, providing richer data samples for subsequent sign language recognition and translation models, which helps improve the performance of downstream models in sentence-level scenarios. It is applicable to multiple practical application scenarios such as education, translation, and barrier-free communication, and has good practical value and promotion prospects. Attached Figure Description

[0066] Figure 1 This is a block diagram of the system structure of the present invention.

[0067] Figure 2 This is a schematic diagram illustrating an application scenario of the present invention.

[0068] Figure 3 This is a schematic diagram of the structure of the long sign language video segmentation network in this invention.

[0069] Figure 4 This is a schematic diagram illustrating the structural principle of the boundary optimization module of the present invention. Detailed Implementation

[0070] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.

[0071] Reference Figures 1-4As shown, the present invention provides a sentence-level sign language video segmentation system based on boundary learning and optimization, comprising: a video data acquisition module, a video preprocessing module, a boundary learning module, a boundary optimization module, and a frame-level label generation module; wherein,

[0072] The video data acquisition module is used to collect long sign language video data and its corresponding annotated text information;

[0073] The video preprocessing module is used to extract frames and standardize the acquired long sign language video data.

[0074] The video preprocessing module includes: a video frame size adjustment module, a random cropping module, an image flipping module, and a timing scaling module.

[0075] The video frame size adjustment module is used to uniformly scale the captured sign language video frames to a size of 256×256 pixels.

[0076] The random cropping module is used to randomly crop the resized sign language video frames to 224×224 pixels.

[0077] The image flipping module is used to horizontally flip the sign language video frame obtained after being processed by the random cropping module with a preset probability in the range of [0,1].

[0078] The timing scaling module is used to adjust the duration of sign language video frame sequences within a preset ratio range (e.g., ±20%), and to expand or compress the number of frames by random frame interpolation or downsampling.

[0079] The boundary learning module is used to extract features from the processed long sign language video frames and labeled text, and to perform boundary discrimination for each frame based on a deep neural network, outputting the corresponding frame-level boundary confidence sequence.

[0080] The boundary learning module includes: a feature extraction module, a dual-path dilated convolution module, a gated cross-attention module, and a boundary prediction module;

[0081] The feature extraction module includes: a visual feature extraction unit and a text feature extraction unit;

[0082] The visual feature extraction unit performs frame-level encoding on the sign language video output by the video preprocessing module. It uses the I3D model to extract the visual features of each frame, outputting a visual feature representation with a feature dimension of 1024. Where V is the number of channels and L is the number of frames;

[0083] The text feature extraction unit encodes the labeled text and uses the BERT model to extract semantic features, resulting in a 768-dimensional text feature representation. Where P is the number of channels and S is the number of tokens in the sentence;

[0084] The visual features and text features are respectively mapped to a unified intermediate feature dimension d through 1×1 convolution, thereby obtaining the visual embedding features. Text embedding features Where d is the dimension of the embedding space;

[0085] The dual-path dilated convolution module consists of two parallel one-dimensional convolutional paths used to extract multi-scale temporal context information. One convolutional path uses an increasing dilation factor sequence of 1, 2, 4, 8, ... to progressively expand the receptive field to capture long-term dependencies, while the other convolutional path uses a decreasing dilation factor sequence of 1024, 512, 256, 128, ... to preserve local fine-grained information. Each convolutional path consists of several dilated convolutional layers, a ReLU activation function, and a normalization layer. The features extracted by the two convolutional paths are concatenated to form a visual multi-scale representation X. DDA ;

[0086] Specifically, the dual-path dilated convolution module consists of dual-path dilated convolution layers and a self-attention module, and the outputs of the two paths are represented as follows:

[0087] X1=Conv1D(X′;d1), X2=Conv1D(X′;d2)

[0088] Where d1 and d2 represent different expansion coefficients. The feature maps extracted from different receptive fields are fused to obtain the visual multi-scale feature representation X. DDA The formula is expressed as follows:

[0089] X DDA =X1⊕X2

[0090] Where ⊕ represents concatenation along the channel dimension, the fused features can simultaneously capture local relationships and global long-term dependencies;

[0091] The gated cross-attention module is used to achieve multimodal fusion between visual and textual features. It employs a gating mechanism to regulate the information flow from different modalities, enhancing semantic alignment capabilities. Its input is the visual multi-scale representation X output by the dual-path dilated convolution module. DDA With the text embedding features T′, cross-modal attention CrossAttn(T′, X) is calculated. DDA This yields a multimodal representation; a learnable gating parameter α is introduced to weight and adjust the multimodal and visual multiscale representations, which are then fused with the visual multiscale representation. The formula is expressed as follows:

[0092] X fused=tanh(α)·CrossAttn(T′,X DDA )+(1-tanh(α))·X DDA

[0093] Where tanh(α) represents the gating weight, used to control the degree to which the text guides the visual features, and the fusion result X fused As input for subsequent boundary prediction;

[0094] The boundary prediction module, consisting of a fully connected layer and a sigmoid activation function, maps the fused features to an original boundary confidence sequence C∈[0,1] with the same length as the input sequence. L , is used to represent the confidence level of each frame as a boundary frame.

[0095] The boundary optimization module is used to smooth, estimate uncertainty, filter boundaries, and adjust the number of boundaries in the boundary confidence sequence output by the boundary learning module to obtain a set of boundary frames.

[0096] The boundary optimization module includes: a confidence smoothing module, an uncertainty assessment module, a boundary merging module, and a boundary adjustment module;

[0097] The confidence smoothing module is used to perform temporal smoothing on the frame-level boundary confidence sequence output by the boundary learning module, suppressing local fluctuations and enhancing the global boundary trend. Specifically, it smooths the original boundary confidence sequence C = [C1, C2, ..., C...]. L After applying Gaussian filtering, the smoothed confidence sequence is: The calculation formula is as follows:

[0098]

[0099] Among them, C t+i This represents the boundary confidence of the (t+i)th frame in the original boundary confidence sequence; i is the time offset relative to the current frame t; Z is the normalization coefficient, used to ensure that the weighted sum of the Gaussian kernel is 1; r is the convolution kernel radius, and the formula for calculating the normalization coefficient Z is as follows:

[0100]

[0101] Where σ is the standard deviation hyperparameter;

[0102] The uncertainty assessment module calculates the boundary score U for each frame based on the smoothed confidence sequence. t This is used to measure the degree of uncertainty in boundary determination of a frame, and thus to evaluate whether each frame is a boundary frame. The calculation formula is as follows:

[0103]

[0104] where ∈ is the local peak weighting coefficient, represents the confidence value of the t-th frame in the smoothed boundary confidence sequence, represents the confidence value of the (t - 1)-th frame in the sequence, represents the confidence value of the (t + 1)-th frame in the sequence;

[0105] The boundary merging module is used to merge consecutive high-confidence frames (i.e., multi-frame boundaries) in the original boundary confidence sequence into single-frame boundaries. In each consecutive boundary segment, the frame with the highest boundary score is selected as the final boundary frame to obtain the boundary set

[0106] The boundary adjustment module is used to perform boundary insertion or deletion according to the difference between the current predicted segmentation number m and the number of labeled text sentences n, so that the final number of boundary frames approaches the reference sentence number, specifically as follows:

[0107] When m < n (under-segmentation), boundary insertion is performed. A number of potential boundary frames are screened out from the current non-boundary frames for supplementation. By selecting the set of frames with a boundary confidence less than 0.5 and a confidence greater than the adjacent frames as the candidate insertion frame set, where the number of candidate boundaries in the candidate insertion frame set is k, and the min(k, n - m) frames with the highest boundary scores are selected and inserted into the boundary set;

[0108] Construct the candidate insertion frame set as follows:

[0109]

[0110] Select from the candidate insertion frame set the min(k, n - m) frames with the highest boundary scores as the final insertion boundary frame set as follows:

[0111]

[0112] When m > n (over-segmentation), boundary deletion is performed. The m - n boundary frames with the lowest boundary scores are screened out from the candidate deletion frame set for elimination. By selecting the set of frames with a boundary confidence greater than 0.5 and a confidence greater than the adjacent frames as the candidate deletion frame set, the candidate deletion frame set is defined as:

[0113]

[0114] Select the m - n frames with the lowest boundary scores in the candidate deletion frame set as the final deletion boundary frame set as follows:

[0115]

[0116] Final boundary frame set Represented as:

[0117]

[0118] Here, \ represents the set difference operation, which means removing the element that is the same as the latter from the former.

[0119] The frame-level label generation module is used to construct a frame-level segmentation label sequence based on the set of boundary frames obtained by the boundary optimization module, and to perform sentence-level segmentation on long sign language videos.

[0120] Based on the boundary set Construct a frame-level segmentation label sequence Y = [y1, y2, ..., y3] L ], where the frame-level segmentation tag value y of frame t. t The definition is as follows:

[0121]

[0122] The output frame-level segmentation label sequence Y is directly used as the sentence-level segment boundary labeling result of the sign language video. Specifically, after binary classification, the output boundary sequence is used to obtain a binary label sequence with the same length as the input frame sequence, where 1 represents a boundary frame and 0 represents an inner frame. The video segments are automatically segmented according to the frame-level segmentation label sequence Y: starting from the first frame, scanning backward from the current position, the first frame with a label of 1 is the end frame of the current sentence segment, and the start point to this frame constitutes a complete sentence segment; the next frame is used as the start point of the next segment, and the above process continues until the entire frame sequence is processed, thereby realizing automatic sentence-level segmentation of long sign language videos.

[0123] This invention also provides a sentence-level sign language video segmentation method based on boundary learning and optimization. Based on the above system, the steps are as follows:

[0124] 1) Video data acquisition: Obtain long sign language videos containing multiple sentence-level sign language segments and their corresponding annotated text information from online platforms or public sign language corpora;

[0125] 2) Video preprocessing: Frame extraction and standardization are performed on the acquired video, including uniform size scaling, random cropping, horizontal flipping, and temporal scaling;

[0126] 3) Visual feature extraction: The pre-trained I3D model is used to encode the pre-processed video frames frame by frame to obtain a frame-level visual feature sequence.

[0127] 4) Text semantic modeling: Input the labeled text corresponding to the video into the BERT model for semantic encoding to obtain text features;

[0128] 5) Boundary learning modeling: Visual features and text features are fused across modalities to extract multi-scale temporal context information, and boundary discrimination is performed on each frame under cross-modal guidance to obtain the corresponding frame-level boundary confidence sequence.

[0129] 6) Boundary smoothing and uncertainty analysis: Gaussian filtering is applied to the boundary confidence sequence to smooth it, suppressing local fluctuations, and frame-level boundary scores are calculated based on the smoothed sequence;

[0130] 7) Boundary quantity optimization: Merge consecutive high-confidence frames in the original boundary confidence sequence into single-frame boundaries to construct a boundary set. Perform boundary insertion or deletion based on the difference between the predicted number of segments and the number of labeled sentences. Construct a candidate insertion frame set or a candidate deletion frame set based on the smoothed boundary confidence. Then calculate the frame-by-frame boundary score to select the final inserted or deleted boundary frame and complete the boundary adjustment.

[0131] 8) Frame tag generation: Based on the final set of boundary frames obtained after optimization of the number of boundaries, construct a binary tag sequence with the same length as the input frame sequence (1 represents a boundary frame and 0 represents an inner frame), and scan the tag sequence in sequence to determine the start and end positions of sentence segments, thereby realizing automatic sentence-level segmentation of long sign language videos.

[0132] This invention has many specific applications. The above description is only a preferred embodiment of this invention. It should be noted that for those skilled in the art, several improvements can be made without departing from the principle of this invention, and these improvements should also be considered within the scope of protection of this invention.

Claims

1. A sentence-level sign language video segmentation system based on boundary learning and optimization, characterized in that, It includes: A video data acquisition module, a video preprocessing module, a boundary learning module, a boundary optimization module, and a frame-level label generation module; The video data acquisition module is used to acquire long sign language video data and its corresponding annotated text information; The video preprocessing module is used to perform frame extraction and normalization processing on the acquired long sign language video data; The boundary learning module is used to extract features from the processed long sign language video frames and annotated text, and perform boundary discrimination on each frame, and output the corresponding frame-level boundary confidence sequence; The boundary optimization module is used to smooth, estimate uncertainty, screen boundaries, and adjust the quantity of the boundary confidence sequence output by the boundary learning module to obtain a boundary frame set; The frame-level label generation module is used to construct a frame-level segmentation label sequence according to the boundary frame set obtained by the boundary optimization module, and perform sentence-level segmentation on the long sign language video. 2.The sentence-level sign language video segmentation system based on boundary learning and optimization of claim 1, wherein, The video preprocessing module includes: a video frame size adjustment module, a random cropping module, an image flipping module, and a temporal scaling module; The video frame size adjustment module is used to uniformly scale the acquired sign language video frames to a size of 256×256 pixels; The random cropping module is used to randomly crop the sign language video frames with adjusted sizes to 224×224 pixels; The image flipping module is used to horizontally flip the sign language video frames obtained after being processed by the random cropping module with a preset probability within the interval [0,1]; The temporal scaling module is used to adjust the duration of the sign language video frame sequence within a preset ratio range, and expand or compress the number of frames by means of random frame interpolation or downsampling. 3.The sentence-level sign language video segmentation system based on boundary learning and optimization of claim 1, wherein, The boundary learning module includes: a feature extraction module, a dual-path dilated convolution module, a gated cross-attention module, and a boundary prediction module; The feature extraction module includes: a visual feature extraction unit and a text feature extraction unit; The visual feature extraction unit is configured to perform frame-level encoding on the sign language video output by the video preprocessing module, extract visual features of each frame by using an I3D model, and output a visual feature representation with a feature dimension of 1024 wherein V is the number of channels, and L is the number of frames. The text feature extraction unit is configured to encode the annotated text, extract semantic features by using a BERT model, and obtain a text feature representation with a dimension of 768, denoted as where P is the number of channels, and S is the number of tokens in a sentence. The visual features and the text features are respectively mapped to a unified intermediate feature dimension d through 1x1 convolution, so as to obtain visual embedding features text embedding features where d is the dimension of the embedding space. The dual-path dilated convolution module is composed of two parallel one-dimensional convolution paths for extracting multi-scale temporal context information, one convolution path adopts an increasing dilated factor sequence 1, 2, 4, 8, … to gradually expand the receptive field to capture long-term dependencies, and the other convolution path adopts a decreasing dilated factor sequence 1024, 512, 256, 128, … to retain local fine-grained information, each convolution path is composed of a plurality of dilated convolution layers, ReLU activation functions and normalization layers, and the features extracted by the two convolution paths are spliced to form a visual multi-scale representation X DDA ; The gate cross-attention module is used to realize the multi-modal fusion between the visual features and the text features, adopts a gate mechanism to regulate the information flow from different modalities, and enhances the semantic alignment capability. The input of the gate cross-attention module is the visual multi-scale representation X output by the double-path dilated convolution module DDA The text embedding feature T' is obtained by calculating the cross-modal attention CrossAttn(T', X DDA ), to obtain a multi-modal representation; and a learnable gate parameter a is introduced to weight and adjust the multi-modal representation and the visual multi-scale representation, and the multi-modal representation is fused with the visual multi-scale representation, and the formula is expressed as follows: X fused = tanh(a) · CrossAttn(T', X DDA +(1 - tanh(a)) · X DDA wherein tanh(a) represents a gating weight for controlling the degree of guidance of the text to the visual features, and the fusion result X fused as input for subsequent boundary prediction; The boundary prediction module includes a full connection layer and a Sigmoid activation function, and maps the fused feature to an original boundary confidence sequence C e [0, 1] consistent with the length of the input sequence, which is used to represent the confidence of each frame being a boundary frame. L , for representing the confidence of each frame being a boundary frame.

4. The boundary learning and optimization based sentence level sign language video segmentation system according to claim 1, wherein, The boundary optimization module includes: a confidence smoothing module, an uncertainty evaluation module, a boundary merging module, and a boundary adjustment module; The confidence smoothing module is configured to perform time smoothing on the frame-level boundary confidence sequence output by the boundary learning module, suppress local fluctuations, and enhance global boundary trends. Specifically, the original boundary confidence sequence C=[C1, C2, …, C L ] is smoothed by applying a Gaussian filter, and the smoothed confidence sequence is The calculation formula is as follows: where C t+i represents the boundary confidence of the t+i frame in the original boundary confidence sequence; i is the time offset relative to the current frame t; Z is a normalization coefficient, used to ensure that the weighted sum of the Gaussian kernel is 1; r is the radius of the convolution kernel, and the calculation formula of the normalization coefficient Z is as follows: where σ is a standard deviation hyperparameter; An uncertainty evaluation module calculates a boundary score U corresponding to each frame based on the smoothed confidence sequence t to measure the uncertainty degree of the frame in boundary determination, and then evaluate whether each frame is a boundary frame, and the calculation formula is as follows: wherein ∈ is a local peak weighting coefficient, denotes a confidence value of the t-th frame in the smoothed border confidence sequence, denotes a confidence value of the t-1-th frame in the sequence, denotes a confidence value of the t+1-th frame in the sequence. a boundary merging module, configured to merge continuous high-confidence frames (i.e. multi-frame boundaries) in the original boundary confidence sequence into single-frame boundaries, in each continuous boundary segment, a frame with the maximum boundary score is selected as a final boundary frame, and a boundary set is obtained The boundary adjustment module is used to perform boundary insertion or deletion according to the difference between the current predicted segmentation number m and the number of sentences n in the annotated text, so that the final number of boundary frames approaches the reference number of sentences, specifically as follows: When m < n, boundary insertion is performed, and several potential boundary frames are selected from the current non-boundary frames for supplementation. By selecting the frame set with a boundary confidence less than 0.5 and greater than the adjacent frames as the candidate insertion frame set, where the number of candidate boundaries in the candidate insertion frame set is k, and the min(k, n - m) frames with the highest boundary scores are selected and inserted into the boundary set; constructing a candidate insertion frame set as follows: selecting the min(k, n-m) frames with the highest boundary scores from the candidate insertion frame set as the final insertion boundary frame set as follows: When m > n, boundary deletion is performed, and the m-n boundary frames with the lowest boundary scores are filtered out from the candidate deletion frame set. The candidate deletion frame set is selected by selecting the frame set with a boundary confidence greater than 0.5 and a confidence greater than that of adjacent frames, and the candidate deletion frame set is defined as: selecting m-n frames with the lowest boundary scores in the candidate deleted frame set as the final deleted boundary frame set As follows: Final boundary frame set is represented as: where \ is the set difference operation, which means removing the elements identical to the latter from the former.

5. The boundary learning and optimization based sentence level sign language video segmentation system according to claim 4, wherein, The frame-level label generation module generates the frame-level label sequence Y = [y1, y2, …, yt, …, yn] according to the boundary set The frame-level segmentation label sequence Y = [y1, y2, …, yt, …, yn] is constructed L , wherein the frame-level segmentation label value yt of the t-th frame is defined as follows: t ​ The output frame-level segmentation label sequence Y is directly used as the sentence-level segment boundary annotation result of the sign language video, specifically: after the output boundary sequence is classified by two categories, a binary label sequence consistent with the length of the input frame sequence is obtained, wherein 1 represents a boundary frame and 0 represents an internal frame; the video paragraph is automatically segmented according to the frame-level segmentation label sequence Y: starting from the first frame, scanning from the current position backward, the first frame with a label of 1 is the end frame of the current sentence segment, and the starting point to the frame constitutes a complete sentence segment; the next frame is used as the starting point of the next segment, and the above process is continued until the entire frame sequence is processed, thereby realizing the automatic sentence-level segmentation of the long sign language video.

6. A method for sentence-level sign language video segmentation based on boundary learning and optimization, based on the system of any of claims 1-5, characterized in that, The method steps are as follows: 1) Video data collection: long sign language videos containing multiple sentence-level sign language segments and their corresponding annotated text information are obtained from a network platform or a public sign language corpus; 2) Video preprocessing: frame extraction and standardization processing are performed on the collected video, including uniform size scaling, random cropping, horizontal flipping and time sequence scaling; 3) Visual feature extraction: the I3D model is used to encode the preprocessed video frames frame by frame to obtain a frame-level visual feature sequence; 4) Text semantic modeling: the annotated text corresponding to the video is input into the BERT model for semantic encoding to obtain a text feature; 5) Boundary learning modeling: the visual feature and the text feature are fused for cross-modal feature fusion, multi-scale temporal context information is extracted, and boundary discrimination is performed on each frame under the guidance of cross-modal to obtain a corresponding frame-level boundary confidence sequence; 6) Boundary smoothing and uncertainty analysis: the boundary confidence sequence is smoothed by Gaussian filtering to suppress local fluctuations, and the frame-level boundary score is calculated based on the smoothed sequence; 7) Boundary number optimization: the continuous high-confidence frames in the original boundary confidence sequence are merged into a single frame boundary to construct a boundary set, boundary insertion or deletion is performed according to the difference between the predicted segmentation number and the annotated sentence number, a candidate insertion frame set or a candidate deletion frame set is constructed based on the smoothed boundary confidence, the frame-by-frame boundary score is calculated to select the final inserted or deleted boundary frame, and boundary adjustment is completed; 8) Frame label generation: according to the final boundary frame set obtained after boundary number optimization, a binary label sequence consistent with the length of the input frame sequence is constructed, and the start and end positions of the sentence segment are determined by sequentially scanning the label sequence, thereby realizing the automatic sentence-level segmentation of the long sign language video.

Images