A deep learning-based continuous sign language recognition method

Abstract

The present application relates to the technical field of sign language recognition, and particularly relates to a continuous sign language recognition method based on deep learning. The method comprises the following steps: extracting human image from a continuous sign language video and performing normalization to construct a multi-modal data input; generating a boundary probability and an action category probability sequence through boundary regression and action regression branches, and determining boundary and non-boundary positions according to the sequence; performing fragment-level reorganization, combining the reorganized fragments with the action category probability sequence, and performing fusion processing on image space features, gesture trajectories, facial key points and skeleton features to generate fragment fusion features and form a fragment fusion feature sequence; finally, generating a final continuous sign language recognition result through sequence learning, connectionist temporal classification decoding, repeated word merging and transition action removal. The present application effectively suppresses the interference of transition actions on the recognition result, and significantly improves the accuracy and semantic coherence of continuous sign language recognition.

Description

Technical Field

[0001] This invention relates to the field of sign language recognition technology, and in particular to a continuous sign language recognition method based on deep learning. Background Technology

[0002] In the field of sign language recognition technology, existing solutions typically revolve around continuous sign language video acquisition, image information processing, key point coordinate extraction, feature fusion, temporal modeling, and decoding output. These solutions suffer from limitations such as difficulty in accurately distinguishing between boundary and non-boundary positions, instability in generating stable fragment fusion feature sequences, and susceptibility of continuous sign language recognition results to interference from transitional movements. Existing methods often directly extract hand key point coordinates, facial key point coordinates, hand skeletal joint coordinates, gesture trajectories, and image spatial features from continuous sign language videos, and then perform unified temporal modeling or unified fusion processing. In continuous sign language recognition scenarios, this often results in the mixing of stable action content with action transitions near boundary positions, making it difficult to effectively distinguish between continuously repeated words and words indicating transitional movements, thus hindering the stable achievement of continuous sign language recognition results. For the joint processing of multimodal data input to achieve continuous sign language recognition through location segmentation information and fragment-level reconstruction, existing technologies generally lack a unified processing link that continues to pass the boundary prediction probability distribution generation, category prediction probability distribution generation, boundary position determination, and non-boundary position determination to adjacent frame extraction, fragment-level reconstruction, and fragment fusion feature sequence generation stages. It is difficult to form a consistent process in continuous sign language recognition application scenarios, including continuous sign language video acquisition, key point coordinate extraction, location segmentation, fragment-level reconstruction, fragment fusion feature sequence generation, and continuous sign language recognition result output. This results in insufficient correspondence between location segmentation information and multimodal data input, and the fragment-level reconstructed data is difficult to stably support subsequent sequence learning module processing and connectionist temporal classification method decoding. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a deep learning-based continuous sign language recognition method, comprising: S100: Acquire continuous sign language video, perform human image extraction and normalization processing, and perform gesture trajectory extraction and image spatial feature extraction processing to obtain multimodal data input; S200. Based on the multimodal data input, perform boundary regression branch construction and boundary prediction probability distribution generation processing, and perform boundary location determination and non-boundary location determination processing to generate location division information; S300: Obtain the location segmentation information and the multimodal data input, perform adjacent frame extraction and fragment-level recombination processing to obtain fragment-level recombination results; based on the fragment-level recombination results and the action category probability sequence, perform image spatial features, gesture trajectory, facial expression key points and skeleton features fusion processing to generate fragment fusion features; based on the fragment fusion features and the location segmentation information, perform global semantic information extraction and temporal feature extraction processing to generate fragment fusion feature sequence; S400. Obtain the fragment fusion feature sequence, process it using the sequence learning module to obtain a word sequence or Gloss sequence; based on the word sequence or Gloss sequence, perform connectionist temporal classification decoding, merge consecutively repeated words, and remove words representing transitional actions to generate processed sign language sentences; based on the processed sign language sentences, perform contextual natural language processing to generate continuous sign language recognition results.

[0004] Furthermore, the process of extracting and normalizing the human body image includes: The human image extraction includes extracting image content including the upper body, hand activity area and face area from continuous sign language video frame by frame, and writing abnormal frame records for frames with human body area that is too small, deviates from the edge or is occluded by more than a preset ratio. The normalization process includes performing size unification, brightness unification, and frame order reordering on the extracted human body image.

[0005] Furthermore, the process of gesture trajectory extraction and image spatial feature extraction includes: The gesture trajectory extraction includes reading the coordinates of key points of the hand and the coordinates of the hand bones and joints in the order of frame number, comparing the displacement changes of adjacent frames to generate movement direction information, and generating trajectory segments based on the displacement direction changes and the distribution of pause frames. The image spatial feature extraction includes synchronously retrieving image information corresponding to key point coordinate information, and performing region mapping and spatial relationship organization from the corresponding human body image to obtain image spatial feature records arranged in frames. The image spatial features include the relative positions of the hands, the relative positions of the face and hands, the extended state of the upper limbs, and the regional distribution relationship; the coordinates of key points of the hands, the coordinates of key points of the face, the coordinates of the hand bones and joints, the gesture trajectory, and the image spatial features are all written into the same data structure.

[0006] Furthermore, the process of constructing the boundary regression branch and generating the boundary prediction probability distribution includes: Read multimodal data input frame by frame; Organize the data groups in the same frame to form a sequence to be judged; Extract the hand gesture trajectory turning segment, the hand bone joint coordinate displacement abrupt segment, the facial key point coordinate expression switching segment, and the relative position change segment of the two hands in the image space features from the sequence to be judged to generate a boundary probability sequence.

[0007] Furthermore, the process of determining boundary and non-boundary locations includes: The boundary position determination process includes selecting frames from the boundary probability sequence whose boundary probability is higher than a preset condition and whose corresponding action category probability sequence shows a trend of switching or transitioning actions as boundary positions. The non-boundary position determination process includes selecting frames from the boundary probability sequence whose boundary probability is lower than a preset condition and whose corresponding action category probability sequence shows a stable action continuation trend as non-boundary positions, and unifying the boundary positions and non-boundary positions into position division information.

[0008] Furthermore, the process of adjacent frame extraction and fragment-level reassembly includes: The adjacent frame extraction includes extracting a group of consecutive frames in the same non-boundary position segment in chronological order. The segment-level reorganization process includes reorganizing the consecutive frames in the same non-boundary position segment into a segment data group and adding adjacent boundary position markers at both ends of the segment. For candidate segments with fewer than a preset number of frames, a segment merging check is performed. When the gesture trajectory direction is continuous, the hand bone joint coordinate changes smoothly, and there is no obvious switching in the image spatial features, the segment merging process is performed. For segments with missing frames, the same type of coordinate information is extracted from the adjacent frames before and after to fill in the missing frames. If there are too many consecutive missing frames, the segment is discarded.

[0009] Furthermore, the process of fusing image spatial features, gesture trajectories, facial expression key points, and skeleton features includes: The fusion process includes reading data groups by segment, aligning the segment data groups with the action category probability sequences on the corresponding frame numbers, identifying stable action segments within the segments and transitional action frames near the boundaries, performing weakening processing on transitional action frames near the boundaries to reduce their proportion in segment fusion, and combining image spatial features, gesture trajectories, facial expression key points, and skeleton features into a segment-level combined data in segment order.

[0010] Furthermore, the process of global semantic information extraction and temporal feature extraction includes: The global semantic information extraction includes reading the boundary position markers at both ends of the segment and the image spatial features and facial expression key points retained near the corresponding boundary positions, and sorting out the switching relationship between segments. The temporal feature extraction includes extracting the continuous unfolding relationship from the stable action segments within the segment and the sequential order between segments, and arranging multiple segment fusion features in chronological order to generate a segment fusion feature sequence.

[0011] Furthermore, the sequence learning module processes the following steps: The sequence learning module includes reading the segment fusion feature sequence, performing sequential association processing on multiple consecutive segments, identifying continuous action segments, switching action segments and pause segments, retaining the main action information in the continuous action segment, and transcribing the transition content in the switched action segment into sequence transition markers.

[0012] Furthermore, the connectionist temporal classification method's decoding, merging of consecutively repeated words, and removal of words representing transitional actions includes the following processes: The connectionist temporal classification method decoding includes aligning and expanding according to the sequence order and eliminating blank positions to obtain the initial sentence sequence. The merging of consecutively recurring words includes merging words with consistent content and temporal continuity in adjacent positions into one word. The removal of words representing transitional actions includes removing words that are transcribed from the transition action segments near the boundary positions and have no stable semantic connection with the main actions before and after from the initial sentence sequence.

[0013] The key innovations of this invention include: (1) Based on the multimodal data input, position division information is generated, and the position division information is used for adjacent frame extraction and fragment-level reconstruction, so that the boundary position and non-boundary position are transformed from the previous judgment result into the organization basis for subsequent fragment-level reconstruction.

[0014] (2) Based on the location segmentation information and the multimodal data input, adjacent frames are extracted and fragments are reassembled. Then, combined with image spatial features, gesture trajectory, facial expression key points and skeleton features, a fragment fusion feature sequence is generated, so that the fragment fusion feature sequence is based on fragment-level reassembly, rather than on the unified feature processing of continuous sign language video.

[0015] (3) Based on the fragment fusion feature sequence, the sequence learning module is processed sequentially, the connectionist temporal classification method is decoded, the words that appear repeatedly are merged, the words that represent transitional actions are removed, and the contextual natural language processing is performed. The fragment-level reorganization, fragment fusion feature sequence generation, and continuous sign language recognition results are organized into a continuous processing link.

[0016] The following are its main beneficial effects: (1) In view of the problem that the boundary position and non-boundary position are difficult to distinguish accurately in the existing scheme and the boundary information is stuck at the segmentation or judgment layer, the position division information is further transmitted to the adjacent frame extraction and fragment-level reconstruction stage, so that the boundary position and non-boundary position in the continuous sign language video participate in the subsequent data organization, the fragment-level reconstruction result maintains the correspondence with the previous judgment result, and the position division information in the continuous sign language recognition link is no longer disconnected from the subsequent processing.

[0017] (2) In response to the problem that the action switching content near the boundary position is easily mixed in after the multimodal data input is unified and fused in the existing scheme, the adjacent frames are extracted and the fragments are recombined first, and then the image spatial features, gesture trajectory, facial expression key points and skeleton features are fused. This makes the fragment fusion feature sequence based on the fragment data after position division. The stable action content in the continuous sign language video and the content near the boundary position are distinguished before entering the sequence learning module.

[0018] (3) In view of the problem that the continuous sign language recognition results in the existing scheme are easily affected by the continuous repetition of words and words representing transitional actions, the sequence learning module is continuously executed after the fusion feature sequence of the segment, the connectionist temporal classification method is decoded, the continuous repetition of words is merged, the words representing transitional actions are removed and the contextual natural language is processed. This ensures that the generation process of the continuous sign language recognition results is consistent with the fusion feature sequence of the preceding segment, thereby reducing the recognition deviation caused by the break between the pre- and post-processing.

[0019] (4) To address the problem of insufficient correspondence between location segmentation information and multimodal data input in the background technology, by organizing the multimodal data input, location segmentation information, fragment-level recombination and fragment fusion feature sequence generation in the same main chain, the continuous sign language video acquisition, key point coordinate extraction, location segmentation, fragment-level recombination and continuous sign language recognition result output form a consistent process, which is convenient for the subsequent sequence learning module to process and call the previous results.

[0020] (5) To address the problem that the fragment fusion feature sequence in the existing scheme is difficult to stably support the subsequent decoding link, by incorporating global semantic information extraction and temporal feature extraction into the fragment fusion feature sequence generation process, the fragment fusion feature sequence carries both internal fragment information and inter-fragment relationship information, so that the subsequent connectionist temporal classification method decoding and contextual natural language processing have a unified input basis. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a deep learning-based continuous sign language recognition method provided in an embodiment of this application. Detailed Implementation

[0022] Example 1: Refer to Figure 1 This is a flowchart illustrating a deep learning-based continuous sign language recognition method provided in an embodiment of the present invention. The process may include at least steps S100-S400: S100: Acquire continuous sign language video, perform human image extraction and normalization processing, and perform gesture trajectory extraction and image spatial feature extraction processing to obtain multimodal data input; S200. Based on the multimodal data input, perform boundary regression branch construction and boundary prediction probability distribution generation processing, and perform boundary location determination and non-boundary location determination processing to generate location division information; S300: Obtain the location segmentation information and the multimodal data input, perform adjacent frame extraction and fragment-level recombination processing to obtain fragment-level recombination results; based on the fragment-level recombination results and the action category probability sequence, perform image spatial features, gesture trajectory, facial expression key points and skeleton features fusion processing to generate fragment fusion features; based on the fragment fusion features and the location segmentation information, perform global semantic information extraction and temporal feature extraction processing to generate fragment fusion feature sequence; S400. Obtain the fragment fusion feature sequence, process it using the sequence learning module to obtain a word sequence or Gloss sequence; based on the word sequence or Gloss sequence, perform connectionist temporal classification decoding, merge consecutively repeated words, and remove words representing transitional actions to generate processed sign language sentences; based on the processed sign language sentences, perform contextual natural language processing to generate continuous sign language recognition results.

[0023] Step S100 includes at least steps S110-S130: S110. Acquire continuous sign language video, perform human image extraction and normalization processing to obtain image information.

[0024] Specifically, the continuous sign language video comes from the continuous acquisition results of the camera device in continuous sign language recognition mode. The acquisition scenario can be a government service window, a hospital guidance terminal, a teaching terminal, or a fixed interactive terminal. The camera device starts acquisition when it detects a human body entering the framing area and the number of consecutive frames reaches a preset number, and writes the acquired video frames into the continuous sign language video buffer.

[0025] The continuous sign language video includes video frames arranged in chronological order, corresponding frame numbers, and acquisition time information. The video frames are used for subsequent human image extraction, the frame numbers are used for subsequent gesture trajectory extraction, and the acquisition time information is used for subsequent adjacent frame sorting.

[0026] Human body image extraction refers to separating the image content containing the upper body, hand movement area and face area from the continuous sign language video. Specifically, the continuous video frames are read one by one, foreground detection and human body region cropping are performed, background images that are off-center from the human movement area are excluded, and images containing the head, shoulders, elbows, wrists and hands are retained. Frames with human body areas that are too small, human body areas that are off-center from the image edge or human body areas that are occluded by more than a preset proportion are written into the abnormal frame record.

[0027] Normalization processing refers to performing size unification, brightness unification, and frame order reordering on the extracted human body image. Size unification is used to adjust the images output by different camera devices to the same resolution, brightness unification is used to eliminate the influence of illumination fluctuations on subsequent key point coordinate extraction, and frame order reordering is used to keep the video frames arranged continuously according to the order of acquisition.

[0028] Understandably, when the number of abnormal frame records exceeds a preset limit, the current video segment is marked as a segment to be re-acquired, and the original frame is retained in the current continuous sign language video buffer for subsequent re-reading; when the number of abnormal frame records does not reach the preset limit, the system continues to perform normalization processing on the valid frames and outputs usable results. After the above processing, the image information is constructed into a unified data structure containing human image, frame number, acquisition time information, and abnormal frame records. The "image information" is written into the step result area as the output field name of this step and is directly called by "image information" in S120. At the same time, the image information is kept traceable within this main step, which facilitates the use of the same source data for subsequent key point coordinate extraction and image spatial feature extraction.

[0029] S120. Based on the image information, perform key point coordinate extraction for the hand, key point coordinate extraction for the face, and key point coordinate extraction for the hand bones and joints to generate key point coordinate information.

[0030] Specifically, the image information serves as the sole input source for the current step. Human body images are retrieved frame by frame according to frame number, and then the hand region localization link, face region localization link, and upper limb skeleton localization link are entered respectively. The hand key point coordinates refer to the position coordinates of the fingertips, knuckle joints, palm reference points, and wrist joints of both hands in each frame of the image. During extraction, the left and right hand regions are first identified from the human body image, and then the hand contour and joint points are searched point by point within the left and right hand regions. The left and right hand key points of each frame are written into the hand key point coordinate table according to the principle of "left hand priority, right hand last, and grouping within the same frame".

[0031] The facial key point coordinates refer to the positional coordinates of facial contour points, eyebrow points, eye points, nose points, and mouth points. During extraction, the head region is first locked, and then local localization is performed on the facial region to obtain a set of key points that reflect changes in facial expressions and mouth shape. This set of key points is then written into a facial key point coordinate table according to frame number correspondence. The hand bone joint coordinates refer to the positional coordinates of the shoulder, elbow, wrist, and bone connection points linked to hand movements. During extraction, based on the upper limb region in the human image, the joint connection sequence from shoulder to wrist is established, and the coordinate results of each connection point are written into a hand bone joint coordinate table.

[0032] Furthermore, during the extraction of the three types of keypoint coordinates, the system performs missing marker processing for situations such as missing left and right hands, excessive facial deflection, and broken bone connections in the same frame image. Missing marker processing includes missing location registration, frame-to-frame interpolation, and abnormal state recording. Frame-to-frame interpolation is only performed on a small number of consecutive missing frames; for a large number of consecutive missing frames, the missing markers are retained and passed to subsequent steps for recognition. This is because the subsequent boundary regression branch and action regression branch both rely on positional changes between consecutive frames. If the current step does not record the source of the missing frames, the missing location, and the source of interpolation, the subsequent gesture trajectory extraction and position segmentation information generation will lack stable input.

[0033] After completing the extraction of hand keypoint coordinates, facial keypoint coordinates, and hand bone joint coordinates, the system organizes the three types of coordinate results according to the same frame number, the same acquisition time information, and the same human image source to obtain the keypoint coordinate information. The keypoint coordinate information includes hand keypoint coordinates, facial keypoint coordinates, hand bone joint coordinates, and corresponding missing marker records. The "keypoint coordinate information" is written into the step result area as the output field name of this step and is directly available for S130 to call the "keypoint coordinate information". At the same time, the keypoint coordinate information maintains a one-to-one correspondence with the aforementioned image information, which facilitates the synchronous call of coordinate information and image source information in the subsequent multimodal data input formation stage.

[0034] S130. Based on the key point coordinate information, perform gesture trajectory extraction and image spatial feature extraction processing to generate multimodal data input.

[0035] Specifically, the key point coordinate information serves as the core input for the current step, first entering the gesture trajectory extraction link. The gesture trajectory refers to the combination of the movement path, movement direction, pause position, and turning position of the key points of both hands in consecutive frames. During extraction, the coordinates of the key points of the hands and the coordinates of the hand bones and joints are read in the order of the frame number. The displacement changes of the wrist connection point, palm reference point, and fingertip point in adjacent frames are compared to obtain the movement direction information between consecutive frames. Then, trajectory segments are generated based on the continuous displacement direction changes, displacement amplitude changes, and pause frame distribution. Multiple trajectory segments are registered separately for the left and right hands, thereby forming a gesture trajectory record that can reflect the continuity of continuous sign language movements.

[0036] The image spatial features refer to the structural content of the human body image from the aforementioned image information in terms of spatial location, including the relative positions of the hands, the relative positions of the face and hands, the extension state of the upper limbs, and the regional distribution relationships in the human body image. Since image spatial features cannot be completely replaced by coordinate tables alone, the current step, while reading the key point coordinate information, simultaneously retrieves the image information corresponding to that key point coordinate information, performs region mapping and spatial relationship organization from the corresponding human body image, and obtains image spatial feature records arranged frame by frame. Here, the relative positions of the hands are used to describe the distance and vertical relationship between the left and right hands in the same frame; the relative positions of the face and hands are used to describe the distance and interaction between the hands and the face; the extension state of the upper limbs is used to describe the spatial extension of the shoulders, elbows, and wrists; and the regional distribution relationships are used to describe the relative layout of the hand region, face region, and upper limb region in the human body image.

[0037] Furthermore, the system sets synchronous triggering conditions during gesture trajectory extraction and image spatial feature extraction. Specifically, the current frame only participates in multimodal data input construction when usable hand keypoint coordinates, facial keypoint coordinates, or hand bone joint coordinates exist within the same frame. For frames with localized gaps but still forming a continuous relationship, the system retains the gap marker and continues assembly. For frames where the keypoint coordinates are completely invalid and the image spatial features cannot be mapped, the system registers them as discardable frames and removes them from the current trajectory segment. After completing gesture trajectory extraction and image spatial feature extraction, the system uniformly writes the hand keypoint coordinates, facial keypoint coordinates, hand bone joint coordinates, gesture trajectory, and image spatial features into the same data structure to generate the multimodal data input.

[0038] The multimodal data input is written into the step result area as the final output field name of this main step and can be directly called by "multimodal data input" in S210. At the same time, the multimodal data input also provides a preceding source for the subsequent action regression branch construction in S220, the adjacent frame extraction and fragment-level reconstruction in S310, and the image spatial features, gesture trajectory, facial expression key points and skeleton feature fusion processing in S320, so that the boundary probability sequence generation, position division information generation and fragment fusion feature generation can share the same set of basic inputs.

[0039] In summary, this step converts continuous sign language video into image information, then into key point coordinate information, and finally into multimodal data input. The hand, face, and upper limb information in the continuous sign language video is uniformly processed before entering the boundary regression branch. Compared to directly feeding the entire continuous video frame into the recognition chain, this step places gesture trajectories and image spatial features within the same preceding data structure. Subsequent boundary probability sequence generation and segment-level reconstruction have a unified input source, and transition frames, missing frames, and abnormal frames in the continuous sign language video are also recorded simultaneously.

[0040] Step S200 includes at least steps S210-S230: S210. Obtain the multimodal data input, perform boundary regression branch construction and boundary prediction probability distribution generation processing to obtain the boundary probability sequence.

[0041] Specifically, the multimodal data input comes from the result generated in S130. This multimodal data input includes hand keypoint coordinates, facial keypoint coordinates, hand bone joint coordinates, gesture trajectory, and image spatial features, maintaining a one-to-one correspondence with the frame numbers in the continuous sign language video. The current step is executed by the boundary regression branch in the continuous sign language recognition main chain. This boundary regression branch is a pre-processing unit located before the action regression branch, and it comprises an input receiving unit, a timing organization unit, a boundary determination unit, and a boundary recording unit.

[0042] The input receiving unit is responsible for reading the multimodal data input frame by frame. The timing processing unit is responsible for organizing the hand keypoint coordinates, facial keypoint coordinates, hand bone joint coordinates, gesture trajectory, and image spatial features of the same frame into a data group for the same frame, and forming a sequence to be judged according to the order of consecutive frames. The boundary judgment unit is used to extract the content related to the switching position of continuous sign language actions from the sequence to be judged. The content related to the switching position specifically includes the turning segment in the gesture trajectory, the displacement change segment in the hand bone joint coordinates, the expression switching segment in the facial keypoint coordinates, and the segment of relative position change of the hands in the image spatial features. The reason for performing this step first is that the boundary position in continuous sign language video is usually accompanied by changes in gesture trajectory direction, changes in bone connection state, and changes in facial expression. If the sequence learning module is directly entered, the transitional actions will be mixed with the stable actions, and the subsequent segment-level reconstruction will lack a reliable basis.

[0043] Furthermore, the boundary regression branch first performs a continuity check on the same frame data group during runtime. The continuity check includes checking whether the frame numbers are consecutive, whether keypoint coordinates are missing, whether the gesture trajectory is interrupted, and whether the image spatial features correspond to the frame numbers. If a small number of frames are missing, the corresponding keypoint coordinates and gesture trajectories are read from adjacent frames to fill in the gaps, and the source of the gaps is registered in the boundary recording unit. If the number of consecutive missing frames exceeds a preset limit, the segment is registered as an abnormal segment, and the boundary determination of the current abnormal segment is paused; only the remaining consecutive segments are processed. After completing the continuity check, the timing processing unit performs position change comparison and state change comparison on each consecutive segment. Position change comparison is used to compare the change magnitude of hand keypoint coordinates and hand bone joint coordinates in adjacent frames. State change comparison is used to compare the change state of facial keypoint coordinates, gesture trajectory direction, and image spatial features in adjacent frames. When trajectory changes, skeletal displacement changes, and facial state changes occur simultaneously near the same location, the boundary determination unit marks the corresponding frame as a boundary candidate frame. When the change only occurs locally and the preceding and following frames remain stable, the boundary determination unit retains the corresponding frame as a non-boundary candidate frame. Subsequently, the boundary determination unit performs boundary prediction probability distribution generation processing on the boundary candidate frames and non-boundary candidate frames, forming a boundary prediction probability distribution arranged by frame. The boundary prediction probability distribution is used to describe the degree to which each frame belongs to the action boundary, and is written into a unified result table by the boundary recording unit according to the frame number.

[0044] Understandably, in the continuous sign language recognition mode at the government service window, the camera continuously collects continuous sign language videos of the applicants. The system first extracts the coordinates of hand key points, facial key points, and image spatial features from the multimodal data input for several consecutive frames. When the hands move from the chest to near the face, and the state of the mouth key points changes synchronously, the boundary regression branch marks the adjacent frames before and after this segment as high boundary candidate areas; when the hands move continuously in the same direction and the facial state remains unchanged, the boundary regression branch marks this segment as a low boundary candidate area. After boundary prediction probability distribution generation processing, the boundary determination results of each frame in the current continuous segment are organized into the boundary probability sequence. The boundary probability sequence is recorded as the output field name of the current step in the step result area and is called by "boundary probability sequence" in S220. At the same time, the boundary probability sequence will continue to participate in the boundary position determination and non-boundary position determination processing in S230, and will serve as the source of input in the subsequent adjacent frame extraction and segment-level reconstruction processing in S310.

[0045] S220. Based on the boundary probability sequence and the multimodal data input, perform action regression branch construction and category prediction probability distribution generation processing to generate action category probability sequence.

[0046] Specifically, the boundary probability sequence comes from the output of S210, and the multimodal data input comes from the output of S130. The current step is executed by the action regression branch in the continuous sign language recognition main chain. The action regression branch is an action recognition unit that runs in parallel with the boundary regression branch and is sequentially followed by it. It consists of an action input processing unit, an action classification unit, a category prediction probability distribution generation unit, and an action category recording unit. The action input processing unit reads the boundary probability sequence and the multimodal data input, and synchronizes the boundary state and image content according to the frame number. Here, the action category is not the final continuous sign language recognition result, but an intermediate result reflecting the action state of the current frame. The action category prediction probability distribution generation process does not directly output the sign language, but provides the action state basis for subsequent boundary position determination, non-boundary position determination, and fragment-level reconstruction.

[0047] Furthermore, the action input processing unit first separately labels the high-boundary-probability frames and low-boundary-probability frames in the boundary probability sequence, and then extracts the image spatial features corresponding to the current frame from the multimodal data input as the main input of the action regression branch. Simultaneously, it combines the gesture trajectory, hand keypoint coordinates, and hand bone joint coordinates as action state correction inputs. This setup is because the action regression branch needs to identify whether the current frame is in a stable action, action transition, or transitional action state. Relying solely on image spatial features can easily misinterpret short-term changes near the boundary positions as stable actions. By inputting the boundary probability sequence together, the action classification unit can reduce the weight of continuous action continuation judgment near high-boundary-probability frames and increase the weight of continuous action continuation judgment in the low-boundary-probability frame range. In specific implementation, the action classification unit performs region correspondence analysis on the image spatial features in consecutive frames, interpreting the action state based on the relative positions of the hands, the relative positions of the hand and face, the upper limb extension state, and the regional distribution relationships in the human body image. Then, it combines the directional continuity in the gesture trajectory and the joint connection changes in the hand bone joint coordinates to generate a candidate set of action states for the current frame. The category prediction probability distribution generation unit performs category prediction probability distribution generation processing on the action state candidate set to obtain the action category probability distribution arranged by frame. The action category probability distribution describes the degree to which the current frame falls into each action category, and the action category recording unit registers it together with the corresponding frame number, the corresponding boundary probability, and the corresponding image spatial features.

[0048] In a feasible engineering embodiment, the hospital's triage terminal continuously receives continuous sign language videos from hearing-impaired individuals and continues running the current step after the boundary probability sequence has been output in S210. For segments with several consecutive frames where both hands remain in front of the chest and the gesture trajectory changes smoothly, the action regression branch categorizes the image spatial features of the segment into the same action category candidate set and outputs a high stable action category prediction probability distribution. For segments where both hands suddenly rise and the facial key point states switch, since the corresponding boundary probability sequence has been marked as a high boundary probability region, the action regression branch will categorize it into the transition action or switching action category candidate set. In this way, the current step forms an intermediate description of the action state of consecutive frames that matches the boundary probability sequence. After the category prediction probability distribution generation process, the action category probability sequence is obtained. The action category probability sequence is recorded as the output field name of the current step in the step result area and is called by "Action Category Probability Sequence" in S230. At the same time, the action category probability sequence is also called by the subsequent image spatial features, gesture trajectory, facial expression key points, and skeleton feature fusion processing in S320, so that the segment fusion features can share the action state information output by the current step.

[0049] S230. Based on the boundary probability sequence and the action category probability sequence, perform boundary position determination and non-boundary position determination processing to generate position division information.

[0050] Specifically, the boundary probability sequence comes from S210, and the action category probability sequence comes from S220. The current step is executed by the position segmentation unit. The position segmentation unit includes a position comparison unit, a boundary position determination unit, a non-boundary position determination unit, and a position segmentation information recording unit. The position comparison unit first reads the boundary probability and action category prediction probability distribution of the same frame, and performs boundary state comparison and action state comparison on each frame simultaneously. The boundary position determination process refers to selecting frames from the boundary probability sequence whose boundary probability is higher than a preset condition and whose corresponding action category prediction probability distribution shows a trend of switching or transitioning actions, and marking them as boundary positions. The non-boundary position determination process refers to selecting frames from the boundary probability sequence whose boundary probability is lower than a preset condition and whose corresponding action category prediction probability distribution shows a trend of stable action continuation, and marking them as non-boundary positions. Here, the boundary positions are used for subsequent global semantic information extraction and suppression of transitional action frames near the boundary positions, while the non-boundary positions are used for subsequent extraction of adjacent frames, fragment-level reconstruction, and extraction of continuous sign language action temporal features.

[0051] Furthermore, when performing boundary and non-boundary position determination, the position comparison unit does not make isolated judgments on single frames, but rather performs continuity judgments on consecutive frame windows. If the boundary probability of a single frame increases but the adjacent frames maintain a stable action category, the frame is registered as a frame to be confirmed, and subsequent consecutive frames are observed. If the boundary probability of several consecutive frames continues to increase, and the action category prediction probability distribution synchronously shows the action switching trend, the center frame or the vicinity of the center frame in this continuous interval is registered as a boundary position. Correspondingly, if the boundary probability of several consecutive frames remains low, and the action category prediction probability distribution maintains the same stable action continuation trend, the interval containing these frames is registered as a non-boundary position. The reason for this is that action switching in continuous sign language videos usually spans multiple adjacent frames. If only single-frame high and low probabilities are used for segmentation, short-term fluctuations are easily mistaken for boundary positions, and transitional actions are easily mixed into the subsequent segment-level reconstruction results. To this end, the position division unit also sorts the adjacency relationship between boundary positions and non-boundary positions, and registers the non-boundary position segments before and after the same boundary position as the front continuous area and the back continuous area, respectively, for direct use by the subsequent adjacent frame extraction and segment-level reconstruction processing of S310.

[0052] Understandably, in the actual scenario of the teaching terminal, when a student makes continuous sign language gestures to the camera device, the boundary regression branch has already provided a sequence of boundary probabilities arranged by frame, and the action regression branch has already provided a sequence of action category probabilities. In the current step, after overlaying and comparing the two frame by frame, when it is detected that the boundary probability continuously increases for several consecutive frames, and the action category changes from a stable action to a switching action, the position partitioning unit registers the specified frame in that segment as a boundary position; when it is detected that the boundary probability remains low for several consecutive frames, and the action category always remains in the same continuous state, the position partitioning unit registers that segment as a non-boundary position. After processing, the boundary positions and the non-boundary positions are uniformly organized into the position partitioning information. The position partitioning information is recorded as the output field name of the current step in the step result area and is called by "position partitioning information" in S310. Simultaneously, this position partitioning information will also participate in the global semantic information extraction and temporal feature extraction processing of S330 together with the multimodal data input. In summary, this step unifies the outputs of the boundary regression branch and the action regression branch into location segmentation information. This allows the boundary probability sequence to move beyond the segmentation result layer and continue into subsequent fragment-level reconstruction and fragment fusion feature formation stages. Compared to methods that only remove transitional action words after decoding, this step pre-classifies boundary and non-boundary locations, providing a direct basis for subsequent adjacent frame extraction and fragment-level reconstruction. The interference of transitional actions on fragment fusion features is reduced in the early stages.

[0053] Step S300 includes at least steps S310-S330: S310. Obtain the location segmentation information and the multimodal data input, perform adjacent frame extraction and fragment-level reconstruction processing, and obtain the fragment-level reconstruction result.

[0054] Specifically, the location segmentation information comes from S230, and the multimodal data input comes from S130. The current step is executed by a segment-level reconstruction unit, which includes a frame reading unit, a location matching unit, an adjacent frame extraction unit, a reconstruction unit, and a recording unit. The frame reading unit retrieves the hand keypoint coordinates, facial keypoint coordinates, hand bone joint coordinates, gesture trajectory, and image spatial features from the multimodal data input according to the frame sequence number. The location matching unit matches the data in the same frame with the boundary and non-boundary positions in the location segmentation information frame by frame. Here, adjacent frame extraction refers to extracting a group of consecutively arranged frames within the same non-boundary position segment in chronological order; segment-level reconstruction refers to reorganizing consecutive frames within the same non-boundary position segment into segment data groups and adding adjacent boundary position markers to both ends of the segments. The reason for this processing is that stable movements in continuous sign language videos often appear in non-boundary position segments, while transitional movements are often present near boundary positions. If location matching is not performed first, subsequent segment fusion features are easily mixed with the content of the switching segments.

[0055] Furthermore, the adjacent frame extraction unit first reads a continuous frame window before and after a boundary position, and then extracts the non-boundary positions within the window to form independent candidate segments. For candidate segments with fewer than a preset number of frames, the reassembly unit performs an adjacency check with the preceding or following candidate segment. When the gesture trajectory direction is continuous, the hand bone joint coordinates change smoothly, and there is no obvious switching in the image spatial features, segment merging is performed. When the facial key point coordinates change abruptly or the gesture trajectory shows a significant turn, the current candidate segment is retained and registered as a short segment. For candidate segments with a preset number of frames, the reassembly unit directly arranges them sequentially according to the frame number to generate segment data groups.

[0056] Regarding anomaly handling, if a candidate segment contains a small number of missing frames, similar coordinate information is extracted from adjacent frames to fill in the missing frames, and the source of the missing frames is written into the recording unit. If a candidate segment contains too many consecutive missing frames, the segment is registered as a discarded segment and not sent to the subsequent fusion process. Understandably, in continuous sign language recognition mode, when a person performs the same action segment consecutively, the system extracts consecutive frames from the non-boundary position segment corresponding to that action segment and reassembles them into a segment data group. When the action changes, frames near the boundary position are marked separately and not merged into the current segment data group. After processing, the reassembly unit outputs the segment-level reassembly result. This result is written into the recording unit as the output field name of the current step and is available for use by S320's "segment-level reassembly result." Simultaneously, this result retains the correspondence with the boundary position, facilitating reading in the subsequent segment fusion feature generation stage.

[0057] S320. Based on the fragment-level recombination result and the action category probability sequence, perform image spatial features, gesture trajectory, facial expression key points and skeleton features fusion processing to generate fragment fusion features.

[0058] Specifically, the fragment-level reconstruction result comes from S310, and the action category probability sequence comes from S220. This current step is executed by a fusion unit, which includes a fragment reading unit, an action alignment unit, a boundary suppression unit, a feature fusion unit, and a fragment fusion recording unit. The fragment reading unit reads the fragment data groups from the fragment-level reconstruction result one by one. The action alignment unit aligns the fragment data group with the action category probability sequence on the corresponding frame number. Here, image spatial features refer to the relative positions of the hands, the relative positions of the hands and face, the upper limb extension state, and the regional distribution relationship; gesture trajectory refers to the movement path, movement direction, and turning point in consecutive frames; facial expression key points refer to the combined data of facial contour points, eyebrow points, eye points, and mouth points; skeletal features refer to the connection state and extension state of the shoulder, elbow, and wrist obtained by organizing the hand bone joint coordinates. All four types of features come from previous steps, but in this step, they are no longer processed in isolation as single frames, but rather processed as a whole around the fragment data group.

[0059] Furthermore, the action alignment unit first identifies the predicted probability distribution of action categories in each frame within the segment data group, and then internally marks them as high-probability action segments, low-probability action segments, and transition action segments. The boundary suppression unit then reads the boundary position markers attached to both ends of the segment and performs weakening processing on transition action frames near the boundary positions. The weakening processing is not deleting the frame, but reducing the proportion of the frame in the segment fusion, while retaining the image spatial features and facial expression key point records of the frame for subsequent global semantic information extraction. The feature fusion unit then performs fusion processing on the remaining frames within the segment. Specifically, it first organizes the image spatial features in the same segment, then arranges the gesture trajectories in order, and then attaches facial expression key points and skeleton features, finally forming a segment-level combined data. The fusion processing here is not a simple splicing, but rather places stable action segments in the center according to the segment order, and places weakened frames near the boundary at the beginning and end of the segment, making the internal structure of the segment closer to the actual unfolding order of continuous sign language actions. In an operable engineering embodiment, after the teaching terminal collects continuous sign language videos of students, the system obtains a segment data group from a certain non-boundary position segment. In this data segment, the first two frames are close to the previous boundary, the middle frames represent stable actions, and the last two frames are close to the next boundary. The boundary suppression unit weakens the first and last frames, and then the feature fusion unit combines the image spatial features, gesture trajectory, facial expression key points, and skeleton features of the stable action segments into a segment-level data. After processing, the segment fusion feature is obtained. The segment fusion feature is written as the output field name of the current step into the segment fusion record unit and can be called by the "segment fusion feature" function of S330. At the same time, this result can also be traced back to the original data segment and the original action category probability sequence by the subsequent sequence learning module.

[0060] S330. Based on the fragment fusion features and the positional segmentation information, perform global semantic information extraction and temporal feature extraction processing to generate a fragment fusion feature sequence.

[0061] Specifically, the segment fusion features come from S320, and the positional segmentation information comes from S230. The current step is executed by the sequence organization unit, which includes a segment sorting unit, a global semantic information extraction unit, a temporal feature extraction unit, a sequence generation unit, and a sequence recording unit. The segment sorting unit first arranges multiple segment fusion features according to the chronological order of the original continuous sign language video, and then restores the boundary positional relationships between each segment based on the positional segmentation information. Here, global semantic information refers to the semantic connection content between segments obtained by combining image spatial features near the boundary positions, facial expression key points, and the relationship between segments; temporal features refer to the sequence of actions within a segment, segment length, the sequential connection relationship between segments, and the pause state between segments. The global semantic information extraction unit first reads the boundary position markers at both ends of each segment, then reads the image spatial features and facial expression key points retained near the corresponding boundary positions, and organizes the switching relationship between segments. The temporal feature extraction unit then extracts the continuous unfolding relationship from the stable action segments within the segments and the sequential order between segments. The reason for this approach is that the fusion features of a single segment can only describe local action content. The subsequent sequence learning module also needs to know how multiple segments are connected in order to generate a continuous word sequence or Gloss sequence.

[0062] Furthermore, the segment sorting unit performs adjacency checks among multiple segment fusion features. If there is a short-term boundary position between two adjacent segments, and the facial expression key points near the boundary position change smoothly and the image spatial features are continuous, the global semantic information extraction unit registers the two segments as an adjacent switching relationship; if there is a significant pause, facial state change, or change in the relative position of the hands between two adjacent segments, they are registered as a separate switching relationship. The temporal feature extraction unit generates continuous segments within segments and switching segments between segments based on the segment length and frame number span. Subsequently, the sequence generation unit concatenates the segment fusion features of each segment with the corresponding global semantic information and temporal features in sequence to form a segment fusion feature sequence unfolded chronologically. In terms of anomaly handling, if a segment lacks a boundary position correspondence, the sequence generation unit reads the position division information from the adjacent segments to fill in the connection relationship and writes the source of the filling into the sequence recording unit; if multiple consecutive segments lack connection relationships, the segment group is registered as a segment group to be reviewed and does not enter the current round of sequence learning module.

[0063] In a complete engineering embodiment, after obtaining multiple segment fusion features from the same continuous sign language video, the government service window terminal first sorts them according to the original frame number order, then reads the boundary positions at both ends of each segment, extracts facial expression key points and image spatial features near the boundary positions, and sorts out the switching relationship between segments; then, it extracts the action continuity relationship within each segment to generate a continuous segment fusion feature sequence. After processing, the segment fusion feature sequence is written into the sequence recording unit as the output field name of the current step and is called by "segment fusion feature sequence" in S410. At the same time, the segment fusion feature sequence is also the prerequisite input basis for subsequent connectionist temporal classification method decoding and contextual natural language processing.

[0064] In summary, this step further refines the fragment fusion features into a fragment fusion feature sequence with global semantic information and temporal features, expanding the continuous sign language recognition object from a single fragment to a sequence of fragments with sequential relationships. Compared to directly feeding a fixed hypergraph or unified fusion result into the sequence learning module, this step first restores the switching relationships between fragments and then organizes the continuous relationships within the fragments. The input link upon which subsequent word sequences or Gloss sequences are based is more complete, and the interference from transitional actions near boundary positions is also lower.

[0065] In one specific embodiment, the multimodal data input and location segmentation information from the previous stage are used to perform structural reorganization and deep fusion of multimodal features on the timeline of continuous sign language videos. The multimodal data input includes hand keypoint coordinates, facial keypoint coordinates, hand skeletal joint coordinates, gesture trajectory, and image spatial features, and each frame of data corresponds to a frame number; the location segmentation information includes frame-level labels for boundary and non-boundary positions, as well as the registration results of continuous segments before and after the boundary positions. In S310, the system first extracts adjacent frames within non-boundary segments, extracting multiple consecutive frames within the same non-boundary segment from the original frame sequence to form independent candidate segments. Then, it performs segment-level reassembly processing on these candidate segments. The reassembly unit performs adjacency checks based on the number of frames in each candidate segment. When the gesture trajectories of two adjacent candidate segments are continuous, the hand bone joint coordinates change smoothly, and there is no significant shift in image spatial features, the candidate segment is merged with the preceding or following candidate segment. If there are abrupt changes in facial keypoint coordinates or significant turns in the gesture trajectory, the segment is retained as a short segment and registered. Adjacent boundary markers are added to both ends of the segment to indicate its transition relationship with the preceding and following segments. If a candidate segment contains a small number of missing frames, the system extracts similar coordinate information from adjacent frames to fill in the missing frames. If the number of consecutive missing frames exceeds a preset limit, the segment is discarded. After completing the above processing, the system outputs the segment-level reassembly result, which contains multiple segment data groups. Each segment data group consists of consecutive frames within the same non-boundary segment, and each segment has boundary markers at both ends for use by S320.

[0066] Formula ① is used to quantify the continuity of gesture trajectories within candidate segments to determine whether the segment merging condition is met: , in: Candidate fragments With candidate fragments Indicators of the continuity of gesture trajectories between individuals; : Index number of the candidate fragment; Candidate fragments and The frame span between adjacent endpoints (i.e., the number of frames used for comparison). : The frame index within the region of adjacent endpoints of two segments, with a value ranging from 1 to ; : No. Frame and the The angular difference in the direction of movement of key hand points between frames; The cosine function maps the angle difference to a measure of directional consistency. : No. The displacement amplitude of the hand bone joint coordinates in the frame (i.e., the Euclidean distance of the key point movement between adjacent frames). : No. The displacement amplitude of the hand bone joint coordinates in the frame; Take the smaller of the two values; Take the larger of the two values; : Vector magnitude operator, used here to calculate displacement magnitude.

[0067] Simple numerical example: If candidate fragments and Frame span between adjacent endpoints The angle differences between the frames are 0.10, 0.12, 0.09, 0.11, and 0.10 radians, respectively, and the displacement amplitude ratios are 0.95, 0.97, 0.96, 0.94, and 0.98, respectively. Therefore, the calculated values ​​are... This indicates that the trajectories of the two segments are highly continuous. This index solves the problem of how to quantitatively determine whether two candidate segments belong to the same stable action segment, providing a numerical basis for segment merging.

[0068] Formula ② is used to interpolate and fill in the keypoint coordinates of missing frames within a candidate segment: , in: ; ; ; ; ; ; .

[0069] Simple numerical example: If a frame is missing Previous valid frames The wrist keypoint coordinates are (120, 240) pixels, and the subsequent effective frames If the wrist keypoint coordinates are (130, 250) pixels, then the interpolation will yield... Pixels. This formula solves the problem of incomplete fragment data groups caused by missing keypoints within candidate fragments, ensuring the continuity of input for subsequent fusion steps. Formula ① As an input for segment merging in the fragment-level recombination results, Formula ② As the coordinate information after being completed within the fragment data group, the two together constitute the fragment-level reconstruction result output by S310, which is then called by the "fragment-level reconstruction result" of S320.

[0070] Furthermore, in S320, the system performs multimodal feature fusion based on the fragment-level recombination results and the action category probability sequence. The fusion unit reads the fragment data groups one by one, and the action alignment unit aligns the fragment data groups with the action category probability sequences on the corresponding frame numbers, identifies the action states of each frame within the fragment, and marks them internally as high-probability action segments, low-probability action segments, and transition action segments. The boundary suppression unit reads the boundary position markers attached to both ends of the fragment, performs weakening processing on transition action frames near the boundary position, reduces the proportion of such frames in the fragment fusion, and retains their image spatial features and facial expression key point records for subsequent use. The feature fusion unit then performs fusion processing on the remaining frames within the fragment, first organizing the image spatial features in the same fragment, then arranging the gesture trajectories, and then attaching facial expression key points and skeleton features, finally forming a fragment-level combined data. This fusion processing is not a simple splicing, but rather places stable action segments in the center according to the fragment order, and places weakened frames near the boundary at the beginning and end of the fragment, making the internal structure of the fragment closer to the actual unfolding order of continuous sign language actions. After processing, the system outputs fragment fusion features, which contain the dominant content of stable action segments within the fragment and supplementary information near the boundaries, for S330 to call.

[0071] Formula ③ is used to calculate the weight coefficients of each frame within a segment during fusion, to reflect the constraint of action category probability on the fusion process: , in: ; ; ; : No. The action category confidence factor for each segment is determined by the average action category probability of stable action segments within that segment, and its value ranges from [value range missing]. ; : No. The first segment The action category probability value of a frame is derived from the action category probability sequence, and its value range is... ; : No. The boundary suppression factor for each segment is determined by the position markers at both ends of the segment, and its value ranges from [value range missing]. ; : No. The first segment The frame boundary state indicator value is 1 if the frame is at a boundary position, and 0 otherwise.

[0072] Simple numerical example: The average probability of a stable action segment in a certain fragment is 0.85. Boundary inhibition factor intermediate frame , ,but Frames near the boundary , ,but The weight of boundary frames is significantly reduced. This formula solves the problem of how to dynamically adjust frame-level fusion weights based on action state and boundary position, achieving pre-suppression of transitional action frames.

[0073] Formula ④ is used to generate the fused feature vector within the fragment: , in: : No. The fusion feature vector of each segment; ;No. The number of frames in each segment; Frame index within the segment, from 1 to ; The first one obtained from formula ③ Frame weighting coefficients; These are modal weighting factors for image spatial features, gesture trajectory, facial expression key points, and skeleton features, respectively, preset by the system; : No. The image spatial feature vector of the frame; : No. The gesture trajectory feature vector of the frame; : No. Facial expression key point feature vectors of the frame; : No. The skeleton feature vector of the frame; : Vector concatenation operator, which joins multiple feature vectors along the feature dimension into a longer vector.

[0074] Simple numerical example: a segment The weights for each frame are 0.6, 1.2, 1.4, 1.3, and 0.7, respectively. The spatial features of each frame are 128-dimensional vectors: gesture trajectory is 64-dimensional, facial keypoints are 96-dimensional, and skeleton features are 32-dimensional. All modal weights are set to 1.0. Therefore, after fusion... for A weighted combination vector of dimensions. This formula addresses the problem of integrating multimodal features into fragment-level representations under action-state constraints, generating structured fragment fusion features. Formula ③ As an input for weight calculation in S320, formula ④ As the core content of fragment fusion features, the two together constitute the fragment fusion features output by S320, which can be called by S330's "fragment fusion features".

[0075] Furthermore, in S330, the system performs global semantic information extraction and temporal feature extraction based on segment fusion features and positional segmentation information. The sequence arrangement unit first arranges multiple segment fusion features according to the chronological order of the original continuous sign language video, and restores the boundary positional relationships between each segment based on the positional segmentation information. The global semantic information extraction unit reads the boundary position markers at both ends of each segment, as well as the image spatial features and facial expression key points retained near the corresponding boundary positions, and organizes the switching relationships between segments, including adjacent switching relationships and separate switching relationships. The temporal feature extraction unit extracts the continuous unfolding relationship from the stable action segments within segments and the chronological order between segments, including the action continuation order within segments, segment length, segment sequential connection relationship, and pause state between segments. Subsequently, the sequence generation unit concatenates the segment fusion features of each segment with the corresponding global semantic information and temporal features in sequence to form a segment fusion feature sequence unfolded chronologically. For segments lacking boundary positional correspondence, the system reads the positional segmentation information from adjacent segments to construct the connection relationship; if multiple consecutive segments lack connection relationships, the segment group is registered as a segment group to be reviewed and does not enter the subsequent sequence learning module. After processing, the system outputs a fragment fusion feature sequence, which arranges the fragment fusion features in chronological order and includes the switching relationship between fragments and the time features within the fragments, for S410 to call.

[0076] Formula ⑤ is used to quantify the switching similarity between adjacent segments in order to determine the type of switching relationship: , in: : No. The segment and the first The similarity index of switching between segments; Fragment index; The number of image spatial feature points used for comparison near the boundary location; The index of spatial feature points in the image, from 1 to... ; : No. The end of the segment and the first The first segment The directional angle difference of each spatial feature point in the image; The cosine function maps angle differences to similarity. : The number of facial expression key points used for comparison near the boundary location; Index of key facial expression points, from 1 to... ; : No. The end of the segment The coordinate modulus of a facial expression key point (i.e., the Euclidean norm of the key point vector). : No. The first segment The coordinates of key facial expression points.

[0077] Simple numerical example: If The angular differences are 0.15, 0.18, and 0.12 radians, respectively. If the facial key point modulus ratios are 0.92 and 0.88 respectively, then... , This indicates that the transition between the two segments is relatively smooth. This formula solves the problem of how to quantitatively describe the transition relationship between segments, providing a numerical basis for global semantic information extraction.

[0078] Formula ⑥ is used to generate the first fragment in the fragment fusion feature sequence. Sequence units: , in: : No. Fragment fusion feature sequence units; Fragment index; The first one obtained from formula ④ The fusion feature vector of each segment; : No. The global semantic information vector of each segment is obtained by organizing the boundary position markers at both ends of the segment and nearby features; : No. The temporal feature vector of each segment contains information such as segment length, internal duration, and pause status between segments; : No. The segment and the previous segment (index) The switching similarity of ) is calculated by formula ⑤ (if (Then it is considered the default value). : No. The first segment and the next segment (index) The switching similarity of ) is calculated by formula ⑤ (if If it is the last segment, it is considered the default value); : Vector concatenation operator.

[0079] Simple numerical example: a segment It is a 1600-dimensional vector. It is 64-dimensional. With a 32-dimensional dimension, plus two switching similarity scalars, then for The formula addresses the problem of integrating fragment fusion features, global semantic information, temporal features, and switching relationships into sequence units, generating a structured fragment fusion feature sequence. Formula ⑤... and As an input to the construction of sequence units in S330, formula ⑥ As the core content of the fragment fusion feature sequence, the two together constitute the fragment fusion feature sequence output by S330, which is called by "fragment fusion feature sequence" of S410.

[0080] This section summarizes the technical effects: S310 reconstructs the continuous time axis into semantic segment units through segment-level recombination; S320 achieves pre-suppression of transitional actions through action state constraint fusion; and S330 extracts global semantic and temporal features to form a structured sequence, making the input link of subsequent sequence learning modules more complete and effectively reducing the interference of transitional actions near the boundary.

[0081] The continuous time axis is reconstructed into semantic segment units through fragment-level recombination in S310, and the transitional action pre-suppression is achieved through action state constraint fusion in S320. The structured sequence is formed through global semantic and temporal feature extraction in S330, making the input link of the subsequent sequence learning module more complete and effectively reducing the interference of transitional actions near the boundary.

[0082] Step S400 includes at least steps S410-S430: S410. Obtain the fragment fusion feature sequence, process it using the sequence learning module, and obtain a word sequence or Gloss sequence.

[0083] Specifically, the segment fusion feature sequence comes from S330. This sequence is arranged chronologically within the continuous sign language video, and each segment carries corresponding global semantic information, temporal features, segment start and end positions, and boundary position association records. The current step is executed by the sequence learning module, a temporal modeling unit positioned after the segment fusion feature sequence and before the connectionist temporal classification method. Its structure includes at least a sequence receiving unit, a sequence organization unit, a temporal learning unit, and a sequence output unit. The sequence receiving unit reads the segment fusion feature sequence. The sequence organization unit uniformly organizes the sequence order, boundary position intervals, and segment lengths of each segment, and writes the switching relationships between adjacent segments into the sequence context area. Here, the word sequence refers to the sign language word results output in the order of continuous sign language movements, and the Gloss sequence refers to the sign language annotation results output in the order of continuous sign language movements; both are candidate output formats for the current step. The reason for processing the fragment fusion feature sequence with the sequence learning module first in this step, instead of directly entering the connectionist temporal classification method, is that although the fragment fusion feature has completed the fragment-level organization, it still needs to learn the continuous relationship and switching relationship between fragments over a longer time chain before a stable sequence output can be formed.

[0084] Furthermore, the temporal learning unit does not isolate and interpret individual segments, but processes multiple consecutive segments together. Specifically, it first reads the segment fusion features of the current segment, then reads the global semantic information and temporal features of the preceding and following segments, grouping these three into a continuous segment group. Subsequently, it performs sequential association processing on the continuous segment group, identifying continuous action segments, switching action segments, and pause segments. It retains the main action information in continuous action segments and transcribes the transition content in switching action segments into sequence transition markers. Here, a continuous action segment refers to a segment where the same action trend continues in adjacent segments; a switching action segment refers to a segment where action transitions occur near the boundary; and a pause segment refers to a segment where the action amplitude between segments weakens and the temporal features show a brief pause. Understandably, in the actual scenario of a government service window, when a person continuously expresses multiple actions such as "getting a number," "processing," and "signing," the system inputs the sequence of multiple segment fusion feature sequences corresponding to these actions into the sequence learning module. The temporal learning unit first identifies the sequential relationship between each segment, then identifies the action continuity state between preceding and following segments, thereby outputting a sequence result that more closely approximates a complete and continuous expression. If the connection information between segments is missing, the sequence sorting unit calls the associated records of the segment's start and end positions and boundary positions to supplement it; if the supplementation fails, the segment is registered as a sequence segment to be reviewed, and its sequence priority is reduced in this round of output. After the above processing, the sequence output unit outputs the word sequence or Gloss sequence, and writes "word sequence or Gloss sequence" as the output field name of the current step into the step result area for S420 to call "word sequence or Gloss sequence". At the same time, this result also serves as the direct input source for the generation of handwritten sentences after subsequent processing.

[0085] S420. Based on the word sequence or Gloss sequence, perform connectionist temporal classification decoding, merge consecutively repeated words, and remove words representing transitional actions to generate the processed handwritten sentence.

[0086] Specifically, the word sequence or Gloss sequence comes from S410. The current step is performed by the alignment translation unit, which includes a sequence reading unit, a connectionist temporal classification method decoding unit, a repeating word sorting unit, a transitional action sorting unit, and a sentence output unit. Here, connectionist temporal classification method is the Chinese expression for Connectionist Temporal Classification, and its full English name is Connectionist Temporal Classification. The connectionist temporal classification method decoding refers to aligning and expanding the word sequence or Gloss sequence and eliminating blank spaces according to the sequence order to obtain an initial sentence sequence distributed continuously over time. Since continuous sign language videos may still retain a small number of repetitive and transitional expressions near segment boundaries, the current step continues to perform continuous repetition word merging and word removal processing after decoding. Continuous repetition word merging means that when adjacent positions contain words with identical content and continuous temporal order, only one word is retained, and the merged position is recorded as a repetition position. Word removal for transition actions refers to removing a word from the initial sentence sequence when it is transcribed from a switching action segment near a boundary position and does not form a stable semantic connection with the preceding and following main actions, and registering the source of the removal in the transition action record area.

[0087] Furthermore, the connectionist temporal classification method decoding unit first reads the segment boundary association information in the current sequence during runtime, and then sequentially expands adjacent results in the word sequence or Gloss sequence. If a result appears repeatedly in multiple consecutive positions, and these positions all come from the same continuous action segment, the repeated word sorting unit merges it into one result; if a result appears near a boundary position, and the preceding and following sequences both show that the main action has switched, the transition action sorting unit marks it as a result to be removed. The reason for this setting is that existing technologies often perform simple deduplication after the final output, and transition actions and repeated actions near the boundary position may still be mixed into the sentence body; the current step synchronously completes alignment, deduplication, and transition sorting within the decoding link, making the processed handwritten sentence structure more regular. In an operable engineering embodiment, after the hospital triage terminal receives the word sequence or Gloss sequence, the connectionist temporal classification method decoding unit first expands it into an initial sentence sequence arranged in order.

[0088] Subsequently, if a word appears repeatedly in three consecutive positions, and all three positions originate from the same continuous action segment, the repeated word sorting unit retains only one word; if a word appears only between two main actions, and the boundary position record indicates that the position belongs to a switching action segment, the transition action sorting unit removes it. After processing, the sentence output unit generates the processed handwritten sentence and writes "processed handwritten sentence" as the output field name of the current step into the step result area for S430 to call. This result also serves as the direct input object for subsequent contextual natural language processing.

[0089] S430. Perform contextual natural language processing on the processed sign language sentences to generate continuous sign language recognition results.

[0090] Specifically, the processed sign language sentence comes from S420. This current step is executed by the contextual natural language processing unit, which includes a sentence reading unit, a context checking unit, a sentence structure organization unit, and a result output unit. Here, contextual natural language processing refers to organizing the semantic connections, sentence component order, and expression completeness within the processed sign language sentence to ensure the output conforms to natural language expression habits. The sentence reading unit first reads the processed sign language sentence sequentially. The context checking unit then checks the connection relationships between words in the sentence to determine if there are any component breaks, sequence misalignments, or short pauses due to continuous sign language switching.

[0091] If component breaks exist, the global semantic information and temporal feature association records retained during the S410 output are used for supplementary judgment; if there are sequence misalignments, the sentence arrangement unit restores the word order according to the original temporal order in the continuous sign language video; if there are short pauses, the sentence arrangement unit deletes redundant connection content at the corresponding pause positions. The reason for this processing is that although the processed sign language sentences have completed the decoding of the connectionist temporal classification method, the merging of consecutively repeated words, and the removal of words representing transitional actions, they may still retain the sequential differences between continuous sign language action expressions and natural language expressions. Only after the current step continues to process them can a continuous sign language recognition result that can be directly output be formed.

[0092] Furthermore, in actual operation, the contextual natural language processing unit does not regenerate sentences, but rather performs sequence and connection organization on the processed sign language sentences. Understandably, in a teaching terminal scenario, after a student completes a continuous sign language expression, the system obtains the processed sign language sentence via S420. If two main words in the sentence lack a connecting element, but the time feature record shows that they come from a continuous action sequence, the context checking unit will output them as adjacent. If the word order at a certain position in the sentence is inconsistent with the sequence of segments in the original continuous sign language video, the sentence organization unit will call the segment start and end position records to restore the original order.

[0093] If a result originates from a sequence segment to be reviewed, the result output unit retains the original sentence body in the current output and registers that position as the position of the sentence to be reviewed, without changing the rest of the confirmed content. After the above processing, the result output unit generates the continuous sign language recognition result. The continuous sign language recognition result is written into the recognition result area as the final output field name of the current step and serves as the end output of the method chain of the present invention. At the same time, the result can also be written back to the interactive terminal display area, broadcast area, or business processing area in the continuous sign language recognition mode, forming a complete closed loop from continuous sign language video to continuous sign language recognition result.

[0094] In summary, this step achieves the following technical results: After fusing the feature sequences of the segments, it sequentially processes the sequence learning module, decodes using the connectionist temporal classification method, and performs contextual natural language processing. The continuous sign language recognition results are no longer limited to local fragments but are transformed into complete sentence-level outputs. Compared to simply feeding the unified fusion result directly into the decoding process, this step incorporates the analysis of segment order, boundary positions, and transitional actions before the final output. Repeatedly occurring words and words indicating transitional actions are processed before output, resulting in continuous sign language recognition results that more closely resemble the actual expression order in continuous sign language videos.

Claims

1. A continuous sign language recognition method based on deep learning, characterized in that, include: S100: Acquire continuous sign language video, perform human image extraction and normalization processing, and perform gesture trajectory extraction and image spatial feature extraction processing to obtain multimodal data input; S200. Based on the multimodal data input, perform boundary regression branch construction and boundary prediction probability distribution generation processing, and perform boundary location determination and non-boundary location determination processing to generate location division information; S300: Obtain the location segmentation information and the multimodal data input, perform adjacent frame extraction and fragment-level reconstruction processing, and obtain the fragment-level reconstruction result; Based on the fragment-level recombination results and the action category probability sequence, image spatial features, gesture trajectory, facial expression key points and skeleton features are fused to generate fragment fusion features; Based on the fragment fusion features and the location segmentation information, global semantic information extraction and temporal feature extraction are performed to generate a fragment fusion feature sequence. S400. Obtain the fragment fusion feature sequence, process it using the sequence learning module, and obtain a word sequence or Gloss sequence. Based on the word sequence or Gloss sequence, the connectionist temporal classification method is used for decoding, continuous repetition of words is merged, and words representing transitional actions are removed to generate processed hand sentences. Based on the processed sign language sentences, contextual natural language processing is performed to generate continuous sign language recognition results.

2. The method according to claim 1, characterized in that, The process of extracting and normalizing human body images includes: The human image extraction includes extracting image content including the upper body, hand activity area and face area from continuous sign language video frame by frame, and writing abnormal frame records for frames with human body area that is too small, deviates from the edge or is occluded by more than a preset ratio. The normalization process includes performing size unification, brightness unification, and frame order reordering on the extracted human body image.

3. The method according to claim 1, characterized in that, The process of gesture trajectory extraction and image spatial feature extraction includes: The gesture trajectory extraction includes reading the coordinates of key points of the hand and the coordinates of the hand bones and joints in the order of frame number, comparing the displacement changes of adjacent frames to generate movement direction information, and generating trajectory segments based on the displacement direction changes and the distribution of pause frames. The image spatial feature extraction includes synchronously retrieving image information corresponding to key point coordinate information, and performing region mapping and spatial relationship organization from the corresponding human body image to obtain image spatial feature records arranged in frames. The image spatial features include the relative positions of the hands, the relative positions of the face and hands, the extended state of the upper limbs, and the regional distribution relationship; the coordinates of key points of the hands, the coordinates of key points of the face, the coordinates of the hand bones and joints, the gesture trajectory, and the image spatial features are all written into the same data structure.

4. The method according to claim 1, characterized in that, The process of constructing boundary regression branches and generating boundary prediction probability distributions includes: Read multimodal data input frame by frame; Organize the data groups in the same frame to form a sequence to be judged; Extract the hand gesture trajectory turning segment, the hand bone joint coordinate displacement abrupt segment, the facial key point coordinate expression switching segment, and the relative position change segment of the two hands in the image space features from the sequence to be judged to generate a boundary probability sequence.

5. The method according to claim 1, characterized in that, The process of determining boundary and non-boundary locations includes: The boundary position determination process includes selecting frames from the boundary probability sequence whose boundary probability is higher than a preset condition and whose corresponding action category probability sequence shows a trend of switching or transitioning actions as boundary positions. The non-boundary position determination process includes selecting frames from the boundary probability sequence whose boundary probability is lower than a preset condition and whose corresponding action category probability sequence shows a stable action continuation trend as non-boundary positions, and unifying the boundary positions and non-boundary positions into position division information.

6. The method according to claim 1, characterized in that, The process of adjacent frame extraction and fragment-level reconstruction includes: The adjacent frame extraction includes extracting a group of consecutive frames in the same non-boundary position segment in chronological order. The segment-level reorganization process includes reorganizing the consecutive frames in the same non-boundary position segment into a segment data group and adding adjacent boundary position markers at both ends of the segment. For candidate segments with fewer than a preset number of frames, a segment merging check is performed. When the gesture trajectory direction is continuous, the hand bone joint coordinate changes smoothly, and there is no obvious switching in the image spatial features, the segment merging process is performed. For segments with missing frames, the same type of coordinate information is extracted from the adjacent frames before and after to fill in the missing frames. If there are too many consecutive missing frames, the segment is discarded.

7. The method according to claim 1, characterized in that, The process of fusing image spatial features, gesture trajectories, facial expression key points, and skeleton features includes: The fusion process includes reading data groups by segment, aligning the segment data groups with the action category probability sequences on the corresponding frame numbers, identifying stable action segments within the segments and transitional action frames near the boundaries, performing weakening processing on transitional action frames near the boundaries to reduce their proportion in segment fusion, and combining image spatial features, gesture trajectories, facial expression key points, and skeleton features into a segment-level combined data in segment order.

8. The method according to claim 1, characterized in that, The process of global semantic information extraction and temporal feature extraction includes: The global semantic information extraction includes reading the boundary position markers at both ends of the segment and the image spatial features and facial expression key points retained near the corresponding boundary positions, and sorting out the switching relationship between segments. The temporal feature extraction includes extracting the continuous unfolding relationship from the stable action segments within the segment and the sequential order between segments, and arranging multiple segment fusion features in chronological order to generate a segment fusion feature sequence.

9. The method according to claim 1, characterized in that, The sequence learning module processes the following steps: The sequence learning module includes reading the segment fusion feature sequence, performing sequential association processing on multiple consecutive segments, identifying continuous action segments, switching action segments and pause segments, retaining the main action information in the continuous action segment, and transcribing the transition content in the switched action segment into sequence transition markers.

10. The method according to claim 1, characterized in that, The connectionist temporal classification method's decoding, merging of consecutively repeated words, and removal of words representing transitional actions includes the following processes: The connectionist temporal classification method decoding includes aligning and expanding according to the sequence order and eliminating blank positions to obtain the initial sentence sequence. The merging of consecutively recurring words includes merging words with consistent content and temporal continuity in adjacent positions into one word. The removal of words representing transitional actions includes removing words that are transcribed from the transition action segments near the boundary positions and have no stable semantic connection with the main actions before and after from the initial sentence sequence.

Images