A sign language recognition method and system

Abstract

The present application relates to the technical field of computer vision, and especially relates to a sign language recognition method and system. The present application obtains initial sign language prediction text based on the pre-processed continuous multiple frames of sign language pictures to be recognized of a user; if the text has a vacancy, the probability that the vacancy part belongs to each candidate word is output by a linear projection layer as visual confidence. When the number of vacancies is less than a preset value, the semantic similarity of the candidate word and the initial sign language prediction text is calculated as text confidence; otherwise, the initial sign language prediction text and the corresponding initial sign language prediction texts of the multiple frames before and after the initial sign language prediction text are spliced into a total prediction text, and the semantic similarity is calculated to obtain text confidence. A dynamic adjustment factor is obtained in combination with the clarity of the sign language picture, the target confidence is obtained by fusing the factor and the two kinds of confidence, the highest one is taken to complete the vacancy, and the final sign language recognition result is obtained. The present application greatly improves the sign language semantic recognition accuracy and enhances the immersion of interaction with the deaf-mute.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a sign language recognition method and system. Background Technology

[0002] With the development of human-computer interaction technology, the demand for immersive real-time interaction among the deaf and mute community is becoming increasingly prominent. Sign language, as the primary means of communication for the deaf and mute, requires accurate recognition and real-time conversion as a core element for achieving barrier-free interaction. In scenarios such as education, healthcare, and public services, sign language recognition can break down communication barriers, helping the deaf and mute quickly express their intentions, obtain information, and thus integrate into digital life.

[0003] Existing sign language recognition methods typically use consecutive frames of sign language images as input, extracting spatiotemporal features through deep learning models and mapping them to text. Specifically, mainstream solutions usually employ 3D convolutional neural networks or temporal modeling models to process consecutive frame sequences: 3D convolutional layers can simultaneously capture spatial features within a single frame (such as hand contours and joint positions) and temporal relationships between frames (such as the start, transition, and end of actions); some methods also incorporate attention mechanisms to enhance feature extraction from key action frames. The extracted features are then mapped to a predefined vocabulary or sentence space via projection layers, and the corresponding text results are output through classification or generation models.

[0004] However, existing methods often suffer from text gaps in their predictions when dealing with complex or ambiguous sign language gestures. This stems primarily from two factors: first, the lack of key information in consecutive frames, such as frame blurring caused by rapid movements or occlusion preventing the capture of crucial gestures, thus hindering the model's ability to extract complete spatiotemporal features; and second, the ambiguity of sign language semantics, where the same gesture can have different meanings in different contexts, making it difficult to determine accurate text based solely on local frame sequences. Directly inputting text with gaps into a large language model for completion presents significant drawbacks. The core of large language models lies in their learning of contextual semantic relationships from massive text corpora. Completion relies solely on grammatical rules and common collocations of the text sequence for inference, potentially leading to grammatically correct but completely irrelevant content. Furthermore, the completion process in large language models is based on statistical guesswork, especially when the number of text gaps is large, resulting in poor sign language recognition accuracy and distorted interactive information. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the defect that existing sign language recognition methods often have text gaps in the prediction results when dealing with complex or ambiguous sign language actions, and that directly using a large language model to fill in the gaps will lead to the distortion of the sign language prediction results.

[0006] To address the aforementioned technical problems, this invention provides a sign language recognition method, comprising: Based on the preprocessed sign language images of the user in multiple consecutive frames, the initial sign language prediction text corresponding to the sign language images in multiple consecutive frames is obtained. Determine whether there are gaps in the initial sign language prediction text corresponding to multiple consecutive frames of sign language images to be recognized. If there are no gaps, use the initial sign language prediction text as the sign language recognition result for the multiple consecutive frames of sign language images to be recognized. If it exists, the sign language image corresponding to the missing part is passed through a linear projection layer, and the preprocessed probability of the missing part belonging to each candidate word is output; the probability of the sign language image corresponding to the missing part belonging to each candidate word is used as the visual confidence of each candidate word. Determine whether the number of missing words in the initial sign language prediction text is greater than the preset number. If it is less, calculate the semantic similarity between each candidate word and the initial sign language prediction text, and use it as the text confidence of each candidate word. If the value is greater than or equal to the value, the initial predicted text corresponding to multiple consecutive frames of sign language images to be identified and the sign language images before and after them will be combined to form the total predicted text; the semantic similarity between each candidate word and the total predicted text will be calculated as the text confidence of each candidate word. Based on the clarity of multiple consecutive frames of sign language images to be recognized, a dynamic adjustment factor is obtained; based on the dynamic adjustment factor, the visual confidence of each candidate word, and the text confidence, the target confidence of each candidate word is obtained; the candidate word with the highest target confidence is taken as the missing text; based on the missing text and the initial sign language prediction text, the sign language recognition results of multiple consecutive frames of sign language images to be recognized are obtained.

[0007] Preferably, the process of acquiring multiple consecutive frames of sign language images to be recognized from the user after preprocessing includes: The system sequentially performs image grayscale processing, multi-scale filtering for noise reduction, edge detection, morphological closing operation, and binarization on multiple consecutive frames of sign language images to be recognized, resulting in preprocessed images of the user's multiple consecutive frames of sign language images to be recognized.

[0008] Preferably, grayscale image processing is performed on multiple consecutive frames of sign language images to be recognized, including: For multiple consecutive frames of sign language images to be recognized, obtain the RGB values ​​of each pixel in each frame of the sign language image to be recognized; Based on the RGB values ​​of each pixel in each frame of the sign language image to be recognized, the channel difference between the G and R channels and the channel difference between the G and B channels of each pixel in each frame of the sign language image to be recognized are calculated using the following formula: , in, The pixels in the current sign language image to be recognized Channel difference between G channel and R channel The pixels in the current sign language image to be recognized Channel difference between G channel and B channel This indicates taking the maximum value. The pixels in the current sign language image to be recognized G channel value, The pixels in the current sign language image to be recognized R channel value, The pixels in the current sign language image to be recognized B channel value, The first sign language image to be recognized Line 1 Column pixels, For row index, For column indexes; Based on the channel difference between the G and R channels and the channel difference between the G and B channels of each pixel in each frame of the sign language image to be recognized, the skin color probability of each pixel in each frame of the sign language image to be recognized is obtained, and the formula is as follows: , in, The pixels in the current sign language image to be recognized The probability of skin color; Based on the skin color probability of each pixel in each frame of the sign language image to be recognized, the RGB values ​​of each pixel in each frame of the sign language image to be recognized are adjusted to obtain a grayscale sign language image for each frame.

[0009] Preferably, multi-scale filtering and denoising are performed on each frame of the grayscale sign language image to be recognized, including: A Gaussian filter whose kernel standard deviation is less than a preset lower limit standard deviation is used as a small-scale Gaussian filter; A Gaussian filter whose Gaussian kernel standard deviation is greater than or equal to the preset lower standard deviation and less than or equal to the preset upper standard deviation is used as a mesoscale Gaussian filter. Each frame of the grayscale sign language image to be recognized is filtered by a small-scale Gaussian filter, a medium-scale Gaussian filter, and an adaptive median filter to obtain the filter map of each frame of the sign language image to be recognized for different filters. Based on the filtered images of each frame of the sign language image to be recognized corresponding to different filters, and the skin color probability of each pixel in each frame of the sign language image to be recognized, the target filtered image of each frame of the sign language image to be recognized is obtained, using the following formula: , in, For the target filtered pixels in the sign language image to be recognized The value, For small-scale Gaussian filters, the corresponding pixels in the filtered image of the sign language image to be recognized are... The value, The pixels in the filtered image of the current sign language image to be recognized are represented by a mesoscale Gaussian filter. The value, The adaptive median filter corresponds to the pixels in the filtered image of the sign language image to be recognized. The value, This indicates taking the maximum value. To set a skin tone threshold, The pixels in the current sign language image to be recognized The probability of skin color, To indicate the first Line 1 Column pixels, For row index, For column indexes.

[0010] Preferably, edge detection is performed on the target filtered image of each frame of the sign language image to be recognized, including: Based on the skin color probability of each pixel in each frame of the sign language image to be recognized, the preset maximum edge detection threshold, and the preset minimum edge detection threshold, the edge detection threshold of each pixel in each frame of the sign language image to be recognized is obtained, and the formula is: , in, The pixels in the current sign language image to be recognized The edge detection threshold, To preset the maximum edge detection threshold, To preset the minimum edge detection threshold, These are the weighting coefficients. The pixels in the current sign language image to be recognized The probability of skin color, Indicates the first Line 1 Column pixels, For row index, For column indexes; Based on the edge detection threshold of each pixel in each frame of the sign language image to be recognized, the edge detection map of each frame of the sign language image to be recognized is obtained.

[0011] Preferably, the edge detection map of each frame of the sign language image to be recognized is binarized, including: The edge detection map of each frame of the sign language image to be recognized is segmented to obtain the knuckle area, palm area and background area of ​​each frame of the sign language image to be recognized. The knuckle area uses pixel blocks of no more than a preset lower limit size to calculate the binarization threshold, the background area uses pixel blocks of no less than a preset upper limit size to calculate the binarization threshold, and the palm area uses pixel blocks between the knuckle area block size and the background area block size to calculate the binarization threshold. For the boundary between different zones, the binarization threshold is calculated by optimizing the threshold through a sliding window with a preset overlap. The optimal threshold for each region of the sign language image to be recognized in each frame is determined by maximizing the inter-class variance. Based on the optimal threshold for each region of the sign language image to be recognized in each frame and the pixel value of each pixel in the sign language image to be recognized in each frame, the user's consecutive multi-frame sign language images to be recognized after preprocessing are obtained.

[0012] Preferably, based on the preprocessed sign language images of the user in multiple consecutive frames to be identified, the sign language text semantic vectors corresponding to the sign language images in multiple consecutive frames to be identified are obtained through the CLIP model. Based on the semantic vectors of sign language text corresponding to multiple consecutive frames of sign language images to be identified, the initial predicted sign language text corresponding to multiple consecutive frames of sign language images to be identified is obtained. Determine if there are any gaps in the initial sign language prediction text. If not, acquire multiple consecutive frames of facial video of the same user. The multiple consecutive frames of facial images and the multiple consecutive frames of sign language images to be recognized are acquired synchronously. Input multiple consecutive frames of facial images from a user into the YOLOv7 model, and output the expression category vectors corresponding to the multiple consecutive frames of facial images. The preprocessed multi-frame sign language images of the user are passed through a multilayer perceptron to obtain the sign language semantic weight vector. The user's continuous multi-frame facial images are processed through a multilayer perceptron to obtain the expression feature weight vector; Based on the sign language semantic weight vector, facial expression feature weight vector, sign language text semantic vector corresponding to multiple consecutive frames of sign language images to be identified, and facial expression category vector corresponding to multiple consecutive frames of facial video, a fused feature vector is obtained, using the following formula: , in, To fuse feature vectors, This refers to the semantic vectors of sign language text corresponding to multiple consecutive frames of sign language images to be recognized. This is a vector representing the expression categories corresponding to multiple consecutive frames of facial video. It is the sigmoid activation function. Scaling factor This is the semantic weight vector for sign language. These are preprocessed, consecutive frames of sign language images from the user to be recognized. For facial expression feature weight vectors, For a series of multiple frames of facial images of the user; Based on the fused feature vector, the sign language recognition results of sign language images to be recognized in multiple consecutive frames are obtained.

[0013] Preferably, the loss function of the YOLOv7 model includes: target confidence loss, classification loss, and facial expression detail perception loss; The formula for loss of facial detail perception is: , in, Loss of perception of facial expression details Weighting loss for facial expression detail perception. A set of pixels representing facial landmarks. For gradient operators, Let (x, y) be the pixel value of pixel (x, y) in the sample image. The pixel value (x, y) in the reconstructed image corresponding to the sample image is given. The process of obtaining the reconstructed image corresponding to the sample image is as follows: the sample image is passed through the backbone network of the YOLOv7 model to obtain basic visual features; the reconstructed image is generated based on the basic visual features. Indicates the first Line 1 Column pixels, For row index, For column indexes, It is the square of the L2 norm.

[0014] Preferably, channel attention and spatial attention modules are set between the Neck module and the Head module of the YOLOv7 model to obtain the target YOLOv7 model; The feature map output by the Neck module in the YOLOv7 model is passed through the channel attention and spatial attention modules to obtain the attention feature map, including: The feature map output by the Neck module is subjected to max pooling to obtain a max pooled feature map; the feature map output by the Neck module is subjected to average pooling to obtain an average pooled feature map. After concatenating the max pooling feature map and the average pooling feature map along the channel dimension, a spatial mask is generated by sequentially passing them through a 3×3 convolution and a Sigmoid activation function. After multiplying the spatial mask element-wise with the feature map output by the Neck module, the channel mask is generated by sequentially applying global average pooling, 1×1 convolution, ReLU activation function, 1×1 convolution, ReLU activation function, and Sigmoid activation function. The attention feature map is obtained by multiplying the channel mask element-wise with the feature map output by the Neck module.

[0015] Preferably, multiple consecutive frames of sign language images to be recognized are input into the image encoder of the CLIP model, and the visual features of the multiple consecutive frames of sign language images to be recognized are output, including: For each frame of the sign language image to be recognized, it is divided into blocks to obtain multiple patches; After recombining all patches corresponding to multiple consecutive frames of sign language images to be recognized into tensors, spatiotemporal features are extracted through 3D convolution. The spatiotemporal features are flattened into embedding vectors and superimposed with positional encodings. Visual features are obtained through a multi-layer Transformer encoder.

[0016] The present invention also provides a sign language recognition system, comprising: A memory for storing computer programs; a processor for executing the computer programs to implement the steps of the aforementioned sign language recognition method.

[0017] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: This invention discloses a sign language recognition method and system. It obtains the visual confidence of candidate words in sign language images corresponding to missing parts through a linear projection layer, associating visual features of the sign language actions to ensure a close connection between candidate words and original sign language actions, effectively avoiding arbitrary completion detached from actual actions. Then, based on the number of gaps, it flexibly selects the total predicted text based on the initial predicted text or its combination with predicted texts from previous and subsequent frames, calculating the semantic similarity between candidate words and corresponding texts as text confidence. This leverages contextual semantic information to improve the matching degree between candidate words and the overall context. Simultaneously, it introduces a dynamic adjustment factor based on image clarity to fuse visual and text confidence, enabling the calculation of target confidence to adapt to sign language images of varying clarity. Even in complex or ambiguous scenes, it accurately balances the weight of visual and semantic information. Finally, it selects the candidate word with the highest target confidence to fill the gaps. Through comprehensive consideration of multi-dimensional confidence, it effectively fills text gaps, ensuring consistency between the filled content and the original sign language actions, as well as the coherence of contextual semantics, thus improving the accuracy of recognizing complex or ambiguous sign language actions.

[0018] Furthermore, to address the shortcomings of existing sign language recognition methods that focus solely on hand information and struggle to accurately interpret the meanings expressed by deaf and mute individuals, resulting in poor sign language recognition accuracy, further research is needed. For instance, when expressing "happy" and "sad" in sign language, the hand movements may be similar, but the facial expressions are drastically different. Relying solely on hand recognition can easily confuse the two meanings. This invention introduces 3D convolution into the image encoder of the CLIP model. Using 3D convolution, it extracts the spatiotemporal features of preprocessed, multi-frame sign language images from the user. These spatiotemporal features are flattened into embedding vectors and superimposed with positional encoding. Through a multi-layer Transformer encoder, visual features are obtained. Based on these visual features, initial predicted sign language text corresponding to the multi-frame images is acquired. This overcomes the limitation of traditional 2D convolution, which only focuses on spatial information, effectively capturing the temporal dynamics of sign language and improving the recognition accuracy of the CLIP model. Furthermore, in the YOLOv7 model, channel attention and spatial attention modules are set before the Head module. Spatial attention strengthens the periorbital and mouth regions, while channel attention strengthens expression-related channels and suppresses irrelevant channels, completing the spatial localization and channel filtering of expression features. This yields expression category vectors corresponding to multi-frame facial images. Through a multimodal fusion mechanism, sign language actions are organically combined with synchronously acquired facial expression information, enabling not only the recognition of sign language content but also the analysis of the emotional information contained within, thus improving sign language recognition accuracy. Attached Figure Description

[0019] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart of the steps of a sign language recognition method according to the present invention.

[0020] Figure 2 It is a comparison image of multiple consecutive frames of sign language images to be recognized before and after preprocessing.

[0021] Figure 3 This is the facial expression prediction confusion matrix of the YOLOv7 model of this invention.

[0022] Figure 4 This is the PR curve of the YOLOv7 model of this invention.

[0023] Figure 5 These are the training and validation loss curves for the CLIP model with 3D convolutional enhancement. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0025] Reference Figure 1 As shown, this embodiment provides a sign language recognition method, including: Step S1: Based on the preprocessed multi-frame sign language images of the user to be recognized, obtain the initial sign language prediction text corresponding to the multi-frame sign language images to be recognized. In this embodiment, specifically, a video stream of the user's hand movements during sign language communication is acquired in real time by a camera. The video data is wirelessly transmitted to a terminal computer for data parsing, and is divided into multiple consecutive frame segments in chronological order. Each consecutive frame segment corresponds to multiple consecutive frames of sign language images to be recognized. The recognition process is performed on each segment, and finally, the recognition results of all segments are spliced ​​together to obtain the sign language recognition text of the complete video.

[0026] In this embodiment, each consecutive frame segment includes ten consecutive frames of sign language images to be recognized.

[0027] In this embodiment, the camera may be either an RGB camera or a depth camera.

[0028] In this embodiment, optionally, multimodal video data of the user's sign language communication is collected in real time by an RGB camera and a depth camera. The video stream is transmitted in real time to a terminal computer equipped with an NVIDIA RTX A5000 GPU via a USB 3.0 interface. During the data parsing stage, the sign language video frames are preprocessed using the OpenCV library.

[0029] like Figure 2 As shown, Figure 2 This is a comparison image of multiple consecutive frames of sign language images to be recognized before and after preprocessing.

[0030] In this embodiment, specifically, the process of acquiring multiple consecutive frames of sign language images to be recognized from the user after preprocessing includes: The system sequentially performs image grayscale processing, multi-scale filtering for noise reduction, edge detection, morphological closing operation, and binarization on multiple consecutive frames of sign language images to be recognized, resulting in preprocessed images of the user's multiple consecutive frames of sign language images to be recognized.

[0031] In this embodiment, preferably, grayscale image processing is performed on multiple consecutive frames of sign language images to be recognized, including: For multiple consecutive frames of sign language images to be recognized, obtain the RGB values ​​of each pixel in each frame of the sign language image to be recognized; Based on the RGB values ​​of each pixel in each frame of the sign language image to be recognized, the channel difference between the G and R channels and the channel difference between the G and B channels of each pixel in each frame of the sign language image to be recognized are calculated using the following formula: , in, The pixels in the current sign language image to be recognized Channel difference between G channel and R channel The pixels in the current sign language image to be recognized Channel difference between G channel and B channel This indicates taking the maximum value. The pixels in the current sign language image to be recognized G channel value, The pixels in the current sign language image to be recognized R channel value, The pixels in the current sign language image to be recognized B channel value, The first sign language image to be recognized Line 1 Column pixels, For row index, For column indexes; Based on the channel difference between the G and R channels and the channel difference between the G and B channels of each pixel in each frame of the sign language image to be recognized, the skin color probability of each pixel in each frame of the sign language image to be recognized is obtained, and the formula is as follows: , in, The pixels in the current sign language image to be recognized The probability of skin color; Based on the skin color probability of each pixel in each frame of the sign language image to be recognized, the RGB values ​​of each pixel in each frame of the sign language image to be recognized are adjusted to obtain a grayscale sign language image for each frame.

[0032] The dominance of the G channel relative to the R and B channels is measured by the skin color probability of each pixel in each frame of the sign language image to be recognized, highlighting the skin color features of the hand. It can reflect the probability that each pixel belongs to the skin color region and dynamically allocate the weights of the R, G, and B channels. For regions with a higher probability of skin color (such as hands), the weight of the G channel is greater to enhance the hand features; while for non-skin color regions (such as the background), the weight of the G channel is appropriately reduced.

[0033] To address the issues of hand features being easily confused with the background and hand features being weakened during grayscale processing in sign language recognition, this invention dynamically adjusts the weights of the RGB channels based on pixel skin color probability. This allows for targeted highlighting of skin-colored features, such as hands, during grayscale processing. By strengthening the weight of the G channel in skin-colored areas, hand features are made clearer in the grayscale image. Simultaneously, by reducing the weight of the G channel in non-skin-colored background areas, background interference is reduced. This enhances the ability of the subsequent sign language recognition model to capture key hand movement features and improves recognition accuracy, making it particularly suitable for scenarios with complex backgrounds or where the hand and background colors are similar.

[0034] In this embodiment, preferably, multi-scale filtering and denoising are performed on each frame of the grayscale sign language image to be recognized, including: A Gaussian filter whose kernel standard deviation is less than a preset lower limit standard deviation is used as a small-scale Gaussian filter; A Gaussian filter whose Gaussian kernel standard deviation is greater than or equal to the preset lower standard deviation and less than or equal to the preset upper standard deviation is used as a mesoscale Gaussian filter. Each frame of the grayscale sign language image to be recognized is filtered by a small-scale Gaussian filter, a medium-scale Gaussian filter, and an adaptive median filter to obtain the filter map of each frame of the sign language image to be recognized for different filters. Based on the filtered images of each frame of the sign language image to be recognized corresponding to different filters, and the skin color probability of each pixel in each frame of the sign language image to be recognized, the target filtered image of each frame of the sign language image to be recognized is obtained, using the following formula: , in, For the target filtered pixels in the sign language image to be recognized The value, For small-scale Gaussian filters, the corresponding pixels in the filtered image of the sign language image to be recognized are... The value, The pixels in the filtered image of the current sign language image to be recognized are represented by a mesoscale Gaussian filter. The value, The adaptive median filter corresponds to the pixels in the filtered image of the sign language image to be recognized. The value, This indicates taking the maximum value. To set a skin tone threshold, The pixels in the current sign language image to be recognized The probability of skin color, To indicate the first Line 1 Column pixels, For row index, For column indexes.

[0035] This invention utilizes three filters in parallel: small-scale, medium-scale, and adaptive median. By leveraging the skin color probability of each pixel in each frame of the sign language image to be recognized, the image is divided into a hand region (high probability) and a background region. Based on the characteristics of each region, the optimal filter output is selected. For the hand region with high skin color probability, a small-scale Gaussian filter is applied, effectively removing subtle noise while preserving the fine details of hand movements to the maximum extent. For the background region with low skin color probability, the maximum value of the medium-scale Gaussian filter and the adaptive median filter is selected. This suppresses medium-scale noise in the background while also handling impulse noise or abrupt changes in the background through adaptive median filtering. This achieves a precise balance between preserving hand details and suppressing background noise during denoising, providing a clearer and more reliable image foundation for subsequent sign language feature extraction. It effectively solves the problem of lost hand details or incomplete background noise removal in grayscale sign language images during denoising.

[0036] In this embodiment, preferably, edge detection is performed on the target filtered image of each frame of the sign language image to be recognized, including: Based on the skin color probability of each pixel in each frame of the sign language image to be recognized, the preset maximum edge detection threshold, and the preset minimum edge detection threshold, the edge detection threshold of each pixel in each frame of the sign language image to be recognized is obtained, and the formula is: , in, The pixels in the current sign language image to be recognized The edge detection threshold, To preset the maximum edge detection threshold, To preset the minimum edge detection threshold, These are weighting coefficients used to control the degree to which skin color probability affects the edge detection threshold. The pixels in the current sign language image to be recognized The probability of skin color, Indicates the first Line 1 Column pixels, For row index, For column indexes; Based on the edge detection threshold of each pixel in each frame of the sign language image to be recognized, the edge detection map of each frame of the sign language image to be recognized is obtained, including: The gradient value of each pixel in each frame of the sign language image to be recognized is calculated using differential methods (such as the Sobel operator). The gradient value objectively reflects the degree of drastic change in pixel grayscale. The larger the value, the higher the probability that there is an edge at that location. Determine the edge detection threshold for each pixel in each frame of the sign language image to be recognized. If the gradient value of a pixel is greater than or equal to its gradient value, the pixel is considered an edge pixel; otherwise, it is considered a non-edge pixel. This invention performs edge detection on the target filtered map of each frame of the sign language image to be recognized. Due to variations in movement and differences in hand skin color, a fixed threshold can easily lead to edge breakage or false detections due to noise, and it is difficult to adapt to different gesture complexities. Therefore, this invention sets adaptive thresholds for gesture environments of varying complexity: a lower threshold for open gestures (complex edges) and a higher threshold for clenched fist gestures (simple edges). By dynamically generating edge detection thresholds based on pixel skin color probability, the threshold can be adaptively adjusted according to gesture complexity. For scenarios with complex edges, such as open gestures, the threshold is automatically lowered to avoid edge breakage; for scenarios with simple edges, such as clenched fist gestures, the threshold is automatically raised to reduce false detections due to noise. Simultaneously, Gaussian smoothing and differential differentiation are combined to ensure smoothing effects when calculating the gradient, resulting in a more complete and accurate hand edge detection map. This lays a more reliable foundation for subsequent sign language action feature extraction and improves overall recognition accuracy.

[0037] In this embodiment, preferably, the edge detection map of each frame of the sign language image to be recognized is binarized, including: The edge detection map of each frame of the sign language image to be recognized is segmented to obtain the knuckle area, palm area and background area of ​​each frame of the sign language image to be recognized; the knuckle area refers to the area where the finger joints are located, the palm area is the area connecting the knuckle area and the wrist, and the background area is the area other than the knuckle area and palm area. The knuckle area uses pixel blocks of no more than a preset lower limit size to calculate the binarization threshold, the background area uses pixel blocks of no less than a preset upper limit size to calculate the binarization threshold, and the palm area uses pixel blocks between the knuckle area block size and the background area block size to calculate the binarization threshold. For the connection boundaries between different areas, the binarization threshold is calculated by optimizing the threshold through a sliding window with a preset overlap. The connection boundaries between different areas refer to the boundary between the knuckle area and the palm area, and the boundary between the palm area and the background area. The optimal threshold for each region of the sign language image to be recognized in each frame is determined by maximizing the inter-class variance method. This includes: statistically analyzing the grayscale distribution of pixel values ​​in each region, calculating the inter-class variance between foreground (edge ​​pixels) and background (non-edge pixels) in each region by traversing possible thresholds, and determining the threshold with the largest inter-class variance as the optimal threshold for that block. Based on the optimal thresholds for each region of each frame of the sign language image to be recognized and the pixel value of each pixel in each frame of the sign language image to be recognized, preprocessed multi-frame sign language images of the user to be recognized are obtained, including: For each pixel in a series of preprocessed sign language images of the user, if its pixel value is greater than the optimal threshold of the region, the pixel is set to 255 (white, preserving edges); if it is less than or equal to the optimal threshold, it is set to 0 (black, suppressing non-edges), thus obtaining a series of preprocessed sign language images of the user.

[0038] In this embodiment, specifically, the fingertip / knuckle area is divided into 4×4 pixel blocks, the palm area into 16×16 pixel blocks, and the background area into 32×32 pixel blocks. Thresholds are calculated independently. For the boundaries of the partitions such as the connection between the fingers and the palm, a sliding window with 50% overlap is used to optimize the threshold calculation and eliminate binarization faults. The optimal threshold for each region is determined by maximizing the inter-class variance method. The image is converted into a binary image of 0 (background) and 255 (foreground) to obtain a preprocessed multi-frame image of the user's sign language to be recognized.

[0039] To solve the edge detection problem of sign language images Figure 2 During binarization, due to the significant differences in texture and brightness features among different regions such as knuckles, palms, and backgrounds, using a uniform block size to calculate the threshold can easily lead to the loss of local details (such as blurring of fine edges on the knuckles) or residual background noise. This invention dynamically adjusts the block size according to regional characteristics—using small blocks for the detail-rich knuckle area to ensure clear preservation of fine edges; using large blocks for the large background area to reduce noise interference; and using medium-sized blocks for the transitional palm area to balance detail and smoothness. Simultaneously, it optimizes the threshold calculation for region boundaries through overlapping sliding windows and combines this with the method of maximizing inter-class variance to determine the optimal threshold for each region. This allows the binarization result to accurately distinguish the edge features of different regions, providing a more reliable binarized image foundation for subsequent sign language motion feature extraction and improving the overall stability of recognition.

[0040] Step S2: Determine whether there are gaps in the initial sign language prediction text corresponding to multiple consecutive frames of sign language images to be recognized. If there are no gaps, use the initial sign language prediction text as the sign language recognition result of the multiple consecutive frames of sign language images to be recognized. In this embodiment, specifically, determining whether there are gaps in the initial sign language prediction text corresponding to multiple consecutive frames of sign language images to be recognized includes: checking the grammatical and semantic integrity of the initial sign language prediction text corresponding to multiple consecutive frames of sign language images to be recognized, and identifying the core components and modifying components of the initial sign language prediction text. The core components include: subject, predicate, and object; the modifying components include attributive and adverbial modifiers; for example: "I want to ____ because it's too hot", "Now it's ____ I want to ____".

[0041] Step S3: If it exists, pass the sign language image corresponding to the missing part through a linear projection layer and output the probability (classification logits, dimension is candidate vocabulary size) of the preprocessed missing part belonging to each candidate word; use the probability of the sign language image corresponding to the missing part belonging to each candidate word as the visual confidence of each candidate word. Step S4: Determine whether the number of missing words in the initial sign language prediction text is greater than the preset number. If it is less, calculate the semantic similarity between each candidate word and the initial sign language prediction text, and use it as the text confidence of each candidate word. Step S5: If it is greater than or equal to, then combine the initial predicted sign language text corresponding to multiple consecutive frames of sign language images to be identified and the sign language images before and after them to form the total predicted text; calculate the semantic similarity between each candidate word and the total predicted text to form the text confidence of each candidate word. When there are too many missing texts in the initial sign language prediction, it is difficult to predict the missing text based solely on multiple consecutive sign language images to be recognized. Therefore, it is necessary to combine multiple consecutive sign language images before and after the missing texts to predict the missing texts, thereby more accurately predicting the text corresponding to multiple missing texts.

[0042] If multiple consecutive frames of sign language images to be identified are at the end of the video segment, and there are only previous frame sign language images but no subsequent frame sign language images, then the initial sign language predicted text corresponding to the multiple consecutive frames of sign language images to be identified and the previous multiple frame sign language images will be combined as the total predicted text. If multiple consecutive frames of sign language images to be identified are located at the beginning of the video, and there are only later frames of sign language images but no earlier frames of sign language images, then the initial sign language predicted text corresponding to the multiple consecutive frames of sign language images to be identified and the subsequent frames of sign language images will be combined to form the total predicted text. If there are multiple consecutive frames of sign language images to be recognized, but no preceding or following sign language image, then the previous and following frames are skipped and the sign language images to be recognized are used only for prediction.

[0043] Step S6: Based on the sharpness of multiple consecutive frames of sign language images to be recognized, obtain a dynamic adjustment factor; based on the dynamic adjustment factor, the visual confidence of each candidate word, and the text confidence, obtain the target confidence of each candidate word; take the candidate word with the highest target confidence as the missing text; based on the missing text and the initial sign language prediction text, obtain the sign language recognition results of multiple consecutive frames of sign language images to be recognized. Integrate the initial sign language prediction text with the missing text obtained by supplementing predictions from previous and subsequent frames to form a complete candidate text sequence.

[0044] In this embodiment, specifically, when the average sharpness of multiple consecutive frames of sign language images to be recognized is higher than a set sharpness value, the weight corresponding to visual confidence is set to be greater than the weight corresponding to text confidence. Conversely, when the average sharpness of multiple consecutive frames of sign language images to be recognized is less than or equal to the set sharpness value, the weight corresponding to visual confidence is set to be less than the weight corresponding to text confidence. This dynamic adjustment of factors achieves adaptive fusion of visual and textual modal information. When image sharpness is high, the accuracy of visual features is fully utilized; when images are blurry, the semantic coherence of the textual context is relied upon. This effectively solves the robustness problem of sign language recognition in complex environments, and significantly improves recognition accuracy, especially under low-quality image conditions.

[0045] To address the limitations of existing sign language recognition methods that focus solely on hand images or joint points, several problems have emerged in practical applications. From a data acquisition perspective, focusing only on hand information leads to the loss of crucial content. Sign language encompasses not only hand movements but also rich facial expressions, head postures, and overall body coordination, all of which are vital components of semantic expression. For example, when expressing "happy" and "sad" in sign language, the hand movements may be similar, but the facial expressions are drastically different. Relying solely on hand recognition easily leads to semantic confusion between the two. In complex scenarios, hand occlusion severely impacts recognition accuracy. Furthermore, individuals differ in the range of motion of their hand joints and gesture habits; some people have large gestures, while others have relatively smaller ones. Relying solely on fixed hand templates or joint point models makes it difficult to adapt to diverse individual differences, resulting in decreased recognition accuracy. In addition, most existing systems struggle to achieve immersive interaction with deaf and mute individuals. Immersive interaction requires the system to respond to user input in real time and naturally, but current mainstream recognition systems often only perform a one-way conversion from sign language to sign language meaning, lacking a two-way interactive feedback mechanism. Simultaneously, these systems exhibit poor adaptability to environmental factors during interaction. Changes in lighting and increased background complexity in real-world scenarios can interfere with the normal operation of the recognition system, thereby disrupting the immersive interactive experience.

[0046] This invention uses the YOLOv7 model to recognize facial expressions during sign language communication by deaf and mute individuals, and the CLIP model to identify the meaning expressed by the sign language. The results of the two models are weighted and summed to obtain the final recognition result. A series of preprocessing steps suitable for sign language communication scenarios are performed on the input sign language images, so that the CLIP model focuses only on the gestures themselves. A facial detail perception loss is redesigned for the YOLOv7 model, which greatly improves the accuracy of facial expression recognition. At the same time, this method uses an RGB camera and a depth camera to wirelessly transmit information to the local computer in real time. The 3D convolution method is used to use 10 consecutive frames of sign language images as input to the CLIP model to better capture the spatial semantic information during communication. The BETR model is used to predict and supplement missing text fields, which greatly improves the accuracy of sign language prediction and makes the interaction with deaf and mute individuals more immersive.

[0047] like Figure 3 As shown, Figure 3 This is the confusion matrix for facial expression prediction in the YOLOv7 model of this invention. Figure 3 Each row (Predicted) represents the class predicted by the model, and the column (True) represents the actual class of the sample. The cell value corresponds to the number of samples at the intersection of the predicted class and the true class. The larger the value, the darker the color.

[0048] like Figure 4 As shown, Figure 4 This is the PR curve of the YOLOv7 model of this invention. The PR curve (Precision-Recall curve) is used to evaluate the performance of the YOLOv7 model. Figure 4 Different colored lines correspond to different expression categories. The higher the curve is and the larger the area, the better the classification performance of that category.

[0049] Depend on Figure 3 , Figure 4 It can be seen that after training for 350 epochs, the YOLOv7 model achieved a total mAP@0.5 score of 0.809 for recognizing the eight basic facial expressions. The model performed best in recognizing the "Happy" expression, with an mAP@0.5 score of 0.947. The "Neutral" expression had a relatively lower mAP@0.5 score of 0.699, indicating that the model is effective in recognizing different facial expressions overall.

[0050] Based on the preprocessed sign language images of the user in multiple consecutive frames, the semantic vector of the sign language text corresponding to the sign language images in multiple consecutive frames is obtained through the CLIP model. Based on the semantic vectors of sign language text corresponding to multiple consecutive frames of sign language images to be identified, the initial predicted sign language text corresponding to multiple consecutive frames of sign language images to be identified is obtained. Determine if there are any gaps in the initial sign language prediction text. If not, acquire multiple consecutive frames of facial video of the same user. The multiple consecutive frames of facial images and the multiple consecutive frames of sign language images to be recognized are acquired synchronously. Input multiple consecutive frames of facial images from a user into the YOLOv7 model, and output the expression category vectors corresponding to the multiple consecutive frames of facial images. The preprocessed multi-frame sign language images of the user are passed through a multilayer perceptron to obtain the sign language semantic weight vector. The user's continuous multi-frame facial images are processed through a multilayer perceptron to obtain the expression feature weight vector; Based on the sign language semantic weight vector, facial expression feature weight vector, sign language text semantic vector corresponding to multiple consecutive frames of sign language images to be identified, and facial expression category vector corresponding to multiple consecutive frames of facial video, a fused feature vector is obtained, using the following formula: , in, To fuse feature vectors, This refers to the semantic vectors of sign language text corresponding to multiple consecutive frames of sign language images to be recognized. This is a vector representing the expression categories corresponding to multiple consecutive frames of facial video. It is the sigmoid activation function. Scaling factor , Used to dynamically reflect the contribution of facial expressions to semantics. This is the semantic weight vector for sign language. These are preprocessed, consecutive frames of sign language images from the user to be recognized. For facial expression feature weight vectors, For a series of multiple frames of facial images of the user, This represents the fusion weights of the CLIP model. This represents the fusion weights of the YOLOv7 model.

[0051] Based on the fused feature vector, the sign language recognition results of sign language images to be recognized in a series of consecutive frames are obtained. This includes: inputting the fused feature vector into a fully connected layer, converting it into a representation that matches the sign language semantic space, and then outputting the probability distribution of each candidate sign language semantic (such as "hello", "thank you", "happy" etc.) through softmax. The softmax will output the highest probability of the corresponding category of the sign language images to be recognized in a series of consecutive frames.

[0052] After independently predicting facial expression type and sign language semantics, this invention employs an attention-weighted fusion strategy to deeply integrate the two types of information, and finally outputs a complete sign language recognition result containing the emotional dimension through softmax.

[0053] In this embodiment, preferably, multiple consecutive frames of sign language images to be recognized are input into the image encoder of the CLIP model, and the visual features of the multiple consecutive frames of sign language images to be recognized are output, including: For each frame of the sign language image to be recognized, it is divided into blocks to obtain multiple patches; After recombining all patches corresponding to multiple consecutive frames of sign language images to be recognized into tensors, spatiotemporal features are extracted through 3D convolution. The spatiotemporal features are flattened into embedding vectors and superimposed with positional encodings. Visual features are obtained through a multi-layer Transformer encoder.

[0054] In this embodiment, specifically, 10 consecutive frames of 224×224 pixel sign language images to be recognized are spatially segmented. Each frame is evenly divided into 7×7 non-overlapping patches of 32×32 pixels. The 10-frame patches are reassembled into tensors of dimensions [10, 7, 7, 32×32×3]. Spatiotemporal features are extracted through 3×3×3 3D convolution. The spatiotemporal feature map is flattened into an embedding vector of dimensions [10×7×7, 768] and positional encoding is superimposed. Then, it is input into a 12-layer Transformer encoder. Each layer calculates the association weights between patches through a multi-head self-attention mechanism. Finally, the output of class_token is taken as a 768-dimensional global visual feature vector and L2 normalized. At the same time, the sentence-level text template is split into sub-word units by the BPE word segmenter, a special token is added, and it is input into the text encoder. After processing by 12 layers of Transformer, the normalized starting token is taken. The output is used as the semantic vector of the sign language text; the model calculates the cosine similarity between the visual features and the predefined candidate text features (such as "hello" and "thank you"), and selects the text with the highest similarity as the initial sign language prediction text corresponding to the sign language images to be recognized in consecutive frames.

[0055] In this embodiment, specifically, the image encoder in the CLIP model extracts the spatiotemporal features of 10 consecutive frames of sign language images through 3D convolution, then flattens the spatiotemporal features into embedding vectors and superimposes positional encoding, and then inputs them into a 12-layer Transformer encoder. Each layer calculates the association weights through a multi-head self-attention mechanism. The CLIP model calculates the cosine similarity between the visual features and the predefined candidate text features, and selects the predefined candidate text feature with the highest similarity as the semantic vector of the sign language text.

[0056] Based on the semantic vectors of sign language text corresponding to multiple consecutive frames of sign language images to be recognized, the initial predicted sign language text corresponding to the multiple consecutive frames of sign language images to be recognized is obtained. It is then determined whether there are gaps in the initial predicted sign language text corresponding to the multiple consecutive frames of sign language images to be recognized. If there are gaps, the sign language images corresponding to the gaps are passed through a linear projection layer, and the probability of the preprocessed gaps belonging to each candidate word is output. The probability of the sign language images corresponding to the gaps belonging to each candidate word is used as the visual confidence of each candidate word. Determine whether the number of missing words in the initial sign language prediction text is greater than the preset number. If it is less, calculate the semantic similarity between each candidate word and the initial sign language prediction text, and use it as the text confidence of each candidate word. If the value is greater than or equal to the value, the initial predicted text corresponding to multiple consecutive frames of sign language images to be identified and the sign language images before and after them will be combined to form the total predicted text; the semantic similarity between each candidate word and the total predicted text will be calculated as the text confidence of each candidate word. Based on the clarity of multiple consecutive frames of sign language images to be recognized, a dynamic adjustment factor is obtained; based on the dynamic adjustment factor, the visual confidence of each candidate word, and the text confidence, the target confidence of each candidate word is obtained; the candidate word with the highest target confidence is taken as the missing text; based on the missing text and the initial sign language prediction text, the sign language recognition results of multiple consecutive frames of sign language images to be recognized are obtained.

[0057] In this embodiment, specifically, the CLIP training process includes: processing the image-sign language text pairs in the Sign Language Detection using Image dataset as follows: First, performing operations such as grayscale conversion, multi-scale filtering for noise reduction, edge detection, morphological closing operation, and binarization on the sign language images in sequence to remove noise from the images and enhance sign language features; then feeding the preprocessed sign language images and corresponding sign language text pairs into the CLIP model for training, and fine-tuning the pre-trained CLIP.

[0058] like Figure 5 As shown, Figure 5 The graph shows the training and validation loss curves for the CLIP model with 3D convolutional enhancement. The horizontal axis represents the training epoch, and the vertical axis represents the loss value, which measures the difference between the model's prediction and the true label. The smaller the value, the more accurate the prediction. The blue line represents the training loss, and the red line represents the validation loss. In the early stages of training (Epoch=1→2), both losses decrease significantly, indicating that the model quickly learns the features of the training data. In the later stages (Epoch≥3), the losses tend to stabilize, indicating that the CLIP model gradually saturates its fit to the training set, and the difference between the training and validation losses is small, showing no obvious overfitting and good generalization ability.

[0059] In this embodiment, specifically, the BERT model is used to calculate semantic similarity. When the number of missing initial sign language prediction texts is greater than a preset number, the input to the BERT model is: the initial sign language prediction texts corresponding to multiple consecutive frames of sign language images to be identified (10 frames), the initial sign language prediction texts corresponding to multiple frames of sign language images before and after the multiple consecutive frames of sign language images to be identified, and the total prediction text (30 frames in total) after being combined. The semantic similarity between each candidate word and the total prediction text is calculated by the BERT model.

[0060] In this embodiment, preferably, the loss function of the YOLOv7 model includes: target confidence loss, classification loss, and facial expression detail perception loss; The formula for loss of facial detail perception is: , in, Loss of perception of facial expression details Weighting loss for facial expression detail perception. A set of pixels representing facial landmarks. For gradient operators, Let (x, y) be the pixel value of pixel (x, y) in the sample image. The pixel value (x, y) in the reconstructed image corresponding to the sample image is given. The process of obtaining the reconstructed image corresponding to the sample image is as follows: the sample image is passed through the backbone network of the YOLOv7 model to obtain basic visual features; the reconstructed image is generated based on the basic visual features. Indicates the first Line 1 Column pixels, For row index, For column indexes, It is the square of the L2 norm, used to measure the squared Euclidean distance between two gradient vectors.

[0061] Facial landmark pixel set It refers to the set of pixels in an image corresponding to predefined key facial features (such as the corners of the eyes, corners of the mouth, wings of the nose, and brow peaks). These key points are determined by facial landmark detection algorithms and cover areas where subtle differences are easily generated when facial expressions change. They are the core pixel areas for capturing subtle dynamics of facial expressions (such as the degree of upward movement of the corners of the mouth and changes in wrinkles at the corners of the eyes).

[0062] The traditional YOLOv7 model's loss function consists of three parts: bounding box loss, target confidence loss, and classification loss. However, in scenarios involving interaction with deaf and mute individuals, the facial expression videos captured by cameras are almost entirely head-to-head videos with minimal interference from other irrelevant background factors. Therefore, the bounding box loss is virtually ineffective in this environment. Consequently, this invention removes the bounding box loss. In contrast, in facial expression recognition scenarios, details in different facial regions (such as furrowed brows and the curvature of the mouth) significantly impact the recognition results. Therefore, this invention additionally designs an expression detail perception loss and conducts comparative experiments to demonstrate the effectiveness of the improved loss function. The loss function designed in this application improves the accuracy of localizing key facial regions (IoU around the eyes increased from 0.75 to 0.88) and the accuracy of rare expression recognition ("surprise" F1 score increased from 0.62 to 0.79).

[0063] To address the issue that traditional YOLOv7 models suffer from low accuracy in facial expression recognition tasks due to their focus solely on target localization and classification losses, resulting in insufficient capture of facial details (such as subtle facial muscle movements and local changes in facial features), particularly in distinguishing between similar expressions (e.g., smiles versus laughter, slight frowns versus pronounced frowns), this invention introduces an expression detail perception loss. This loss focuses on the facial keypoint pixel set—a region sensitive to facial changes—and calculates the gradient difference between the sample image and the reconstructed image in this region. This enhances the model's ability to learn subtle facial features, enabling it to more accurately capture dynamic changes in key areas such as the corners of the eyes and mouth. By forcing the model's predicted facial region gradient to match the gradient of the original image, it ensures that micro-expression-related texture features (such as skin wrinkles caused by muscle movements and subtle deformations at the corners of the eyes) are not ignored by the model. The invention also eliminates the original bounding box loss, reducing interference from non-expression-related features and thus improving the precision and accuracy of facial expression recognition, particularly in distinguishing between similar expressions and recognizing subtle expressions.

[0064] In this embodiment, preferably, the traditional YOLOv7 model consists of a Backbone, a Neck (feature fusion network), and a Head connected sequentially. This invention adds channel attention and spatial attention modules between the Neck and Head modules of the YOLOv7 model to obtain the target YOLOv7 model. Spatial attention enhances the periorbital and oral regions; channel attention enhances expression-related channels and suppresses irrelevant channels, completing the spatial localization and channel selection of expression features, which are then passed to the Head module to output the classification results.

[0065] The feature map output by the Neck module in the YOLOv7 model is passed through the channel attention and spatial attention modules to obtain the attention feature map, including: The feature map output by the Neck module is subjected to max pooling to obtain a max pooled feature map; the feature map output by the Neck module is subjected to average pooling to obtain an average pooled feature map, compressing the channels to 1 dimension. After concatenating the max-pooling feature map and the average-pooling feature map along the channel dimension, the channel becomes 2D. Then, a spatial mask is generated by sequentially passing the mask through a 3×3 convolution and a sigmoid activation function. ; The spatial mask is multiplied element-wise with the feature map output by the Neck module to enhance key areas such as the eye area and mouth, thus obtaining the spatial attention feature map. The spatial attention feature map is compressed using global average pooling to extract global features for each channel. Channel dependencies are then learned through two 1×1 convolutions, with ReLU activation introduced after the first convolution to add non-linearity. Finally, Sigmoid activation is applied to normalize the convolution results to [0, 1] to generate a channel mask. ; The channel mask is multiplied element-wise with the feature map output by the Neck module to enhance expression-related channels and suppress irrelevant channels, thereby completing the spatial localization and channel filtering of expression features and obtaining the attention feature map.

[0066] In this embodiment, preferably, a series of parameter improvements are made to the YOLOv7 model to improve its inference speed. The preprocessed deaf-mute facial expressions are set to a resolution of 640×640, which ensures image clarity while reducing model adaptation overhead. The model depth scaling factor depth_multiple is set to 0.3, which reduces the number of network layers and avoids the gradient vanishing problem caused by excessively deep networks. The model channel scaling factor width_multiple is set to 0.5, which reduces the number of channels, reduces feature redundancy, and focuses on key facial expression features. The hyperparameters of the YOLOv7 network are adjusted using Adam, with a learning rate momentum of 0.928 to accelerate convergence. At the same time, the gradient weight decay coefficient is 0.0005 during optimization to suppress model overfitting through regularization. The output layer uses the Softmax activation function to convert the output category into a probability score value.

[0067] In this embodiment, specifically, a camera is used to acquire several facial images of the user communicating in sign language. The facial images are then preprocessed to remove images with unclear facial expressions and images with excessive background noise. The preprocessed facial images were manually labeled using labelImg. The bounding box annotation function of labelImg was used to accurately select key facial regions. Each annotation box was assigned a corresponding expression category label, such as happy, sad, angry, and surprise. The dataset was then divided into training, test, and validation sets using the train_test_split function of sklearn in an 8:1:1 ratio. Stratified random partitioning ensured that the expression category distribution within each subset remained consistent. The training set contained 5406 images of 8 expression categories, while the test and validation sets each contained 676 images of each expression category.

[0068] Using the aforementioned training set, test set, and validation set, the YOLOv7 model was trained, tested, and validated, respectively.

[0069] This second embodiment provides a sign language recognition system, including: A memory for storing computer programs; a processor for executing the computer programs to implement the steps of the aforementioned sign language recognition method.

[0070] In this embodiment, specifically, user communication information is wirelessly transmitted to the local computer in real time through an RGB camera and a depth camera. The feature information of the facial expressions and sign language of the deaf-mute person is weighted and fused for output, which not only improves the accuracy of sign language prediction, but also greatly enhances the real-time performance of the interaction.

[0071] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0072] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0073] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0074] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0075] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A sign language recognition method, characterized in that, include: Based on preprocessed, consecutive frames of sign language images from the user, the initial predicted sign language text corresponding to these frames is obtained, including: Based on the preprocessed sign language images of the user in multiple consecutive frames, the semantic vector of the sign language text corresponding to the sign language images in multiple consecutive frames is obtained through the CLIP model. Based on the semantic vectors of sign language text corresponding to multiple consecutive frames of sign language images to be identified, the initial predicted sign language text corresponding to multiple consecutive frames of sign language images to be identified is obtained. Determine whether there are gaps in the initial sign language prediction text corresponding to multiple consecutive frames of sign language images to be recognized. If not, use the initial sign language prediction text as the sign language recognition result for the multiple consecutive frames of sign language images to be recognized. Alternatively, the steps to obtain the sign language recognition result include: Acquire consecutive multi-frame facial videos of the same user; wherein, consecutive multi-frame facial images and consecutive multi-frame sign language images to be recognized are acquired synchronously; Input multiple consecutive frames of facial images from a user into the YOLOv7 model, and output the expression category vectors corresponding to the multiple consecutive frames of facial images. The preprocessed multi-frame sign language images of the user are passed through a multilayer perceptron to obtain the sign language semantic weight vector. The user's continuous multi-frame facial images are processed through a multilayer perceptron to obtain the expression feature weight vector; Based on the sign language semantic weight vector, facial expression feature weight vector, sign language text semantic vector corresponding to multiple consecutive frames of sign language images to be identified, and facial expression category vector corresponding to multiple consecutive frames of facial video, a fused feature vector is obtained, using the following formula: ， in, To fuse feature vectors, This refers to the semantic vectors of sign language text corresponding to multiple consecutive frames of sign language images to be recognized. This is a vector representing the expression categories corresponding to multiple consecutive frames of facial video. It is the sigmoid activation function. Scaling factor This is the semantic weight vector for sign language. These are preprocessed, consecutive frames of sign language images from the user to be recognized. For facial expression feature weight vectors, For a series of multiple frames of facial images of the user; Based on the fused feature vector, the sign language recognition results of sign language images to be recognized in multiple consecutive frames are obtained; If it exists, the sign language image corresponding to the missing part is passed through a linear projection layer, and the preprocessed probability of the missing part belonging to each candidate word is output; the probability of the sign language image corresponding to the missing part belonging to each candidate word is used as the visual confidence of each candidate word. Determine whether the number of missing words in the initial sign language prediction text is greater than the preset number. If it is less, calculate the semantic similarity between each candidate word and the initial sign language prediction text, and use it as the text confidence of each candidate word. If the value is greater than or equal to the value, the initial predicted text corresponding to multiple consecutive frames of sign language images to be identified and the sign language images before and after them will be combined to form the total predicted text; the semantic similarity between each candidate word and the total predicted text will be calculated as the text confidence of each candidate word. Based on the clarity of multiple consecutive frames of sign language images to be recognized, a dynamic adjustment factor is obtained; based on the dynamic adjustment factor, the visual confidence of each candidate word, and the text confidence, the target confidence of each candidate word is obtained; the candidate word with the highest target confidence is taken as the missing text; based on the missing text and the initial sign language prediction text, the sign language recognition results of multiple consecutive frames of sign language images to be recognized are obtained.

2. The sign language recognition method according to claim 1, characterized in that, The process of acquiring multiple consecutive frames of sign language images to be recognized from the user after preprocessing includes: The system sequentially performs image grayscale processing, multi-scale filtering for noise reduction, edge detection, morphological closing operation, and binarization on multiple consecutive frames of sign language images to be recognized, resulting in preprocessed images of the user's multiple consecutive frames of sign language images to be recognized.

3. The sign language recognition method according to claim 2, characterized in that, Image grayscale processing is performed on multiple consecutive frames of sign language images to be recognized, including: For multiple consecutive frames of sign language images to be recognized, obtain the RGB values ​​of each pixel in each frame of the sign language image to be recognized; Based on the RGB values ​​of each pixel in each frame of the sign language image to be recognized, the channel difference between the G and R channels and the channel difference between the G and B channels of each pixel in each frame of the sign language image to be recognized are calculated using the following formula: ， in, The pixels in the current sign language image to be recognized Channel difference between G channel and R channel The pixels in the current sign language image to be recognized Channel difference between G channel and B channel This indicates taking the maximum value. The pixels in the current sign language image to be recognized G channel value, The pixels in the current sign language image to be recognized R channel value, The pixels in the current sign language image to be recognized B channel value, The first sign language image to be recognized Line number Column pixels, For row index, For column indexes; Based on the channel difference between the G and R channels and the channel difference between the G and B channels of each pixel in each frame of the sign language image to be recognized, the skin color probability of each pixel in each frame of the sign language image to be recognized is obtained, and the formula is as follows: ， in, The pixels in the current sign language image to be recognized The probability of skin color; Based on the skin color probability of each pixel in each frame of the sign language image to be recognized, the RGB values ​​of each pixel in each frame of the sign language image to be recognized are adjusted to obtain a grayscale sign language image for each frame.

4. The sign language recognition method according to claim 2, characterized in that, Multi-scale filtering and denoising are performed on each frame of the grayscale sign language image to be recognized, including: A Gaussian filter whose kernel standard deviation is less than a preset lower limit standard deviation is used as a small-scale Gaussian filter; A Gaussian filter whose Gaussian kernel standard deviation is greater than or equal to the preset lower standard deviation and less than or equal to the preset upper standard deviation is used as a mesoscale Gaussian filter. Each frame of the grayscale sign language image to be recognized is filtered by a small-scale Gaussian filter, a medium-scale Gaussian filter, and an adaptive median filter to obtain the filter map of each frame of the sign language image to be recognized for different filters. Based on the filtered images of each frame of the sign language image to be recognized corresponding to different filters, and the skin color probability of each pixel in each frame of the sign language image to be recognized, the target filtered image of each frame of the sign language image to be recognized is obtained, using the following formula: ， in, For the target filtered pixels in the sign language image to be recognized The value, For small-scale Gaussian filters, the corresponding pixels in the filtered image of the sign language image to be recognized are... The value, The pixels in the filtered image of the current sign language image to be recognized are represented by a mesoscale Gaussian filter. The value, The adaptive median filter corresponds to the pixels in the filtered image of the sign language image to be recognized. The value, This indicates taking the maximum value. To set a skin tone threshold, The pixels in the current sign language image to be recognized The probability of skin color, To indicate the first Line number Column pixels, For row index, For column indexes.

5. The sign language recognition method according to claim 2, characterized in that, Edge detection is performed on the target filtered map of each frame of the sign language image to be recognized, including: Based on the skin color probability of each pixel in each frame of the sign language image to be recognized, the preset maximum edge detection threshold, and the preset minimum edge detection threshold, the edge detection threshold of each pixel in each frame of the sign language image to be recognized is obtained, and the formula is: ， in, The pixels in the current sign language image to be recognized The edge detection threshold, To preset the maximum edge detection threshold, To preset the minimum edge detection threshold, These are the weighting coefficients. The pixels in the current sign language image to be recognized The probability of skin color, Indicates the first Line number Column pixels, For row index, For column indexes; Based on the edge detection threshold of each pixel in each frame of the sign language image to be recognized, the edge detection map of each frame of the sign language image to be recognized is obtained.

6. The sign language recognition method according to claim 2, characterized in that, Binarization is performed on the edge detection map of each frame of the sign language image to be recognized, including: The edge detection map of each frame of the sign language image to be recognized is segmented to obtain the knuckle area, palm area and background area of ​​each frame of the sign language image to be recognized. The knuckle area uses pixel blocks of no more than a preset lower limit size to calculate the binarization threshold, the background area uses pixel blocks of no less than a preset upper limit size to calculate the binarization threshold, and the palm area uses pixel blocks between the knuckle area block size and the background area block size to calculate the binarization threshold. For the boundary between different zones, the binarization threshold is calculated by optimizing the threshold through a sliding window with a preset overlap. The optimal threshold for each region of the sign language image to be recognized in each frame is determined by maximizing the inter-class variance. Based on the optimal threshold for each region of the sign language image to be recognized in each frame and the pixel value of each pixel in the sign language image to be recognized in each frame, the user's consecutive multi-frame sign language images to be recognized after preprocessing are obtained.

7. The sign language recognition method according to claim 1, characterized in that, The loss functions of the YOLOv7 model include: target confidence loss, classification loss, and facial detail perception loss; The formula for loss of facial detail perception is: ， in, Loss of perception of facial expression details Weighting loss for facial expression detail perception. A set of pixels representing facial landmarks. For gradient operators, Let (x, y) be the pixel value of pixel (x, y) in the sample image. The pixel value (x, y) in the reconstructed image corresponding to the sample image is given. The process of obtaining the reconstructed image corresponding to the sample image is as follows: the sample image is passed through the backbone network of the YOLOv7 model to obtain basic visual features; the reconstructed image is generated based on the basic visual features. Indicates the first Line number Column pixels, For row index, For column indexes, It is the square of the L2 norm.

8. The sign language recognition method according to claim 1, characterized in that, Between the Neck and Head modules of the YOLOv7 model, channel attention and spatial attention modules are set to obtain the target YOLOv7 model. The feature map output by the Neck module in the YOLOv7 model is passed through the channel attention and spatial attention modules to obtain the attention feature map, including: The feature map output by the Neck module is subjected to max pooling to obtain a max pooled feature map; the feature map output by the Neck module is subjected to average pooling to obtain an average pooled feature map. After concatenating the max pooling feature map and the average pooling feature map along the channel dimension, a spatial mask is generated by sequentially passing the 3×3 convolution and the Sigmoid activation function. After multiplying the spatial mask element-wise with the feature map output by the Neck module, the channel mask is generated by sequentially applying global average pooling, 1×1 convolution, ReLU activation function, 1×1 convolution, ReLU activation function, and Sigmoid activation function. The attention feature map is obtained by multiplying the channel mask element-wise with the feature map output by the Neck module.

9. The sign language recognition method according to claim 1, characterized in that, The CLIP model's image encoder takes multiple consecutive frames of sign language images to be recognized and outputs the visual features of these images, including: For each frame of the sign language image to be recognized, it is divided into blocks to obtain multiple patches; After recombining all patches corresponding to multiple consecutive frames of sign language images to be recognized into tensors, spatiotemporal features are extracted through 3D convolution. The spatiotemporal features are flattened into embedding vectors and superimposed with positional encodings. Visual features are obtained through a multi-layer Transformer encoder.

10. A sign language recognition system, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the sign language recognition method according to any one of claims 1 to 9.

Images