Sign language recognition method and device, and readable storage medium

Abstract

The present application relates to the technical field of sign language recognition, and discloses a sign language recognition method and device, and a readable storage medium. The method comprises: acquiring video data to undergo sign language recognition, wherein the video data comprises a plurality of image frames; for each image frame, calculating a first similarity between the image frame and a preset reference image, and determining, on the basis of the first similarity, an image value corresponding to the image frame, wherein the image value is positively correlated with the first similarity, and the preset reference image is an image containing effective sign language content; and selecting, from among the respective image frames, a keyframe image having an image value greater than a preset threshold, and inputting the keyframe image into a preset sign language recognition model to obtain a sign language recognition result. The present application improves the accuracy of sign language recognition results.

Description

Sign language recognition methods, devices and readable storage media

[0001] This application claims priority to Chinese Patent Application No. 202411205689.1, filed on August 29, 2024, entitled “Sign Language Recognition Method, Device and Readable Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of sign language recognition technology, and in particular to a sign language recognition method, device and readable storage medium. Background Technology

[0003] Sign language is an important way for deaf and mute people to obtain and express information. They typically communicate through sign language, which requires both parties to understand it. However, learning sign language requires a significant investment of time and effort for most hearing people, resulting in low sign language proficiency and difficulty communicating with deaf and mute individuals. Therefore, intelligent sign language recognition is of great significance for communication among deaf and mute people.

[0004] Traditional sign language recognition schemes collect video data of sign language users, then use deep learning models to extract sign language features from the video, and finally predict results based on these extracted features. However, because the collected sign language video data may contain some unintentional gesture interference frames, such as frames of hands lowered, raised, or blank frames, the accuracy of the above-mentioned sign language recognition schemes is relatively low.

[0005] Therefore, how to improve the accuracy of sign language recognition results is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] The main objective of this application is to provide a sign language recognition method, device, and readable storage medium, aiming to solve the technical problem of how to improve the accuracy of sign language recognition results.

[0007] To achieve the above objectives, this application provides a sign language recognition method, which includes:

[0008] Acquire video data to be used for sign language recognition, wherein the video data includes multiple frames of images;

[0009] For each frame of image, a first similarity is calculated between the image and a preset reference image. Based on the first similarity, the image value corresponding to the image is determined, wherein the image value is positively correlated with the first similarity, and the preset reference image is an image containing valid sign language content.

[0010] Select keyframe images with image values ​​greater than a preset threshold from each of the images, and input the keyframe images into a preset sign language recognition model to obtain sign language recognition results.

[0011] In one embodiment, the step of determining the image value corresponding to the image based on the first similarity includes:

[0012] Obtain the adjacent frame images corresponding to the image, and calculate the second similarity between the image and the adjacent frame images, wherein the adjacent frame images are the previous frame image or the next frame image;

[0013] Calculate the weighted sum of the first similarity and the second similarity, and determine the image value corresponding to the image by the weighted sum.

[0014] In one embodiment, the step of calculating a first similarity between the image and a preset reference image includes:

[0015] Calculate the Pearson correlation coefficient between the image and the preset reference image, and determine the Pearson correlation coefficient between the image and the preset reference image as the first similarity.

[0016] In one embodiment, after acquiring the video data to be recognized for sign language, the method further includes:

[0017] The images are obtained by arranging them in chronological order to form an image sequence, and the middle position of the image sequence is determined.

[0018] The image corresponding to the middle position of the image sequence is selected as the preset reference image.

[0019] In one embodiment, the sign language recognition model includes an input module, an encoding module, a decoding module, and a prediction head connected in sequence. The step of inputting the keyframe image into the preset sign language recognition model to obtain the sign language recognition result includes:

[0020] The input module performs embedding and position encoding processing on the keyframe image to obtain the input vector.

[0021] The input vector is input into the encoding module to obtain an encoding matrix, wherein the encoding module is used to encode the input vector;

[0022] The encoding matrix is ​​input into the decoding module to obtain the decoding matrix, wherein the decoding module is used to perform decoding processing on the encoding matrix;

[0023] The decoding matrix is ​​input into the prediction head to obtain the sign language recognition result.

[0024] In one embodiment, the input module includes an image embedding module, a keypoint embedding module, and a position encoding module. The step of obtaining an input vector by embedding and position encoding the keyframe image through the input module includes:

[0025] The keyframe image is embedded using the image embedding module to obtain an image vector;

[0026] The key points in the keyframe image are extracted by the key point embedding module, and the key points are embedded to obtain a key point vector.

[0027] The image vector and the key point vector are positionally encoded by the position encoding module to obtain an input vector, wherein the input vector includes an image position encoding vector and a key point position encoding vector.

[0028] In one embodiment, the encoding module includes an image encoder stack, a keypoint encoder stack, and a fusion module. The step of inputting the input vector into the encoding module to obtain the encoding matrix includes:

[0029] The image position encoding vector in the input vector is input into the image encoder stack to obtain the image encoding matrix;

[0030] The keypoint location encoding vector in the input vector is input into the keypoint encoder stack to obtain the keypoint encoding matrix;

[0031] The image encoding matrix and the keypoint encoding matrix are weighted and fused by the fusion module to obtain the encoding matrix.

[0032] In one embodiment, the prediction head includes a pooling layer, multiple regularization layers, and an output layer. The step of inputting the decoding matrix into the prediction head to obtain the sign language recognition result includes:

[0033] The decoding matrix is ​​input into the pooling layer to obtain the fusion vector;

[0034] The fusion vector is input into each of the regularization layers to obtain the regularization vector;

[0035] All the regularized vectors are input into the output layer to obtain the sign language recognition result.

[0036] In addition, to achieve the above objectives, this application also provides a sign language recognition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the sign language recognition method as described above.

[0037] In addition, to achieve the above objectives, this application also provides a readable storage medium, which is a computer-readable storage medium, storing a program that implements the sign language recognition method. The program that implements the sign language recognition method is executed by a processor to implement the steps of the sign language recognition method as described above.

[0038] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the sign language recognition method described above.

[0039] One or more technical solutions proposed in this application have at least the following technical effects:

[0040] The process involves acquiring video data for sign language recognition, which includes multiple frames of images. For each frame, a first similarity is calculated between the image and a preset reference image. The image value is determined based on the first similarity, where the image value is positively correlated with the first similarity. The preset reference image is an image containing valid sign language content. Keyframe images with image values ​​greater than a preset threshold are selected from the images, and these keyframe images are input into a preset sign language recognition model to obtain the sign language recognition result. Considering that sign language primarily involves finger and palm movements, the similarity between images is typically high, with only small areas of the fingers and / or palms showing differences. Conversely, when not using sign language, the hands are usually lowered, resulting in lower similarity. Therefore, in this embodiment, a first similarity (or similarity score) is calculated between each image and a preset benchmark image containing valid sign language content. The image value of each image is then evaluated based on this first similarity score, which is positively correlated with the first similarity score. This evaluation effectively represents the probability that each image contains valid sign language content. Keyframe images with a value greater than or equal to a preset threshold are then selected from all images. Sign language recognition is performed based on these keyframe images to obtain the recognition result. In other words, high-quality keyframe images with a high probability of containing valid sign language content are selected from the video data, while low-value, meaningless images are removed, thus reducing the interference of meaningless images on the recognition result and improving the accuracy of the sign language recognition. Furthermore, compared to the method of calculating the similarity between each frame image and adjacent frame images separately, the method of calculating the similarity between each frame image and a preset reference frame image in this embodiment can reduce the influence of initial blank frames and ending blank frames (blank frames refer to meaningless image frames without actual sign language meaning). If the adjacent frame image is a blank frame, the similarity between the current frame image and the adjacent frame image cannot effectively evaluate the image value of the current frame image, thereby further improving the accuracy of the sign language recognition results. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 is a flowchart illustrating the first embodiment of the sign language recognition method of this application;

[0044] Figure 2 is a schematic diagram of the keyframe image screening process of the sign language recognition method of this application;

[0045] Figure 3 is a schematic diagram of the image embedding module processing flow of the sign language recognition method of this application;

[0046] Figure 4 is a schematic diagram of key hand points involved in an embodiment of the sign language recognition method of this application;

[0047] Figure 5 is a schematic diagram of the key point embedding module processing flow of the sign language recognition method of this application;

[0048] Figure 6 is a schematic diagram of the overall structure of the sign language recognition model involved in an embodiment of the sign language recognition method of this application;

[0049] Figure 7 is another schematic diagram of the overall structure of the sign language recognition model involved in an embodiment of the sign language recognition method of this application;

[0050] Figure 8 is a schematic diagram of the hardware operating environment of the sign language recognition device in the embodiments of this application.

[0051] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Sign language videos can be viewed as ordered sequences containing a large amount of redundant information. Currently, sign language recognition for deaf and mute individuals mainly relies on single-image information. This method uses deep learning models to identify sign language information from images. However, this approach is often heavily influenced by the quality of the sign language expression and struggles to recognize less-than-perfect expressions. Another method is sensor-based sign language recognition. This method identifies hand trajectories and matches them against predefined templates to achieve sign language recognition. However, this method typically requires building a large number of highly standardized sign language trajectory templates and often necessitates the use of external devices such as data gloves, making it highly dependent on the hardware environment.

[0054] Currently, Shot Boundary Detection (SBD) is a keyframe extraction algorithm used to extract key frames from sign language videos. This allows for the selection of critical image frames for sign language recognition. SBD is a widely used and mature algorithm for determining key frames in videos. The basic idea of ​​this algorithm is to determine key frames by detecting changes in visual content. However, sign language videos often contain numerous blank frames at the beginning and end. Since the SBD algorithm simply compares the current frame with the previous frame, it is heavily influenced by these initial and final blank frames, leading to wasted computational resources and inefficient keyframe calculation, resulting in lower accuracy in sign language recognition.

[0055] Based on this, the main solution of this application is: to acquire video data to be recognized by sign language, wherein the video data includes multiple frames of images; for each frame of image, to calculate the first similarity between the image and a preset reference image, and to determine the image value corresponding to the image based on the first similarity, wherein the image value is positively correlated with the first similarity, and the preset reference image is an image containing valid sign language content; to select key frame images with image values ​​greater than a preset threshold from each image, and to input the key frame images into a preset sign language recognition model to obtain the sign language recognition result.

[0056] This application calculates the similarity between each image and a preset benchmark image, i.e., a first similarity, and evaluates the image value of each image based on this first similarity. Since image value is positively correlated with the first similarity, the evaluated image value effectively represents the probability that each image contains valid sign language content. Then, keyframe images with an image value greater than or equal to a preset threshold are selected from all images. Sign language recognition is performed based on these keyframe images to obtain the sign language recognition result. In other words, high-quality keyframe images with a high probability of containing valid sign language content are selected from the video data, while low-value, meaningless images are deleted, thereby reducing the interference of meaningless images on the recognition result and improving the accuracy of the sign language recognition result. Furthermore, the similarity calculation between each frame image and the preset benchmark frame image in this application reduces the influence of initial and ending blank frames, further improving the accuracy of sign language recognition.

[0057] It should be noted that the execution subject of each embodiment of the sign language recognition method of this application can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a sign language recognition device capable of realizing the above functions, such as AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, smart glasses, or a robot. For example, AR glasses will be used as the execution subject in the description and explanation of each embodiment of the sign language recognition method of this application.

[0058] Based on this, this application proposes a sign language recognition method according to a first embodiment. Referring to Figure 1, the sign language recognition method includes steps S10 to S30:

[0059] Step S10: Obtain video data to be used for sign language recognition, wherein the video data includes multiple frames of images;

[0060] The video data specifically refers to the captured sign language video. In other words, the video is composed of multiple frames of images. For example, when a camera captures video, it captures images at a preset frame rate to obtain video data.

[0061] Furthermore, after acquiring the video data, image preprocessing can be performed on all images included in the video data. Subsequent steps are then executed based on the preprocessed images to further improve image quality. Image preprocessing includes, but is not limited to, deleting images with a sharpness below a certain preset threshold, deleting images with an illumination intensity below a certain preset threshold, deleting a preset frame image at a starting position (deleting the first preset frame image in the video data), and deleting a preset frame image at an ending position (deleting the last preset frame image in the video data). This embodiment does not impose specific limitations on these steps.

[0062] Step S20: For each frame of image, calculate the first similarity between the image and a preset reference image, and determine the image value corresponding to the image based on the first similarity, wherein the image value is positively correlated with the first similarity, and the preset reference image is an image containing effective sign language content;

[0063] After acquiring the video data, for each frame of the video data, calculate the first similarity between the frame image (hereinafter referred to as the current frame image) and the preset reference image. The first similarity is the similarity between the current frame image and the preset reference image.

[0064] The preset reference image is an image containing valid sign language content. Further, the preset reference image is one or more frames containing valid sign language content. If the preset reference image includes multiple frames, the first similarity between the image and the preset reference image can specifically be the average similarity, maximum similarity, or median similarity between the image and all images in the preset reference image, etc., without limitation. To reduce computational load, the preset reference frame is a single frame. Therefore, the embodiments of this application are described and illustrated using the preset reference frame as a single frame.

[0065] It should be noted that an image containing valid sign language content means that the hand gestures in the image are sign language postures with actual sign language meaning.

[0066] Regarding the setting of the preset reference image, as one implementation method, relevant personnel can set the preset reference image in advance. As another implementation method, one frame containing valid sign language content can be selected from multiple frames included in the video data as the preset reference image.

[0067] In a first implementation method, after acquiring the video data for sign language recognition, the method further includes: (The method involves selecting one frame containing valid sign language content from multiple frames of video data as a preset reference image.)

[0068] Step S201: Obtain an image sequence by arranging the images in chronological order, and determine the middle position of the image sequence.

[0069] Considering that sign language gestures typically have a start and end time, with the middle time usually consisting of continuous gestures, the middle position of this sequence can be any position other than a predetermined number of start and end positions. For example, if the image sequence length is 400, the middle position can be any position from the 50th element to the 300th element. Here, the sequence length of the image sequence is the number of image frames included in the sequence.

[0070] Furthermore, the middle position of the sequence can specifically be the very center of the image sequence. When the sequence length of the image sequence is even, the very center position can be any one of the two very center positions. For example, when the image sequence length is 5, the middle position of the sequence can be the position of the 3rd element. Or, when the image sequence length is 6, the middle position of the sequence can be either the position of the 3rd element or the position of the 4th element.

[0071] Step S202: Select the image corresponding to the middle position of the image sequence as the preset reference image.

[0072] By selecting the image corresponding to the middle position of the sequence as the reference image, or in other words, selecting the image at the middle position of the sequence as the preset reference image, the efficient selection of the preset reference image is achieved.

[0073] In the second implementation, considering that the signer's arm and body form a certain angle during sign language expression, let (x1, y1) be the position coordinates of the elbow joint and (x2, y2) be the position coordinates of the wrist. The angle s formed by these two points and the angle in the opposite direction of gravitational acceleration can represent the on / off state of the sign language gesture. When sign language is being used, s is usually less than a certain value (such as 150°, 160°, etc.), while when no sign language is being used, s is usually greater than or equal to this certain value. Based on this, the s value of the image can be calculated, and a frame with an s value less than a preset threshold can be selected as the preset reference frame image.

[0074] As a third implementation, a preset gesture detection algorithm, such as support vector machine, random forest, or deep learning model, can be used to determine whether the gestures in an image have actual sign language meaning. If the preset gesture recognition algorithm detects that a gesture in a frame of an image has actual sign language meaning, then that frame of an image is selected as a preset reference image. It should be noted that, considering the generally long duration of sign language videos, which include a large number of images, and the high complexity and computational cost of current gesture detection algorithms, inputting all images into the preset gesture detection algorithm and then filtering for high-value keyframe images would require significant computational resources and be time-consuming. Therefore, images can be input into the preset gesture detection algorithm sequentially according to time order until a frame with actual sign language meaning is detected, at which point the detection process ends.

[0075] It should be noted that the above are only four feasible implementation methods for setting a preset reference image provided in this embodiment. This embodiment does not impose specific limitations on the specific implementation methods for setting the preset reference image.

[0076] It should be noted that, unless otherwise specified, the images described in the embodiments of this application refer only to the images in the acquired video data.

[0077] Step S30: Select keyframe images whose image value is greater than a preset threshold from the images, and input the keyframe images into a preset sign language recognition model to obtain sign language recognition results.

[0078] Select keyframe images whose image value is greater than a preset threshold from among the images. Specifically, the keyframe images can be all images whose image value is greater than the preset threshold. The preset threshold can be any value set in advance, and this embodiment does not impose any specific restrictions on it.

[0079] After selecting keyframe images, these images are input into a pre-defined sign language recognition model to obtain the sign language recognition result, which can be the final output of the sign language recognition model. By using a sign language recognition model for sign language recognition, compared to sensor-based sign language recognition methods, there is no need to rely on additional sensor devices such as sign language gloves, nor is it necessary to pre-build a large number of sign language trajectory templates, thus reducing the implementation cost of sign language recognition.

[0080] In this embodiment, video data to be recognized for sign language is acquired, wherein the video data includes multiple frames of images; for each frame of image, a first similarity between the image and a preset reference image is calculated, and the image value corresponding to the image is determined based on the first similarity, wherein the image value is positively correlated with the first similarity, and the preset reference image is an image containing valid sign language content; key frame images with image values ​​greater than a preset threshold are selected from each image, and the key frame images are input into a preset sign language recognition model to obtain the sign language recognition result. Considering that sign language primarily involves finger and palm movements, the differences between images are typically limited to small areas of the fingers and / or palms. However, when the signer is not using sign language, their hands are usually lowered, resulting in lower similarity to images used in sign language. Therefore, this embodiment calculates the similarity (first similarity) between each image and a preset benchmark image containing valid sign language content. The image value of each image is then evaluated based on this first similarity, which is positively correlated with the first similarity. This evaluation effectively represents the probability that each image contains valid sign language content. Keyframe images with a value greater than or equal to a preset threshold are then selected from all images. Sign language recognition is performed based on these keyframe images to obtain the recognition result. In other words, high-quality keyframe images with a high probability of containing valid sign language content are selected from the video data, while low-value, meaningless images are removed. This reduces the interference of meaningless images on the recognition result and improves the accuracy of the sign language recognition. Furthermore, compared to the method of calculating the similarity between each frame image and adjacent frame images separately, this embodiment calculates the similarity between each frame image and a preset reference frame image, which can reduce the influence of initial blank frames and ending blank frames. If the adjacent frame is a blank frame, the similarity between the current frame image and the adjacent frame image cannot effectively evaluate the image value of the current frame image, further improving the accuracy of sign language recognition results.

[0081] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, the step of determining the image value corresponding to the image based on the first similarity includes:

[0082] Step A10: Obtain the adjacent frame image corresponding to the image, and calculate the second similarity between the image and the adjacent frame image, wherein the adjacent frame image is the previous frame image or the next frame image;

[0083] It should be noted that a second similarity is calculated between the current frame image and the adjacent frame images, and this second similarity is the similarity between the current frame image and the adjacent frame images.

[0084] The adjacent frame image can be either the previous frame image or the next frame image; this embodiment does not impose any specific restrictions on this.

[0085] Step A20: Calculate the weighted sum of the first similarity and the second similarity, and determine the weighted sum as the image value corresponding to the image.

[0086] Calculate the weighted sum of the first similarity and the second similarity, and determine this weighted sum as the image value corresponding to the current frame image. Specifically, this weighted sum can be obtained by weighted summation of the first similarity and the second similarity.

[0087] The weights corresponding to the first similarity and the second similarity can be set in advance, and the weighted sum value can be obtained by weighted summation based on the weights corresponding to the first similarity and the second similarity.

[0088] Furthermore, let the weight corresponding to the first similarity be denoted as the first weight, and the weight corresponding to the second similarity be denoted as the second weight. Then, the sum of the first weight and the second weight can be one. The first weight and the second weight can be the same or different, and this embodiment does not impose specific restrictions on this.

[0089] In one possible implementation, the step of calculating a first similarity between the image and a preset reference image includes:

[0090] Step B10: Calculate the Pearson correlation coefficient between the image and the preset reference image, and determine the Pearson correlation coefficient between the image and the preset reference image as the first similarity.

[0091] The Pearson Correlation Coefficient (PCC), also known as the Pearson product-moment correlation coefficient, is a statistical indicator that measures the degree of linear correlation between two variables.

[0092] Specifically, the Pearson correlation coefficient can be expressed by the formula: Where PCC(P, Q) represents the Pearson correlation coefficient between images P and Q, N represents the number of pixels in the horizontal direction of the image, M represents the number of pixels in the vertical direction of the image, P(i,j) represents the pixel value of the i-th row and j-th column pixel of image P, Q(i,j) represents the pixel value of the i-th row and j-th column pixel of image P, and μ p μ represents the average pixel value of all pixels in image P. q σ represents the average pixel value of all pixels in image Q. p σ represents the mean squared error of the pixel values ​​of all pixels in image P. q This represents the mean squared error of the pixel values ​​of all pixels in image Q.

[0093] Furthermore, referring to Figure 2, considering that the pixel value of each frame image can be decomposed into three channels: R, G, and B, the PCC calculation formula described above can be used to calculate the PCC value of each channel between two frames. Let the PCC value between the R channels of two frames be PCC. R The PCC value between the G channels of two frames is PCC. G The PCC value between the B channels of two frames is PCC. B The PCC value between two frames can specifically be PCC. R PCC G With PCC B The average value.

[0094] For example, to help understand the technical concept or principle of the sign language recognition method after combining this embodiment with the first embodiment, please refer to Figure 2. The specific sign language recognition process is as follows:

[0095] 1. Obtain the video data to be used for sign language recognition. The video data includes t frames, namely [F1, F2, F3...Fi...Ft], where F...Ft is the number of frames in the video. i Let represent the i-th frame image, with the middle frame of the video data as the preset reference image FF.

[0096] 2. Calculate the image F of the i-th frame. i The PCC value between the current frame and the preset reference image FF is denoted as PCC1. The next frame image F is then selected. i+1 As the adjacent frame image of the i-th frame image, calculate the next frame F. i+1 With F i The PCC value between them is denoted as PCC2. It should be noted that for the last frame, the previous frame can be selected as the adjacent frame.

[0097] 3. Calculate PCC by weighted summation of PCC1 and PCC2. i Specifically, it can be expressed by the formula: PCC i =αPCC(i,FF)+(1-α)PCC(i,F) i+1 ), where PCC(i, FF) represents F i The PCC value between FF, also known as PCC1, PCC(i, F) i+1 ) represents F i With F i+1 The PCC value between them, also known as PCC2, where α represents the first weight.

[0098] 4. Determine the image value of the i-th frame as PCC. i If PCC i If the value is greater than the preset threshold τ, then select the frame image F. iFor keyframe images, otherwise determine F. i For non-keyframe images, the final keyframe sequence is extracted, which is a sequence composed of all keyframe images.

[0099] 5. Input the keyframe sequence into the pre-set sign language recognition model and output the sign language recognition result.

[0100] It should be noted that the above examples are only used to help understand this application and do not constitute a limitation on the sign language recognition method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0101] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the first and second embodiments described above can be referred to the above description and will not be repeated hereafter. Based on this, the sign language recognition model includes an input module, an encoding module, a decoding module, and a prediction head connected in sequence. The step of inputting the keyframe image into the preset sign language recognition model to obtain the sign language recognition result includes:

[0102] Step C10: The keyframe image is embedded and positionally encoded by the input module to obtain the input vector;

[0103] Considering that the Transformer model (a deep learning model based on self-attention) has achieved significant breakthroughs compared to traditional recurrent neural networks in handling long sequence tasks, it introduces a self-attention mechanism that allows the model to capture global dependencies and distribute information across multiple subspaces through multi-head attention techniques, thereby significantly enhancing the model's expressive and understanding capabilities. The architectural advantage of the Transformer model lies in its equal treatment of all elements within a sequence, enabling the model to recognize and understand the rich semantic relationships between elements, especially excelling in understanding relationships between distant sequence elements. Therefore, in this embodiment, the sign language recognition model is specifically a Transformer model.

[0104] Specifically, the Transformer model includes an input module, an encoding module, a decoding module, and a prediction head connected in sequence. The input module is used to embed and positionally encode the keyframe images.

[0105] Embedding is the process of converting input data (such as words, characters, sentences, or any serializable entity) into a fixed-length vector representation. Embedding is the foundation for a model to understand input data.

[0106] Positional encoding refers to adding information about the position of each word in a sentence, also known as "information fragment". Specifically, absolute positional encoding, relative positional encoding, etc. can be used, but this embodiment does not impose any specific restrictions on this.

[0107] Step C20: Input the input vector into the encoding module to obtain an encoding matrix, wherein the encoding module is used to encode the input vector;

[0108] The input vector output from the keyframe image after passing through the input module is transmitted to the encoding module. The encoding module encodes the input vector, converting it into a series of feature vectors, i.e., the encoding matrix, and then inputs the encoding matrix into the decoding module.

[0109] Personnel can design the encoding module according to actual needs, and the encoding matrix is ​​the output of the encoding module. In one specific implementation, the encoding module can be an encoder stack consisting of one or more encoders connected in sequence. In this case, the encoding matrix is ​​the output of the encoder stack, that is, the output of the last encoder in the encoder stack.

[0110] Step C30: Input the encoding matrix into the decoding module to obtain the decoding matrix, wherein the decoding module is used to perform decoding processing on the encoding matrix;

[0111] The decoding module receives the encoding matrix input from the encoding module, decodes the encoding matrix to obtain the output sequence, i.e., the decoding matrix, and inputs the decoding matrix into the prediction head.

[0112] Personnel can design decoding modules according to actual needs, and the decoding matrix is ​​the output of the decoding module. In one specific implementation, the decoding module can be a decoder stack consisting of one or more decoders connected in sequence. In this case, the decoding matrix is ​​the output of the decoder stack, that is, the output of the last layer of the decoder in the decoder stack.

[0113] Step C40: Input the decoding matrix into the prediction head to obtain the sign language recognition result.

[0114] It should be noted that the sign language recognition result is the output of the prediction head.

[0115] In the Transformer model, the prediction head is used to convert the model's internal representation into the final prediction output. In this embodiment, the prediction head is used to convert the decoding matrix into the final prediction output, which is the sign language recognition result.

[0116] Personnel can design a prediction head according to actual needs. In one specific implementation, the prediction head includes a pooling layer, a regularization layer (dropout), a fully connected layer and a softmax function connected in sequence, and finally predicts the output through the softmax function.

[0117] In one possible implementation, the input module includes an image embedding module, a keypoint embedding module, and a position encoding module. The step of obtaining an input vector by embedding and position encoding the keyframe image through the input module includes:

[0118] Step D10: The keyframe image is embedded using the image embedding module to obtain an image vector;

[0119] The image embedding module is used to embed keyframe images to obtain image vectors. Specifically, referring to Figure 3, the embedding process is as follows:

[0120] (1) First, scale the image to a uniform size, such as 224×224×3. Then, divide the image into small patches, each patch having the same size, such as 16×16×3. Then, arrange the small patches.

[0121] (2) Perform an Embedding operation on the arranged patches to obtain the image vector.

[0122] Step D20: Extract key points from the keyframe image through the key point embedding module, and perform embedding processing on the key points to obtain key point vectors;

[0123] The keypoint embedding module is used to extract keypoints (actually, coordinates) from keyframe images. Specifically, it can extract keypoints of the face and hands from keyframe images. Keypoints of the hands are denoted as hand keypoints, and keypoints of the face as facial keypoints. Specifically, a preset number of hand keypoints and a preset number of facial keypoints can be extracted. The preset number of facial keypoints may include eye keypoints (keypoints for the glasses), nose keypoints (keypoints for the nose), mouth keypoints (keypoints for the mouth), and contour keypoints (keypoints for the facial contour). In one specific embodiment, referring to Figure 4, 21 hand keypoints are extracted from a single hand based on the keyframe image, for a total of 42 hand keypoints from both hands. Simultaneously, 58 facial keypoints are extracted, specifically including 17 contour keypoints, 12 eye keypoints, 9 nose keypoints, and 20 mouth keypoints.

[0124] Understandably, a key point can be represented by coordinates, thereby extracting the actual coordinates of each key point.

[0125] For example, referring to FIG5, key points are extracted from the keyframe image, specifically hand key points and facial key points. The extracted hand key points and facial key points are embedded to obtain key point vectors, that is, the key point coordinates are embedded. Then, the key point vectors are input to the position encoding module for position encoding.

[0126] Step D30: The image vector and the key point vector are positionally encoded by the position encoding module to obtain an input vector, wherein the input vector includes an image position encoding vector and a key point position encoding vector.

[0127] The input vector is obtained by performing positional encoding on the image vector and the key point vector respectively through the positional encoding module. The input vector includes the image positional encoding vector and the key point positional encoding vector. Specifically, the image positional encoding vector is the vector obtained by positional encoding the image vector, and the key point positional encoding vector is the vector obtained by positional encoding the key point vector.

[0128] In this embodiment, to allow the model to focus more on the hand and facial information of the sign language interpreter during the expression of sentences, a key point embedding module is set up to extract key points of the hands and face in the image, thereby improving the performance of the model. At the same time, to avoid the influence of noise on key points, an image embedding module is set up to embed key frame images, thereby fusing data from both the image and key points (actually text data in coordinate form) modalities for sign language recognition. By utilizing the complementary advantages of the two modalities of data, features are learned together, further improving the accuracy of sign language recognition results.

[0129] For example, to help understand the technical concept or principle of the sign language recognition model in this embodiment, please refer to Figure 6. The sign language recognition model specifically includes an input module, an encoding module, a decoding module, and a prediction head connected in sequence. The encoding module includes n layers of encoders connected in sequence, the decoding module includes m layers of decoders connected in sequence, and the input module includes an image embedding module and a key point embedding module. The image embedding module and the key point embedding module are respectively connected to the position encoding module.

[0130] Based on this sign language recognition model, after extracting keyframe images from the video data (i.e., Key Frame Extraction (KEF) as shown in Figure 6), the extracted keyframe images are input into the input module of the sign language recognition model. The image embedding module and keypoint embedding module in the input module embed the images and keypoints respectively, then concatenate the outputs of the two embeddings. This output is then sequentially passed through position encoding, an n-layer encoder connected in sequence, an m-layer decoder connected in sequence, and a prediction head to obtain the model's final output, which is the sign language recognition result. It should be noted that the final output of the n-layer encoder is input into each of the m-layer decoders.

[0131] It should be noted that if the current process is training a sign language model, the training result label (as shown in Figure 6, "business license"), which is the expected output of the model, will also be embedded and positionally encoded in sequence before being input into the decoding module. At the same time, the keyframe image corresponding to the result label will be input into the input module. Based on the input of the result label and the input of the image data, the sign language recognition model will be trained to obtain the trained sign language recognition model.

[0132] It should be noted that the above examples are only used to help understand this application and do not constitute a limitation on the structure of the sign language recognition model of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0133] Based on the first, second, and / or third embodiments of this application, in the fourth embodiment of this application, the content that is the same as or similar to the above-described embodiments one, two, and three can be referred to the above description and will not be repeated hereafter. Based on this, the encoding module includes an image encoder stack, a keypoint encoder stack, and a fusion module. The step of inputting the input vector into the encoding module to obtain the encoding matrix includes:

[0134] Step E10: Input the image position encoding vector in the input vector into the image encoder stack to obtain the image encoding matrix;

[0135] It should be noted that this image encoder stack consists of one or more encoders connected in sequence. The image encoding matrix is ​​the output of the image encoder stack, that is, the output of the last encoder layer in the image encoder stack.

[0136] Step E20: Input the key point position encoding vector in the input vector into the key point encoder stack to obtain the key point encoding matrix;

[0137] It should be noted that the keypoint encoder stack consists of one or more encoders connected in sequence, and the keypoint encoding matrix is ​​the output of the keypoint encoder stack, that is, the output of the last encoder layer in the keypoint encoder stack.

[0138] The number of encoder layers in the keypoint encoder stack can be the same as or different from the number of encoder layers in the image encoder stack; this embodiment does not impose a specific limitation on this. In a preferred embodiment, the number of encoder layers in the image encoder is the same as the number of encoder layers in the keypoint encoding stack.

[0139] Step E30: The image coding matrix and the key point coding matrix are weighted and fused by the fusion module to obtain the coding matrix.

[0140] The fusion module is connected to both the image encoder stack and the keypoint encoder stack to fuse the outputs of the image encoder stack and the keypoint encoder stack.

[0141] Furthermore, the image encoding matrix and the keypoint encoding matrix can be weighted and fused based on their respective fusion weights to obtain the encoding matrix. Regarding the setting of the fusion weights, relevant personnel can set the fusion weights corresponding to the image encoding matrix and the keypoint encoding matrix separately, or the fusion weights can be set as model parameters to optimize the fusion weights during the sign language recognition model training process. This embodiment does not impose specific limitations on this.

[0142] Let p be the fusion weight corresponding to the image coding matrix, IA be the image coding matrix, q be the fusion weight corresponding to the keypoint coding matrix, KA be the keypoint coding matrix, and EA be the coding matrix. Then EA can be expressed by the formula EA = p * IA + q * KA. In one specific implementation, the sum of p and q is one.

[0143] In this embodiment, an image encoder stack and a keypoint encoder stack are set to encode the image encoding vector and the keypoint encoding vector respectively. This is equivalent to using encoder stacks to encode image data and text data separately. As a result, the characteristics of each data type can be better utilized, the model performance can be improved, the computational cost can be reduced, and the behavior of each encoder stack can be more easily understood and explained, thereby enhancing the interpretability and customizability of the model. Furthermore, the image and text encoders can be tuned separately, which helps to stabilize the training process.

[0144] For example, to help understand the technical concept or principle of the sign language recognition model after combining this embodiment and the third embodiment, please refer to Figure 7. The sign language recognition model specifically includes an input module, an encoding module, a decoding module, and a prediction head connected in sequence. The encoding module includes an image encoder stack, a keypoint encoder stack, and a fusion module. The image encoder stack includes n layers of encoders (Encoder VD) connected in sequence, and the keypoint encoder stack includes n layers of encoders (Encoder KP) connected in sequence. The decoding module includes m layers of decoders connected in sequence. The input module includes an image embedding module and a keypoint embedding module, which are respectively connected to the position encoding module.

[0145] The fusion module performs weighted fusion of the outputs of the image encoder stack and the keypoint encoder stack, and inputs the fusion result into each decoder of the decoder module. Specifically, the fusion weight corresponding to the image encoding matrix output by the image encoder stack is α, and the fusion weight corresponding to the keypoint encoding matrix output by the keypoint encoder stack is 1-α. The image encoding matrix and the keypoint encoding matrix are weighted and fused based on their respective fusion weights.

[0146] The processing mechanisms of other modules can be referred to the description in Figure 6 above, and will not be repeated here.

[0147] It should be noted that the above examples are only used to help understand this application and do not constitute a limitation on the structure of the sign language recognition model of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0148] Based on the first, second, third, and / or fourth embodiments of this application, in the fifth embodiment of this application, the content that is the same as or similar to the above-described embodiments one, two, three, and four can be referred to the above description and will not be repeated hereafter. Based on this, the prediction head includes a pooling layer, multiple regularization layers, and an output layer. The step of inputting the decoding matrix into the prediction head to obtain the sign language recognition result includes:

[0149] Step F10: Input the decoding matrix into the pooling layer to obtain the fusion vector;

[0150] It should be noted that this fusion vector is the output of the pooling layer.

[0151] This pooling layer fuses the vector sequences in the feature matrix to aggregate them into a fused vector with fixed dimensions and a fixed length.

[0152] Pooling layers aggregate the output sequences. In the Transformer model, after the sequence is processed by multiple layers of self-attention mechanisms and feedforward neural networks in the encoder and decoder, a series of hidden states are obtained, which are representation vectors at each position. These representation vectors contain the semantic information of the input sequence. In order to aggregate these vectors into a vector representation with fixed dimensions and fixed length, a pooling operation is required.

[0153] The pooling layer can specifically be an average pooling layer, a max pooling layer, a global pooling layer, etc., and this embodiment does not impose any specific limitations on it. Considering that compared with other pooling operations, the average pooling operation can preserve the overall characteristics of the sequence because it performs an averaging operation on the sequence, it not only considers the maximum and minimum features in the sequence, but also the overall trend and distribution. Average pooling can also enhance robustness and reduce overfitting. Based on this, in a preferred embodiment, the pooling layer is specifically an average pooling layer.

[0154] Step F20: Input the fusion vector into each of the regularization layers to obtain the regularization vector;

[0155] It should be noted that this regularization vector is the output of the regularization layer.

[0156] Dropout is a technique that randomly "deactivates" (i.e. temporarily ignores) a portion of neurons in a neural network during training. Deactivated neurons do not participate in computation during the current forward and backward propagation. By randomly "deactivating" a portion of neurons in the network during training, the network's dependence on specific training samples is reduced, thereby improving the model's ability to generalize on unseen data.

[0157] The fusion vector is obtained by randomly deactivating the fusion vector through multiple regularization layers (Dropout), that is, the fusion vector is input into each regularization layer.

[0158] It should be noted that the dropout ratios of each regularization layer can be the same or different, and this embodiment does not impose specific restrictions on this. Considering that different dropout ratios can provide different regularization strengths, thereby improving the model's generalization ability, and that applying different dropout ratios can increase model diversity, allowing the model to be trained under different configurations, this diversity helps the model to better adapt to unknown data during use, and allows for customization of different dropout ratios, thus enabling more flexible control of the model's training process. This helps in finding the optimal training strategy to balance the model's accuracy and generalization ability. Therefore, in a preferred embodiment, the dropout ratios of each regularization layer are different.

[0159] The dropout ratio of the regularization layer refers to the proportion of neurons dropped by the regularization layer.

[0160] Step F30: Input all the regularized vectors into the output layer to obtain the sign language recognition result.

[0161] It should be noted that the sign language recognition result is the output of the output layer.

[0162] Relevant personnel can design the output layer according to actual needs. In one specific implementation, the output layer includes a fully connected layer and a softmax function connected in sequence.

[0163] In one possible implementation, the output layer includes multiple fully connected layers and a prediction layer, with each fully connected layer corresponding to a regularization layer. The step of converting all regularization vectors into target probability distribution vectors through the output layer and outputting gesture recognition results based on the target probability distribution vectors includes:

[0164] Step F301: Input each of the regularized vectors into the corresponding connected fully connected layer to obtain a linear vector;

[0165] It should be noted that the linear vector is the output of the fully connected vector.

[0166] The number of fully connected layers and regularization layers is the same, and they are connected one-to-one. Each fully connected layer performs feature integration processing on the input regularization vector to obtain a linear vector, and then outputs this linear vector. For example, suppose the i-th fully connected layer is connected to the i-th regularization layer, and the regularization vector output by the i-th regularization layer is Di. Then Di is input to the i-th fully connected layer, which processes the input Di and outputs a linear vector Li. Here, i can take values ​​from i to N, where N is the total number of regularization layers or fully connected layers.

[0167] Step F302: The prediction layer converts each linear vector into a probability distribution vector, fuses the probability distribution vectors to obtain a target probability distribution vector, and outputs the gesture recognition result based on the target probability distribution vector.

[0168] It should be noted that this prediction layer includes a softmax function. After a linear vector is input into the softmax function, it is converted into a probability distribution vector. Having obtained the probability distribution vectors corresponding to each linear vector, a weighted fusion of all probability distribution vectors can be performed to obtain the target probability distribution vector. For example, the average vector of all probability distribution vectors can be calculated (the average value of the elements at the same position in all probability distribution vectors is used to obtain the vector). This average vector is then determined as the target probability distribution vector. After obtaining this target probability distribution vector, the final result, i.e., the gesture recognition result, can be output based on the target probability distribution vector.

[0169] In this embodiment, multiple regularization layers are set in the prediction head, and multiple different sub-network models are generated using multiple regularization layers, thereby improving the generalization ability of the sign language recognition algorithm model and thus improving the stability of the sign language recognition results.

[0170] Furthermore, this application also proposes a sign language recognition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the steps of the sign language recognition method described above.

[0171] Referring to Figure 8, a schematic diagram of a structure suitable for implementing the sign language recognition device of this application embodiment is shown. The sign language recognition device in this application embodiment may also include, but is not limited to, mobile terminals such as AR glasses, robots, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. The sign language recognition device shown in Figure 8 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0172] As shown in Figure 8, the sign language recognition device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the sign language recognition device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the sign language recognition device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show sign language recognition devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0173] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0174] The sign language recognition device provided in this application, employing the sign language recognition method described in the above embodiments, can solve the technical problem of how to improve the accuracy of sign language recognition results. Compared with the prior art, the beneficial effects of the sign language recognition device provided in this application are the same as those of the sign language recognition method provided in the above embodiments, and other technical features of this sign language recognition device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0175] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0176] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0177] In addition, to achieve the above objectives, embodiments of this application also provide a readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the sign language recognition method in the above embodiments.

[0178] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0179] The aforementioned computer-readable storage medium may be included in the sign language recognition device; or it may exist independently and not be assembled into the sign language recognition device.

[0180] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a sign language recognition device, cause the sign language recognition device to: acquire video data to be recognized for sign language, wherein the video data includes multiple frames of images; for each frame of image, calculate a first similarity between the image and a preset reference image, determine the image value corresponding to the image based on the first similarity, wherein the image value is positively correlated with the first similarity, and the preset reference image is an image containing valid sign language content; select keyframe images with image values ​​greater than a preset threshold from the images, and input the keyframe images into a preset sign language recognition model to obtain a sign language recognition result.

[0181] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0182] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0183] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0184] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described sign language recognition method, and is capable of solving the technical problem of sign language recognition. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the sign language recognition method provided in the above embodiments, and will not be repeated here.

[0185] Furthermore, this application also proposes a computer program product, including a sign language recognition program, which, when executed by a processor, implements the steps of the sign language recognition method described above.

[0186] The specific implementation of the computer program product in this application is basically the same as the various embodiments of the sign language recognition method described above, and will not be repeated here.

[0187] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0188] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0189] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software sensor. This computer software sensor is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0190] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A sign language recognition method, characterized in that, The sign language recognition method includes the following steps: Acquire video data to be used for sign language recognition, wherein the video data includes multiple frames of images; For each frame of image, a first similarity is calculated between the image and a preset reference image. Based on the first similarity, the image value corresponding to the image is determined, wherein the image value is positively correlated with the first similarity, and the preset reference image is an image containing valid sign language content. Select keyframe images with image values ​​greater than a preset threshold from each of the images, and input the keyframe images into a preset sign language recognition model to obtain sign language recognition results.

2. The sign language recognition method as described in claim 1, characterized in that, The step of determining the image value corresponding to the image based on the first similarity includes: Obtain the adjacent frame images corresponding to the image, and calculate the second similarity between the image and the adjacent frame images, wherein the adjacent frame images are the previous frame image or the next frame image; Calculate the weighted sum of the first similarity and the second similarity, and determine the image value corresponding to the image by the weighted sum.

3. The sign language recognition method as described in claim 1, characterized in that, The step of calculating the first similarity between the image and the preset reference image includes: Calculate the Pearson correlation coefficient between the image and the preset reference image, and determine the Pearson correlation coefficient between the image and the preset reference image as the first similarity.

4. The sign language recognition method as described in claim 1, characterized in that, After acquiring the video data to be used for sign language recognition, the method further includes: The images are obtained by arranging them in chronological order to form an image sequence, and the middle position of the image sequence is determined. The image corresponding to the middle position of the image sequence is selected as the preset reference image.

5. The sign language recognition method according to any one of claims 1 to 4, characterized in that, The sign language recognition model includes an input module, an encoding module, a decoding module, and a prediction head connected in sequence. The step of inputting the keyframe image into the preset sign language recognition model to obtain the sign language recognition result includes: The input module performs embedding and position encoding processing on the keyframe image to obtain the input vector. The input vector is input into the encoding module to obtain an encoding matrix, wherein the encoding module is used to encode the input vector; The encoding matrix is ​​input into the decoding module to obtain the decoding matrix, wherein the decoding module is used to perform decoding processing on the encoding matrix; The decoding matrix is ​​input into the prediction head to obtain the sign language recognition result.

6. The sign language recognition method as described in claim 5, characterized in that, The input module includes an image embedding module, a keypoint embedding module, and a position encoding module. The step of obtaining an input vector by embedding and position encoding the keyframe image through the input module includes: The keyframe image is embedded using the image embedding module to obtain an image vector; The key points in the keyframe image are extracted by the key point embedding module, and the key points are embedded to obtain a key point vector. The image vector and the key point vector are positionally encoded by the position encoding module to obtain an input vector, wherein the input vector includes an image position encoding vector and a key point position encoding vector.

7. The sign language recognition method as described in claim 6, characterized in that, The encoding module includes an image encoder stack, a keypoint encoder stack, and a fusion module. The step of inputting the input vector into the encoding module to obtain the encoding matrix includes: The image position encoding vector in the input vector is input into the image encoder stack to obtain the image encoding matrix; The keypoint location encoding vector in the input vector is input into the keypoint encoder stack to obtain the keypoint encoding matrix; The image encoding matrix and the keypoint encoding matrix are weighted and fused by the fusion module to obtain the encoding matrix.

8. The sign language recognition method as described in claim 5, characterized in that, The prediction head includes a pooling layer, multiple regularization layers, and an output layer. The step of inputting the decoding matrix into the prediction head to obtain the sign language recognition result includes: The decoding matrix is ​​input into the pooling layer to obtain the fusion vector; The fusion vector is input into each of the regularization layers to obtain the regularization vector; All the regularized vectors are input into the output layer to obtain the sign language recognition result.

9. A sign language recognition device, characterized in that, The sign language recognition device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the sign language recognition method as described in any one of claims 1 to 8.

10. A readable storage medium, characterized in that, The readable storage medium is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the sign language recognition method as described in any one of claims 1 to 8.

Images