Sign language recognition method and device and readable storage medium

By calculating the similarity between the image and the preset reference image, the low accuracy problem caused by unconscious interference with the image frame in sign language recognition is solved, and the accuracy of sign language recognition is improved.

CN119007298APending Publication Date: 2024-11-22GEER TECH CO LTD

Patent Information

Application Number
CN202411205689.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing sign language recognition technology has low accuracy in recognition results due to unconscious gestures interfering with image frames, making it difficult to effectively filter out keyframe images containing effective sign language content.

Method used

By calculating the similarity between each frame of image and the preset reference image, the keyframe images with image value greater than the threshold are selected, and input them into the sign language recognition model for recognition, reducing the interference of meaningless images and improving the recognition results Accuracy.

Benefits of technology

Effectively filter out high-quality keyframe images containing effective sign language content, reduce the interference of meaningless images on recognition results, improve the accuracy of sign language recognition results, and reduce the impact of initial and end blank frames.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007298A_ABST
    Figure CN119007298A_ABST
Patent Text Reader

Abstract

The invention discloses a sign language recognition method and device and a readable storage medium, and relates to the technical field of sign language recognition, and the method comprises the steps: obtaining video data to be subjected to sign language recognition, and the video data comprises multiple frames of images; for each frame of image, a first similarity between the image and a preset reference image is calculated, an image value corresponding to the image is determined based on the first similarity, the image value is in positive correlation with the first similarity, and the preset reference image is an image containing effective sign language content; and selecting a key frame image of which the image value is greater than a preset threshold value from the images, and inputting the key frame image into a preset sign language recognition model to obtain a sign language recognition result, thereby improving the accuracy of the sign language recognition result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of sign language recognition, and in particular to a sign language recognition method, device and readable storage medium. Background Art

[0002] Sign language is an important way for the deaf-mute to obtain and express information. The deaf-mute usually communicate with others through sign language, which requires both parties to understand sign language. However, it takes a lot of time and energy for normal people to learn sign language, so most normal people have low sign language skills and find it difficult to communicate with the deaf-mute. Therefore, intelligent sign language recognition is of great significance to the communication of the deaf-mute.

[0003] In traditional sign language recognition schemes, after collecting the sign language video data of the signer, a deep learning model is used to extract the sign language features in the sign language video, and then the results are predicted based on these extracted features. However, since there may be some unconscious gestures interfering with the image frames in the collected sign language video data, such as image frames of lowering the hand, image frames of raising the hand, and blank image frames, the above sign language recognition scheme results in low accuracy of sign language recognition results.

[0004] Therefore, how to improve the accuracy of sign language recognition results is a technical problem that needs to be solved urgently in this technical field. Summary of the invention

[0005] The main purpose of this application is to provide a sign language recognition method, device and readable storage medium, aiming to solve the technical problem of how to improve the accuracy of sign language recognition results.

[0006] To achieve the above objectives, the present application provides a sign language recognition method, which includes:

[0007] Acquire video data to be subjected to sign language recognition, wherein the video data includes multiple frames of images;

[0008] For each frame of image, a first similarity between the image and a preset reference image is calculated, and an image value corresponding to the image is determined based on the first similarity, wherein the image value is positively correlated with the first similarity, and the preset reference image is an image containing valid sign language content;

[0009] A key frame image whose image value is greater than a preset threshold is selected from each of the images, and the key frame image is input into a preset sign language recognition model to obtain a sign language recognition result.

[0010] In one embodiment, the step of determining the image value corresponding to the image based on the first similarity includes:

[0011] Acquire an adjacent frame image corresponding to the image, and calculate a second similarity between the image and the adjacent frame image, wherein the adjacent frame image is a previous frame image or a next frame image;

[0012] A weighted sum of the first similarity and the second similarity is calculated, and the weighted sum is determined as an image value corresponding to the image.

[0013] In one embodiment, the step of calculating a first similarity between the image and a preset reference image includes:

[0014] A Pearson correlation coefficient between the image and a preset reference image is calculated, and the Pearson correlation coefficient between the image and the preset reference image is determined as a first similarity.

[0015] In one embodiment, after the step of obtaining the video data to be subjected to sign language recognition, the method further comprises:

[0016] Acquire the images and arrange them in time order to obtain an image sequence, and determine the middle position of the image sequence;

[0017] An image corresponding to a middle position of the sequence in the image sequence is selected as a preset reference image.

[0018] In one embodiment, the sign language recognition model includes an input module, an encoding module, a decoding module and a prediction head connected in sequence, and the step of inputting the key frame image into a preset sign language recognition model to obtain a sign language recognition result includes:

[0019] The key frame image is embedded and position-encoded by the input module to obtain an input vector;

[0020] Inputting the input vector into the encoding module to obtain an encoding matrix, wherein the encoding module is used to perform encoding processing on the input vector;

[0021] Inputting the encoding matrix into the decoding module to obtain a decoding matrix, wherein the decoding module is used to perform decoding processing on the encoding matrix;

[0022] The decoding matrix is ​​input into the prediction head to obtain a sign language recognition result.

[0023] In one embodiment, the input module includes an image embedding module, a key point embedding module and a position encoding module, and the step of embedding and position encoding the key frame image through the input module to obtain an input vector includes:

[0024] Performing embedding processing on the key frame image by the image embedding module to obtain an image vector;

[0025] Extracting key points in the key frame image by the key point embedding module, and embedding the key points to obtain key point vectors;

[0026] The position encoding module performs position encoding on the image vector and the key point vector to obtain an input vector, wherein the input vector includes an image position encoding vector and a key point position encoding vector.

[0027] In one embodiment, the encoding module includes an image encoder stack, a key point encoder stack and a fusion module, and the step of inputting the input vector into the encoding module to obtain an encoding matrix includes:

[0028] Inputting the image position encoding vector in the input vector into the image encoder stack to obtain an image encoding matrix;

[0029] Inputting the key point position encoding vector in the input vector into the key point encoder stack to obtain a key point encoding matrix;

[0030] The image coding matrix and the key point coding matrix are weightedly fused by the fusion module to obtain a coding matrix.

[0031] In one embodiment, the prediction head includes a pooling layer, a plurality of regularization layers and an output layer, and the step of inputting the decoding matrix into the prediction head to obtain the sign language recognition result includes:

[0032] Inputting the decoding matrix into the pooling layer to obtain a fusion vector;

[0033] Inputting the fused vector into each of the regularization layers respectively to obtain a regularization vector;

[0034] All the regularized vectors are input into the output layer to obtain the sign language recognition result.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a sign language recognition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the sign language recognition method as described above.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a readable storage medium, which is a computer-readable storage medium. A program for implementing the sign language recognition method is stored on the computer-readable storage medium. The program for implementing the sign language recognition method is executed by a processor to implement the steps of the sign language recognition method as described above.

[0037] The present application also provides a computer program product, including a computer program, which implements the steps of the sign language recognition method as described above when the computer program is executed by a processor.

[0038] One or more technical solutions proposed in this application have at least the following technical effects:

[0039] Obtain video data for sign language recognition, wherein the video data includes multiple frames of images; for each frame of image, calculate a first similarity between the image and a preset reference image, and determine an image value corresponding to the image based on the first similarity, wherein the image value is positively correlated with the first similarity, and the preset reference image is an image containing valid sign language content; select a key frame image whose image value is greater than a preset threshold from each image, and input the key frame image into a preset sign language recognition model to obtain a sign language recognition result. Considering that when signing, it is mainly the fingers and palm parts that move. Therefore, when signing, usually only small areas of the fingers and / or palm parts are different, and the similarity between images is high. When not signing, the hands are usually put down, and the image similarity with the signing is low. Based on this, in the embodiment of the present application, based on a preset reference image containing valid sign language content, the similarity between each image and the preset reference image, that is, the first similarity, is calculated, and the image value of each image is evaluated based on the first similarity, and the image value is positively correlated with the first similarity. The image value evaluated in this way can effectively characterize the probability that each image contains valid sign language content, and then key frame images with image values ​​greater than or equal to a preset threshold are screened from all images, and sign language recognition is performed based on the key frame images to obtain a sign language recognition result, that is, high-quality key frame images with a high probability of containing valid sign language content are screened out from the video data, and low-value meaningless images are deleted, thereby reducing the interference of meaningless images on the recognition results, and then improving the accuracy of the sign language recognition results. In addition, compared to the method of calculating the similarity between each frame image and the adjacent frame image, the similarity between each frame image and the preset reference frame image is calculated in the embodiment of the present application, which can reduce the influence of the initial blank frame and the ending blank frame (blank frame refers to a meaningless image frame without actual sign language meaning). For example, when the adjacent frame image is a blank frame, the similarity between the current frame image and the adjacent frame image cannot effectively evaluate the image value of the current frame image, thereby further improving the accuracy of the sign language recognition result. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] Figure 1 This is a flow chart of the first embodiment of the sign language recognition method of the present application;

[0043] Figure 2 This is a schematic diagram of the key frame image screening process of the sign language recognition method of this application;

[0044] Figure 3 This is a schematic diagram of the processing flow of the image embedding module of the sign language recognition method of this application;

[0045] Figure 4 A schematic diagram of key points of a hand involved in an embodiment of a sign language recognition method of the present application;

[0046] Figure 5 This is a schematic diagram of the key point embedding module processing flow of the sign language recognition method of this application;

[0047] Figure 6 This is a schematic diagram of the overall structure of a sign language recognition model involved in an embodiment of a sign language recognition method of the present application;

[0048] Figure 7 Another schematic diagram of the overall structure of the sign language recognition model involved in an embodiment of the sign language recognition method of the present application;

[0049] Figure 8 Schematic diagram of the device structure of the hardware operating environment involved in the sign language recognition device in the embodiment of the present application.

[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0052] Sign language video can be seen as an ordered sequence with a lot of redundant information. At present, the sign language recognition of the deaf-mute mainly includes sign language recognition based on single image information. This method uses a deep learning model to recognize the sign language information of the deaf-mute in the image. Usually, this method is greatly affected by the quality of sign language expression, and it is difficult to recognize sign language that is not particularly standardized. In addition, there is also sensor-based sign language recognition, which will identify the trajectory curve of the hand and match it with a pre-defined template to achieve sign language recognition, but this method usually requires the construction of enough very standard sign language trajectory templates, and usually requires the use of peripherals such as data gloves, which is highly dependent on the hardware environment.

[0053] At present, the key frames in sign language videos can be extracted based on the shot boundary key frame extraction algorithm (Shot Boundary Detection, SBD) to screen key image frames for sign language recognition. The SBD algorithm is an algorithm for determining key frames in videos, and it is also a classic and mature method that is widely used. The basic idea of ​​the algorithm is to determine key frames by detecting changes in visual content. However, there are a large number of blank frames at the beginning and end of sign language videos. In the SBD algorithm, simply comparing the current frame image with the previous frame image will be greatly affected by the initial and ending blank frames, which will cause a waste of resource calculations and inefficient key frame calculations, resulting in low accuracy of sign language recognition results.

[0054] Based on this, the main solution of the present application is: obtain video data to be used for sign language recognition, wherein the video data includes multiple frames of images; for each frame of image, calculate the first similarity between the image and a preset reference image, and determine the image value corresponding to the image based on the first similarity, wherein the image value is positively correlated with the first similarity, and the preset reference image is an image containing valid sign language content; select a key frame image whose image value is greater than a preset threshold from each image, and input the key frame image into a preset sign language recognition model to obtain a sign language recognition result.

[0055] The present application calculates the similarity between each image and the preset reference image, that is, the first similarity, and then evaluates the image value of each image based on the first similarity. The image value is positively correlated with the first similarity. The image value evaluated in this way can effectively represent the probability that each image contains valid sign language content. Then, key frame images with image values ​​greater than or equal to the preset threshold are screened from all images, and sign language recognition is performed based on the key frame images to obtain sign language recognition results. That is, high-quality key frame images with a high probability of containing valid sign language content are screened from the video data, and low-value meaningless images are deleted, thereby reducing the interference of meaningless images on the recognition results, thereby improving the accuracy of the sign language recognition results. In addition, in the present application, the similarity between each frame image and the preset reference frame image is calculated, which can reduce the influence of the initial blank frame and the end blank frame, further improving the accuracy of sign language recognition.

[0056] It should be noted that the execution subject of each embodiment of the sign language recognition method of the present application may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a sign language recognition device capable of realizing the above functions, such as AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, smart glasses, robots, etc. Exemplarily, AR glasses are used as the execution subject to illustrate and describe each embodiment of the sign language recognition method of the present application.

[0057] Based on this, the present application proposes a sign language recognition method of the first embodiment, please refer to Figure 1 The sign language recognition method includes steps S10 to S30:

[0058] Step S10, obtaining video data to be subjected to sign language recognition, wherein the video data includes multiple frames of images;

[0059] The video data is specifically a captured sign language video. It can be understood that a video is composed of multiple frames of images. For example, when a camera captures a video, it captures images at a preset frame rate to obtain video data.

[0060] Furthermore, after acquiring the video data, all images included in the video data may be preprocessed, and subsequent steps may be performed based on the preprocessed images to further improve the image quality. The image preprocessing includes, but is not limited to, deleting images whose clarity is less than a certain preset threshold, deleting images whose illumination intensity is less than a certain preset threshold, deleting the preset frame image at the start position (deleting the starting preset frame image in the video data), deleting the preset frame image at the end position (deleting the last preset frame image in the video data), etc., and this embodiment does not impose specific restrictions on this.

[0061] Step S20, for each frame of image, calculating a first similarity between the image and a preset reference image, and determining an image value corresponding to the image based on the first similarity, wherein the image value is positively correlated with the first similarity, and the preset reference image is an image containing valid sign language content;

[0062] After the video data is acquired, for each frame image in the video data, a first similarity between the frame image (hereinafter referred to as the current frame image) and a preset reference image is calculated, where the first similarity is the similarity between the current frame image and the preset reference image.

[0063] The preset reference image is an image containing valid sign language content. Further, the preset reference image is one or more frames of images containing valid sign language content. If the preset reference image includes multiple frames of images, the first similarity between the image and the preset reference image can be the average similarity, maximum similarity or median similarity between the image and all images in the preset reference image, etc., which is not limited here. In order to reduce the amount of calculation, the preset reference frame is a frame of image. Therefore, the description and explanation of each embodiment of the present application are carried out with the preset reference frame as a frame of image.

[0064] It should be noted that an image containing valid sign language content means that the hand gestures in the image are sign language gestures with actual sign language meanings.

[0065] Regarding the setting of the preset reference image, as one implementation, the relevant personnel may set the preset reference image in advance. As another implementation, an image frame containing valid sign language content may be selected from multiple frames of images included in the video data as the preset reference image.

[0066] Regarding the selection method of selecting a frame of image containing valid sign language content from multiple frames of image included in the video data as the preset reference image, as a first implementation, after the step of obtaining the video data to be subjected to sign language recognition, the method further includes:

[0067] Step S201, acquiring each of the images and arranging them in time order to obtain an image sequence, and determining the middle position of the image sequence;

[0068] Considering that when signing, there is usually a start time and an end time of the sign language action, and the middle time is usually in continuous signing. Based on this, the middle position of the sequence can be any position except the starting preset number of positions and the ending preset number of positions. For example, if the sequence length of the image sequence is 400, the middle position of the sequence can be any position from the 50th element to the 300th element. Among them, the sequence length of the image sequence is the number of image frames included in the image sequence.

[0069] Further, the middle position of the sequence may be the middle position of the image sequence, and when the sequence length of the image sequence is an even number, the middle position may be any one of the two middle positions. For example, when the sequence length of the image sequence is 5, the middle position of the sequence may be the position of the third element, and when the sequence length of the image sequence is 6, the middle position of the sequence may be the position of the third element or the position of the fourth element.

[0070] Step S202: Selecting an image corresponding to a middle position of the sequence in the image sequence as a preset reference image.

[0071] The image corresponding to the middle position of the sequence is selected as the reference image, that is, the image at the middle position of the sequence is selected as the preset reference image, thereby achieving efficient selection of the preset reference image.

[0072] As a second implementation method, considering that during the process of signing, the arm and body of the signer will form a certain angle, (x1, y1) is the position coordinate of the elbow joint of the arm, (x2, y2) is the position coordinate of the wrist, and the angle formed by these two points and the angle s in the opposite direction of gravity acceleration can represent the switch status of the sign language action. When signing, s is usually less than a certain value (such as 150, 160, etc.), and when not signing, s is usually greater than or equal to the certain value. Based on this, the s value of the image can be calculated, and a frame of images with an s value less than a preset threshold is selected from each image as the preset reference frame image.

[0073] As a third implementation, it is possible to determine whether the gesture in the image has actual sign language meaning based on a preset gesture detection algorithm, such as a support vector machine, a random forest or a deep learning model. If the gesture in a frame of the image is detected to have actual sign language meaning based on the preset gesture recognition algorithm, the frame of the image is selected as the preset reference image. It should be noted that, considering that the duration of general sign language videos is relatively long, including a large number of images, and the current gesture detection algorithm is highly complex and consumes a lot of computing resources, if all images are input into the preset gesture detection algorithm, and the high-value key frame images are screened by the preset gesture detection algorithm, the screening of the key frame images will require a lot of computing resources and the screening time will also be long. Based on this, the images can be input into the preset gesture detection algorithm in sequence according to the time sequence until a frame of image with actual sign language meaning is detected and the detection is terminated.

[0074] It should be noted that the above are only four feasible implementation methods for setting the preset reference image provided by this embodiment, and this embodiment does not specifically limit the specific implementation method of setting the preset reference image.

[0075] It should be noted that, unless otherwise specified, the embodiments of the present application only describe images in the acquired video data.

[0076] Step S30, selecting a key frame image whose image value is greater than a preset threshold from each of the images, and inputting the key frame image into a preset sign language recognition model to obtain a sign language recognition result.

[0077] A key frame image having an image value greater than a preset threshold is selected from each image, and the key frame image may specifically be an image in each image having an image value greater than the preset threshold. The preset threshold is an arbitrary value set in advance, and this embodiment does not specifically limit this.

[0078] After the key frame images are screened out, the key frame images are input into a preset sign language recognition model to obtain a sign language recognition result, which may be the final output of the sign language recognition model. By performing sign language recognition through the sign language recognition model, compared with the sensor-based sign language recognition method, there is no need to rely on additional sensor devices such as sign language gloves, nor is there a need to build a large number of sign language trajectory templates in advance, thereby reducing the implementation cost of sign language recognition.

[0079] In this embodiment, video data to be used for sign language recognition is obtained, wherein the video data includes multiple frames of images; for each frame of image, a first similarity between the image and a preset reference image is calculated, and an image value corresponding to the image is determined based on the first similarity, wherein the image value is positively correlated with the first similarity, and the preset reference image is an image containing valid sign language content; a key frame image whose image value is greater than a preset threshold is selected from each image, and the key frame image is input into a preset sign language recognition model to obtain a sign language recognition result. Considering that when signing, it is mainly the fingers and palms that move. Therefore, when signing, usually only small areas of the fingers and / or palms are different, and the similarity between images is high. When the relevant person is not signing, the hands are usually put down, and the similarity between the images when signing is low. Based on this, in this embodiment, based on a preset reference image containing valid sign language content, the similarity between each image and the preset reference image, that is, the first similarity, is calculated, and the image value of each image is evaluated based on the first similarity, and the image value is positively correlated with the first similarity. The image value evaluated in this way can effectively represent the probability that each image contains valid sign language content, and then key frame images with image values ​​greater than or equal to a preset threshold are screened from all images, and sign language recognition is performed based on the key frame images to obtain a sign language recognition result, that is, high-quality key frame images with a high probability of containing valid sign language content are screened out from the video data, and low-value meaningless images are deleted, thereby reducing the interference of meaningless images on the recognition results, and then improving the accuracy of the sign language recognition results. In addition, compared to the method of calculating the similarity between each frame image and the adjacent frame image, the similarity between each frame image and the preset reference frame image is calculated in this embodiment, which can reduce the influence of the initial blank frame and the ending blank frame. For example, when the adjacent frames are blank frames, the similarity between the current frame image and the adjacent frame image cannot effectively evaluate the image value of the current frame image, thereby further improving the accuracy of the sign language recognition results.

[0080] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description and will not be described in detail later. On this basis, the step of determining the image value corresponding to the image based on the first similarity includes:

[0081] Step A10, obtaining an adjacent frame image corresponding to the image, and calculating a second similarity between the image and the adjacent frame image, wherein the adjacent frame image is a previous frame image or a next frame image;

[0082] It should be noted that the second similarity between the current frame image and the adjacent frame image is calculated, and the second similarity is the similarity between the current frame image and the adjacent frame image.

[0083] The adjacent frame image may specifically be a previous frame image or a next frame image, and this embodiment does not impose any specific limitation on this.

[0084] Step A20, calculating a weighted sum of the first similarity and the second similarity, and determining the weighted sum as an image value corresponding to the image.

[0085] The weighted sum of the first similarity and the second similarity is calculated, and the weighted sum is determined as the image value corresponding to the current frame image. The weighted sum can be a calculation result obtained by performing a weighted sum calculation on the first similarity and the second similarity.

[0086] The weights corresponding to the first similarity and the second similarity may be set in advance, and a weighted sum value may be obtained by performing a weighted sum calculation based on the weights corresponding to the first similarity and the second similarity.

[0087] Further, the weight corresponding to the first similarity is recorded as the first weight, and the weight corresponding to the second similarity is recorded as the second weight, and the sum of the first weight and the second weight may be 1. The first weight and the second weight may be the same or different, and this embodiment does not make any specific limitation on this.

[0088] In a possible implementation, the step of calculating a first similarity between the image and a preset reference image includes:

[0089] Step B10, calculating the Pearson correlation coefficient between the image and a preset reference image, and determining the Pearson correlation coefficient between the image and the preset reference image as a first similarity.

[0090] The Pearson Correlation Coefficient (PCC), also known as the Pearson product-moment correlation coefficient, is a statistical indicator that measures the degree of linear correlation between two variables.

[0091] Specifically, the Pearson correlation coefficient can be expressed as: Where PCC(P, Q) represents the Pearson correlation coefficient between images P and Q, N represents the number of pixels in the horizontal direction of the image, M represents the number of pixels in the vertical direction of the image, P(i, j) represents the pixel value of the pixel in the i-th row and j-th column of image P, Q(i, j) represents the pixel value of the pixel in the i-th row and j-th column of image P, μ p Represents the average pixel value of all pixels in image P, μ q represents the average pixel value of all pixels in image Q, σ p Represents the mean square error of the pixel values ​​of all pixels in image P, σ q Represents the mean square error of the pixel values ​​of all pixels in image Q.

[0092] Further, refer to Figure 2 As shown, considering that the pixel value of each frame image can be decomposed into three channels of R, G, and B, based on this, the above PCC calculation formula can be used to calculate the PCC value of each channel between two frames of images, and the PCC value between the R channel of the two frames of images is PCC R , the PCC value between the G channels of the two frames is PCC G , the PCC value between the B channels of the two frames is PCC B , then the PCC value between two frames of images can be specifically PCC R 、PCC G With PCC B The average value of .

[0093] For example, to help understand the technical concept or technical principle of the sign language recognition method after the present embodiment is combined with the first embodiment, please refer to Figure 2 As shown, the specific sign language recognition process is as follows:

[0094] 1. Obtain the video data to be used for sign language recognition. The video data includes t frames of images, namely [F1, F2, F3...Fi...Ft], F i It represents the i-th frame image, and the middle frame of the video data is used as the preset reference image FF.

[0095] 2. Calculate the i-th frame image F i The PCC value between the preset reference image FF is recorded as PCC1, and the next frame image F is selected. i+1 As the adjacent frame image of the i-th frame image, calculate the next frame F i+1 With F i The PCC value between is recorded as PCC2. It should be noted that, for the last frame image, the previous frame image can be selected as the adjacent frame image of the image.

[0096] 3. Take the weighted sum of PCC1 and PCC2 to get PCC i , specifically, it can be expressed as: PCC i =αPCC(i,FF)+(1-α)PCC(i,F i+1 ), where PCC(i,FF) represents F i The PCC value between FF, that is, PCC1, PCC(i,F i+1 ) indicates F i With F i+1 The PCC value between , that is, PCC2, α represents the first weight.

[0097] 4. Determine the image value of the i-th frame as PCC i , if PCC iIf it is greater than the preset threshold τ, the frame image F is selected i is the key frame image, otherwise determine F i is a non-key frame image, and finally a complete key frame sequence is extracted, which is a sequence composed of all key frame images.

[0098] 5. Input the key frame sequence into a preset sign language recognition model and output the sign language recognition result.

[0099] It should be noted that the above examples are only used to assist in understanding the present application and do not constitute a limitation on the sign language recognition method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0100] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first and second embodiments can be referred to the above description, and will not be described in detail later. On this basis, the sign language recognition model includes an input module, an encoding module, a decoding module and a prediction head connected in sequence, and the step of inputting the key frame image into the preset sign language recognition model to obtain the sign language recognition result includes:

[0101] Step C10, embedding and position encoding the key frame image through the input module to obtain an input vector;

[0102] Considering that the Transformer model (a deep learning model based on the self-attention mechanism) has achieved significant breakthroughs in processing long sequence tasks compared to traditional recurrent neural networks, it introduces a self-attention mechanism, which enables the model to capture global dependencies and disperse information into multiple subspaces through multi-head attention technology, thereby significantly enhancing the model's expression and understanding capabilities. The architectural advantage of the Transformer model lies in its equal treatment of all elements in the sequence, which enables the model to recognize and understand the rich semantic relationships between the elements in the sequence, especially in understanding the relationship between distant sequence elements. Based on this, in this embodiment, the sign language recognition model is specifically a Transformer model.

[0103] Specifically, the Transformer model includes an input module, an encoding module, a decoding module and a prediction head connected in sequence. The input module is used to embed and positionally encode key frame images.

[0104] Embedding refers to the process of converting input data (such as words, characters, sentences, or any serializable entity) into a fixed-length vector representation. Embedding processing is the basis for the model to understand the input data.

[0105] Position coding refers to adding a piece of information about the position of each word in the sentence, also called "information fragment". Specifically, absolute position coding, relative position coding, etc. can be used, and this embodiment does not impose specific restrictions on this.

[0106] Step C20, inputting the input vector into the encoding module to obtain an encoding matrix, wherein the encoding module is used to perform encoding processing on the input vector;

[0107] The input vector output by the key frame image after passing through the input module is transmitted to the encoding module, and the input vector is encoded by the encoding module, and the input vector is converted into a series of feature vectors, that is, the encoding matrix, and the encoding matrix is ​​input into the decoding module.

[0108] Relevant personnel can design the encoding module according to actual needs, and the encoding matrix is ​​the output of the encoding module. In a specific implementation, the encoding module can be an encoder stack composed of one or more encoders connected in sequence, and the encoding matrix is ​​the output of the encoder stack, that is, the output of the last layer of encoders in the encoder stack.

[0109] Step C30, inputting the encoding matrix into the decoding module to obtain a decoding matrix, wherein the decoding module is used to perform decoding processing on the encoding matrix;

[0110] The decoding module receives the encoding matrix input by the encoding module, and decodes the encoding matrix to obtain an output sequence, namely, a decoding matrix, and inputs the decoding matrix into the prediction head.

[0111] Relevant personnel can design a decoding module according to actual needs, and the decoding matrix is ​​the output of the decoding module. In a specific implementation, the decoding module can be a decoder stack composed of one or more decoders connected in sequence, and the decoding matrix is ​​the output of the decoder stack, that is, the output of the last layer of decoders in the decoder stack.

[0112] Step C40, inputting the decoding matrix into the prediction head to obtain a sign language recognition result.

[0113] It should be noted that the sign language recognition result is the output of the prediction head.

[0114] The prediction head in the Transformer model is used to convert the internal representation of the model into the final prediction output. In this embodiment, the prediction head is used to convert the decoding matrix into the final prediction output, that is, the sign language recognition result.

[0115] Relevant personnel can design a prediction head according to actual needs. In a specific implementation, the prediction head includes a pooling layer, a regularization layer (dropout), a fully connected layer and a softmax function connected in sequence, and finally predicts the output through the softmax function.

[0116] In a possible implementation, the input module includes an image embedding module, a key point embedding module and a position encoding module, and the step of embedding and position encoding the key frame image through the input module to obtain an input vector includes:

[0117] Step D10, embedding the key frame image through the image embedding module to obtain an image vector;

[0118] The image embedding module is used to embed the key frame image to obtain an image vector. Figure 3 As shown, the embedding process is as follows:

[0119] (1) First, scale the image to a uniform size, such as 224×224×3, and then divide the image into small patches. Each patch size is also the same, such as 16×16×3, and then arrange the small patches;

[0120] (2) Perform an Embedding operation on the arranged patches to obtain an image vector.

[0121] Step D20, extracting key points in the key frame image by the key point embedding module, and embedding the key points to obtain key point vectors;

[0122] The key point embedding module is used to extract key points (actually extract coordinates) in the key frame image. Specifically, the key points of the face and hands in the key frame image can be extracted. The key points of the hand are referred to as hand key points, and the key points of the face are referred to as facial key points. Specifically, a preset number of hand key points and a preset number of facial key points can be extracted. The preset number of facial key points may include eye key points (key points of the glasses), nose key points (key points of the nose), mouth key points (key points of the mouth), contour key points (key points of the facial contour), etc. In a specific embodiment, refer to Figure 4 As shown, based on the key frame image, 21 hand key points are extracted for one hand, and a total of 42 hand key points are extracted for both hands. At the same time, 58 facial key points are extracted. The 58 facial key points specifically include 17 contour key points, 12 eye key points, 9 nose key points and 20 mouth key points.

[0123] It can be understood that a key point can be represented by coordinates, so that each key point is actually represented as a coordinate.

[0124] For example, refer to Figure 5 As shown, key point extraction is performed on the key frame image, specifically hand key points and facial key points are extracted, and the extracted hand key points and facial key points are respectively embedded to obtain key point vectors, that is, the key point coordinates are embedded, and then the key point vectors are input into the position encoding module for position encoding processing.

[0125] Step D30, position encoding the image vector and the key point vector by the position encoding module to obtain an input vector, wherein the input vector includes an image position encoding vector and a key point position encoding vector.

[0126] The image vector and the key point vector are position-encoded respectively by the position coding module to obtain an input vector, wherein the input vector includes an image position coding vector and a key point position coding vector. Specifically, the image position coding vector is a vector obtained by position coding the image vector, and the key point position coding vector is a vector obtained by position coding the key point vector.

[0127] In this embodiment, in order to make the model pay more attention to the hand and facial information of the signer in the process of expressing sentences, a key point embedding module is set to extract the key points of the hands and face in the image, thereby improving the performance of the model. At the same time, in order to avoid the influence of noise on the key points, an image embedding module is set to embed the key frame image, thereby fusing the two modal data of image and key point (actually text data in coordinate form) for sign language recognition, utilizing the complementary advantages of the two modal data to jointly learn features, thereby further improving the accuracy of the sign language recognition results.

[0128] For example, to help understand the technical concept or technical principle of the sign language recognition model in this embodiment, please refer to Figure 6 As shown, the sign language recognition model specifically includes an input module, an encoding module, a decoding module and a prediction head connected in sequence, wherein the encoding module includes n layers of encoders (Encoder) connected in sequence, the decoding module includes m layers of decoders (Decoder) connected in sequence, and the input module includes an image embedding (Embedding) module and a key point embedding module, and the image embedding module and the key point embedding module are respectively connected to the position encoding module.

[0129] Based on this sign language recognition model, after extracting the key frame images in the video data, that is, Figure 6The key frame extraction (KEF) shown in FIG. 1 inputs the extracted key frame image into the input module of the sign language recognition model. The image embedding module and the key point embedding module in the input module embed the image and the key point respectively, and then splices the outputs of the two embeddings. After passing through position encoding, n layers of sequentially connected encoders, m layers of sequentially connected decoders and prediction heads, the final output of the model, i.e., the sign language recognition result, is obtained. It should be noted that the final output of the n layers of sequentially connected encoders is respectively input into each decoder of the m layers of sequentially connected decoders.

[0130] It should be noted that if the sign language model is currently being trained, the training result labels (such as Figure 6 The "business license" shown in the figure, which is the expected output of the model, is embedded and positionally encoded in sequence and then input into the decoding module. At the same time, the key frame image corresponding to the result label is input into the input module. The sign language recognition model is trained based on the input of the result label and the input of the image data to obtain a trained sign language recognition model.

[0131] It should be noted that the above examples are only used to assist in understanding the present application and do not constitute a limitation on the sign language recognition model structure of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0132] Based on the first embodiment, the second embodiment and / or the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the above-mentioned embodiments 1, 2 and 3 can be referred to the above introduction, and will not be repeated later. On this basis, the encoding module includes an image encoder stack, a key point encoder stack and a fusion module, and the step of inputting the input vector into the encoding module to obtain the encoding matrix includes:

[0133] Step E10, inputting the image position encoding vector in the input vector into the image encoder stack to obtain an image encoding matrix;

[0134] It should be noted that the image encoder stack is composed of one or more encoders connected in sequence. The image encoding matrix is ​​the output of the image encoder stack, that is, the output of the last layer of encoders in the image encoder stack.

[0135] Step E20, inputting the key point position encoding vector in the input vector into the key point encoder stack to obtain a key point encoding matrix;

[0136] It should be noted that the key point encoder stack is composed of one or more encoders connected in sequence, and the key point encoding matrix is ​​the output of the key point encoder stack, that is, the output of the last layer of encoder in the key point encoder stack.

[0137] The number of layers of encoders in the key point encoder stack may be the same as or different from the number of layers of encoders in the image encoder stack, and this embodiment does not impose any specific restrictions on this. In a preferred embodiment, the number of layers of encoders in the image encoder is the same as the number of layers of encoders in the key point encoder stack.

[0138] Step E30: Performing weighted fusion on the image coding matrix and the key point coding matrix through the fusion module to obtain a coding matrix.

[0139] The fusion module is connected to the image encoder stack and the key point encoder stack respectively to fuse the output of the image encoder stack with the output of the key point encoder stack.

[0140] Furthermore, the image coding matrix and the key point coding matrix may be weightedly fused based on the fusion weights corresponding to the image coding matrix and the key point coding matrix to obtain the coding matrix. For setting the fusion weights, the relevant personnel may set the fusion weights corresponding to the image coding matrix and the key point coding matrix respectively, or may set the fusion weights corresponding to the image coding matrix and the key point coding matrix as model parameters, and optimize the fusion weights corresponding to the image coding matrix and the key point coding matrix during the sign language recognition model training process, and this embodiment does not impose specific restrictions on this.

[0141] Let the fusion weight corresponding to the image coding matrix be p, the image coding matrix be IA, the fusion weight corresponding to the key point coding matrix be q, the key point coding matrix be KA, and the coding matrix be EA, then EA can be expressed by the formula EA=p*IA+q*KA. In a specific implementation, the sum of p and q is one.

[0142] In this embodiment, an image encoder stack and a key point encoder stack are set to encode the image coding vector and the key point coding vector respectively, which is equivalent to using the encoder stack to encode the image data and the text data respectively. As a result, the characteristics of each data type can be better utilized to improve model performance and reduce computing costs. It is also easier to understand and explain the behavior of each encoder stack to enhance the interpretability and customizability of the model, and the image and text encoders can be tuned separately, which helps to stabilize the training process.

[0143] For example, to help understand the technical concept or technical principle of the sign language recognition model after the present embodiment is combined with the third embodiment, please refer to Figure 7As shown, the sign language recognition model specifically includes an input module, an encoding module, a decoding module and a prediction head connected in sequence, wherein the encoding module includes an image encoder stack, a key point encoder stack and a fusion module, the image encoder stack includes n layers of encoders (Encoder VD) connected in sequence, the key point encoder stack includes n layers of encoders (Encoder KP) connected in sequence, the decoding module includes m layers of decoders (Decoder) connected in sequence, the input module includes an image embedding (Embedding) module and a key point embedding module, and the image embedding module and the key point embedding module are respectively connected to the position encoding module.

[0144] The fusion module performs weighted fusion on the outputs of the image encoder stack and the key point encoder stack, and inputs the fusion results into each decoder of the decoder module. Specifically, the fusion weight corresponding to the image encoding matrix output by the image encoder stack is α, and the fusion weight corresponding to the key point encoding matrix output by the key point encoder stack is 1-α. The image encoding matrix and the key point encoding matrix are weightedly fused based on their respective fusion weights.

[0145] The processing mechanisms of other modules can refer to the above Figure 6 The description in will not be repeated here.

[0146] It should be noted that the above examples are only used to assist in understanding the present application and do not constitute a limitation on the sign language recognition model structure of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0147] Based on the first embodiment, the second embodiment, the third embodiment and / or the fourth embodiment of the present application, in the fifth embodiment of the present application, the same or similar contents as those in the first embodiment, the second embodiment, the third embodiment and the fourth embodiment can be referred to the above introduction, and will not be repeated later. On this basis, the prediction head includes a pooling layer, multiple regularization layers and an output layer, and the step of inputting the decoding matrix into the prediction head to obtain the sign language recognition result includes:

[0148] Step F10, inputting the decoding matrix into the pooling layer to obtain a fusion vector;

[0149] It should be noted that the fused vector is the output of the pooling layer.

[0150] The vector sequences in the feature matrix are fused through the pooling layer to aggregate the vectors into a fused vector of fixed dimension and fixed length.

[0151] The pooling layer aggregates the output sequence. After the sequence in the Transformer model is processed by the multi-layer self-attention mechanism and feedforward neural network in the encoder and decoder, a series of hidden states, that is, the representation vector of each position, are obtained. These representation vectors contain the semantic information of the input sequence. In order to aggregate these vectors into a vector representation of fixed dimension and fixed length, pooling operation is required.

[0152] The pooling layer may be an average pooling layer, a maximum pooling layer, a global pooling layer, etc., and this embodiment does not impose any specific restrictions on this. Considering that compared with other pooling operations, the average pooling operation performs an average operation on the sequence, so the overall characteristics of the sequence can be retained, not only the maximum and minimum characteristics in the sequence are considered, but also the overall trend and distribution are considered. Average pooling can also enhance robustness and reduce overfitting. Based on this, in a preferred embodiment, the pooling layer is specifically an average pooling layer.

[0153] Step F20, inputting the fused vector into each of the regularization layers to obtain a regularized vector;

[0154] It should be noted that the regularization vector is the output of the regularization layer.

[0155] Regularization (Dropout) is a technique that randomly "inactivates" (temporarily ignores) some neurons in a neural network during training. The inactivated neurons do not participate in the calculation during the current forward propagation and backpropagation processes. By randomly "inactivating" some neurons in the network during training, the network's dependence on specific training samples is reduced, thereby improving the model's generalization ability on unseen data.

[0156] The fusion vector is randomly inactivated through multiple regularization layers (Dropout) to obtain a regularized vector, that is, the fusion vector is input into each regularization layer respectively.

[0157] It should be noted that the discard ratio of each regularization layer may be the same or different, and this embodiment does not impose specific restrictions on this. Considering that different discard ratios can provide different regularization strengths, thereby improving the generalization ability of the model, and applying different discard ratios can increase the diversity of the model, so that the model can be trained under different configurations. This diversity helps the model to have better adaptability to unknown data when used, and different discard ratios can be customized to more flexibly control the training process of the model, which helps to find the best training strategy to balance the accuracy and generalization ability of the model. Based on this, in a preferred embodiment, the discard ratio of each regularization layer is different.

[0158] Among them, the discard ratio of the regularization layer refers to the proportion of neurons discarded by the regularization layer.

[0159] Step F30, inputting all the regularized vectors into the output layer to obtain the sign language recognition result.

[0160] It should be noted that the sign language recognition result is the output of the output layer.

[0161] Relevant personnel can design the output layer according to actual needs. In a specific implementation, the output layer includes a fully connected layer and a softmax function connected in sequence.

[0162] In a possible implementation, the output layer includes a plurality of fully connected layers and prediction layers, the fully connected layers are connected to the regularization layers in a one-to-one correspondence, and the step of converting all the regularization vectors into target probability distribution vectors through the output layer and outputting the gesture recognition result based on the target probability distribution vector includes:

[0163] Step F301, inputting each of the regularized vectors into the correspondingly connected fully connected layers to obtain a linear vector;

[0164] It should be noted that the linear vector is the output of the full connection.

[0165] The number of fully connected layers is the same as that of regularized layers, and they are connected one-to-one. The fully connected layer performs feature integration processing on the regularized vector input therein to obtain a linear vector, and outputs the linear vector. Exemplarily, assuming that the i-th fully connected layer is connected to the i-th regularized layer, and the regularized vector output by the i-th regularized layer is Di, then Di is input to the i-th fully connected layer, and the i-th fully connected layer performs relevant processing on the input Di and outputs a linear vector Li. Among them, the value of i ranges from i to N, and N is the total number of regularized layers or fully connected layers.

[0166] Step F302 , converting each of the linear vectors into a probability distribution vector through the prediction layer, fusing each of the probability distribution vectors to obtain a target probability distribution vector, and outputting a gesture recognition result based on the target probability distribution vector.

[0167] It should be noted that the prediction layer includes a softmax function. After the linear vector is input into the softmax function, the softmax function will convert it into a probability distribution vector. After obtaining the probability distribution vectors corresponding to all linear vectors, all probability distribution vectors can be weighted and fused to obtain the target probability distribution vector. For example, the average value vector of all probability distribution vectors can be calculated (the average value of the element values ​​at the same element position in all probability distribution vectors is taken to obtain the vector), and the average value vector is determined to be the target probability distribution vector. After obtaining the target probability distribution vector, the final result can be output based on the target probability distribution vector, that is, the gesture recognition result.

[0168] In this embodiment, multiple regularization layers are set in the prediction head, and multiple different sub-network models are generated by using the multiple regularization layers, thereby improving the generalization ability of the sign language recognition algorithm model, and further improving the stability of the sign language recognition results.

[0169] In addition, an embodiment of the present application also proposes a sign language recognition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the sign language recognition method as described above.

[0170] refer to Figure 8 , which shows a schematic diagram of the structure of a sign language recognition device suitable for implementing the embodiment of the present application. The sign language recognition device in the embodiment of the present application may also include but is not limited to mobile terminals such as AR glasses, robots, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The sign language recognition device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0171] like Figure 8As shown, the sign language recognition device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the sign language recognition device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the sign language recognition device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a sign language recognition device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0172] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0173] The sign language recognition device provided in the embodiment of the present application adopts the sign language recognition method in the above embodiment, which can solve the technical problem of how to improve the accuracy of the sign language recognition result. Compared with the prior art, the beneficial effects of the sign language recognition device provided in the present application are the same as the beneficial effects of the sign language recognition method provided in the above embodiment, and the other technical features in the sign language recognition device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0174] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0175] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0176] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also provides a readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, and the computer-readable program instructions are used to execute the sign language recognition method in the above-mentioned embodiment.

[0177] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0178] The computer-readable storage medium may be included in the sign language recognition device; or may exist independently without being assembled into the sign language recognition device.

[0179] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the sign language recognition device, the sign language recognition device: obtains video data to be subjected to sign language recognition, wherein the video data includes multiple frames of images; for each frame of image, calculates a first similarity between the image and a preset reference image, and determines an image value corresponding to the image based on the first similarity, wherein the image value is positively correlated with the first similarity, and the preset reference image is an image containing valid sign language content; selects a key frame image whose image value is greater than a preset threshold from each of the images, and inputs the key frame image into a preset sign language recognition model to obtain a sign language recognition result.

[0180] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0181] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0182] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0183] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned sign language recognition method, and can solve the technical problems of sign language recognition. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the sign language recognition method provided in the above-mentioned embodiment, and will not be elaborated here.

[0184] In addition, an embodiment of the present application also proposes a computer program product, including a sign language recognition program, which implements the steps of the above sign language recognition method when executed by a processor.

[0185] The specific implementation methods of the computer program product of the present application are basically the same as the above-mentioned embodiments of the sign language recognition method, and will not be repeated here.

[0186] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0187] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0188] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software sensor, which is stored in a storage medium (such as ROM / RAM, disk, CD) as above, including a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of each embodiment of the present application.

[0189] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A sign language recognition method, It is characterized in that The sign language recognition method comprises the following steps: Acquire video data to be subjected to sign language recognition, wherein the video data includes multiple frames of images; For each frame of image, a first similarity between the image and a preset reference image is calculated, and an image value corresponding to the image is determined based on the first similarity, wherein the image value is positively correlated with the first similarity, and the preset reference image is an image containing valid sign language content; A key frame image whose image value is greater than a preset threshold is selected from each of the images, and the key frame image is input into a preset sign language recognition model to obtain a sign language recognition result.

2. The sign language recognition method according to claim 1, It is characterized in that The step of determining the image value corresponding to the image based on the first similarity comprises: Acquire an adjacent frame image corresponding to the image, and calculate a second similarity between the image and the adjacent frame image, wherein the adjacent frame image is a previous frame image or a next frame image; A weighted sum of the first similarity and the second similarity is calculated, and the weighted sum is determined as an image value corresponding to the image.

3. The sign language recognition method according to claim 1, It is characterized in that The step of calculating a first similarity between the image and a preset reference image comprises: A Pearson correlation coefficient between the image and a preset reference image is calculated, and the Pearson correlation coefficient between the image and the preset reference image is determined as a first similarity.

4. The sign language recognition method according to claim 1, It is characterized in that After the step of acquiring the video data to be subjected to sign language recognition, the method further comprises: Acquire the images and arrange them in time order to obtain an image sequence, and determine the middle position of the image sequence; An image corresponding to a middle position of the sequence in the image sequence is selected as a preset reference image.

5. The sign language recognition method according to any one of claims 1 to 4, It is characterized in that The sign language recognition model includes an input module, an encoding module, a decoding module and a prediction head connected in sequence, and the step of inputting the key frame image into a preset sign language recognition model to obtain a sign language recognition result includes: The key frame image is embedded and position-encoded by the input module to obtain an input vector; Inputting the input vector into the encoding module to obtain an encoding matrix, wherein the encoding module is used to perform encoding processing on the input vector; Inputting the encoding matrix into the decoding module to obtain a decoding matrix, wherein the decoding module is used to perform decoding processing on the encoding matrix; The decoding matrix is ​​input into the prediction head to obtain a sign language recognition result.

6. The sign language recognition method according to claim 5, It is characterized in that The input module includes an image embedding module, a key point embedding module and a position encoding module. The step of embedding and position encoding the key frame image through the input module to obtain an input vector includes: Performing embedding processing on the key frame image by the image embedding module to obtain an image vector; Extracting key points in the key frame image by the key point embedding module, and embedding the key points to obtain key point vectors; The position encoding module performs position encoding on the image vector and the key point vector to obtain an input vector, wherein the input vector includes an image position encoding vector and a key point position encoding vector.

7. The sign language recognition method according to claim 6, It is characterized in that The encoding module includes an image encoder stack, a key point encoder stack and a fusion module, and the step of inputting the input vector into the encoding module to obtain an encoding matrix includes: Inputting the image position encoding vector in the input vector into the image encoder stack to obtain an image encoding matrix; Inputting the key point position encoding vector in the input vector into the key point encoder stack to obtain a key point encoding matrix; The image coding matrix and the key point coding matrix are weightedly fused by the fusion module to obtain a coding matrix.

8. The sign language recognition method according to claim 5, It is characterized in that The prediction head includes a pooling layer, a plurality of regularization layers and an output layer. The step of inputting the decoding matrix into the prediction head to obtain a sign language recognition result includes: Inputting the decoding matrix into the pooling layer to obtain a fusion vector; Inputting the fused vector into each of the regularization layers respectively to obtain a regularization vector; All the regularized vectors are input into the output layer to obtain the sign language recognition result.

9. A sign language recognition device, It is characterized in that The sign language recognition device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the sign language recognition method according to any one of claims 1 to 8.

10. A readable storage medium, It is characterized in that The readable storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the sign language recognition method according to any one of claims 1 to 8 are implemented.

Citation Information

Patent Citations

  • Sign language video key frame sampling method based on DTW distance

    CN113312959A

  • Image processing method, an image processing device, terminal and nonvolatile computer readable storage medium

    CN113676659A

Cited By

  • Sign language recognition method and device, and readable storage medium

    WO2026045363A1