Method for recognizing hand language actions and method for training discrete coding model

Abstract

This application discloses a method for recognizing sign language gestures and a training method for a discretized coding model. The method includes: acquiring a set of image frames during the process of a biological object outputting a sign language gesture to be recognized; discretizing and encoding the image frames in the image frame set to obtain discrete features of the sign language gesture to be recognized, wherein the discrete features are used to characterize the features of the vocabulary represented by the sign language gesture; classifying the discrete features of the sign language gesture to be recognized to obtain a recognition result, wherein the recognition result is used to characterize the category of the vocabulary to be recognized. This application solves the technical problem of low accuracy in recognizing sign language gestures in related technologies and improves the interactive experience.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to a method for recognizing sign language gestures and a method for training a discretized coding model. Background Technology

[0002] Currently, sign language translation can help hearing-impaired people communicate better with others. Sign language translation can convert sign language into natural language. The first step in sign language translation is sign language recognition, which aims to identify the corresponding words for each sign language in the input video. However, the recognition methods currently used have low accuracy in recognizing sign language actions in videos.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method for recognizing sign language gestures and a method for training a discretized coding model, so as to at least solve the technical problem of low recognition accuracy of sign language gestures in related technologies.

[0005] According to one aspect of the embodiments of this application, a method for recognizing sign language actions is provided, comprising: acquiring a set of image frames during the process of a biological object outputting a sign language action to be recognized; discretizing and encoding the image frames in the set of image frames to obtain discrete features of the sign language action to be recognized, wherein the discrete features are used to characterize the features of the vocabulary to be recognized represented by the sign language action; classifying the discrete features of the sign language action to be recognized to obtain a recognition result of the sign language action to be recognized, wherein the recognition result is used to characterize the category of the vocabulary to be recognized.

[0006] According to one aspect of the embodiments of this application, a training method for a discretized coding model is provided, comprising: acquiring a training image frame set, wherein the training image frame set contains training sign language actions output by the same biological object; discretizing the training image frames in the training image frame set using a discretized coding model to obtain discrete features of the training sign language actions; reconstructing images from the discrete features of the training sign language actions using a decoder model to obtain a reconstructed image frame set; and adjusting the model parameters of the discretized coding model and the decoder model based on the training image frame set and the reconstructed image frame set.

[0007] According to one aspect of the embodiments of this application, a method for recognizing sign language actions is provided, comprising: responding to an input command applied to an operating interface, displaying a set of image frames on the operating interface, wherein the set of image frames is acquired during the process of a biological object outputting a sign language action to be recognized; responding to a recognition command applied to the operating interface, displaying a recognition result of the sign language action to be recognized on the operating interface, wherein the recognition result is used to characterize the category of the word to be recognized, the recognition result is obtained by classifying the discrete features of the sign language action to be recognized, the discrete features of the sign language action to be recognized are obtained by discretizing and encoding the image frames in the set of image frames, and the discrete features are used to characterize the features of the word to be recognized represented by the sign language action to be recognized.

[0008] According to one aspect of the embodiments of this application, a method for recognizing sign language gestures is provided, comprising: displaying a set of image frames on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the set of image frames is acquired during the process of a biological object outputting a sign language gesture to be recognized; discretizing and encoding the image frames in the set of image frames to obtain discrete features of the sign language gesture to be recognized, wherein the discrete features are used to characterize the features of the vocabulary to be recognized represented by the sign language gesture; classifying the discrete features of the sign language gesture to be recognized to obtain a recognition result of the sign language gesture to be recognized, wherein the recognition result is used to characterize the category of the vocabulary to be recognized; and driving the VR device or AR device to render and display the recognition result of the sign language gesture to be recognized.

[0009] According to one aspect of the embodiments of this application, a method for recognizing sign language actions is provided, comprising: acquiring an image frame set by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter being the image frame set, the image frame set being acquired during the process of a biological object outputting a sign language action to be recognized; discretizing and encoding the image frames in the image frame set to obtain discrete features of the sign language action to be recognized, wherein the discrete features are used to characterize the features of the vocabulary to be recognized represented by the sign language action; classifying the discrete features of the sign language action to be recognized to obtain a recognition result of the sign language action to be recognized, wherein the recognition result is used to characterize the category of the vocabulary to be recognized; and outputting the recognition result of the sign language action to be recognized by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter being the recognition result of the sign language action to be recognized.

[0010] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform any of the methods described above.

[0011] According to one aspect of the embodiments of this application, an electronic device includes: a data acquisition device for acquiring a set of image frames during the process of a biological object outputting a sign language action to be recognized; and a processor for running a program, wherein the program executes any of the methods described above on data output from the data acquisition device during runtime.

[0012] In this embodiment, a set of image frames is first acquired during the process of a biological object outputting a sign language action to be recognized. The image frames in the set are then discretized and encoded to obtain discrete features of the sign language action to be recognized. These discrete features characterize the features of the vocabulary represented by the sign language action. The discrete features of the sign language action are then classified to obtain the recognition result. This recognition result characterizes the category of the vocabulary to be recognized, thereby improving the accuracy of the sign language action recognition result. It is noteworthy that discretizing and encoding the image frames in the set to obtain discrete features of the sign language action allows for better modeling of features at the granularity of sign language vocabulary, resulting in more accurate recognition of the sign language action. This solves the technical problem of low recognition accuracy of sign language actions in related technologies and improves the interactive experience. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0014] Figure 1 This is a schematic diagram of the hardware environment of a virtual reality device for a sign language gesture recognition method according to an embodiment of this application;

[0015] Figure 2 This is a structural block diagram of the computational environment for a sign language gesture recognition method according to an embodiment of this application;

[0016] Figure 3 This is a flowchart of a sign language gesture recognition method according to Embodiment 1 of this application;

[0017] Figure 4 This is a schematic diagram of a sign language discretization representation learning according to an embodiment of this application;

[0018] Figure 5 This is a schematic diagram of a sign language recognition process according to an embodiment of this application;

[0019] Figure 6 This is a flowchart of a model training method according to Embodiment 2 of this application;

[0020] Figure 7This is a flowchart of a sign language gesture recognition method according to Embodiment 3 of this application;

[0021] Figure 8 This is a flowchart of a sign language gesture recognition method according to Embodiment 4 of this application;

[0022] Figure 9 This is a flowchart of a sign language gesture recognition method according to Embodiment 5 of this application;

[0023] Figure 10 This is a schematic diagram of a sign language gesture recognition device according to Embodiment 6 of this application;

[0024] Figure 11 This is a schematic diagram of a model training device according to Embodiment 7 of this application;

[0025] Figure 12 This is a schematic diagram of a model training device according to Embodiment 8 of this application;

[0026] Figure 13 This is a schematic diagram of a sign language recognition device according to Embodiment 9 of this application;

[0027] Figure 14 This is a schematic diagram of a sign language gesture recognition device according to Embodiment 10 of this application;

[0028] Figure 15 This is a structural block diagram of a computer terminal according to an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0032] Sign language recognition: Identify the corresponding sign language vocabulary category based on continuous sign language videos;

[0033] Image representation learning: Learning latent variable representations containing semantic information from high-dimensional input to an image.

[0034] Currently, the common approach is to sample sign language videos to generate image frames, extract image features from each image frame, and then decode the image features to obtain the sign language video recognition result. However, this approach has low accuracy and poor model robustness.

[0035] This application provides a method for recognizing sign language gestures, which can better model the features at the granularity of sign language vocabulary through discretization, and obtain better recognition results.

[0036] Example 1

[0037] According to an embodiment of this application, a method for recognizing sign language gestures is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0038] The disclosed method embodiments can be executed on a mobile terminal, computer terminal, or similar computing device. Taking the operation on a mobile terminal as an example, the mobile terminal can be a smartphone, tablet computer, PDA, mobile internet device, PAD, game console, or other terminal device. Figure 1 This is a schematic diagram of the hardware environment of a virtual reality device according to a sign language gesture recognition method based on an embodiment of this application. Figure 1As shown, the virtual reality device 104 is connected to the terminal 106, and the terminal 106 is connected to the server 102 via a network. The virtual reality device 104 is not limited to: virtual reality helmets, virtual reality glasses, virtual reality all-in-one machines, etc. The terminal 104 is not limited to PCs, mobile phones, tablets, etc. The server 102 can be a server corresponding to a media file operator. The network includes, but is not limited to: wide area network, metropolitan area network, or local area network.

[0039] Optionally, the virtual reality device 104 in this embodiment includes a memory, a processor, and a transmission device. The memory stores an application program that can be used to execute actions such as: acquiring a set of image frames during the process of a biological object outputting a sign language gesture to be recognized; discretizing and encoding the image frames in the set of image frames to obtain discrete features of the sign language gesture to be recognized, wherein the discrete features are used to characterize the features of the vocabulary represented by the sign language gesture; and classifying the discrete features of the sign language gesture to be recognized to obtain a recognition result of the sign language gesture to be recognized, wherein the recognition result is used to characterize the category of the vocabulary to be recognized, thereby solving the technical problem of low recognition accuracy of sign language gestures in related technologies.

[0040] The terminal in this embodiment can be used to display the recognition results on the display screen of a Virtual Reality (VR) device or an Augmented Reality (AR) device. The steps are as follows: displaying a set of image frames on the display screen of the VR or AR device; discretizing and encoding the image frames in the set to obtain discrete features of the sign language gesture to be recognized; classifying the discrete features of the sign language gesture to obtain the recognition result; and driving the VR or AR device to display the recognition result of the sign language gesture.

[0041] In some alternative embodiments primarily focused on user interaction, the aforementioned device may also provide a human-computer interaction interface with a touch-sensitive surface. This interface can sense finger contact and / or gestures to interact with a graphical user interface (GUI). The human-computer interaction functions may include the following: creating web pages, drawing, word processing, creating electronic documents, playing games, video conferencing, instant messaging, sending and receiving emails, call interfaces, playing digital videos, playing digital music, and / or web browsing, etc. Executable instructions for performing the aforementioned human-computer interaction functions are configured / stored in a processor-executable computer program product or a readable storage medium.

[0042] Figure 1 The hardware structure block diagram shown can serve not only as an exemplary block diagram of the aforementioned AR / VR device (or mobile device), but also as an exemplary block diagram of the aforementioned server. In one optional embodiment, Figure 2 The use of the above is illustrated in a block diagram. Figure 1 The AR / VR device (or mobile device) shown is an example of a computing node in computing environment 201.

[0043] Figure 2 This is a structural block diagram of the computational environment for a sign language gesture recognition method according to an embodiment of this application, such as... Figure 2 As shown, computing environment 201 includes multiple computing nodes (such as servers) running on a distributed network (shown as 210-1, 210-2, ..., in the diagram). Each computing node contains local processing and memory resources, and end user 202 can remotely run applications or store data within computing environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3, and 220-4 within computing environment 201, representing services "A", "D", "E", and "H", respectively.

[0044] End user 202 can provide and access services through a web browser or other software application on a client. In some embodiments, the provisioning and / or requests of end user 202 can be provided to ingress gateway 230. Ingress gateway 230 may include a corresponding agent to handle provisioning and / or requests for service 220 (one or more services provided in computing environment 201).

[0045] Service 220 is provided or deployed based on various virtualization technologies supported by computing environment 201. In some embodiments, service 220 may be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization may involve simulating a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the machine is virtualized by a virtual machine, container-based virtualization may launch containers to virtualize an entire operating system (OS), allowing multiple workloads to run on a single OS instance.

[0046] In one embodiment based on container virtualization, several containers of service 220 can be assembled into a POD (e.g., a Kubernetes POD). For example, such as Figure 2As shown, service 220-2 can be equipped with one or more PODs 240-1, 240-2, ..., 240-N (collectively referred to as POD 240). Each POD 240 can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers 242). One or more containers 242 in POD 240 handle requests related to one or more corresponding functions of the service, and the proxy 245 typically controls service-related network functions such as routing and load balancing. Other services 220 can also be equipped with PODs similar to POD 240.

[0047] During operation, executing a user request from end user 202 may require calling one or more services 220 in computing environment 201. Executing one or more functions of one service 220 requires calling one or more functions of another service 220. For example... Figure 2 As shown, service "A" 220-1 receives user requests from terminal user 202 from ingress gateway 230. Service "A" 220-1 can call service "D" 220-2, and service "D" 220-2 can request service "E" 220-3 to perform one or more functions.

[0048] The aforementioned computing environment can be a cloud computing environment, where resource allocation is managed by cloud services, allowing functionality development without needing to consider implementation, adjustment, or server scaling. This computing environment allows developers to execute event-responsive code without building or maintaining complex infrastructure. Services can be partitioned into a set of functions that can automatically and independently scale, rather than scaling a single hardware device to handle potential loads.

[0049] Under the aforementioned operating environment, this application provides the following: Figure 3 The method for recognizing sign language gestures shown is illustrated. It should be noted that the sign language gesture recognition method in this embodiment can be derived from... Figure 1 The mobile terminal in the illustrated embodiment is executed. Figure 3 This is a flowchart of a sign language gesture recognition method according to Embodiment 1 of this application. Figure 3 As shown, the method may include the following steps:

[0050] Step S302: Collect a set of image frames of the biological object during the process of the sign language action to be recognized.

[0051] The biological object mentioned above can be a person, an animal, etc., and there is no limitation on the biological object here. This application uses the biological object of a sign language teacher for illustration, but it is not limited to this.

[0052] The aforementioned sign language gestures to be identified can be sign language gestures expressed by biological objects through their limbs.

[0053] The sign language action to be recognized can be a sign language action corresponding to a word, and a sign language action corresponding to a word can contain multiple sign language actions. The image frames corresponding to a sign language action corresponding to a word can be used as a set of image frames.

[0054] The sign language action to be recognized can also be a sign language action corresponding to multiple words. The sign language action corresponding to multiple words can contain multiple sign language actions. The image frames corresponding to the sign language actions corresponding to multiple words can be combined into a set of image frames.

[0055] The sign language action to be identified can also be a complete sentence or multiple complete sentences corresponding to the sign language action to be identified. A complete sentence or multiple complete sentences corresponding to the sign language action to be identified can contain multiple sign language actions. The image frames corresponding to a complete sentence or multiple complete sentences can be combined into a single image frame set.

[0056] In one optional embodiment, video images of the biological object outputting the sign language action to be recognized can be acquired. By acquiring the video images at preset intervals, a set of image frames of the biological object outputting the sign language action to be recognized can be obtained, wherein the set of image frames may contain multiple consecutive image frames.

[0057] Step S304: Discretize and encode the image frames in the image frame set to obtain discrete features of the sign language actions to be recognized.

[0058] Among them, discrete features are used to characterize the features of the vocabulary to be identified represented by the sign language actions to be identified.

[0059] The aforementioned discretization encoding converts sign language gestures in an image frame into discrete features of the corresponding words to be recognized. This improves the granularity of recognition and thus enhances the accuracy of recognition.

[0060] The aforementioned discrete features can be represented by numbers, letters, symbols, etc., to indicate the words to be recognized corresponding to the sign language actions. By obtaining the discrete features of the sign language actions to be recognized, the memory resources occupied when recognizing the sign language actions can be reduced.

[0061] In one optional embodiment, feature extraction can be performed on the image frames in the image frame set to obtain a feature sequence of the image frames. This feature sequence can be converted into a discrete index sequence. The discrete features corresponding to the index sequence can be obtained by looking up a table in the codebook. These discrete features are used to represent the features of the words to be recognized corresponding to the features of the sign language actions to be recognized.

[0062] Step S306: Classify the discrete features of the sign language gestures to be recognized to obtain the recognition results of the sign language gestures to be recognized.

[0063] The recognition results are used to characterize the category of the words to be recognized.

[0064] In one alternative embodiment, since discrete features occupy very little memory and computing resources, classifying the discrete features of the sign language actions to be recognized can improve the recognition speed of the sign language actions and thus increase the efficiency of recognizing the sign language actions.

[0065] In another optional embodiment, the above-described scheme can be applied to a virtual reality sign language translation scenario. The recognition result of the sign language action to be recognized can be displayed in the sign language action video of the biological object, so that other users can simultaneously see the category of the word to be recognized represented by the sign language action and the specific meaning of the sign language action when watching the sign language action of the biological object.

[0066] Taking a sign language teacher as an example, the process involves collecting a set of image frames from the teacher's output of the sign language gesture to be recognized. These image frames are then discretized to obtain discrete features representing the teacher's gesture. These discrete features can be represented numerically, but are not limited to this. Optionally, the corresponding discrete features can be found in a codebook based on the image frame's characteristics. The discrete features of the gesture can be classified to obtain the recognition result, which can be the category of the vocabulary the teacher intends to express through sign language.

[0067] Taking the communication process of a hearing-impaired person as an example, an image frame set can be collected during the process of the hearing-impaired person outputting the sign language action to be recognized. The image frames in the image frame set are discretized and encoded to obtain the discrete features of the sign language action to be recognized. These discrete features are used to represent the discrete features of the hearing-impaired person's sign language action to be recognized. These discrete features can be represented by numbers, but are not limited to this. Optionally, the discrete features corresponding to the features of the image frames can be searched in the Codebook. The discrete features of the sign language action to be recognized can be classified to obtain the recognition result of the hearing-impaired person's sign language action to be recognized. This recognition result can be the category of the words that the hearing-impaired person wants to express.

[0068] In this embodiment, a set of image frames is first acquired during the process of a biological object outputting a sign language action to be recognized. The image frames in the set are then discretized and encoded to obtain discrete features of the sign language action to be recognized. These discrete features characterize the features of the vocabulary represented by the sign language action. The discrete features of the sign language action are then classified to obtain the recognition result. This recognition result characterizes the category of the vocabulary to be recognized, thereby improving the accuracy of the sign language action recognition result. It is noteworthy that discretizing and encoding the image frames in the set to obtain discrete features of the sign language action allows for better modeling of features at the granularity of sign language vocabulary, resulting in more accurate recognition of the sign language action and thus solving the technical problem of low recognition accuracy of sign language actions in related technologies.

[0069] In the above embodiments of this application, the image frames in the image frame set are discretized and encoded to obtain discrete features of the sign language action to be recognized, including: using a discretization coding model to discretize and encode the image frames to obtain discrete features of the sign language action to be recognized.

[0070] The discretization coding model mentioned above can be Vector Quantized Variational Autoencoder (VQ-VAE), but it is not limited to this and can also be other models capable of performing discretization coding.

[0071] In one alternative embodiment, a discretization coding model can be used to discretize the image frame based on the sign language action to be recognized in the image frame, thereby obtaining the discrete features of the sign language action to be recognized.

[0072] In the above embodiments of this application, the discretization coding model includes an encoder model and a discretization vocabulary. The discretization coding model is used to discretize and encode image frames to obtain discrete features of the sign language actions to be recognized. This includes: extracting features from the image frames using the encoder model to obtain image features of the image frames; discretizing the image features of the image frames to obtain intermediate features corresponding to the image features; and obtaining preset features corresponding to the intermediate features from the discretization vocabulary to obtain discrete features of the sign language actions to be recognized. The discretization vocabulary stores a preset number of preset features, and different combinations of these preset features are used to characterize the vocabulary represented by different sign language actions.

[0073] The encoder model described above can be an encoder network.

[0074] The aforementioned discretized vocabulary can serve as a codebook.

[0075] The aforementioned intermediate features can be discrete index sequences, where each index can represent a sign language gesture.

[0076] The aforementioned preset features can be word features, but are not limited to this. The discretized vocabulary can store word features representing words for different sign language actions. The aforementioned preset features can also be sentence features; by arranging and combining different numbers of sentence features, words representing different sign language actions can be obtained.

[0077] In one optional embodiment, an encoder model can be used to sequentially extract features from the image frames in the image frame set to obtain image features for each image frame in the image frame set. These image features contain features of the sign language action to be recognized. The image features of each image frame in the image frame set can be discretized to obtain discretized intermediate features. A preset feature corresponding to the intermediate feature can be searched from a discretized vocabulary to obtain the discrete features of the sign language action to be recognized. By discretizing the image features of the image frames, the search process uses discretized features that consume less memory and computing resources. Therefore, the speed of searching for the preset features corresponding to the sign language action to be recognized in the image frame can be improved, thereby improving the efficiency of recognizing the sign language action.

[0078] In the above embodiments of this application, the model parameters of the discretization coding model are adjusted based on a first loss value. The first loss value is constructed based on a set of training image frames and a set of reconstructed image frames. The set of reconstructed image frames is obtained by reconstructing images using spliced ​​features using a decoder model. The spliced ​​features are obtained by splicing the image features of reference image frames in the set of training image frames with the discrete features of training sign language actions contained in the set of training image frames. The image features of reference image frames are obtained by extracting features from reference image frames using a feature extraction model. The discrete features of training sign language actions are obtained by discretizing and encoding training image frames in the set of training image frames using a discretization coding model.

[0079] The aforementioned training image frame set can be a set of image frames corresponding to image frames containing the sign language actions to be recognized.

[0080] The aforementioned set of reconstructed image frames can be obtained by discretizing the training image frames into a set using a discretization coding model to obtain discrete features, and then reconstructing the image frames based on these discrete features.

[0081] A first loss value can be constructed by using the training image frame set and the reconstructed image frame set. If the difference between the reconstructed image frame set and the training image frame set is large, it indicates that the discretization coding accuracy of the discretization coding model is low. In this case, the model parameters of the discretization coding model need to be adjusted based on the first loss value to improve the accuracy of the discretization coding model. If the reconstructed image frame set and the training image frame set are consistent or have little difference, it indicates that the discretization coding accuracy of the discretization coding model is high. The first loss value mentioned above can be the loss value of the normalized loss function (L2norm), or it can be the loss value of any other type of loss function; there are no restrictions here.

[0082] The aforementioned reference image frame can be a high-resolution reference image frame or one that contains relatively complete sign language gestures from the training image frame set. The reference image frame can be any image frame from the training image frame set. Alternatively, the reference image frame can be an image frame determined by the user from the training image frame set.

[0083] The feature extraction model described above can be a Transformer model that uses an attention mechanism to improve the training speed of the model.

[0084] In one optional embodiment, a discretization coding model can be used to discretize and encode the training image frames in the training image frame set to obtain discrete features of the training sign language actions. Reference image frames can be extracted from the training image frame set, and the image features of the reference image frames can be concatenated with the discrete features of the training sign language actions to obtain concatenated features. Since the concatenated features contain the image features of the reference images, when reconstructing images from the concatenated features, a reconstructed image frame set with higher accuracy can be obtained.

[0085] In another alternative embodiment, a set of training image frames can be displayed to the user, who can determine a reference image frame from the set of training image frames, extract features from the reference image frame to obtain image features of the reference image frame, and then concatenate the image features of the reference image frame with the discrete features of the training sign language actions contained in the set of training image frames to obtain concatenated features.

[0086] Figure 4This is a schematic diagram of a sign language discretization representation learning according to an embodiment of this application. The input can be isolated sign language vocabulary data. Each video can correspond to a sign language vocabulary. The sign language vocabulary can be represented as training image frames in a training image frame set. The training image frames in the training image frame set can be discretized and encoded using a discretization coding model to obtain discrete features (latent vectors) of the sign language action to be recognized. Optionally, the image features of the training image frames can be discretized to obtain intermediate features corresponding to the image features. The word features corresponding to the intermediate features can be obtained from the discretized vocabulary to obtain the discrete features of the sign language action to be recognized. Optionally, the word features can be concatenated with the image features of the reference image frame to obtain concatenated features. The length of the image frame can be obtained by sampling through a decoder model, and then the image frames in the image frame set can be reconstructed through multiple deconvolutional neural network layers to obtain a reconstructed image frame set. The model parameters of the discretization coding model can be adjusted according to the first loss value of the training image frame set and the reconstructed image frame set.

[0087] In the above embodiments of this application, classifying the discrete features of the sign language action to be recognized to obtain the recognition result of the sign language action to be recognized includes: classifying the discrete features of the sign language action to be recognized using a recognition model to obtain the recognition result.

[0088] The aforementioned recognition model can be a classification network (Transformer), which may contain a classification layer (classifier).

[0089] In one alternative embodiment, a recognition model can be used to classify the discrete features of the sign language gestures to be recognized. By classifying the discrete features, the efficiency of classification can be improved, and the efficiency of obtaining recognition results can be improved.

[0090] In the above embodiments of this application, the model parameters of the recognition model are adjusted based on a second loss value, which is determined based on the recognition result of the sign language action and the received feedback result. The feedback result is obtained by modifying the recognition result after the recognition result is output.

[0091] In one optional embodiment, the recognition result of the sign language gesture can be output to the user's client first. If the accuracy of the recognition result is low, the user can modify the recognition result. After modifying the recognition result, a feedback result can be obtained. A second loss value can be determined based on the recognition result and the feedback result. The model parameters of the recognition model can be adjusted based on the second loss value to improve the model recognition performance. The second loss value can be the loss value of the Connectionist Temporal Classification Loss (CTCloss), or it can be the loss value of any other type of loss function; there are no limitations on this.

[0092] For example, if the sign language gesture needs to express "hello", but the output sign language gesture recognition result is "bye-bye", then the user can change the result "hello" to "bye-bye", and determine the second loss value based on "hello" and "bye-bye", and adjust the model parameters of the recognition model according to the second loss value.

[0093] Figure 5 This is a schematic diagram of a sign language recognition process according to an embodiment of this application. A continuous sign language video containing multiple sign language words can be input. The sign language video can be processed through a discretization encoding model to obtain discrete features of the sign language actions to be recognized. These discrete features can be classified using a recognition model to obtain the category of the sign language action corresponding to the word to be recognized, i.e., the recognition result. Optionally, the discrete features can be input into a Transformer for word classification, thereby outputting a sequence of word categories to complete the classification task. After obtaining the recognition result, the sign language action recognition result can be output; feedback results obtained from modifying the recognition result can be received; a second loss value can be determined based on the recognition result and the feedback result; and the model parameters of the recognition model can be adjusted based on the second loss value.

[0094] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0095] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0096] Example 2

[0097] According to an embodiment of this application, a training method for a discretized coding model is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0098] Figure 6 This is a flowchart of a training method for a discretized coding model according to Embodiment 2 of this application, as shown below. Figure 6 As shown, the method may include the following steps:

[0099] Step S602: Obtain the training image frame set.

[0100] The training image frame set contains training sign language actions output by the same biological object.

[0101] The aforementioned identical biological objects can refer to biological objects of the same type, where biological objects can refer to humans, animals of the same kind, etc.

[0102] The aforementioned training image frame set can be a set of image frames corresponding to image frames containing the sign language actions to be recognized.

[0103] Step S604: Discrete encoding is performed on the training image frames in the training image frame set using a discretization encoding model to obtain the discrete features of the training sign language actions.

[0104] Step S606: Use the decoder model to reconstruct the discrete features of the training sign language actions to obtain a set of reconstructed image frames.

[0105] Step S608: Adjust the model parameters of the discretized coding model and the decoder model based on the training image frame set and the reconstructed image frame set.

[0106] Through the above steps, the training image frames in the training image frame set can be discretized using a discretization coding model to obtain discrete features of the training sign language actions. This allows for the representation of the training sign language actions with less resource consumption. The decoder model can then be used to reconstruct the images from the discrete features of the training sign language actions, resulting in a reconstructed image frame set. The model parameters of the discretization coding model and the decoder model can be adjusted based on the training image frame set and the reconstructed image frame set, thereby improving the accuracy of the discretization coding model in discretizing the training image frames and the accuracy of the decoder model in reconstructing the discrete features.

[0107] In the above embodiments of this application, the discrete features of training sign language actions are used to reconstruct images to obtain a set of reconstructed image frames using a decoder model. This includes: using a feature extraction model to extract features from reference image frames in the set of training image frames to obtain image features of the reference image frames; concatenating the image features of the reference image frames with the discrete features of the training sign language actions to obtain concatenated features; and using a decoder model to reconstruct images from the concatenated features to obtain a set of reconstructed image frames.

[0108] In one optional embodiment, a feature extraction model can be used to extract features from reference image frames in the training image frame set to obtain image features of the reference image frames. The image features of the reference image frames can be concatenated with the discrete features of the training sign language actions to obtain concatenated features. Since the concatenated features contain the image features of the reference images, when reconstructing images from the concatenated features, a reconstructed image frame set with higher accuracy can be obtained.

[0109] In the above embodiments of this application, the discretization coding model includes: an encoder model and a discretization vocabulary. The discretization coding model is used to discretize and encode training image frames in the training image frame set to obtain discrete features of training sign language actions. This includes: extracting features from the training image frames using the encoder model to obtain image features of the training image frames; performing a discretization operation on the image features of the training image frames to obtain intermediate features corresponding to the image features; and obtaining preset features corresponding to the intermediate features from the discretization vocabulary. Different combinations of preset features are used to characterize the vocabulary represented by different sign language actions.

[0110] By discretizing the image features of the image frame, the search process uses discretized features that consume less memory and computational resources. Therefore, the speed of searching for the preset features corresponding to the sign language action to be recognized in the image frame can be improved, thereby improving the efficiency of recognizing the sign language action.

[0111] In the above embodiments of this application, the model parameters of the discretized coding model and the decoder model are adjusted based on the training image frame set and the reconstructed image frame set, including: constructing the loss value of a preset loss function based on the training image frame set and the reconstructed image frame set; and adjusting the model parameters of the discretized coding model and the decoder model based on the loss value.

[0112] The preset loss function mentioned above can be L2norm.

[0113] In one optional embodiment, a loss value of a preset loss function can be constructed based on the training image frame set and the reconstructed image frame set. The model parameters of the discretized coding model and the decoder model can be adjusted based on the loss value, thereby improving the accuracy of the discretized coding model and the decoder model.

[0114] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0115] Example 3

[0116] According to an embodiment of this application, a method for recognizing sign language gestures is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0117] Figure 7 This is a flowchart of a sign language gesture recognition method according to Embodiment 3 of this application, as follows: Figure 7 As shown, the method may include the following steps:

[0118] Step S702: In response to the input command applied to the operation interface, display a set of image frames on the operation interface.

[0119] The image frame set is collected during the process of the biological object outputting the sign language action to be recognized.

[0120] The above-mentioned input commands can be obtained by the user through operation of the interface.

[0121] Step S704: In response to the recognition command applied to the operation interface, display the recognition result of the sign language action to be recognized on the operation interface.

[0122] The recognition result is used to characterize the category of the word to be recognized. The recognition result is obtained by classifying the discrete features of the sign language action to be recognized. The discrete features of the sign language action to be recognized are obtained by discretizing and encoding the image frames in the image frame set. The discrete features are used to characterize the features of the word to be recognized represented by the sign language action.

[0123] The aforementioned recognition instructions can be obtained by the user through operations on the user interface.

[0124] Through the above steps, in response to input commands applied to the user interface, a set of image frames can be displayed on the interface so that the user can view the set of image frames and determine whether recognition is required. If recognition is required, the image frame set can be discretized and encoded on the user interface to obtain discrete features of the sign language action to be recognized. The recognition result is obtained by classifying the discrete features, and then displayed on the user interface. Processing the image frame set interactively to obtain the recognition result of the sign language action improves the convenience of user operation.

[0125] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0126] Example 4

[0127] According to an embodiment of this application, a method for recognizing sign language gestures is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0128] Figure 8 This is a flowchart of a sign language gesture recognition method according to Embodiment 4 of this application, as follows: Figure 8 As shown, the method may include the following steps:

[0129] Step S802: Display a set of image frames on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device.

[0130] The image frame set is collected during the process of the biological object outputting the sign language action to be recognized.

[0131] Displaying a set of image frames on the screen of a virtual reality (VR) device or an augmented reality (AR) device can improve the applicable scenarios for recognizing the set of image frames.

[0132] Step S804: Discretize and encode the image frames in the image frame set to obtain discrete features of the sign language actions to be recognized.

[0133] Among them, discrete features are used to characterize the features of the vocabulary to be identified represented by the sign language actions to be identified.

[0134] Step S806: Classify the discrete features of the sign language gestures to be recognized to obtain the recognition results of the sign language gestures to be recognized.

[0135] The recognition results are used to characterize the category of the words to be recognized;

[0136] Step S808: Drive the VR device or AR device to render and display the recognition result of the sign language action to be recognized.

[0137] Through the above steps, by displaying a set of image frames on the screen of a virtual reality (VR) device or an augmented reality (AR) device, the image frames in the set can be discretized and encoded to obtain discrete features of the sign language gesture to be recognized. These discrete features are then classified to obtain the recognition result, which in turn drives the VR or AR device to render and display the result. In other words, by discretizing and encoding the image frames in the set to obtain discrete features of the sign language gesture, it is possible to better model the features at the granularity of sign language vocabulary, resulting in more accurate recognition of the gesture and thus solving the technical problem of low accuracy in sign language gesture recognition in related technologies.

[0138] Optionally, in this embodiment, the above-described sign language recognition method can be applied to a hardware environment consisting of a server and a virtual reality device. The VR or AR device is controlled to perform human-computer interaction operations corresponding to the sign language recognition results. The server can be a server corresponding to a media file operator. The aforementioned network includes, but is not limited to, a wide area network (WAN), a metropolitan area network (MAN), or a local area network (LAN). The aforementioned virtual reality device is not limited to, a virtual reality headset, virtual reality glasses, or a standalone virtual reality device.

[0139] Optionally, the virtual reality device includes: a memory, a processor, and a transmission device. The memory stores a program that can be used to execute: displaying a set of image frames on the presentation screen of the virtual reality (VR) device or augmented reality (AR) device; discretizing and encoding the image frames in the set of image frames to obtain discrete features of the sign language gesture to be recognized; classifying the discrete features of the sign language gesture to be recognized to obtain the recognition result of the sign language gesture; and driving the VR or AR device to render and display the recognition result of the sign language gesture to be recognized.

[0140] It should be noted that the sign language broadcasting method described above in this embodiment, applied to VR or AR devices, may include, for example: Figure 8 The method of the illustrated embodiment is used to control a VR device or AR device to perform human-computer interaction operations corresponding to the sign language gestures to be recognized.

[0141] Optionally, the processor in this embodiment can invoke the application stored in the memory via the transmission device to execute the above steps. The transmission device can receive a set of image frames sent by a server via a network, or it can be used for data transmission between the processor and the memory.

[0142] Optionally, in a virtual reality device, there is a head-mounted display with eye tracking. The screen in the HMD is used to display the video footage. The eye tracking module in the HMD is used to acquire the real-time movement trajectory of the user's eyes. The tracking system is used to track the user's position and movement information in real three-dimensional space. The computing and processing unit is used to acquire the user's real-time position and movement information from the tracking system and calculate the three-dimensional coordinates of the user's head in the virtual three-dimensional space, as well as the user's field of vision orientation in the virtual three-dimensional space.

[0143] In this embodiment, the virtual reality device can be connected to a terminal, and the terminal can be connected to a server via a network. The virtual reality device is not limited to virtual reality headsets, virtual reality glasses, virtual reality all-in-one machines, etc., and the terminal is not limited to PCs, mobile phones, tablets, etc. The server can be the server corresponding to the image frame collection operator. The network includes, but is not limited to, wide area networks, metropolitan area networks, or local area networks.

[0144] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0145] Example 5

[0146] According to an embodiment of this application, a method for recognizing sign language gestures is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0147] Figure 9 This is a flowchart of a sign language gesture recognition method according to Embodiment 5 of this application, as follows: Figure 9 As shown, the method may include the following steps:

[0148] Step S902: Obtain the image frame set by calling the first interface.

[0149] The first interface includes a first parameter, the value of which is a set of image frames, which are collected during the process of the biological object outputting the sign language action to be recognized.

[0150] The first interface mentioned above can be an interface for data interaction between the server and the client. The client can pass a set of image frames to the interface function as the first parameter of the interface function to achieve the purpose of uploading the set of image frames to the cloud server.

[0151] Step S904: Discretize and encode the image frames in the image frame set to obtain discrete features of the sign language actions to be recognized.

[0152] Among them, discrete features are used to characterize the features of the vocabulary to be identified represented by the sign language actions to be identified.

[0153] Step S906: Classify the discrete features of the sign language gestures to be recognized to obtain the recognition results of the sign language gestures to be recognized.

[0154] The recognition results are used to characterize the category of the words to be recognized.

[0155] Step S908: Output the recognition result of the sign language action to be recognized by calling the second interface.

[0156] The second interface includes a second parameter, the value of which is the recognition result of the sign language action to be recognized.

[0157] The second interface in the above steps can be an interface for data exchange between the cloud server and the client. The cloud server can pass the recognition result of the sign language action to be recognized into the interface function as the second parameter of the interface function, so as to achieve the purpose of sending the recognition result of the sign language action to be recognized to the client.

[0158] Through the above steps, an image frame set is obtained by calling a first interface, where the first interface includes a first parameter whose value is the image frame set, which is collected during the process of a biological object outputting a sign language action to be recognized. The image frames in the image frame set are then discretized and encoded to obtain discrete features of the sign language action to be recognized, where the discrete features characterize the features of the vocabulary represented by the sign language action. The discrete features of the sign language action to be recognized are then classified to obtain the recognition result of the sign language action, where the recognition result characterizes the category of the vocabulary to be recognized. Finally, the recognition result of the sign language action to be recognized is output by calling a second interface, where the second interface includes a second parameter whose value is the recognition result of the sign language action. In other words, by discretizing and encoding the image frames in the image frame set to obtain discrete features of the sign language action to be recognized, it is possible to better model the features at the granularity of sign language vocabulary, enabling more accurate recognition of the sign language action and thus solving the technical problem of low recognition accuracy of sign language actions in related technologies.

[0159] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0160] Example 6

[0161] According to an embodiment of this application, a sign language recognition device is also provided for implementing the above-described sign language recognition method. Figure 10 This is a schematic diagram of a sign language gesture recognition device according to Embodiment 6 of this application, as shown below. Figure 10 As shown, the device 1000 includes: a data acquisition module 1002, an encoding module 1004, and a classification module 1006.

[0162] The acquisition module is used to acquire a set of image frames during the process of biological objects outputting sign language actions to be recognized; the encoding module is used to discretize and encode the image frames in the image frame set to obtain discrete features of the sign language actions to be recognized, wherein the discrete features are used to characterize the features of the vocabulary to be recognized represented by the sign language actions; the classification module is used to classify the discrete features of the sign language actions to be recognized to obtain the recognition result of the sign language actions to be recognized, wherein the recognition result is used to characterize the category of the vocabulary to be recognized.

[0163] It should be noted that the acquisition module 1002, encoding module 1004, and classification module 1006 mentioned above correspond to steps S302 to S306 in Embodiment 1. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run on the computer terminal provided in Embodiment 1.

[0164] In the above embodiments of this application, the encoding module is used to discretize the image frame using a discretization encoding model to obtain the discrete features of the sign language action to be recognized.

[0165] In the above embodiments of this application, the discretization coding model includes: an encoder model and a discretization vocabulary, wherein the coding module includes: an extraction unit, an operation unit, and an acquisition unit.

[0166] The extraction unit is used to extract features from the image frame using the encoder model to obtain the image features of the image frame; the operation unit is used to discretize the image features of the image frame to obtain the intermediate features corresponding to the image features; the acquisition unit is used to acquire the preset features corresponding to the intermediate features from the discretized vocabulary to obtain the discrete features of the sign language action to be recognized. The discretized vocabulary stores a preset number of preset features, and the permutation and combination of different numbers of preset features are used to represent the vocabulary represented by different sign language actions.

[0167] In the above embodiments of this application, the model parameters of the discretization coding model are adjusted based on a first loss value. The first loss value is constructed based on a set of training image frames and a set of reconstructed image frames. The set of reconstructed image frames is obtained by reconstructing images using spliced ​​features using a decoder model. The spliced ​​features are obtained by splicing the image features of reference image frames in the set of training image frames with the discrete features of training sign language actions contained in the set of training image frames. The image features of reference image frames are obtained by extracting features from reference image frames using a feature extraction model. The discrete features of training sign language actions are obtained by discretizing and encoding training image frames in the set of training image frames using a discretization coding model.

[0168] In the above embodiments of this application, the classification module is used to classify the discrete features of the sign language actions to be recognized using a recognition model to obtain the recognition result.

[0169] In the above embodiments of this application, the model parameters of the recognition model are adjusted based on a second loss value, which is determined based on the recognition result of the sign language action and the received feedback result. The feedback result is obtained by modifying the recognition result after the recognition result is output.

[0170] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0171] Example 7

[0172] According to an embodiment of this application, a training apparatus for a discretized coding model corresponding to the training method for implementing the above-described discretized coding model is also provided. Figure 11 This is a schematic diagram of a training device for a discretized coding model according to Embodiment 7 of this application, as shown below. Figure 11 As shown, the device 1100 includes: an acquisition module 1102, an encoding module 1104, a reconstruction module 1106, and an adjustment module 1108.

[0173] The acquisition module acquires a set of training image frames, which contains training sign language actions output by the same biological object. The encoding module uses a discretization encoding model to discretize the training image frames in the training image frame set to obtain discrete features of the training sign language actions. The reconstruction module uses a decoder model to reconstruct the images from the discrete features of the training sign language actions to obtain a set of reconstructed image frames. The adjustment module adjusts the model parameters of the discretization encoding model and the decoder model based on the training image frame set and the reconstructed image frame set.

[0174] It should be noted that the acquisition module 1102, encoding module 1104, reconstruction module 1106, and adjustment module 1108 mentioned above correspond to steps S702 to S708 in Embodiment 2. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run on the computer terminal provided in Embodiment 1.

[0175] In the above embodiments of this application, the reconstruction module includes: an extraction unit, a splicing unit, and a reconstruction unit.

[0176] The extraction unit is used to extract features from the reference image frames in the training image frame set using a feature extraction model to obtain the image features of the reference image frames; the stitching unit is used to stitch the image features of the reference image frames with the discrete features of the training sign language actions to obtain stitched features; and the reconstruction unit is used to reconstruct the image using the stitched features using a decoder model to obtain the reconstructed image frame set.

[0177] In the above embodiments of this application, the discretization coding model includes: an encoder model and a discretization vocabulary, wherein the coding module includes: a coding unit, an operation unit, and an acquisition unit.

[0178] The encoding unit is used to extract features from the training image frames using an encoder model to obtain image features of the training image frames; the operation unit is used to discretize the image features of the training image frames to obtain intermediate features corresponding to the image features; the acquisition unit is used to acquire preset features corresponding to the intermediate features from the discretized vocabulary to obtain discrete features of the training sign language actions. The discretized vocabulary stores a preset number of preset features, and the permutations and combinations of different numbers of preset features are used to represent the vocabulary represented by different sign language actions.

[0179] In the above embodiments of this application, the adjustment module includes: a construction unit and an adjustment unit.

[0180] The construction unit is used to construct the loss value of the preset loss function based on the training image frame set and the reconstructed image frame set; the adjustment unit is used to adjust the model parameters of the discretized encoding model and the decoder model based on the loss value.

[0181] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0182] Example 8

[0183] According to an embodiment of this application, a sign language recognition device is also provided for implementing the above-described sign language recognition method. Figure 12 This is a schematic diagram of a model training device according to Embodiment 8 of this application, as shown below. Figure 12 As shown, the device 1200 includes: a first display module 1202 and a second display module 1204.

[0184] The first display module responds to input commands applied to the operation interface and displays a set of image frames on the operation interface. The set of image frames is acquired during the process of the biological object outputting the sign language action to be recognized. The second display module responds to recognition commands applied to the operation interface and displays the recognition result of the sign language action to be recognized. The recognition result is used to characterize the category of the word to be recognized. The recognition result is obtained by classifying the discrete features of the sign language action to be recognized. The discrete features of the sign language action to be recognized are obtained by discretizing and encoding the image frames in the set of image frames. The discrete features are used to characterize the features of the word to be recognized represented by the sign language action.

[0185] It should be noted that the first display module 1202 and the second display module 1204 mentioned above correspond to steps S802 to S804 in Embodiment 3. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal provided in Embodiment 1.

[0186] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0187] Example 9

[0188] According to an embodiment of this application, a sign language recognition device is also provided for implementing the above-described sign language recognition method. Figure 13 This is a schematic diagram of a sign language gesture recognition device according to Embodiment 9 of this application, as shown below. Figure 13 As shown, the device 1300 includes: a display module 1302, an encoding module 1304, a classification module 1306, and a driving module 1308.

[0189] The system comprises the following modules: a display module for displaying a set of image frames on the screen of a virtual reality (VR) or augmented reality (AR) device, wherein the set of image frames is acquired during the process of a biological object outputting a sign language gesture to be recognized; an encoding module for discretizing and encoding the image frames in the set of image frames to obtain discrete features of the sign language gesture to be recognized, wherein the discrete features are used to characterize the features of the vocabulary to be recognized represented by the sign language gesture; a classification module for classifying the discrete features of the sign language gesture to be recognized to obtain the recognition result of the sign language gesture, wherein the recognition result is used to characterize the category of the vocabulary to be recognized; and a driving module for driving the VR or AR device to render and display the recognition result of the sign language gesture to be recognized.

[0190] It should be noted that the display module 1302, encoding module 1304, classification module 1306, and driving module 1308 mentioned above correspond to steps S902 to S908 in Embodiment 5. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run on the computer terminal provided in Embodiment 1.

[0191] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0192] Example 10

[0193] According to an embodiment of this application, a sign language recognition device is also provided for implementing the above-described sign language recognition method. Figure 14 This is a schematic diagram of a sign language gesture recognition device according to Embodiment 10 of this application, as shown below. Figure 14 As shown, the device 1400 includes: an acquisition module 1402, an encoding module 1404, a classification module 1406, and an output module 1408.

[0194] The system comprises the following modules: an acquisition module for acquiring a set of image frames by calling a first interface, wherein the first interface includes a first parameter whose value is the set of image frames acquired during the process of a biological object outputting a sign language gesture to be recognized; an encoding module for discretizing and encoding the image frames in the set of image frames to obtain discrete features of the sign language gesture to be recognized, wherein the discrete features are used to characterize the features of the vocabulary to be recognized represented by the sign language gesture; a classification module for classifying the discrete features of the sign language gesture to be recognized to obtain the recognition result of the sign language gesture, wherein the recognition result is used to characterize the category of the vocabulary to be recognized; and an output module for outputting the recognition result of the sign language gesture to be recognized by calling a second interface, wherein the second interface includes a second parameter whose value is the recognition result of the sign language gesture to be recognized.

[0195] It should be noted that the acquisition module 1402, encoding module 1404, classification module 1406, and output module 1408 mentioned above correspond to steps S1002 to S1008 in Embodiment 6. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run on the computer terminal provided in Embodiment 1.

[0196] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0197] Example 11

[0198] Embodiments of this application may provide an electronic device, which may be any one of a group of electronic devices. Optionally, in this embodiment, the aforementioned electronic device may also be replaced by a terminal device such as a mobile terminal.

[0199] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0200] In this embodiment, the above-mentioned electronic device can execute the program code of the following steps in the sign language action recognition method: acquiring a set of image frames during the process of a biological object outputting a sign language action to be recognized; discretizing and encoding the image frames in the image frame set to obtain discrete features of the sign language action to be recognized, wherein the discrete features are used to characterize the features of the vocabulary to be recognized represented by the sign language action; classifying the discrete features of the sign language action to be recognized to obtain the recognition result of the sign language action to be recognized, wherein the recognition result is used to characterize the category of the vocabulary to be recognized.

[0201] Optionally, Figure 15 This is a structural block diagram of a computer terminal according to an embodiment of this application. Figure 15 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors and memory.

[0202] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the sign language recognition method and device in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned sign language recognition method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0203] The processor can access the information and application program stored in the memory via the transmission device to perform the following steps: acquiring a set of image frames during the process of a biological object outputting a sign language action to be recognized; discretizing and encoding the image frames in the image frame set to obtain discrete features of the sign language action to be recognized, wherein the discrete features are used to characterize the features of the vocabulary to be recognized represented by the sign language action; classifying the discrete features of the sign language action to be recognized to obtain the recognition result of the sign language action to be recognized, wherein the recognition result is used to characterize the category of the vocabulary to be recognized.

[0204] Optionally, the processor may also execute program code that performs the following steps: discretizes the image frame using a discretization coding model to obtain discrete features of the sign language action to be recognized.

[0205] Optionally, the processor may also execute program code for the following steps: extracting features from the image frame using an encoder model to obtain image features of the image frame; discretizing the image features of the image frame to obtain intermediate features corresponding to the image features; obtaining preset features corresponding to the intermediate features from the discretized vocabulary to obtain discrete features for training sign language actions, wherein the discretized vocabulary stores a preset number of preset features, and the permutations and combinations of different numbers of preset features are used to represent the vocabulary represented by different sign language actions.

[0206] Optionally, the processor may also execute program code with the following steps: the model parameters of the discretization coding model are adjusted based on a first loss value, which is constructed based on a set of training image frames and a set of reconstructed image frames. The set of reconstructed image frames is obtained by reconstructing the spliced ​​features using a decoder model. The spliced ​​features are obtained by splicing the image features of reference image frames in the set of training image frames with the discrete features of the training sign language actions contained in the set of training image frames. The image features of reference image frames are obtained by extracting features from reference image frames using a feature extraction model. The discrete features of the training sign language actions are obtained by discretizing and encoding the training image frames in the set of training image frames using a discretization coding model.

[0207] Optionally, the processor may also execute program code that performs the following steps: classifying the discrete features of the sign language gestures to be recognized using a recognition model to obtain the recognition result.

[0208] Optionally, the processor may also execute program code that performs the following steps: the model parameters of the recognition model are adjusted based on a second loss value, which is determined based on the recognition result of the sign language action and the received feedback result, and the feedback result is obtained by modifying the recognition result after the recognition result is output.

[0209] The processor can access information and applications stored in memory via a transmission device to perform the following steps: acquiring a set of training image frames, wherein the set of training image frames contains training sign language actions output by the same biological object; discretizing the training image frames in the set of training image frames using a discretization coding model to obtain discrete features of the training sign language actions; reconstructing the images using the discrete features of the training sign language actions using a decoder model to obtain a set of reconstructed image frames; and adjusting the model parameters of the discretization coding model and the decoder model based on the set of training image frames and the set of reconstructed image frames.

[0210] Optionally, the processor may also execute program code for the following steps: extracting features from reference image frames in the training image frame set using a feature extraction model to obtain image features of the reference image frames; concatenating the image features of the reference image frames with the discrete features of the training sign language actions to obtain concatenated features; and reconstructing the images using a decoder model to obtain a reconstructed image frame set.

[0211] Optionally, the processor may also execute program code for the following steps: extracting features from training image frames using an encoder model to obtain image features of the training image frames; discretizing the image features of the training image frames to obtain intermediate features corresponding to the image features; obtaining preset features corresponding to the intermediate features from the discretized vocabulary to obtain discrete features of the training sign language actions, wherein the discretized vocabulary stores a preset number of preset features, and the permutations and combinations of different numbers of preset features are used to represent the vocabulary represented by different sign language actions.

[0212] Optionally, the processor may also execute program code that performs the following steps: constructing a loss value for a preset loss function based on the training image frame set and the reconstructed image frame set; and adjusting the model parameters of the discretized coding model and the decoder model based on the loss value.

[0213] The processor can invoke information and application programs stored in memory via a transmission device to perform the following steps: In response to an input command applied to the operating interface, display a set of image frames on the operating interface, wherein the set of image frames is acquired during the process of a biological object outputting a sign language action to be recognized; In response to a recognition command applied to the operating interface, display the recognition result of the sign language action to be recognized on the operating interface, wherein the recognition result is used to characterize the category of the word to be recognized, and the recognition result is obtained by classifying the discrete features of the sign language action to be recognized, the discrete features of the sign language action to be recognized are obtained by discretizing and encoding the image frames in the set of image frames, and the discrete features are used to characterize the features of the word to be recognized represented by the sign language action to be recognized.

[0214] The processor can access information and applications stored in memory via a transmission device to perform the following steps: displaying a set of image frames on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the set of image frames is acquired during the process of a biological object outputting a sign language gesture to be recognized; discretizing and encoding the image frames in the set of image frames to obtain discrete features of the sign language gesture to be recognized, wherein the discrete features are used to characterize the features of the vocabulary to be recognized represented by the sign language gesture; classifying the discrete features of the sign language gesture to be recognized to obtain the recognition result of the sign language gesture to be recognized, wherein the recognition result is used to characterize the category of the vocabulary to be recognized; and driving the VR or AR device to render and display the recognition result of the sign language gesture to be recognized.

[0215] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: acquiring a set of image frames by calling a first interface, wherein the first interface includes a first parameter, the parameter value of which is the set of image frames acquired during the process of a biological object outputting a sign language action to be recognized; discretizing and encoding the image frames in the set of image frames to obtain discrete features of the sign language action to be recognized, wherein the discrete features are used to characterize the features of the vocabulary to be recognized represented by the sign language action; classifying the discrete features of the sign language action to be recognized to obtain a recognition result of the sign language action to be recognized, wherein the recognition result is used to characterize the category of the vocabulary to be recognized; and outputting the recognition result of the sign language action to be recognized by calling a second interface, wherein the second interface includes a second parameter, the parameter value of which is the recognition result of the sign language action to be recognized.

[0216] In this embodiment, a set of image frames is first acquired during the process of a biological object outputting a sign language action to be recognized. The image frames in the set are then discretized and encoded to obtain discrete features of the sign language action to be recognized. These discrete features characterize the features of the vocabulary represented by the sign language action. The discrete features of the sign language action are then classified to obtain the recognition result. This recognition result characterizes the category of the vocabulary to be recognized, thereby improving the accuracy of the sign language action recognition result. It is noteworthy that discretizing and encoding the image frames in the set to obtain discrete features of the sign language action allows for better modeling of features at the granularity of sign language vocabulary, resulting in more accurate recognition of the sign language action and thus solving the technical problem of low recognition accuracy of sign language actions in related technologies.

[0217] Those skilled in the art will understand that Figure 15 The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a mobile internet device (MID), a PAD, and other terminal devices. Figure 15 This does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include components that are more advanced than those described above. Figure 15 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 15 The different configurations shown.

[0218] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0219] Example 12

[0220] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the aforementioned computer-readable storage medium can be used to store the program code executed by the sign language gesture recognition method provided in Embodiment 1.

[0221] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in the AR / VR device terminal group in the AR / VR device network, or in any mobile terminal in the mobile terminal group.

[0222] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring a set of image frames during the process of a biological object outputting a sign language action to be recognized; discretizing and encoding the image frames in the set of image frames to obtain discrete features of the sign language action to be recognized, wherein the discrete features are used to characterize the features of the vocabulary to be recognized represented by the sign language action; classifying the discrete features of the sign language action to be recognized to obtain the recognition result of the sign language action to be recognized, wherein the recognition result is used to characterize the category of the vocabulary to be recognized.

[0223] Optionally, the storage medium is also configured to store program code for performing the following steps: discretizing the image frame using a discretization coding model to obtain discrete features of the sign language action to be recognized.

[0224] Optionally, the aforementioned storage medium is further configured to store program code for performing the following steps: extracting features from an image frame using an encoder model to obtain image features of the image frame; discretizing the image features of the image frame to obtain intermediate features corresponding to the image features; obtaining preset features corresponding to the intermediate features from a discretized vocabulary to obtain discrete features of the sign language action to be recognized, wherein the discretized vocabulary stores a preset number of preset features, and the permutations and combinations of different numbers of preset features are used to characterize the vocabulary represented by different sign language actions.

[0225] Optionally, the aforementioned storage medium is further configured to store program code for performing the following steps: the model parameters of the discretization coding model are adjusted based on a first loss value, which is constructed based on a set of training image frames and a set of reconstructed image frames. The set of reconstructed image frames is obtained by reconstructing images from spliced ​​features using a decoder model. The spliced ​​features are obtained by splicing the image features of reference image frames in the set of training image frames with the discrete features of training sign language actions contained in the set of training image frames. The image features of reference image frames are obtained by extracting features from reference image frames using a feature extraction model. The discrete features of training sign language actions are obtained by discretizing and encoding training image frames in the set of training image frames using a discretization coding model.

[0226] Optionally, the aforementioned storage medium is also configured to store program code for performing the following steps: classifying the discrete features of the sign language actions to be recognized using a recognition model to obtain recognition results.

[0227] Optionally, the aforementioned storage medium is further configured to store program code for performing the following steps: adjusting the model parameters of the recognition model based on a second loss value, which is determined based on the recognition result of the sign language action and the received feedback result, which is obtained by modifying the recognition result after the recognition result is output.

[0228] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining a set of training image frames, wherein the set of training image frames contains training sign language actions output by the same biological object; using a discretization coding model to discretize and encode the training image frames in the set of training image frames to obtain discrete features of the training sign language actions; using a decoder model to reconstruct the images from the discrete features of the training sign language actions to obtain a set of reconstructed image frames; and adjusting the model parameters of the discretization coding model and the decoder model based on the set of training image frames and the set of reconstructed image frames.

[0229] Optionally, the storage medium is further configured to store program code for performing the following steps: extracting features from reference image frames in the training image frame set using a feature extraction model to obtain image features of the reference image frames; concatenating the image features of the reference image frames with discrete features of training sign language actions to obtain concatenated features; and reconstructing images from the concatenated features using a decoder model to obtain a reconstructed image frame set.

[0230] Optionally, the aforementioned storage medium is further configured to store program code for performing the following steps: extracting features from training image frames using an encoder model to obtain image features of the training image frames; discretizing the image features of the training image frames to obtain intermediate features corresponding to the image features; obtaining preset features corresponding to the intermediate features from a discretized vocabulary to obtain discrete features of the training sign language actions, wherein the discretized vocabulary stores a preset number of preset features, and the permutations and combinations of different numbers of preset features are used to characterize the vocabulary represented by different sign language actions.

[0231] Optionally, the storage medium is also configured to store program code for performing the following steps: constructing the loss value of a preset loss function based on the training image frame set and the reconstructed image frame set; and adjusting the model parameters of the discretized encoding model and the decoder model based on the loss value.

[0232] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: in response to an input command applied to the operation interface, displaying a set of image frames on the operation interface, wherein the set of image frames is acquired during the process of the biological object outputting a sign language action to be recognized; in response to a recognition command applied to the operation interface, displaying the recognition result of the sign language action to be recognized on the operation interface, wherein the recognition result is used to characterize the category of the word to be recognized, and the recognition result is obtained by classifying the discrete features of the sign language action to be recognized, wherein the discrete features of the sign language action to be recognized are obtained by discretizing and encoding the image frames in the set of image frames, and the discrete features are used to characterize the features of the word to be recognized represented by the sign language action to be recognized.

[0233] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: displaying a set of image frames on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the set of image frames is acquired during the process of a biological object outputting a sign language action to be recognized; discretizing and encoding the image frames in the set of image frames to obtain discrete features of the sign language action to be recognized, wherein the discrete features are used to characterize the features of the vocabulary to be recognized represented by the sign language action; classifying the discrete features of the sign language action to be recognized to obtain the recognition result of the sign language action to be recognized, wherein the recognition result is used to characterize the category of the vocabulary to be recognized; and driving the VR device or AR device to render and display the recognition result of the sign language action to be recognized.

[0234] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining an image frame set by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter being the image frame set, the image frame set being acquired during the process of a biological object outputting a sign language action to be recognized; discretizing and encoding the image frames in the image frame set to obtain discrete features of the sign language action to be recognized, wherein the discrete features are used to characterize the features of the vocabulary to be recognized represented by the sign language action; classifying the discrete features of the sign language action to be recognized to obtain a recognition result of the sign language action to be recognized, wherein the recognition result is used to characterize the category of the vocabulary to be recognized; and outputting the recognition result of the sign language action to be recognized by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter being the recognition result of the sign language action to be recognized.

[0235] In this embodiment, a set of image frames is first acquired during the process of a biological object outputting a sign language action to be recognized. The image frames in the set are then discretized and encoded to obtain discrete features of the sign language action to be recognized. These discrete features characterize the features of the vocabulary represented by the sign language action. The discrete features of the sign language action are then classified to obtain the recognition result. This recognition result characterizes the category of the vocabulary to be recognized, thereby improving the accuracy of the sign language action recognition result. It is noteworthy that discretizing and encoding the image frames in the set to obtain discrete features of the sign language action allows for better modeling of features at the granularity of sign language vocabulary, resulting in more accurate recognition of the sign language action and thus solving the technical problem of low recognition accuracy of sign language actions in related technologies.

[0236] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0237] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0238] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0239] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0240] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0241] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0242] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method of recognizing a sign language action, characterized by, include: A set of image frames is collected during the process of a biological object outputting a sign language action to be recognized, wherein the sign language action to be recognized is the sign language action expressed by the biological object through its limbs; Discretize and encode the image frames in the image frame set to obtain discrete features of the sign language action to be recognized. The discrete features are used to characterize the features of the vocabulary to be recognized represented by the sign language action. The discrete features of the sign language action to be recognized are obtained from preset features corresponding to intermediate features obtained from the discretized vocabulary. The intermediate features are obtained by discretizing the image features of the image frames. The intermediate features are discrete index sequences. Different indices in the index sequence represent different sign language actions. The preset features are word features and / or sentence features. The discrete features of the sign language gesture to be identified are classified to obtain the recognition result of the sign language gesture to be identified, wherein the recognition result is used to characterize the category of the word to be identified.

2. The method of claim 1, wherein, Discretize and encode the image frames in the image frame set to obtain discrete features of the sign language action to be recognized, including: The image frame is discretized using a discretization coding model to obtain the discrete features of the sign language action to be recognized.

3. The method of claim 2, wherein, The discretization coding model includes an encoder model and a discretization vocabulary. The image frames are discretized using the discretization coding model to obtain discrete features of the sign language actions to be recognized, including: The image frame is used to extract features from the image frame using the encoder model to obtain the image features of the image frame; Discretize the image features of the image frame to obtain intermediate features corresponding to the image features; The discrete features of the sign language action to be identified are obtained by retrieving the preset features corresponding to the intermediate features from the discretized vocabulary. The discretized vocabulary stores a preset number of preset features, and the permutations and combinations of different numbers of preset features are used to characterize the vocabulary represented by different sign language actions.

4. The method according to claim 2, characterized in that, The model parameters of the discretization coding model are adjusted based on a first loss value, which is constructed based on a set of training image frames and a set of reconstructed image frames. The set of reconstructed image frames is obtained by reconstructing images using spliced ​​features using a decoder model. The spliced ​​features are obtained by splicing the image features of reference image frames in the set of training image frames with the discrete features of training sign language actions contained in the set of training image frames. The image features of the reference image frames are obtained by extracting features from the reference image frames using a feature extraction model. The discrete features of training sign language actions are obtained by discretizing and encoding the training image frames in the set of training image frames using the discretization coding model.

5. The method according to claim 1, characterized in that, The discrete features of the sign language gesture to be recognized are classified to obtain the recognition result of the sign language gesture to be recognized, including: The discrete features of the sign language gestures to be recognized are classified using a recognition model to obtain the recognition result.

6. The method according to claim 5, characterized in that, The model parameters of the recognition model are adjusted based on a second loss value, which is determined based on the recognition result of the sign language action and the received feedback result. The feedback result is obtained by modifying the recognition result after the recognition result is output.

7. A training method for a discretized coding model, characterized in that, include: Obtain a set of training image frames, wherein the set of training image frames contains training sign language actions output by the same biological object, and the training sign language actions are training sign language actions corresponding to at least one word; The training image frames in the training image frame set are discretized using a discretization coding model to obtain discrete features of the training sign language actions. The discrete features of the training sign language actions are obtained from preset features corresponding to intermediate features obtained from a discretized vocabulary. The intermediate features are obtained by discretizing the image features of the training image frames. The intermediate features are discrete index sequences. Different indices in the index sequence represent different sign language actions. The preset features are word features and / or sentence features. The discrete features of the trained sign language actions are used to reconstruct images using a decoder model, resulting in a set of reconstructed image frames. The model parameters of the discretized encoding model and the decoder model are adjusted based on the training image frame set and the reconstructed image frame set.

8. The method according to claim 7, characterized in that, Using a decoder model, discrete features of the trained sign language actions are reconstructed into images, resulting in a set of reconstructed image frames, including: The feature extraction model is used to extract features from the reference image frames in the training image frame set to obtain the image features of the reference image frames; The image features of the reference image frame are concatenated with the discrete features of the training sign language movements to obtain the concatenated features; The image is reconstructed using the stitched features by the decoder model to obtain the set of reconstructed image frames.

9. The method according to claim 7, characterized in that, The discretization coding model includes an encoder model and a discretization vocabulary. The discretization coding model is used to discretize and encode the training image frames in the training image frame set to obtain the discrete features of the trained sign language actions, including: The encoder model is used to extract features from the training image frames to obtain the image features of the training image frames. The image features of the training image frames are discretized to obtain intermediate features corresponding to the image features; The discrete features of the training sign language actions are obtained by retrieving preset features corresponding to the intermediate features from the discretized vocabulary. The discretized vocabulary stores a preset number of preset features, and the permutations and combinations of different numbers of preset features are used to characterize the vocabulary represented by different sign language actions.

10. A method for recognizing sign language gestures, characterized in that, include: In response to an input command applied to the operation interface, a set of image frames is displayed on the operation interface, wherein the set of image frames is acquired during the process of a biological object outputting a sign language action to be recognized, and the sign language action to be recognized is a sign language action corresponding to at least one word. In response to a recognition command applied to the operation interface, the recognition result of the sign language action to be recognized is displayed on the operation interface. The recognition result is used to characterize the category of the word to be recognized. The recognition result is obtained by classifying the discrete features of the sign language action to be recognized. The discrete features of the sign language action to be recognized are obtained by discretizing and encoding the image frames in the image frame set. The discrete features are used to characterize the features of the word to be recognized represented by the sign language action. The discrete features of the sign language action to be recognized are obtained from preset features corresponding to intermediate features obtained from a discretized vocabulary. The intermediate features are obtained by discretizing the image features of the image frames. The intermediate features are discrete index sequences, where different indices represent different sign language actions. The preset features are word features and / or sentence features.

11. A method for recognizing sign language gestures, characterized in that, include: A set of image frames is displayed on the screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the set of image frames is collected during the process of a biological object outputting a sign language action to be recognized, and the sign language action to be recognized is a sign language action corresponding to at least one word. Discretize and encode the image frames in the image frame set to obtain discrete features of the sign language action to be recognized, wherein the discrete features are used to characterize the features of the vocabulary to be recognized represented by the sign language action; The discrete features of the sign language action to be identified are classified to obtain the recognition result of the sign language action to be identified. The recognition result is used to characterize the category of the word to be identified. The discrete features of the sign language action to be identified are obtained from preset features corresponding to intermediate features obtained from a discretized vocabulary. The intermediate features are obtained by discretizing the image features of the image frame. The intermediate features are discrete index sequences. Different indices in the index sequence represent different sign language actions. The preset features are word features and / or sentence features. The VR device or AR device is driven to render and display the recognition result of the sign language gesture to be recognized.

12. A method for recognizing sign language gestures, characterized in that, include: A set of image frames is obtained by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the set of image frames, the set of image frames is collected during the process of a biological object outputting a sign language action to be recognized, and the sign language action to be recognized is a sign language action corresponding to at least one word. Discretize and encode the image frames in the image frame set to obtain discrete features of the sign language action to be recognized. The discrete features are used to characterize the features of the vocabulary to be recognized represented by the sign language action. The discrete features of the sign language action to be recognized are obtained from preset features corresponding to intermediate features obtained from the discretized vocabulary. The intermediate features are obtained by discretizing the image features of the image frames. The intermediate features are discrete index sequences. Different indices in the index sequence represent different sign language actions. The preset features are word features and / or sentence features. The discrete features of the sign language action to be identified are classified to obtain the recognition result of the sign language action to be identified, wherein the recognition result is used to characterize the category of the word to be identified; The recognition result of the sign language action to be recognized is output by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the recognition result of the sign language action to be recognized.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 12.

14. An electronic device, characterized in that, include: The acquisition device is used to acquire a set of image frames during the process of a biological object outputting a sign language action to be recognized; A processor that runs a program, wherein the program, when running, performs the method of any one of claims 1 to 12 on data output from the acquisition device.

Images