Sign language recognition model training method and device and readable storage medium
By optimizing the sign language recognition model with double loss value, the problem of excessive gradient and overfitting caused by a single loss value is solved, and the stability of the model and the accuracy of the recognition results are improved.
Patent Information
- Application Number
- CN202411203872.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-19
AI Technical Summary
The existing sign language recognition model training methods are too small in the backpropagation process, which makes it difficult for the network to train, and may overfit specific data distributions or scenarios, resulting in poor stability of the recognition results.
The dual loss value optimization method is used to calculate the loss values of the hidden layer and the output layer and perform weighted summing to optimize the model to ensure that the parameter update amplitude of the model is appropriate during the training process and reduce overfitting.
The overall data learning ability of the sign language recognition model is improved, the stability and accuracy of the recognition results are enhanced, and the overfitting problem caused by the optimization of single loss value is avoided.
Smart Images

Figure CN118982873A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of sign language recognition, and in particular to a sign language recognition model training method, device and readable storage medium. Background Art
[0002] Sign language is an important way for the deaf-mute to obtain and express information. The deaf-mute usually communicate with others through sign language, which requires both parties to understand sign language. However, it takes a lot of time and energy for normal people to learn sign language, so most normal people have low sign language skills and find it difficult to communicate with the deaf-mute. Therefore, intelligent sign language recognition is of great significance to the communication of the deaf-mute.
[0003] At present, sign language recognition technology has been developed on the basis of deep neural networks, and is mainly divided into traditional machine learning methods and deep learning methods. Traditional sign language recognition methods rely on artificially designed features, which require repeated verification, and the preprocessing process is cumbersome and inefficient. The development of deep learning technology has greatly promoted the development of sign language recognition technology. Sign language recognition methods based on deep learning require the sign language recognition model to be trained in advance. However, the current model training method mainly uses the output of the last network to calculate the function value of the loss function (hereinafter referred to as the loss value), and optimizes the model based on this loss value. However, this single loss value is often the case in the back propagation process. The network layer closer to the input end will not easily learn the characteristics of the data because the gradient is too small, and the parameter update amplitude is small, which makes the network difficult to train. In addition, this single loss value may overfit certain specific data distributions or scenarios, which ultimately leads to poor stability of the sign language recognition results when sign language recognition is performed based on the trained sign language recognition model.
[0004] Therefore, how to improve the stability of sign language recognition results is a technical problem that needs to be solved urgently in this technical field. Summary of the invention
[0005] The main purpose of this application is to provide a sign language recognition model training method, device and readable storage medium, aiming to solve the technical problem of how to improve the stability of sign language recognition results.
[0006] To achieve the above objectives, the present application provides a sign language recognition model training method, the sign language recognition model training method comprising:
[0007] Acquire a training data set and a sign language recognition model to be trained, wherein the training data set includes sign language data and result labels corresponding to the sign language data;
[0008] Inputting the sign language data into the sign language recognition model to obtain a first prediction result and a second prediction result, wherein the first prediction result is an output of a hidden layer in the sign language recognition model, and the second prediction result is an output of an output layer in the sign language recognition model;
[0009] Calculate a first loss value based on the first prediction result and the result label, and calculate a second loss value based on the second prediction result and the result label;
[0010] The sign language recognition model is optimized based on the first loss value and the second loss value, and the step of inputting the sign language data into the sign language recognition model to obtain the first prediction result and the second prediction result is returned to execute until the preset training end condition is met, thereby obtaining the trained sign language recognition model.
[0011] In one embodiment, the sign language recognition model includes at least one encoder stack, the first prediction result is an output of each encoder stack, and the step of calculating the first loss value based on the first prediction result and the result label includes:
[0012] For each of the first prediction results, a first loss value corresponding to the first prediction result is calculated based on the first prediction result and the result label.
[0013] In one embodiment, the step of optimizing the sign language recognition model based on the first loss value and the second loss value includes:
[0014] Calculating a target loss value by performing a weighted summation on each of the first loss value and the second loss value;
[0015] The sign language recognition model is optimized based on the target loss value.
[0016] In one embodiment, the step of calculating the target loss value by weighted summing the first loss values and the second loss values comprises:
[0017] respectively setting the loss weights corresponding to the first loss values and the second loss values;
[0018] The target loss value is calculated by performing weighted summation of the first loss values and the second loss values based on the loss weights.
[0019] In one embodiment, the loss weight corresponding to the second loss value is 0.5, the loss weight corresponding to each of the first loss values is greater than or equal to 0.1 and less than or equal to 0.4, and the loss weight corresponding to each of the first loss values is iteratively optimized during the model training process.
[0020] In one embodiment, the step of calculating a first loss value based on the first prediction result and the result label includes:
[0021] Inputting the first prediction result into a preset result conversion head to obtain a third prediction result, wherein the preset result conversion head includes a pooling layer, a fully connected layer and a softmax function connected in sequence;
[0022] The third prediction result and the result label are input into a preset loss function to obtain a first loss value, wherein the preset loss function includes a cross entropy loss function.
[0023] In one embodiment, after the step of obtaining the trained sign language recognition model, the method further includes:
[0024] Performing model quantization processing on the trained sign language recognition model to obtain a lightweight sign language recognition model;
[0025] The lightweight sign language recognition model is deployed to a preset edge device so that the preset edge device performs sign language recognition based on the lightweight sign language recognition model, wherein the preset edge device includes an extended reality device.
[0026] In one embodiment, the step of performing model quantization processing on the trained sign language recognition model to obtain a lightweight sign language recognition model includes:
[0027] Converting the trained sign language recognition model into a preset standard format, wherein the preset standard format includes an ONNX format;
[0028] The sign language recognition model converted into a preset standard format is quantized to obtain a lightweight sign language recognition model.
[0029] In addition, to achieve the above-mentioned purpose, the present application also provides a sign language recognition model training device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the sign language recognition model training method as described above.
[0030] In addition, to achieve the above-mentioned purpose, the present application also provides a readable storage medium, which is a computer-readable storage medium, and a program for implementing the sign language recognition model training method is stored on the computer-readable storage medium. The program for implementing the sign language recognition model training method is executed by a processor to implement the steps of the sign language recognition model training method as described above.
[0031] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned sign language recognition model training method when executed by a processor.
[0032] One or more technical solutions proposed in this application have at least the following technical effects:
[0033] A training data set and a sign language recognition model to be trained are obtained, wherein the training data set includes sign language data and result labels corresponding to the sign language data; the sign language data is input into the sign language recognition model to obtain a first prediction result and a second prediction result, wherein the first prediction result is the output of a hidden layer in the sign language recognition model, and the second prediction result is the output of an output layer in the sign language recognition model; a first loss value is calculated based on the first prediction result and the result label, and a second loss value is calculated based on the second prediction result and the result label; the sign language recognition model is optimized based on the first loss value and the second loss value, and the step of inputting the sign language data into the sign language recognition model to obtain a first prediction result and a second prediction result is returned to execute, until a preset training end condition is met, and a trained sign language recognition model is obtained. In this way, the embodiment of the present application calculates a first loss value based on the first prediction result output by the encoder in the model, and calculates a second loss value based on the second prediction result output by the prediction head in the model, and optimizes the training of the sign language recognition model based on the first loss value and the second loss value. It can be understood that the output of the hidden layer is the intermediate output of the model, and the output of the output layer is the final output of the model. Therefore, the loss value is calculated based on the final output of the model, and the loss value is also calculated based on the intermediate output of the model, and the model is optimized based on these two loss values, rather than just optimizing the model after calculating the loss value based on the final output of the model. The first loss value calculated based on the intermediate output of the model can characterize the learning situation of the shallow network, and based on the second loss value After optimizing the model with the first loss value, the parameter update gradient of the network layer close to the input end is not too small, and the parameter update amplitude is increased accordingly. The shallow network in the model can better learn the characteristic distribution of the data. At the same time, the final output calculation of the model improves the second loss value to characterize the learning situation of the deep network. After optimizing the model based on the second loss value, the deep network in the model can better learn the characteristic distribution of the data. In this way, the overall network of the model has better data learning ability, that is, the overall data learning ability of the model is improved, and the phenomenon of overfitting certain specific data distributions or scenarios is reduced, so that the recognition results of the trained sign language recognition model have higher stability. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0036] Figure 1 This is a flow chart of the first embodiment of the sign language recognition model training method of the present application;
[0037] Figure 2 A schematic diagram of the loss function design involved in an embodiment of the sign language recognition model training method of the present application;
[0038] Figure 3 This is a schematic diagram of the structure of a sign language recognition model involved in an embodiment of a sign language recognition model training method of the present application;
[0039] Figure 4 Another schematic diagram of the sign language recognition model structure involved in an embodiment of the sign language recognition model training method of the present application;
[0040] Figure 5 Schematic diagram of the equipment structure of the hardware operating environment involved in the sign language recognition model training device in the embodiment of the present application.
[0041] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0042] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.
[0043] At present, loss function plays a very important role in the training process of sign language recognition models. Choosing a suitable loss function can greatly affect the training effect and recognition performance of the model. In the traditional sign language recognition model training method, the model mainly uses the output of the last network, and uses the cross entropy loss function as the loss function of the overall model for calculation according to the task category. However, during the back propagation process, this single loss function often makes it difficult to learn the characteristics of the data in the network layer closer to the input end because the gradient is too small, and the parameter update amplitude is small, making the network difficult to train. At the same time, this single loss function may overfit certain specific data distributions or scenarios, and fail to take into account other important performance indicators, which ultimately leads to poor stability of sign language recognition results.
[0044] Based on this, the main solution of the present application is: obtain a training data set and a sign language recognition model to be trained, wherein the training data set includes sign language data and result labels corresponding to the sign language data; input the sign language data into the sign language recognition model to obtain a first prediction result and a second prediction result, wherein the first prediction result is the output of a hidden layer in the sign language recognition model, and the second prediction result is the output of an output layer in the sign language recognition model; calculate a first loss value based on the first prediction result and the result label, and calculate a second loss value based on the second prediction result and the result label; optimize the sign language recognition model based on the first loss value and the second loss value, and return to execute the step of inputting the sign language data into the sign language recognition model to obtain a first prediction result and a second prediction result, until the preset training end condition is met, and a trained sign language recognition model is obtained.
[0045] The present application calculates a first loss value based on a first prediction result output by an encoder in the model, and calculates a second loss value based on a second prediction result output by a prediction head in the model, and optimizes and trains a sign language recognition model based on the first loss value and the second loss value. It can be understood that the output of the hidden layer is an intermediate output of the model, and the output of the output layer is the final output of the model. Thus, the loss value is calculated based on the final output of the model, and the loss value is also calculated based on the intermediate output of the model. The model is optimized based on these two loss values, rather than just optimizing the model after calculating the loss value based on the final output of the model. The first loss value calculated based on the intermediate output of the model can characterize the learning situation of the shallow network. Based on the first loss value After optimizing the model based on the second loss value, the parameter update gradient of the network layer close to the input end is not too small, and the parameter update amplitude increases accordingly. The shallow network in the model can better learn the characteristic distribution of the data. At the same time, the final output calculation of the model obtains an improved second loss value that can characterize the learning situation of the deep network. After optimizing the model based on the second loss value, the deep network in the model can better learn the characteristic distribution of the data. In this way, the overall network of the model has better data learning ability, that is, it improves the overall data learning ability of the model, reduces the phenomenon of overfitting certain specific data distributions or scenarios, and thus makes the recognition results of the trained sign language recognition model have higher stability.
[0046] It should be noted that the execution subject of each embodiment of the sign language recognition model training method of the present application can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a sign language recognition model training device that can realize the above functions. The embodiments of the sign language recognition model training method of the present application do not make specific restrictions on this.
[0047] Based on this, this application proposes a sign language recognition model training method of the first embodiment, please refer to Figure 1 The sign language recognition model training method includes steps S10 to S50:
[0048] Step S10, obtaining a training data set and a sign language recognition model to be trained, wherein the training data set includes sign language data and result labels corresponding to the sign language data;
[0049] The training data set includes sign language data and result labels corresponding to the sign language data. Specifically, relevant personnel can collect multiple sign language data (such as videos and images of sign language), and mark the result labels corresponding to each sign language data, so as to obtain a training data set for training the sign language recognition model.
[0050] The sign language recognition model can be any model built in advance, such as a deep learning model. Considering that the Transformer model (deep learning model based on self-attention mechanism) has achieved significant breakthroughs in processing long sequence tasks compared to traditional recurrent neural networks, it introduces a self-attention mechanism, which enables the model to capture global dependencies and disperse information into multiple subspaces through multi-head attention technology, thereby significantly enhancing the expression and understanding capabilities of the model. The architectural advantage of the Transformer model lies in its equal treatment of all elements in the sequence, which enables the model to recognize and understand the rich semantic relationships between the elements in the sequence, especially in understanding the relationship between distant sequence elements. Based on this, in the present embodiment, the sign language recognition model is specifically a Transformer model (deep learning model based on self-attention mechanism).
[0051] Based on this, the various embodiments of the sign language recognition model training method of the present application are explained and illustrated by taking the sign language recognition model as the Transformer model.
[0052] Step S20, inputting the sign language data into the sign language recognition model to obtain a first prediction result and a second prediction result, wherein the first prediction result is the output of a hidden layer in the sign language recognition model, and the second prediction result is the output of an output layer in the sign language recognition model;
[0053] It can be understood that in the Transformer model, the hidden layer usually includes at least one encoder stack, which is composed of at least one encoder connected in sequence, and the encoder is a component composed of multiple self-attention layers and feedforward neural network layers, which is used to process sequence data and generate representations rich in contextual information. Therefore, the output of the hidden layer can specifically be the output of the encoder stack.
[0054] It can be understood that in the Transformer model, the output layer is usually included in the prediction head (also called the decoder head or output head), which is the last layer of the prediction head responsible for generating the final output. Therefore, the output of the prediction head is consistent with the output of the output layer, and the output of the output layer can also be understood as the output of the prediction head.
[0055] Based on this, the various embodiments of the sign language recognition model training method of the present application are explained and illustrated with the first prediction result being the output of each encoder stack in the Transformer model and the second prediction result being the output of the prediction head in the Transformer model.
[0056] Specifically, the output of the encoder stack may be the output of the last layer of encoder in the encoder stack.
[0057] Step S30, calculating a first loss value based on the first prediction result and the result label, and calculating a second loss value based on the second prediction result and the result label;
[0058] The first loss value is calculated based on the first prediction result and the result label. Specifically, a loss function (hereinafter referred to as the first preset loss function) can be set in advance, and the first loss value is calculated based on the first preset loss function, the first prediction result and the result label. Similarly, the second loss value is calculated based on the second prediction result and the result label. Specifically, a loss function (hereinafter referred to as the second preset loss function) can be set in advance, and the second loss value is calculated based on the second preset loss function, the second prediction result and the result label. Specifically, the prediction result and the result label can be input into the second preset loss function to output the loss value.
[0059] It should be noted that the first preset loss function and the second preset loss function may be the same or different, and the first preset loss function and the second preset loss function may be any loss function set in advance, such as a cross loss function, a logarithmic loss function, etc. This embodiment does not impose any specific restrictions on this.
[0060] Step S40: optimizing the sign language recognition model based on the first loss value and the second loss value.
[0061] If the preset training end condition is met, step S50 is executed: obtaining the trained sign language recognition model.
[0062] If the preset training receiving condition is not met, the process returns to step S20.
[0063] The training end condition may be a condition set in advance, such as reaching a predetermined number of iterations, the loss function value being lower than a predetermined threshold, computing resources being exhausted, reaching a time limit, a predetermined accuracy threshold, etc. This embodiment does not impose specific restrictions on this.
[0064] After calculating the first loss value and the second loss value, the sign language recognition model is optimized based on the first loss value and the second loss value. Specifically, the first loss value and the second loss value can be reduced as the optimization target, and the model parameters in the sign language recognition model are updated. For example, the gradient values corresponding to the model parameters in the sign language recognition model can be calculated using a gradient descent algorithm based on the first loss value and the second loss value, and the model parameters in the sign language recognition model are updated according to the gradient values, thereby achieving the purpose of optimizing the sign language recognition model.
[0065] The first prediction result and the second prediction result are respectively the intermediate result and the final result output by the sign language recognition model for the same input sign language data. The first loss value represents the difference between the first prediction result and the result label. The greater the difference, the greater the first loss value. By reducing the first loss value, the problem of excessive attention to the loss of the deep network in the sign language recognition model can be avoided, so that the sign language recognition model can better retain the influence of the shallow network on the model. It can be understood that the shallow network in the model is mainly used to extract features, while the deep network is mainly based on the feature prediction results extracted by the shallow network. Therefore, by reducing the first loss value, the learning of the shallow network in the sign language recognition model can be strengthened, so that the sign language recognition model can more accurately capture the important features in the sign language recognition task, thereby improving the performance of the sign language recognition model and the accuracy of the recognition results.
[0066] It should be noted that the shallow network described in this embodiment may refer to the network that outputs the first prediction result in the sign language recognition model (hereinafter referred to as the target network, such as the encoder that outputs the first prediction result) and the network before the target network, and the deep network may refer to the network after the target network in the sign language recognition model.
[0067] This embodiment calculates a first loss value based on a first prediction result output by the encoder in the model, and calculates a second loss value based on a second prediction result output by the prediction head in the model, and optimizes and trains the sign language recognition model based on the first loss value and the second loss value. It can be understood that the output of the encoder is the intermediate output of the model, and the output of the prediction head is the final output of the model. Therefore, the loss value is calculated based on the final output of the model, and the loss value is also calculated based on the intermediate output of the model, and the model is optimized based on these two loss values, rather than just optimizing the model after calculating the loss value based on the final output of the model. The first loss value calculated based on the intermediate output of the model can characterize the learning situation of the shallow network. Based on the first loss value After optimizing the model based on the second loss value, the parameter update gradient of the network layer close to the input end is not too small, and the parameter update amplitude increases accordingly. The shallow network in the model can better learn the characteristic distribution of the data. At the same time, the final output calculation of the model obtains an improved second loss value that can characterize the learning situation of the deep network. After optimizing the model based on the second loss value, the deep network in the model can better learn the characteristic distribution of the data. In this way, the overall network of the model has better data learning ability, that is, it improves the overall data learning ability of the model, reduces the phenomenon of overfitting certain specific data distributions or scenarios, and thus makes the recognition results of the trained sign language recognition model have higher stability.
[0068] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated later. On this basis, the sign language recognition model is a Transformer model, the Transformer model includes at least one encoder stack, the first prediction result is the output of each encoder stack, the first prediction result is the output of each encoder stack, and the step of calculating the first loss value based on the first prediction result and the result label includes:
[0069] Step A10: For each of the first prediction results, a first loss value corresponding to the first prediction result is calculated based on the first prediction result and the result label.
[0070] The sign language recognition model includes at least one encoder stack, and the first prediction result is the output of the encoder in each encoder stack. It can be understood that the encoder stack is composed of at least one encoder connected in sequence. Based on this, the first prediction result can specifically be the output of the last layer of encoder in each encoder stack, that is, the output of the encoder stack.
[0071] It should be noted that the output of an encoder in the encoder stack is a first prediction result. Based on this, for each first prediction result, the first loss value corresponding to the first prediction result is calculated, that is, the first loss value corresponds to the first prediction result one-to-one.
[0072] Furthermore, in the sign language recognition model, corresponding encoder stacks can be set to process input data of different modalities. By setting different encoder stacks to process data of different modalities respectively, the characteristics of each data type can be better utilized, the model performance can be improved, the computational cost can be reduced, and the behavior of each encoder stack can be easier to understand and explain to enhance the interpretability and customizability of the model. The image and text encoders can also be tuned separately, which helps to stabilize the training process.
[0073] Exemplarily, assuming that the sign language recognition model includes two encoder stacks, one encoder stack is used to process data in image modality, denoted as the image encoder stack, and one encoder stack is used to process data in text modality (key point coordinates in the image), denoted as the key point encoder stack. Then the output of the last layer of encoder in the image encoder stack is a first prediction result, and the output of the last layer of encoder in the key point encoder stack is a first prediction result.
[0074] In this embodiment, the corresponding first loss value is calculated based on the first prediction result output by each encoder stack, that is, the loss value corresponding to the encoder stack is calculated for each encoder stack. In this way, the learning status of each encoder stack can be characterized by each first loss value, so that after the model is subsequently optimized based on each first loss value, each encoder stack in the model can better learn the characteristic distribution of the data, thereby further improving the overall performance of the sign language recognition model.
[0075] In a possible implementation, the step of optimizing the sign language recognition model based on the first loss value and the second loss value includes:
[0076] Step B10, performing weighted sum calculation on each of the first loss value and the second loss value to obtain a target loss value;
[0077] The target loss value is calculated by weighted summing up the first loss values and the second loss values. It should be noted that the loss weights corresponding to the first loss values and the second loss values may be the same or different, and no specific limitation is made here.
[0078] Step B20: optimizing the sign language recognition model based on the target loss value.
[0079] In this embodiment, the sign language recognition model is optimized based on the target loss value, and the target loss value integrates the loss of the prediction result output by the encoder and the loss of the prediction result output by the prediction head, so that the shallow network and the deep network can make certain contributions in the final loss function. This avoids the problem of excessive attention to the loss of the deep network, and enables the entire model to better retain the impact of the shallow network data features on the model. By strengthening the learning of the shallow network, the model can more accurately capture the important features in the sign language recognition task, and improve the performance and accuracy of the model.
[0080] For example, refer to Figure 2 As shown, assuming that the encoder stack includes an image encoder stack and a key point encoder stack, the image encoder stack is used to process image modal data, including n encoders connected in sequence, divided into [EncoderVD1, Encoder VD2...Encoder VDn], and the key point encoder stack is used to process text modal data of key point coordinates in the image, including n encoders connected in sequence, divided into [Encoder KP1, Encoder KP2...EncoderKPn]. Then the first loss value includes the loss value corresponding to the first prediction result output by the image encoder stack, that is, Figure 2 The loss 1 shown in , corresponds to the loss value of the first prediction result output by the keypoint encoder stack, that is, Figure 2The loss 2 shown in FIG. 2 is obtained by comparing the first loss value with the second loss value ( Figure 2 The target loss value is calculated by weighted summing the loss 3) shown in , and the sign language recognition model is optimized based on this target loss value, which can avoid the problem of excessive attention to the loss of the deep network, so that the entire model can better retain the impact of key points and image data features on the model. By strengthening the learning of key point information and image data, the model can more accurately capture the important features in the sign language recognition task and improve the performance and accuracy of the sign language recognition model.
[0081] In a possible implementation, the step of calculating the target loss value by weighted summing the first loss values and the second loss values includes:
[0082] Step C10, respectively setting the loss weights corresponding to the first loss value and the second loss value;
[0083] Step C20, based on each of the loss weights, weighted sum of each of the first loss values and the second loss values is calculated to obtain a target loss value.
[0084] The loss weights corresponding to each first loss value (hereinafter referred to as the first loss weight) and the loss weights corresponding to the second loss value (hereinafter referred to as the second loss weight) are set respectively. It should be noted that the first loss weight and / or the second loss weight are iteratively optimized during the model training process. Thus, during the sign language recognition model training process, the loss weights of the two groups of branches can be balanced by adjusting the first loss weight and / or the second loss weight. The first loss weight and / or the second loss weight can be adaptively learned based on the features of the trained sign language data, so that the model can better adapt to the feature weights of different parts.
[0085] In one possible implementation, the loss weight corresponding to the second loss value is 0.5, the loss weight corresponding to each of the first loss values is greater than or equal to 0.1 and less than or equal to 0.4, and the loss weight corresponding to each of the first loss values is iteratively optimized during the model training process.
[0086] In this embodiment, the second loss weight is 0.5, and each first loss weight is greater than or equal to 0.1 and less than or equal to 0.4. In this way, on the one hand, the second loss weight is fixed and does not need to be iteratively optimized during model training, which can reduce the number of model parameters. On the other hand, considering that the information learned by the data after passing through the deep network will be better, the second loss weight is set to be greater than each first loss weight. Each first loss weight is greater than or equal to 0.1 and less than or equal to 0.4, which can prevent the loss weight of a branch from being too small. Each first loss weight is iteratively optimized during the model training process. By learning and adjusting each first loss weight, the model can automatically adjust the degree of attention to different parts of the shallow network, so as to better adapt to the feature distribution of the training data set and the needs of the sign language recognition task.
[0087] In a possible implementation, the step of calculating a first loss value based on the first prediction result and the result label includes:
[0088] Step D10, inputting the first prediction result into a preset result conversion head to obtain a third prediction result, wherein the preset result conversion head includes a pooling layer, a fully connected layer and a softmax function connected in sequence;
[0089] It can be understood that the preset result conversion head includes a pooling layer, a fully connected layer and a softmax function connected in sequence, so that the first prediction result is input into the preset result conversion head and then the pooling layer, the fully connected layer and the softmax function in sequence, and the third prediction result in the form of probability distribution is output.
[0090] It should be noted that the third prediction result is the output of the preset result conversion head. If there are multiple first prediction results, each first prediction result is input into the preset result conversion head respectively, and the third prediction result corresponding to each first prediction result is output.
[0091] Step D20, inputting the third prediction result and the result label into a preset loss function to obtain a first loss value, wherein the preset loss function includes a cross entropy loss function.
[0092] The third prediction result and the result label are input into the preset loss function (ie, the first preset loss function) to obtain a first loss value. If there are multiple third prediction results, each third prediction result is input into the preset loss function to obtain a corresponding first loss value.
[0093] Similarly, the second prediction result and the result label are input into a preset loss function (ie, a second pre-loss function) to obtain a second loss value.
[0094] Based on the first embodiment, the second embodiment and / or the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the first embodiment, the second embodiment and the third embodiment can be referred to the above introduction, and will not be repeated later. On this basis, after the step of obtaining the trained sign language recognition model, the method further includes:
[0095] Step E10, performing model quantization processing on the trained sign language recognition model to obtain a lightweight sign language recognition model;
[0096] Model quantization aims to reduce the model's demand for storage space and computing resources. This can be achieved by converting the model's weights and activation values from floating-point representations to lower-precision representations. In deep learning, common floating-point representations are usually represented using 32 bits (single precision) or 16 bits (half precision). Model quantization converts these floating-point parameters into integer or fixed-point representations with lower digits, such as 8-bit integers or 4-bit fixed-point numbers. By reducing the number of bits used to represent values, model quantization can significantly reduce the model's demand for storage space and computing resources, thereby improving the model's reasoning efficiency.
[0097] Based on this, the trained sign language recognition model is quantized to obtain a lightweight sign language recognition model that has low requirements for storage space and computing resources, so that the lightweight sign language recognition model can be deployed in devices with limited storage resources or low computing resources.
[0098] Step E20, deploying the lightweight sign language recognition model to a preset edge device, so that the preset edge device performs sign language recognition based on the lightweight sign language recognition model, wherein the preset edge device includes an extended reality device.
[0099] The extended reality device includes but is not limited to AR (Augmented Reality) glasses, and may also include VR (Virtual Reality) glasses, robots, etc., which are not specifically limited in this embodiment.
[0100] In this embodiment, the sign language recognition model is quantized, and the lightweight sign language recognition model obtained after quantization is deployed to a preset edge device, so as to ensure that sign language recognition can be performed locally on the preset edge device with lower computing power, without uploading local data to other devices with higher computing power (such as the server), thereby ensuring the security of local data on the preset edge device.
[0101] In a possible implementation, the step of performing model quantization processing on the trained sign language recognition model to obtain a lightweight sign language recognition model includes:
[0102] Step F10, converting the trained sign language recognition model into a preset standard format, wherein the preset standard format includes an ONNX format;
[0103] Step F20, quantizing the sign language recognition model converted into a preset standard format to obtain a lightweight sign language recognition model.
[0104] The ONNX format (Open Neural Network Exchange, an open deep learning model exchange format) supports cross-platform deployment. ONNX provides a high-performance inference engine, ONNX Runtime, which can achieve fast and efficient model inference on various hardware platforms. These inference engines are optimized and adjusted for specific hardware, which can make full use of the parallel computing capabilities of the hardware and provide fast inference speed. Based on this, this embodiment converts the sign language recognition model into the ONNX format.
[0105] Based on the first embodiment, the second embodiment and / or the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the first embodiment, the second embodiment and the third embodiment can be referred to the above introduction, and will not be repeated later. On this basis, the sign language recognition model includes an input module, an encoding module, a decoding module and a prediction head connected in sequence.
[0106] The input module is used to embed and position code the input sign language data. It can be understood that the sign language data may include one or more frames of images, and embedding and position coding the sign language data specifically includes embedding and position coding the images in the sign language data.
[0107] Furthermore, after acquiring the sign language data, key frame images in the sign language data can also be extracted, such as by extracting key frame images in the sign language data based on a shot boundary key frame extraction algorithm (Shot Boundary Detection, SBD), retaining key frame images in the sign language data and deleting non-key frame images to improve the image quality in the sign language data. The sign language data retaining the key frame images is subsequently processed.
[0108] Embedding refers to the process of converting input data (such as words, characters, sentences, or any serializable entity) into a fixed-length vector representation. Embedding processing is the basis for the model to understand the input data.
[0109] Position coding refers to adding a piece of information about the position of each word in the sentence, also called "information fragment". Specifically, absolute position coding, relative position coding, etc. can be used, and this embodiment does not impose specific restrictions on this.
[0110] The input vector output by the sign language data after passing through the input module is transmitted to the encoding module, and the input vector is encoded by the encoding module to convert the input vector into a series of feature vectors, that is, the encoding matrix, and the encoding matrix is input into the decoding module.
[0111] Relevant personnel may design the encoding module according to actual needs. In a specific implementation, the encoding module may be an encoder stack composed of one or more encoders connected in sequence.
[0112] The decoding module receives the input of the encoding module, decodes the input to obtain an output sequence, and inputs the output sequence into the prediction head.
[0113] Relevant personnel can design the decoding module according to actual needs. In a specific implementation, the decoding module can be a decoder stack composed of one or more decoders connected in sequence.
[0114] The prediction head in the Transformer model is used to convert the internal representation of the model into the final prediction output.
[0115] Relevant personnel can design a prediction head according to actual needs. In a specific implementation, the prediction head includes a pooling layer, a regularization layer (dropout), a fully connected layer and a softmax function connected in sequence, and finally predicts the output through the softmax function.
[0116] In a possible implementation, the input module includes an image embedding module, a key point embedding module and a position encoding module, and the position encoding module is connected to the image embedding module and the key point embedding module respectively.
[0117] The image embedding module is used to embed the sign language data. Specifically, the embedding process is as follows:
[0118] (1) First, scale the image to a uniform size, such as 224×224×3, and then divide the image into small patches. Each patch size is also the same, such as 16×16×3, and then arrange the small patches;
[0119] (2) Perform an Embedding operation on the arranged patches to obtain an image vector.
[0120] The key point embedding module is used to extract key points in the image (actually extract coordinates). Specifically, the key points of the face and hands in the image can be extracted. The key points of the hand are recorded as hand key points, and the key points of the face are recorded as facial key points. Specifically, a preset number of hand key points and a preset number of facial key points can be extracted. The preset number of facial key points can specifically include eye key points (key points of the glasses part), nose key points (key points of the nose part), mouth key points (key points of the mouth part), contour key points (key points of the face contour part), etc. In a specific implementation, 21 hand key points are extracted from one hand based on the image, and a total of 42 hand key points are extracted from both hands. At the same time, 58 facial key points are extracted. The 58 facial key points specifically include 17 contour key points, 12 eye key points, 9 nose key points and 20 mouth key points.
[0121] It can be understood that a key point can be represented by coordinates, so that each key point is actually represented as a coordinate.
[0122] The position encoding module is used to perform position encoding processing on the outputs of the image embedding module and the key point embedding module respectively.
[0123] In this embodiment, in order to make the model pay more attention to the hand and facial information of the signer in the process of expressing sentences, a key point embedding module is set to extract the key points of the hands and face in the image, thereby improving the performance of the model. At the same time, in order to avoid the influence of noise on the key points, an image embedding module is set to embed the image, thereby fusing the two modalities of image and key point (actually text data in coordinate form) for sign language recognition. The complementary advantages of the two modalities of data are utilized to jointly learn features, thereby further improving the accuracy of the sign language recognition results.
[0124] For example, to help understand the technical concept or technical principle of the sign language recognition model in this embodiment, please refer to Figure 3 As shown, the sign language recognition model specifically includes an input module, an encoding module, a decoding module and a prediction head connected in sequence, wherein the encoding module includes n layers of encoders (Encoder) connected in sequence, the decoding module includes m layers of decoders (Decoder) connected in sequence, and the input module includes an image embedding (Embedding) module and a key point embedding module, and the image embedding module and the key point embedding module are respectively connected to the position encoding module.
[0125] Based on this sign language recognition model, after extracting the key frame images in the sign language data, that is, Figure 3The key frame extraction (KEF) shown in FIG. 1 inputs the extracted key frame image into the input module of the sign language recognition model. The image embedding module and the key point embedding module in the input module embed the image and the key point respectively, and then splices the outputs of the two embeddings. After passing through position encoding, n layers of sequentially connected encoders, m layers of sequentially connected decoders and prediction heads, the final output of the model, i.e., the sign language recognition result, is obtained. It should be noted that the final output of the n layers of sequentially connected encoders is respectively input into each decoder of the m layers of sequentially connected decoders.
[0126] It should be noted that when the sign language model is trained, the training result labels (such as Figure 3 The "business license" shown in the figure, which is the expected output of the model, is embedded and positionally encoded in sequence and then input into the decoding module. The sign language recognition model is trained based on the input of the result label and the input of the sign language data to obtain a trained sign language recognition model.
[0127] It should be noted that the above examples are only used to assist in understanding the present application and do not constitute a limitation on the sign language recognition model structure of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.
[0128] Based on the first embodiment, the second embodiment, the third embodiment and / or the fourth embodiment of the present application, in the fifth embodiment of the present application, the same or similar contents as those in the above-mentioned embodiments 1, 2, 3 and 4 can be referred to the above introduction, and will not be repeated later. On this basis, the encoding module includes an image encoder stack, a key point encoder stack and a fusion module, and the fusion module is connected to the image encoder stack and the key point encoder stack respectively.
[0129] Among them, the image encoder stack consists of one or more encoders connected in sequence, and the key point encoder stack consists of one or more encoders connected in sequence.
[0130] The number of layers of encoders in the key point encoder stack may be the same as or different from the number of layers of encoders in the image encoder stack, and this embodiment does not impose any specific restrictions on this. In a preferred embodiment, the number of layers of encoders in the image encoder is the same as the number of layers of encoders in the key point encoder stack.
[0131] The fusion module is connected to the image encoder stack and the key point encoder stack respectively to fuse the output of the image encoder stack with the output of the key point encoder stack. For the convenience of subsequent elaboration and explanation, the output of the image encoder stack is referred to as the image encoding matrix, the output of the key point encoder stack is referred to as the key point encoding matrix, and the output of the fusion module is referred to as the encoding matrix.
[0132] Furthermore, the image coding matrix and the key point coding matrix may be weightedly fused based on the fusion weights corresponding to the image coding matrix and the key point coding matrix to obtain the coding matrix. For setting the fusion weights, the relevant personnel may set the fusion weights corresponding to the image coding matrix and the key point coding matrix respectively, or may set the fusion weights corresponding to the image coding matrix and the key point coding matrix as model parameters, and optimize the fusion weights corresponding to the image coding matrix and the key point coding matrix during the sign language recognition model training process, and this embodiment does not impose specific restrictions on this.
[0133] Let the fusion weight corresponding to the image coding matrix be p, the image coding matrix be IA, the fusion weight corresponding to the key point coding matrix be q, the key point coding matrix be KA, and the coding matrix be EA, then EA can be expressed by the formula EA=p*IA+q*KA. In a specific implementation, the sum of p and q is one.
[0134] In this embodiment, an image encoder stack and a key point encoder stack are set to encode the image coding vector and the key point coding vector respectively, which is equivalent to using the encoder stack to encode the image data and the text data respectively. As a result, the characteristics of each data type can be better utilized to improve model performance and reduce computing costs. It is also easier to understand and explain the behavior of each encoder stack to enhance the interpretability and customizability of the model, and the image and text encoders can be tuned separately, which helps to stabilize the training process.
[0135] For example, to help understand the technical concept or technical principle of the sign language recognition model training method after combining this embodiment with the first embodiment, the second embodiment and the third embodiment, please refer to Figure 4 As shown, the sign language recognition model specifically includes an input module, an encoding module, a decoding module and a prediction head connected in sequence, wherein the encoding module includes an image encoder stack, a key point encoder stack and a fusion module, the image encoder stack includes n layers of encoders (Encoder VD) connected in sequence, the key point encoder stack includes n layers of encoders (Encoder KP) connected in sequence, the decoding module includes m layers of decoders (Decoder) connected in sequence, the input module includes an image embedding (Embedding) module and a key point embedding module, and the image embedding module and the key point embedding module are respectively connected to the position encoding module.
[0136] The fusion module performs weighted fusion on the outputs of the image encoder stack and the key point encoder stack, and inputs the fusion results into each decoder of the decoder module. Specifically, the fusion weight corresponding to the image encoding matrix output by the image encoder stack is α, and the fusion weight corresponding to the key point encoding matrix output by the key point encoder stack is 1-α. The image encoding matrix and the key point encoding matrix are weightedly fused based on their respective fusion weights.
[0137] During the training of the sign language recognition model, the first loss value corresponding to the first prediction result output by the image encoder stack is calculated ( Figure 4 The loss shown in 1) is denoted as CEloss VD , based on the first prediction result output by the key point encoder stack, the corresponding first loss value is calculated ( Figure 4 The loss shown in 2) is denoted as CEloss KP , based on the second prediction result output by the prediction head, the corresponding second loss value is calculated ( Figure 4 The loss shown in 3) is denoted as CEloss decoder , set CEloss VD The corresponding loss weight is β, CEloss KP The corresponding loss weight is 1 / 2-β, CEloss decoder The corresponding loss weight is 1 / 2. Based on this loss weight setting, the target loss value can be calculated. The target loss value is LOSS, which can be expressed as:
[0138]
[0139] The sign language recognition model is optimized based on the calculated target loss value until the preset training end conditions are met, thereby obtaining a trained sign language recognition model.
[0140] It should be noted that the above examples are only used to assist in understanding the present application and do not constitute a limitation on the sign language recognition model training method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.
[0141] Based on the first embodiment, the second embodiment, the third embodiment, the fourth embodiment and / or the fifth embodiment of the present application, in the sixth embodiment of the present application, the same or similar contents as those in the above-mentioned embodiments 1, 2, 3, 4 and 5 can be referred to the above introduction, and will not be repeated later. On this basis, the prediction head includes a pooling layer, multiple regularization layers and an output layer, the pooling layer is respectively connected to each of the regularization layers, and each of the regularization layers is respectively connected to the output layer.
[0142] The pooling layer is used to aggregate the input sequence. After the sequence in the Transformer model is processed by the multi-layer self-attention mechanism and feedforward neural network in the encoder and decoder, a series of hidden states, that is, the representation vector of each position, are obtained. These representation vectors contain the semantic information of the input sequence. In order to aggregate these vectors into a vector representation of fixed dimension and fixed length, pooling operations are required.
[0143] The pooling layer may be an average pooling layer, a maximum pooling layer, a global pooling layer, etc., and this embodiment does not impose any specific restrictions on this. Considering that compared with other pooling operations, the average pooling operation performs an average operation on the sequence, so the overall characteristics of the sequence can be retained, not only the maximum and minimum characteristics in the sequence are considered, but also the overall trend and distribution are considered. Average pooling can also enhance robustness and reduce overfitting. Based on this, in a preferred embodiment, the pooling layer is specifically an average pooling layer.
[0144] Regularization (Dropout) is a technique that randomly "inactivates" (i.e. temporarily ignores) a portion of neurons in a neural network during training. The inactivated neurons do not participate in the calculation during the current forward propagation and backpropagation process. By randomly "inactivating" a portion of neurons in the network during training, the network's dependence on specific training samples is reduced, thereby improving the model's generalization ability on unseen data. Therefore, in this embodiment, multiple regularization layers (Dropout) are used before the output layer to perform random inactivation processing.
[0145] It should be noted that the discard ratio of each regularization layer may be the same or different, and this embodiment does not impose specific restrictions on this. Considering that different discard ratios can provide different regularization strengths, thereby improving the generalization ability of the model, and applying different discard ratios can increase the diversity of the model, so that the model can be trained under different configurations. This diversity helps the model to have better adaptability to unknown data when used, and different discard ratios can be customized to more flexibly control the training process of the model, which helps to find the best training strategy to balance the accuracy and generalization ability of the model. Based on this, in a preferred embodiment, the discard ratio of each regularization layer is different. Among them, the discard ratio of the regularization layer refers to the proportion of neurons discarded by the regularization layer.
[0146] Relevant personnel can design the output layer according to actual needs. In a specific implementation, the output layer includes a fully connected layer and a softmax function connected in sequence.
[0147] The number of fully connected layers is the same as that of regularized layers, and they are connected one-to-one. The fully connected layer performs feature integration processing on the data input therein to obtain a linear vector and outputs a linear vector. For example, assuming that the i-th fully connected layer is connected to the i-th regularization layer, and the output of the i-th regularization layer is Di, then Di is input to the i-th fully connected layer, and the i-th fully connected layer performs relevant processing on the input Di and outputs a linear vector Li. Among them, the value of i ranges from i to N, and N is the total number of regularization layers or fully connected layers.
[0148] It should be noted that the prediction layer includes a softmax function. After the linear vector is input into the softmax function, the softmax function will convert it into a probability distribution vector. At this time, each probability distribution vector is recorded as a third prediction result, and the second prediction result is a set of third prediction results. After obtaining each third prediction result, the corresponding third loss value can be calculated based on each third prediction result and the result label, and the average loss value of all third loss values is determined as the second loss value.
[0149] In this embodiment, multiple regularization layers are set in the prediction head, and multiple different sub-network models are generated by using the multiple regularization layers, thereby improving the generalization ability of the sign language recognition algorithm model, and further improving the stability of the sign language recognition results.
[0150] In addition, an embodiment of the present application also proposes a sign language recognition model training device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the sign language recognition model training method as described above.
[0151] refer to Figure 5 , which shows a schematic diagram of the structure of a sign language recognition model training device suitable for implementing the embodiment of the present application. The sign language recognition model training device in the embodiment of the present application may also include but is not limited to mobile terminals such as laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The sign language recognition model training device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0152] like Figure 5As shown, the sign language recognition model training device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the sign language recognition model training device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the sign language recognition model training device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a sign language recognition model training device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.
[0153] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0154] The sign language recognition model training device provided in the embodiment of the present application adopts the sign language recognition model training method in the above embodiment, which can solve the technical problem of how to improve the stability of the sign language recognition result. Compared with the prior art, the beneficial effects of the sign language recognition model training device provided in the present application are the same as the beneficial effects of the sign language recognition model training method provided in the above embodiment, and the other technical features in the sign language recognition model training device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0155] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0156] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0157] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also provides a readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the sign language recognition model training method in the above-mentioned embodiment.
[0158] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.
[0159] The computer-readable storage medium may be included in the sign language recognition model training device; or may exist independently without being assembled into the sign language recognition model training device.
[0160] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the sign language recognition model training device, the sign language recognition model training device: obtains a training data set and a sign language recognition model to be trained, wherein the training data set includes sign language data and result labels corresponding to the sign language data; inputs the sign language data into the sign language recognition model to obtain a first prediction result and a second prediction result, wherein the first prediction result is the output of a hidden layer in the sign language recognition model, and the second prediction result is the output of an output layer in the sign language recognition model; calculates a first loss value based on the first prediction result and the result label, and calculates a second loss value based on the second prediction result and the result label; optimizes the sign language recognition model based on the first loss value and the second loss value, and returns to execute the step of inputting the sign language data into the sign language recognition model to obtain a first prediction result and a second prediction result, until a preset training end condition is met, and the trained sign language recognition model is obtained.
[0161] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0162] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0163] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0164] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned sign language recognition model training method, and can solve the technical problem of how to improve the stability of the sign language recognition results. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the sign language recognition model training method provided in the above-mentioned embodiment, and will not be repeated here.
[0165] In addition, an embodiment of the present application also proposes a computer program product, including a sign language recognition model training program, which, when executed by a processor, implements the steps of the sign language recognition model training method described above.
[0166] The specific implementation methods of the computer program product of the present application are basically the same as the embodiments of the above-mentioned sign language recognition model training method, and will not be repeated here.
[0167] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0168] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0169] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software sensor, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, including a number of instructions for a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the methods described in each embodiment of the present application.
[0170] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A sign language recognition model training method, characterized in that: The sign language recognition model training method comprises the following steps: Acquire a training data set and a sign language recognition model to be trained, wherein the training data set includes sign language data and result labels corresponding to the sign language data; Inputting the sign language data into the sign language recognition model to obtain a first prediction result and a second prediction result, wherein the first prediction result is an output of a hidden layer in the sign language recognition model, and the second prediction result is an output of an output layer in the sign language recognition model; Calculate a first loss value based on the first prediction result and the result label, and calculate a second loss value based on the second prediction result and the result label; The sign language recognition model is optimized based on the first loss value and the second loss value, and the step of inputting the sign language data into the sign language recognition model to obtain the first prediction result and the second prediction result is returned to execute until the preset training end condition is met, thereby obtaining the trained sign language recognition model.
2. The sign language recognition model training method according to claim 1, characterized in that: The sign language recognition model is a Transformer model, the Transformer model includes at least one encoder stack, the first prediction result is the output of each encoder stack, and the step of calculating the first loss value based on the first prediction result and the result label includes: For each of the first prediction results, a first loss value corresponding to the first prediction result is calculated based on the first prediction result and the result label.
3. The sign language recognition model training method according to claim 1, characterized in that: The step of optimizing the sign language recognition model based on the first loss value and the second loss value includes: Calculating a target loss value by performing a weighted summation on each of the first loss value and the second loss value; The sign language recognition model is optimized based on the target loss value.
4. The sign language recognition model training method according to claim 3, characterized in that: The step of calculating the target loss value by weighted summing the first loss values and the second loss values comprises: respectively setting the loss weights corresponding to the first loss values and the second loss values; The target loss value is calculated by performing weighted summation of the first loss values and the second loss values based on the loss weights.
5. The sign language recognition model training method according to claim 4, characterized in that: The loss weight corresponding to the second loss value is 0.5, and the loss weights corresponding to each of the first loss values are greater than or equal to 0.1 and less than or equal to 0.
4. The loss weights corresponding to each of the first loss values are iteratively optimized during the model training process.
6. The sign language recognition model training method according to claim 1, characterized in that: The step of calculating a first loss value based on the first prediction result and the result label includes: Inputting the first prediction result into a preset result conversion head to obtain a third prediction result, wherein the preset result conversion head includes a pooling layer, a fully connected layer and a softmax function connected in sequence; The third prediction result and the result label are input into a preset loss function to obtain a first loss value, wherein the preset loss function includes a cross entropy loss function.
7. The sign language recognition model training method according to any one of claims 1 to 6, characterized in that: After the step of obtaining the trained sign language recognition model, the method further includes: Performing model quantization processing on the trained sign language recognition model to obtain a lightweight sign language recognition model; The lightweight sign language recognition model is deployed to a preset edge device so that the preset edge device performs sign language recognition based on the lightweight sign language recognition model, wherein the preset edge device includes an extended reality device.
8. The sign language recognition model training method according to claim 7, characterized in that: The step of performing model quantization processing on the trained sign language recognition model to obtain a lightweight sign language recognition model includes: Converting the trained sign language recognition model into a preset standard format, wherein the preset standard format includes an ONNX format; The sign language recognition model converted into a preset standard format is quantized to obtain a lightweight sign language recognition model.
9. A sign language recognition model training device, characterized in that: The sign language recognition model training device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the sign language recognition model training method as described in any one of claims 1 to 8.
10. A readable storage medium, characterized in that: The readable storage medium is a computer-readable storage medium, and a computer program is stored on the readable storage medium. When the computer program is executed by a processor, the steps of the sign language recognition model training method according to any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Sign language recognition method and device, computer device and storage medium
CN110363077A
Sign language recognition model training method and device
CN115601831A
Sign language recognition method and device
CN117037263A
On-body sensor system and method for automatic interpretation of visual body signals
US20230358848A1
