A sign language emotion classification method and system based on machine learning

Abstract

The present invention discloses a sign language emotion classification method and system based on machine learning. First, a frame extraction is performed on a sign language video to obtain an image set, and then the gesture features are recognized to obtain text. Based on the text, the prediction of the first emotion label and the analysis of the text semantic features are performed. Among them, the first emotion label includes various categories of emotion types, and each emotion type also includes multiple subcategories of specific emotion words. By calculating the matching degree between the emotion words in the first emotion label and the text, the second emotion label is determined. The sign language emotion classification is more accurate, which significantly improves the richness of the emotion description and the accuracy of the classification, and meets the needs of diversified scenarios. Compared with the existing technology that can only perform simple emotion classification (such as positive, negative or neutral), the present invention combines image recognition, text recognition and semantic analysis technology, and uses the matching degree calculation of emotion words and text semantic feature vectors to achieve accurate division of emotion labels.

Description

Technical Field

[0001] The present invention relates to the technical field of sign language animation, and in particular to a sign language emotion classification method based on machine learning and a system thereof. Background Art

[0002] Sign language is a body language that uses gestures to express information. The shape, position, direction and movement trajectory of the hand can be expressed as a sign language word. Sign language recognition is not only the main means of communication for the hearing-impaired, but effective sign language recognition technology will also alleviate the distress caused by communication difficulties for the hearing-impaired. Therefore, sign language recognition technology has important research significance and has been widely used in human-computer interaction, computer games, health care and other fields.

[0003] However, the existing technology for the emotions implied in sign language is still limited to the interpretation of gesture expression, and can only achieve simple emotion classification, such as positive, negative or neutral, and it is difficult to clearly and accurately express the user's true emotions.

[0004] Therefore, it is necessary to provide a sign language emotion classification method and system based on machine learning to solve the above technical problems. Summary of the invention

[0005] The present invention overcomes the deficiencies of the prior art and provides a sign language emotion classification method and system based on machine learning.

[0006] To achieve the above object, the technical solution adopted by the present invention is: a sign language emotion classification method based on machine learning, comprising the following steps:

[0007] Get the original sign language video;

[0008] Converting the acquired original sign language video into an image sequence, performing frame extraction, and obtaining a sign language image set;

[0009] Extracting gesture features from the sign language image set, identifying text corresponding to the gesture features, and the emotional tendency of some paragraphs;

[0010] Match all the feature words in the recognized text and convert them into vector values;

[0011] The processed vector value is input into the sentiment classification model to predict the first sentiment label corresponding to the text;

[0012] The processed vector value is input into the semantic analysis model to perform deep semantic analysis of the text;

[0013] Combined with the semantic analysis results, the matching degree between the sentiment words in the first sentiment tag and the text is calculated, and the second sentiment tag is determined based on the matching degree maximization strategy.

[0014] In a preferred embodiment of the present invention, when performing frame extraction on the original sign language video, FFmpeg or OpenCV is used to extract several frames of images of the original sign language video, including the first frame of the original sign language video, and then a frame of image is extracted every 20-30 frames.

[0015] In a preferred embodiment of the present invention, gesture features are extracted and recognized from the sign language image set using a pre-trained deep learning model;

[0016] The text interpretation obtained by the deep learning model recognizing the gesture features is the interpretation of all gesture expressions in the sign language image set, including texts of various parts of speech.

[0017] In a preferred embodiment of the present invention, the sentiment tendency analysis process includes:

[0018] According to the extracted gesture features, query the corresponding several frames of images;

[0019] Detect key points of gesture features, including: motion nodes of limb joints, ends, bones, and head;

[0020] The position changes of key points between consecutive frame images are calculated, and the optical flow algorithm is used to estimate the speed and direction of gesture movements, identify the acceleration or slowdown of gesture rhythm, and then judge the emotional tendency of these paragraphs.

[0021] In a preferred embodiment of the present invention, a method for converting all feature words in a text into vector values ​​includes:

[0022] Let V be the vocabulary, w j For the words in the vocabulary, further build the vocabulary:

[0023]

[0024] in, For the preprocessed text set, the UniqueWords function is used to extract all unique words;

[0025] Let X BOW or X TF-IDF is the feature matrix of the text, the vectorization process is expressed as:

[0026]

[0027] Among them, Vectorize BOW / TF-IDF The function converts the preprocessed text set T′ into a feature matrix according to the vocabulary V;

[0028] At the same time, let E be the word embedding matrix, e j For word wj The word embedding process is: , where e j Obtained through the pre-trained word embedding model Word2Vec;

[0029] For a single text , whose word embedding sequence is expressed as:

[0030] in, is text The vocabulary index corresponding to the word in;

[0031] Furthermore, the word embedding sequence is padded to obtain a fixed-length input sequence. The padding process is: , where maxlen is the maximum length of the sequence.

[0032] In a preferred embodiment of the present invention, the converted X BOW or X TF-IDF As feature input to the sentiment classification model.

[0033] In a preferred embodiment of the present invention, the first emotion tag includes: a plurality of different emotion types, each emotion type also includes emotion words of different expression levels, namely, the second emotion tag.

[0034] In a preferred embodiment of the present invention, the semantic analysis model is a bidirectional LSTM network structure, and the specific semantic analysis process includes:

[0035] The converted As the input of the LSTM network, set , where a t Represents the input of the LSTM network at time t, specifically , σ in this formula is the activation function sigmoid, is the weight matrix, b a is the bias term, Et is the input embedding vector at time step t;

[0036] Calculate the output of the LSTM network at the previous time step , is the previous moment of the LSTM network To the current time The weight matrix of is the bias term at the current moment, The calculation process is consistent, and the calculation of the latter LSTM is consistent with the former LSTM;

[0037] The hidden layer of the previous LSTM at the last moment is merged with the first dimension of the next LSTM at the last moment: ,in, is the output of the semantic analysis model.

[0038] In a preferred embodiment of the present invention, before calculating the matching degree between the emotion words in the first emotion tag and the text, several emotion words in the first emotion tag are converted into vector representation: let the emotion word be w, and use the conversion tool used when converting text features to convert it into a vector representation .

[0039] In a preferred embodiment of the present invention, the sentiment vocabulary vector and the text semantic feature vector are calculated Degree of match:

[0040]

[0041] in, Representation vector and The dot product of Represents vectors and The modulus, i.e., the length;

[0042] Further, yes Perform a linear transformation to obtain the matching degree value:

[0043] , where T represents the text, and the calculated matching degree value ranges from 0 to 1. When the result is close to 1, it means that the matching degree between the emotional vocabulary in the first emotional tag and the text is high.

[0044] A sign language emotion classification system based on machine learning, executing any one of the sign language emotion classification methods based on machine learning as described above, comprising:

[0045] A video acquisition module is used to acquire original sign language video;

[0046] An image extraction module, used to convert the acquired original sign language video into an image sequence, perform frame extraction, and obtain a sign language image set;

[0047] A text recognition module extracts gesture features from the sign language image set, identifies the text corresponding to the gesture features, and the emotional tendency of some paragraphs;

[0048] The vector conversion module is used to match and convert all the feature words in the recognized text into vector values;

[0049] The sentiment prediction module is used to input the processed vector value into the sentiment classification model to predict the first sentiment label corresponding to the text;

[0050] The semantic analysis module is used to input the processed vector value into the semantic analysis model to perform deep semantic analysis of the text;

[0051] The sentiment classification module is used to combine the semantic analysis results, calculate the matching degree between the sentiment words in the first sentiment tag and the text, and determine the second sentiment tag based on the matching degree maximization strategy.

[0052] In a preferred embodiment of the present invention,.

[0053] The present invention solves the defects existing in the background technology and has the following beneficial effects:

[0054] The present invention proposes a sign language emotion classification method based on machine learning. This method significantly improves the richness of emotion description and the accuracy of classification by carefully classifying emotions into first emotion labels (major categories) and second emotion labels (minor categories), meeting the needs of diverse scenarios. Compared with the existing technology that can only perform simple emotion classification (such as positive, negative or neutral), the present invention combines image recognition, text recognition and semantic analysis technology, and uses the matching degree calculation of emotion vocabulary and text semantic feature vectors to achieve accurate division of emotion labels, thereby greatly improving the level of refinement of emotion classification.

[0055] In the process of identifying gesture features, the rhythm of all gesture features is obtained through key point detection and calculation of key point position changes, and the differences between gesture rhythms are used to divide the original sign language video into normal tone segments and emotional fluctuation segments. Furthermore, the emotional trend and intensity of the metaphor in the emotional fluctuation segment are judged based on the degree of difference in the rhythm of the emotional fluctuation segment relative to the rhythm of the normal tone segment. The present invention not only focuses on gesture features, but also takes into account the different ways of expression of users under the influence of emotions into consideration for emotion classification. This comprehensive analysis method can identify information beyond the gesture features themselves, thereby providing strong evidence for the emotion classification results.

[0056] In the present invention, steps S6 and S7 work together to perform deep correction of the sentiment classification of the sign language text obtained by image recognition. Step S6 first performs preliminary classification, and then step S7 not only refers to this result, but also conducts a more in-depth semantic analysis of the entire sign sentence. It calculates the degree of match between the sentiment vocabulary and the text content in a specific semantic environment, accurately capturing the meaning and usage of the sentiment vocabulary in the context and its close connection with the overall sentiment tendency. More importantly, combined with the semantic analysis results of step S6, step S7 further designates the sentiment vocabulary with the highest degree of match as the second sentiment tag of the corresponding text, thereby accurately subdividing the first sentiment tag. Not only does it improve the accuracy of sentiment classification, but it also significantly enhances the classification efficiency, achieving a deep understanding and accurate conversion of the emotional expression of sign language videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 is a diagram of a sign language emotion classification method according to an embodiment of the present invention;

[0059] Figure 2 is a diagram of a sign language emotion classification system according to an embodiment of the present invention;

[0060] Figure 3 It is an example of deep learning model parameters of an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0063] Application Overview:

[0064] In the prior art, the sign language emotion classification method for sign language videos is mainly based on gesture features. Specifically, first, gesture features (including gesture shape, motion trajectory, speed, etc.) are extracted from sign language animations through image processing and computer vision technology. These features can reflect the emotional expression of gestures. The extracted gesture features are then sent to a classifier, such as a support vector machine (SVM), neural network, etc., for classification. By training these classifiers, gesture features can be mapped to specific emotions, thereby achieving emotion classification.

[0065] The present invention first extracts frames from a sign language video to obtain an image set, then uses a deep learning model to identify all gesture features in the image set to obtain text, and then predicts the first emotion label and analyzes the semantic features of the text based on the text. The first emotion label includes various emotion types, and each emotion type also includes multiple sub-categories of specific emotion words, which are the second emotion labels of the present invention. By calculating the matching degree between the emotion words in the first emotion label and the text, and based on the matching degree maximization strategy, the second emotion label is determined, and the sign language emotion classification is more accurate.

[0066] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0067] Exemplary methods:

[0068] Figure 1 A method for sign language emotion classification based on machine learning according to an embodiment of the present invention is shown, comprising the following steps:

[0069] S1, obtain the original sign language video;

[0070] S2, converting the acquired original sign language video into an image sequence, performing frame extraction, and obtaining a sign language image set;

[0071] S3, extracting gesture features from the sign language image set, identifying the text corresponding to the gesture features, and the emotional tendency of some paragraphs;

[0072] S4, matching all the feature words in the recognized text and converting them into vector values;

[0073] S5, inputting the processed vector value into the sentiment classification model to predict the first sentiment label corresponding to the text;

[0074] S6, inputting the processed vector value into the semantic analysis model to perform deep semantic analysis of the text;

[0075] S7. Based on the semantic analysis result, the matching degree between the sentiment words in the first sentiment tag and the text is calculated, and based on the matching degree maximization strategy, the second sentiment tag is determined.

[0076] The present invention proposes an innovative sign language emotion classification method based on machine learning. This method significantly improves the richness of emotion description and the accuracy of classification by carefully classifying emotions into the first emotion label (major category) and the second emotion label (minor category), meeting the needs of diverse scenarios. Compared with the existing technology that can only perform simple emotion classification (such as positive, negative or neutral), the present invention combines image recognition, text recognition and semantic analysis technology, and uses the matching degree calculation of emotion vocabulary and text semantic feature vector to achieve accurate division of emotion labels, thereby greatly improving the level of refinement of emotion classification.

[0077] Below, each step will be described in detail.

[0078] Step S1, obtaining the original sign language video.

[0079] That is, when a continuous sign language video needs to be sentimentally classified, the original sign language video that needs to be sentimentally classified is first obtained. The original sign language video can be a complete sign language dialogue video or a sign language dialogue video segment.

[0080] Step S2: convert the acquired original sign language video into an image sequence, perform frame extraction, and obtain a sign language image set.

[0081] That is, after obtaining the original sign language video, use FFmpeg or OpenCV to extract several frames of the original sign language video, including the first frame of the original sign language video, and then extract a frame every 20-30 frames. The extracted multiple frames are marked in order and summarized to obtain a sign language image set.

[0082] Furthermore, during the frame extraction process, the image is preprocessed, including resizing, cropping, rotating, denoising, etc., to highlight the gestures in the image to improve the accuracy of subsequent feature extraction.

[0083] Further, the processed frame image is saved as an image file in the sign language image set, such as JPEG or PNG format.

[0084] Through the above steps, the acquired original sign language video can be converted into a series of images, which are used for subsequent feature extraction and emotion classification.

[0085] Step S3: extracting gesture features from the sign language image set, identifying the text corresponding to the gesture features, and the emotional tendency of some paragraphs.

[0086] That is to say, the pre-trained deep learning model is used to extract gesture features from the sign language image set, and then the text interpretations corresponding to these gestures are identified. It is worth noting that the text interpretations obtained by identifying gestures are the interpretations of all gesture expressions in the sign language image set, including various types of text such as subjects, verbs, and nouns.

[0087] Furthermore, the sentiment analysis process of some paragraphs includes:

[0088] S311. Query several corresponding frames of images based on the gesture features extracted by the deep learning model.

[0089] S312: Detect key points of the gesture features, preferably using the MediaPipe tool, and annotate the motion nodes of the limb joints, ends, bones, and head included in the gesture features.

[0090] S313. Calculate the position changes of key points between consecutive frame images, use the optical flow algorithm to estimate the speed and direction of the gesture, identify the acceleration or deceleration of the gesture rhythm, and then determine the emotional tendency of these paragraphs.

[0091] Specifically, the optical flow algorithm is based on the assumption of constant brightness, that is, the brightness of a pixel in consecutive frames is constant. By comparing the brightness changes of the same point in consecutive frames, the motion vector (speed and direction) of the point is estimated. For each pair of consecutive frame images, the position change process of the key point is calculated as follows:

[0092] 1) Use the Sobel filter to calculate the gradient of the image (I x , I y ), which is the rate of change of the image brightness in the x and y directions.

[0093] 2) Calculate the time gradient I t , which is the change in brightness between consecutive frames.

[0094] 3) According to the basic equation of optical flow , solve for the velocity vectors u and v, where u and v represent the velocity in the x and y directions respectively. Calculate the average velocity vector of a gesture as the velocity estimate of the gesture.

[0095] 4) The speed estimate of the gesture actions with similar data and the largest number is used as the standard speed (taking the average value).

[0096] 5) Based on the difference between the estimated speed of the gesture and the standard speed, the degree of change of the gesture rhythm of the corresponding paragraphs is calculated, and then the emotional tendency of these paragraphs is determined.

[0097] It is worth noting that when the mood is low, the rhythm of the sign language movements will slow down. On the contrary, when the mood is high (happy / angry), the rhythm will speed up. The present invention judges the positive / negative (positive, neutral, negative) and intensity (very, slightly) of the emotion according to the rhythm of the gesture movements of the paragraph.

[0098] In the process of identifying gesture features, the rhythm of all gesture features is obtained through key point detection and calculation of key point position changes, and the differences between gesture rhythms are used to divide the original sign language video into normal tone segments and emotional fluctuation segments. Furthermore, the emotional trend and intensity of the metaphor in the emotional fluctuation segment are judged based on the degree of difference in the rhythm of the emotional fluctuation segment relative to the rhythm of the normal tone segment. The present invention not only focuses on gesture features, but also takes into account the different ways of expression of users under the influence of emotions into consideration for emotion classification. This comprehensive analysis method can identify information beyond the gesture features themselves, thereby providing strong evidence for the emotion classification results.

[0099] Further, the training process of the deep learning model:

[0100] S321, image preprocessing: First, image edge detection is performed based on the color mutation, spatial texture and geometric shape characteristics of the gesture image. Then the gesture image is converted into a grayscale image. The processed image size is adjusted to 64×64 and normalized. During the model training process, the original image is translated to achieve data expansion to avoid overfitting problems.

[0101] S322. Model construction: The deep learning model is a deep convolutional neural network (DCNN) structure, which includes 12 deep neural layers, the first 9 of which are convolutional layers and the last 3 are fully connected layers. The output of the last fully connected layer is used to classify the gesture image through the Softmax classifier.

[0102] S323, Model training and optimization: During the network training process, the network parameters are first randomly initialized, and then the loss function is calculated based on the network output label and the real label of the sample, and the network parameters are continuously updated. In the network structure, except for the last layer, all other layers use Leaky ReLU as the activation function, and the mean square error is used as the loss function. The model is trained using the small batch stochastic gradient descent algorithm, and the size of the small batch is 256. Two Dropout layers are added to the hidden layer for regularization, and each layer performs batch normalization on the input of the hidden layer.

[0103] like Figure 3 As shown, it is the deep learning model information of an instance.

[0104] The texts recognized by the above deep learning model are then annotated with the sentiment analysis results in the corresponding paragraphs of the text. Paragraphs without emotional expression are normal tone paragraphs and are not labeled. Paragraphs with emotional tendency are emotional fluctuation paragraphs.

[0105] Step S4: Match all the feature words in the recognized text and convert them into vector values.

[0106] Before conversion, the text recognized from the sign language image set is preprocessed, including text deduplication, denoising, word segmentation, and stop word removal. Specifically:

[0107] 1) Text deduplication: There are repeated words in the preprocessed text, which causes storage redundancy and also increases the computational burden. The present invention adopts a traversal method to effectively remove these duplicate items to reduce redundancy.

[0108] 2) Text denoising: denoising some interfering text or garbled characters in the text to reduce the computational complexity and improve the accuracy and efficiency of subsequent analysis.

[0109] 3) Word segmentation: split the words in the text and assign the corresponding part of speech to each word in combination with the dictionary. In English, each word is separated by a space, so the processing is relatively straightforward; while Chinese word segmentation requires a high degree of comprehensive accuracy of the dictionary, so this article uses the jieba word segmentation in the python language environment.

[0110] 4) Remove stop words. Stop words refer to words that appear frequently in the text but are of no substantial help to data analysis, such as modal particles, adverbs, prepositions, conjunctions, etc. These words are summarized in the stop word list and removed in the preprocessing stage to purify the text data and improve the analysis quality.

[0111] The recognized text is preprocessed through the above steps, making the text data an object that can be operated on in subsequent analysis.

[0112] Furthermore, the preprocessed text is processed into vector values ​​for subsequent operations, specifically:

[0113] Let T be the original text set, t i is a single text in the collection, the preprocessing process is expressed as: , where the Preprocess function includes steps such as word segmentation, stop word removal, and punctuation removal. For Chinese text, the Jieba word segmentation tool is used.

[0114] Let V be the vocabulary, w j For the words in the vocabulary, further build the vocabulary:

[0115]

[0116] in, For the preprocessed text set, the UniqueWords function is used to extract all unique words.

[0117] Let X BOW or X TF-IDF is the feature matrix of the text, the vectorization process is expressed as:

[0118]

[0119] Among them, Vectorize BOW / TF-IDF The function converts the preprocessed text set T′ into a feature matrix according to the vocabulary V.

[0120] At the same time, let E be the word embedding matrix, e j For word w j The word embedding process is: , where e j Obtained through the pre-trained word embedding model Word2Vec.

[0121] For a single text , whose word embedding sequence is expressed as:

[0122] in, is text The vocabulary index corresponding to the word in .

[0123] Furthermore, the word embedding sequence is padded to obtain a fixed-length input sequence. The padding process is: , where maxlen is the maximum length of the sequence.

[0124] Step S5: input the processed vector value into the sentiment classification model to predict the first sentiment labels corresponding to all paragraphs in the text.

[0125] That is to say, the X obtained by step S4 is BOW or X TF-IDF As the feature input of the sentiment classification model, a preliminary prediction is made on the sentiment contained in the text to obtain the first sentiment label.

[0126] The sentiment classification model is a model based on machine learning that uses known data to predict the sentiment of unknown data. Specifically: first, use the machine learning algorithm to train the text data that has been annotated by feature engineering to derive the sentiment classification model; then, input new text data with the same structure into the sentiment classification model to predict the sentiment type of the new text; finally, compare the true label of the new text data with the predicted value label to derive the accuracy of the label in the data set.

[0127] The present invention preferably uses a text sentiment classification method based on SVM, comprising the following steps:

[0128] S51, the X obtained by the transformation in step S4 BOW or X TF-IDF As feature input of support vector machine.

[0129] S51. Use the trained support vector machine to make a preliminary prediction on the sentiment contained in the text to obtain a first sentiment label.

[0130] It is worth noting that during the training process of the support vector machine, the sentiment types of the text annotations such as the training set and the test set are all within the first sentiment label, and the rest of the support vector machine training methods are existing technologies and will not be elaborated here.

[0131] The present invention obtains text corresponding to the sign language by performing gesture recognition on a sign language image set, and then performs emotion classification on the text to obtain a first emotion label. That is to say, after steps S1-S5, a preliminary text and emotion label based on gesture expression are obtained.

[0132] It is worth noting that the first emotion label includes: a number of different emotion types, each emotion type also includes emotion words with different expression levels, that is, the second emotion label. As shown in Table 1, examples of the correspondence between the first emotion label and the second emotion label. Table 1 only gives examples of a few types, and there are also types such as envy, embarrassment, fear, shame, pity and anger, which will not be listed one by one.

[0133] Table 1 Examples of the correspondence between the first emotion label and the second emotion label

[0134]

[0135] That is to say, the first emotion label is a general term for a class of second emotion labels.

[0136] The present invention divides emotions into first emotion labels (major categories), which are further divided into multiple second emotion labels (minor categories). Because the same type of emotion will express different degrees of emotion in different scenarios. For example, the emotion type "like" uses different emotion labels according to different scenarios, and also expresses different meanings in a sentence.

[0137] Step S6: input the processed vector value into the semantic analysis model to perform deep semantic analysis of the text.

[0138] It is worth noting that the semantic analysis model is a bidirectional LSTM network structure. The specific semantic analysis process includes:

[0139] The converted As the input of the LSTM network, set , where a t Represents the input of the LSTM network at time t, specifically , σ in this formula is the activation function sigmoid, is the weight matrix, b a is the bias term, and Et is the input embedding vector at time step t.

[0140] Calculate the output of the LSTM network at the previous time step , is the previous moment of the LSTM network To the current time The weight matrix of is the bias term at the current moment, The calculation process is consistent, and the calculation of the latter LSTM is consistent with the former LSTM.

[0141] The hidden layer of the previous LSTM at the last moment is merged with the first dimension of the next LSTM at the last moment: ,in, is the output of the LSTM network.

[0142] After step S6, the deep meaning of the text content recognized from the sign language video can be analyzed and understood, including identifying key information such as themes, emotions, relationships, events, etc. in the text, and inferring the implicit meaning and contextual relationship in the text.

[0143] Step S7: Calculate the matching degree between the sentiment words in the first sentiment tag and the text in combination with the semantic analysis result, and determine the second sentiment tag based on the matching degree maximization strategy.

[0144] That is to say, combined with the semantic analysis result of step S6, the text content in the corresponding semantic environment is calculated, and the matching degree of several emotional words in the first emotional label is calculated, and the emotional word with the greatest matching degree is used as the second emotional label of the corresponding text. The role of this step is to further accurately divide the first emotional label and achieve more accurate and efficient sign language emotion classification.

[0145] First, convert several sentiment words in the first sentiment tag into vector representation: Let the sentiment word be w, and use the conversion tool in step S4 to convert it into a vector representation .

[0146] Further, calculate the sentiment vocabulary vector and text semantic feature vector Degree of match:

[0147]

[0148] in, Representation vector and The dot product of Represents vectors and The modulus (i.e. length) of .

[0149] Further, yes Perform a linear transformation to obtain the matching degree value:

[0150] , where T represents the text, and the calculated matching degree value ranges from 0 to 1. When the result is close to 1, it means that the matching degree between the emotional vocabulary in the first emotional tag and the text is high.

[0151] The sentiment word with the highest matching degree is selected to replace the first sentiment label and annotated on the text as the second sentiment label.

[0152] The prior art can only perform simple emotion classification, such as positive, negative or neutral, etc. However, the present invention divides emotions into several major categories (first emotion labels), which are further subdivided into multiple subcategories (second emotion labels), which can describe and classify emotions in more detail.

[0153] In the present invention, steps S6 and S7 work together to perform deep correction of the sentiment classification of the sign language text obtained by image recognition. Step S6 first performs preliminary classification, and then step S7 not only refers to this result, but also conducts a more in-depth semantic analysis of the entire sign sentence. It calculates the degree of match between the sentiment vocabulary and the text content in a specific semantic environment, accurately capturing the meaning and usage of the sentiment vocabulary in the context and its close connection with the overall sentiment tendency. More importantly, combined with the semantic analysis results of step S6, step S7 further designates the sentiment vocabulary with the highest degree of match as the second sentiment tag of the corresponding text, thereby accurately subdividing the first sentiment tag. Not only does it improve the accuracy of sentiment classification, but it also significantly enhances the classification efficiency, achieving a deep understanding and accurate conversion of the emotional expression of sign language videos.

[0154] Example systems:

[0155] Below, reference Figure 2 To describe a sign language emotion classification system based on machine learning according to an embodiment of the present invention.

[0156] like Figure 2As shown, the present invention is a sign language emotion classification system based on machine learning, including: a video acquisition module, used to acquire original sign language video; an image extraction module, used to convert the acquired original sign language video into an image sequence, perform frame extraction, and obtain a sign language image set; a text recognition module, used to extract gesture features from the sign language image set, identify the text corresponding to the gesture features, and the emotional tendency of some paragraphs; a vector conversion module, used to match and convert all feature words in the recognized text into vector values; a sentiment prediction module, used to input the processed vector values ​​into a sentiment classification model, and predict the first sentiment label corresponding to the text; a semantic analysis module, used to input the processed vector values ​​into a semantic analysis model, and perform deep semantic analysis of the text; a sentiment classification module, used to calculate the matching degree between the sentiment vocabulary in the first sentiment label and the text in combination with the semantic analysis result, and determine the second sentiment label based on the matching degree maximization strategy.

[0157] In one embodiment, the video acquisition module can acquire an original sign language video that needs to be sentimentally classified. The original sign language video can be a complete sign language dialogue video or a sign language dialogue video segment.

[0158] In one embodiment, the image extraction module uses FFmpeg or OpenCV to extract several frames of the original sign language video, including the first frame of the original sign language video, and then extracts a frame of image every 20-30 frames. The extracted multiple frames of image are marked in order and summarized to obtain a sign language image set.

[0159] In one embodiment, the text recognition module uses a pre-trained deep learning model to extract gesture features from the sign language image set, and then identifies the text interpretations corresponding to these gestures. The text interpretations obtained by identifying the gestures are the interpretations of all gesture expressions in the sign language image set, including various types of text such as subjects, verbs, and nouns.

[0160] In one embodiment, before the vector conversion module performs vector conversion on the text, it first pre-processes the text, including de-duplication, de-noising, word segmentation and stop word removal. BOW or X TF-IDF， as well as Vector form for subsequent analysis.

[0161] In one embodiment, the sentiment prediction module converts the converted X BOW or X TF-IDF As the feature input of the sentiment classification model, the sentiment contained in the text is preliminarily predicted to obtain the first sentiment label. The first sentiment label includes: a number of different sentiment types, each sentiment type also includes sentiment words with different expression levels, which is the second sentiment label.

[0162] In one embodiment, the semantic analysis module converts the As the input of the semantic analysis model, deep semantic analysis of the text is performed to analyze and understand the deep meaning of the text from the text content recognized in the sign language video, including identifying key information such as themes, emotions, relationships, events, etc. in the text, and inferring the implicit meaning and contextual relationships in the text.

[0163] In one embodiment, the sentiment classification module combines the semantic analysis results, calculates the matching degree between the text content in the corresponding semantic environment and several sentiment words in the first sentiment tag, and uses the sentiment word with the greatest matching degree as the second sentiment tag of the corresponding text. The role of this step is to further accurately divide the first sentiment tag and achieve more accurate and efficient sign language sentiment classification.

[0164] As described above, the sign language emotion classification system based on machine learning according to the embodiment of the present invention can be implemented in a movable device or a stand-alone device independent of the movable device.

[0165] In one example, the sign language emotion classification system based on machine learning according to an embodiment of the present invention can be integrated into the mobile device as a software module and / or a hardware module. For example, the sign language emotion classification system based on machine learning can be a software module in the operating system of the mobile device, or can be an application developed for the mobile device; of course, the sign language emotion classification system based on machine learning can also be one of the many hardware modules of the mobile device.

[0166] Alternatively, in another example, the machine learning-based sign language emotion classification system and the movable device may also be separate devices, and the machine learning-based sign language emotion classification system may be connected to the sound processing device via a wired and / or wireless network and transmit interaction information in accordance with an agreed data format.

[0167] The above is based on the ideal embodiment of the present invention. Through the above description, relevant personnel can make various changes and modifications without departing from the technical concept of the present invention. The technical scope of the present invention is not limited to the content in the specification, and the technical scope must be determined according to the scope of the claims.

Claims

1. A sign language emotion classification method based on machine learning, characterized in that: The following steps are involved: Get the original sign language video; Converting the acquired original sign language video into an image sequence, performing frame extraction, and obtaining a sign language image set; Extracting gesture features from the sign language image set, identifying text corresponding to the gesture features, and the emotional tendency of some paragraphs; Match all the feature words in the recognized text and convert them into vector values; The processed vector value is input into the sentiment classification model to predict the first sentiment label corresponding to the text; The processed vector value is input into the semantic analysis model to perform deep semantic analysis of the text; Combined with the semantic analysis results, the matching degree between the sentiment words in the first sentiment tag and the text is calculated, and based on the matching degree maximization strategy, the second sentiment tag is determined; The first emotion label includes: a plurality of different emotion types, each emotion type also includes emotion words of different expression levels, namely, the second emotion label.

2. The method for sign language emotion classification based on machine learning according to claim 1, characterized in that: When performing frame extraction on the original sign language video, FFmpeg or OpenCV is used to extract several frames of images of the original sign language video, including the first frame of the original sign language video, and then one frame of image is extracted every 20-30 frames.

3. The method for sign language emotion classification based on machine learning according to claim 1, characterized in that: Extracting and recognizing gesture features from the sign language image set using a pre-trained deep learning model; The text interpretation obtained by the deep learning model recognizing the gesture features is the interpretation of all gesture expressions in the sign language image set, including texts of various parts of speech.

4. The method for sign language emotion classification based on machine learning according to claim 1, characterized in that: The sentiment analysis process includes: According to the extracted gesture features, query the corresponding several frames of images; Detect key points of gesture features, including: motion nodes of limb joints, ends, bones, and head; The position changes of key points between consecutive frame images are calculated, and the optical flow algorithm is used to estimate the speed and direction of gesture movements, identify the acceleration or slowdown of gesture rhythm, and then judge the emotional tendency of these paragraphs.

5. The method for sign language emotion classification based on machine learning according to claim 1, characterized in that: Methods for converting all feature words in a text into vector values ​​include: Let V be the vocabulary, w j For the words in the vocabulary, further build the vocabulary: ,in, For the preprocessed text set, the UniqueWords function is used to extract all unique words; Let X BOW or X TF-IDF is the feature matrix of the text, the vectorization process is expressed as: , where Vectorize BOW / TF-IDF The function converts the preprocessed text set T′ into a feature matrix according to the vocabulary V; At the same time, let E be the word embedding matrix, e j For word w j The word embedding process is: , where e j Obtained through the pre-trained word embedding model Word2Vec; For a single text , whose word embedding sequence is expressed as: ,in, is text The vocabulary index corresponding to the word in; Furthermore, the word embedding sequence is padded to obtain a fixed-length input sequence. The padding process is: , where maxlen is the maximum length of the sequence.

6. The method for sign language emotion classification based on machine learning according to claim 5, characterized in that: The semantic analysis model is a bidirectional LSTM network structure, and the specific semantic analysis process includes: The converted As the input of the LSTM network, set , where a t Represents the input of the LSTM network at time t, specifically , σ in this formula is the activation function sigmoid, is the weight matrix, b a is the bias term, Et is the input embedding vector at time step t; Calculate the output of the LSTM network at the previous time step , From the previous moment to the current moment of the LSTM network The weight matrix of is the bias term at the current moment, The calculation process is consistent, and the calculation of the latter LSTM is consistent with the former LSTM; The hidden layer of the previous LSTM at the last moment is merged with the first dimension of the next LSTM at the last moment: ,in, is the output of the semantic analysis model.

7. The method for sign language emotion classification based on machine learning according to claim 6, characterized in that: Before calculating the matching degree between the sentiment words in the first sentiment tag and the text, convert several sentiment words in the first sentiment tag into vector representation: Let the sentiment word be w, and use the conversion tool used when converting text features to convert it into a vector representation .

8. The method for sign language emotion classification based on machine learning according to claim 7, characterized in that: Calculate sentiment word vectors and text semantic feature vectors Degree of match: ,in, Representation vector and The dot product of Represents vectors and The modulus, i.e., the length; Further, yes Perform a linear transformation to obtain the matching degree value: , where T represents the text, and the calculated matching degree value ranges from 0 to 1. When the result is close to 1, it means that the matching degree between the emotional vocabulary in the first emotional tag and the text is high.

9. A sign language emotion classification system based on machine learning, which executes a sign language emotion classification method based on machine learning as claimed in any one of claims 1 to 8, characterized in that: include: A video acquisition module is used to acquire original sign language videos; An image extraction module, used to convert the acquired original sign language video into an image sequence, perform frame extraction, and obtain a sign language image set; A text recognition module extracts gesture features from the sign language image set, identifies the text corresponding to the gesture features, and the emotional tendency of some paragraphs; The vector conversion module is used to match and convert all the feature words in the recognized text into vector values; The sentiment prediction module is used to input the processed vector value into the sentiment classification model to predict the first sentiment label corresponding to the text; The semantic analysis module is used to input the processed vector value into the semantic analysis model to perform deep semantic analysis of the text; The sentiment classification module is used to combine the semantic analysis results, calculate the matching degree between the sentiment words in the first sentiment tag and the text, and determine the second sentiment tag based on the matching degree maximization strategy.

Images