Tablet computer control method and system based on expression recognition

By combining facial expressions and voice information, using trained models for expression and speech recognition, the problems of insufficient emotional state recognition and unnatural interaction in traditional tablet control methods are solved, and more efficient user interaction experience and personalization are achieved response.

CN119046927AInactive Publication Date: 2024-11-29SHENZHEN ZHIWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411361078.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-27
Publication Date
2024-11-29
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional tablet computer control methods rely on a single input method and cannot fully understand the user's emotional state. There is a high misidentification rate, unintuitive, long response time, difficulty in adapting to different user needs, and lack of multi-dimensional analysis of user emotions, which limits interaction. Depth and nature.

Method used

Using a method based on expression recognition, face images and voice data are collected through cameras and voice acquisition devices, combined with trained models for expression and voice recognition, match the user's expression types to generate control commands, and achieve more natural and intuitive user interaction. .

Benefits of technology

It improves the accurate recognition of user emotional state, enhances the interactive experience, adapts to the personalized needs of different users, reduces the false recognition rate, and improves the system response accuracy and user operation convenience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046927A_ABST
    Figure CN119046927A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computer interaction control, in particular to a tablet computer control method and system based on expression recognition, and the method comprises the steps: carrying out the face image collection of a target user according to a preset sampling period based on a camera and voice collection equipment integrated in a tablet computer, obtaining a face image set and a voice set corresponding to the target user; performing image preprocessing on the human face image set to obtain a human face standard image set, sorting the human face standard image set according to a time sequence to obtain a human face standard image sequence, and sorting the voice set according to the time sequence to obtain a voice sequence. By combining the facial expression and the voice information, the accurate recognition of the emotional state of the user can be improved, the interaction experience is enhanced, a more natural and visual tablet personal computer control mode can be realized based on the expression instruction of the user, and the convenience of user operation is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer interactive control, and in particular to a tablet computer control method and system based on expression recognition. Background Art

[0002] A tablet computer is a portable computing device, usually with a touch screen interface, that combines the features of a smartphone and a laptop. It is usually lightweight and suitable for carrying around. It is mainly used for browsing the web, watching videos, reading e-books, playing games, and processing documents.

[0003] Traditional methods usually rely on only one input method, resulting in incomplete information acquisition and inability to fully understand the user's emotional state. When facing a noisy environment or a complex background, the single modality of the traditional method may produce a high misrecognition rate, affecting the performance of the system. Traditional methods often require users to input clear commands, which is not intuitive enough and reduces the naturalness and fluency of the interaction. Traditional methods have weak adaptability to the personalized needs of different users and find it difficult to handle the habits and styles of different users. They lack multi-dimensional analysis of user emotions and find it difficult to capture subtle emotional changes, limiting the depth of interaction. In addition, due to reliance on clear command input, they often lead to long response times, affecting user experience. Summary of the invention

[0004] The purpose of the present invention is to solve the problems existing in the background technology and to provide a tablet computer control method and system based on expression recognition.

[0005] The technical solution of the present invention is a tablet computer control method based on expression recognition, comprising:

[0006] Based on the camera and voice collection device integrated in the tablet computer, facial images of the target user are collected according to a preset sampling period to obtain a facial image set and a voice set corresponding to the target user;

[0007] The face image set is subjected to image preprocessing to obtain a standard face image set.

[0008] The standard image set is sorted according to the time series to obtain a standard face image sequence, and the speech set is sorted according to the time series to obtain a speech sequence;

[0009] Performing expression recognition on the standard face image sequence based on the trained expression recognition model to obtain a facial expression type sequence, and performing speech recognition on the speech sequence based on the trained speech recognition model to obtain a speech text sequence;

[0010] Acquire the expression type of the target user based on the facial expression type sequence and the voice text sequence;

[0011] The expression type of the target user is matched with a preset command association table to obtain a control command corresponding to the expression type of the target user, and the tablet computer is controlled based on the control command.

[0012] Preferably, the facial expression type sequence corresponds to the voice text sequence, and the command association table includes the expression type of the target user and the control command corresponding to each expression type.

[0013] Preferably, acquiring the expression type of the target user based on the facial expression type sequence and the voice text sequence includes:

[0014] Determine whether each voice text in the voice text sequence is an empty text, and if the voice text is an empty text, use the facial expression type as the expression type of the target user;

[0015] If the speech text is not an empty text, the speech text sequence is matched with a preset expression keyword table to obtain an expression keyword corresponding to each speech text in the speech text sequence;

[0016] Acquire a facial expression image corresponding to the expression keyword corresponding to the voice text based on a facial expression image set pre-stored in a database;

[0017] The similarity between the standard face image corresponding to the speech text and the facial expression image is calculated based on the similarity model. If the similarity is greater than a preset similarity threshold, the speech text is

[0018] The facial expression type corresponding to the expression keyword or the facial standard image corresponding to the voice text is used as the expression type of the target user.

[0019] Preferably, acquiring the expression type of the target user based on the facial expression type sequence and the voice text sequence further includes:

[0020] If the similarity is not greater than a preset similarity threshold, the speech text is subjected to expression classification based on a trained expression classification model to obtain an emotion label corresponding to the speech text, wherein the emotion classification model includes a vector embedding layer, an encoder, a hybrid attention mechanism and a decoder, the vector embedding layer adopts a BER model, the encoder includes a bidirectional gated recurrent unit and a long short-term memory network, the hybrid attention mechanism is used to fuse the encoder output results, and the decoder is used to map the fused encoder output results to the emotion label;

[0021] Matching the emotion label corresponding to the speech text with the facial expression type corresponding to the standard face image corresponding to the speech text to obtain a matching degree between the two. If the matching degree is less than a preset matching degree threshold, the emotion label corresponding to the speech text is used as the expression type of the target user.

[0022] If the matching degree is not less than a preset matching degree threshold, the facial expression type corresponding to the standard facial image corresponding to the speech text is used as the expression type of the target user.

[0023] Preferably, the empty text determination rule includes:

[0024] Counting the number of words in the speech text to obtain the number of words in the text corresponding to the speech text, and if the number of words in the text corresponding to the speech text is less than a preset word count threshold, determining that the speech text is an empty text;

[0025] If the number of words in the text corresponding to the speech text is not less than the preset word count threshold, the sentiment score of each sentiment word in the speech text is calculated to obtain the sentiment score corresponding to the speech text.

[0026] The emotion score set corresponding to the speech text is summed up, and the sum result is normalized to obtain the comprehensive emotion score corresponding to the speech text. If the comprehensive emotion score corresponding to the speech text is less than the preset emotion score threshold, the speech text is determined to be an empty text.

[0027] Preferably, calculating the sentiment score of each sentiment word in the speech text to obtain a sentiment score set corresponding to the speech text includes:

[0028] Pre-constructing a sentiment dictionary, wherein the sentiment dictionary includes sentiment words, sentiment polarity of the sentiment words, and sentiment intensity of the sentiment words;

[0029] Matching the speech text with the emotion dictionary to obtain the emotion words in the speech text, and obtaining the word segments before the emotion words in the emotion words;

[0030] The word segment before the emotion word in the speech text is matched with the preset degree adverb table. If the word segment matches the degree adverb table, the emotion intensity of the word segment is obtained based on the emotion dictionary, and the first emotion score of the emotion word is calculated based on the emotion intensity of the word segment. Otherwise, the second emotion score of the emotion word is calculated based on the type of the word segment. The calculation formula of the first emotion score is as follows:

[0031] S i =p wi ·swi ·d i ;

[0032] Among them, S i represents the first sentiment score of the sentiment word, p wi Indicates the sentiment polarity of sentiment words, s wi Indicates the sentiment intensity of sentiment words, d i Indicates the emotional intensity of the word segment;

[0033] The calculation formula of the second sentiment score is as follows:

[0034]

[0035] in, Indicates the second sentiment score of the sentiment word. When the type of the word segment is a negative word, n i =1, when the type of the word segment is a negative word, n i =1, when the type of the word segment is not a negative word, n i =0;

[0036] The sentiment score corresponding to the sentiment word is acquired based on the first sentiment score and the second sentiment score of each sentiment word in the speech text.

[0037] Preferably, performing image preprocessing on the face image set to obtain a standard face image set includes:

[0038] A sliding window is set, the face image is traversed based on the sliding window, the fluctuation coefficient in the sliding window is calculated, and the clarity of all pixels in the sliding window is characterized based on the fluctuation coefficient. The calculation formula of the fluctuation coefficient is as follows:

[0039]

[0040] Among them, V ar (x,y) represents the fluctuation degree of the pixel with coordinates (x,y) in the face image, I(x,y) represents the gray value of the pixel with coordinates (x,y) in the face image, u(x,y) represents the mathematical expectation of the pixel, and

[0041] Counting the fluctuation coefficients of all sliding windows in the face image to obtain a fluctuation set, and traversing each fluctuation coefficient in the fluctuation set to obtain the maximum fluctuation coefficient in the fluctuation set;

[0042] The pixel value of the sliding window corresponding to the maximum fluctuation coefficient is assigned as the pixel value of each pixel point in the face image.

[0043] Preferably, performing expression classification on the speech text based on a trained expression classification model to obtain an emotion label corresponding to the speech text includes:

[0044] Inputting the speech text into the BER model, and converting the speech text into a target word vector through the BER model;

[0045] Inputting the target word vector into the bidirectional gated recurrent unit and the long short-term memory network respectively, obtaining the inter-sentence semantic features of the target word vector through the bidirectional gated recurrent unit, and converting the inter-sentence semantic features into an inter-sentence semantic feature vector;

[0046] Acquire the intra-sentence semantic features of the target word vector through the long short-term memory network, and convert the intra-sentence semantic features into an intra-sentence semantic feature vector;

[0047] The inter-sentence semantic feature vector and the intra-sentence semantic feature vector are fused by the hybrid attention mechanism to obtain a first fused vector, and the inter-sentence semantic feature vector, the intra-sentence semantic feature vector and the first fused vector are fused to obtain a second fused vector;

[0048] The second fusion vector is input into the decoder, and the second fusion vector is mapped to the emotion type by the decoder to obtain the emotion label corresponding to the speech text.

[0049] Preferably, calculating the similarity between the standard face image corresponding to the speech text and the face expression image based on a similarity model includes:

[0050] Extracting the standard face image corresponding to the speech text based on the deep learning model to obtain the face image features corresponding to the standard face image;

[0051] Extracting image features of the facial expression image based on a deep learning model to obtain expression image features corresponding to the facial expression image;

[0052] The similarity between the facial image features corresponding to the standard facial image and the expression image features corresponding to the facial expression image is calculated based on the similarity model.

[0053] The technical solution of the present invention is a tablet computer control system based on expression recognition, which is applicable to the tablet computer control method based on expression recognition, comprising:

[0054] A data acquisition unit, the data acquisition unit is used to collect facial images of a target user according to a preset sampling period based on a camera and a voice acquisition device integrated in the tablet computer, so as to obtain a facial image set and a voice set corresponding to the target user;

[0055] A sequence processing unit, the sequence processing unit is used to perform image preprocessing on the face image set to obtain a standard face image set, sort the standard face image set according to a time series to obtain a standard face image sequence, and sort the voice set according to a time series to obtain a voice sequence;

[0056] An expression recognition unit, the expression recognition unit is used to perform expression recognition on the standard face image sequence based on a trained expression recognition model to obtain a facial expression type sequence, and to perform speech recognition on the speech sequence based on a trained speech recognition model to obtain a speech text sequence;

[0057] A comprehensive recognition unit, the comprehensive recognition unit is used to obtain the expression type of the target user based on the facial expression type sequence and the voice text sequence;

[0058] A computer control unit is used to match the expression type of the target user with a preset command association table to obtain a control command corresponding to the expression type of the target user, and control the tablet computer based on the control command.

[0059] Compared with the prior art, the above technical solution of the present invention has the following beneficial technical effects:

[0060] 1. The present invention can improve the accurate recognition of the user's emotional state and enhance the interactive experience by combining facial expressions and voice information. Based on the user's expression instructions, a more natural and intuitive tablet computer control method can be achieved, improving the convenience of user operation. By learning and matching the facial and voice features of different users, the system can adapt to the personalized needs of different users. Moreover, by using the trained model and similarity calculation, the user's emotional state can be effectively distinguished and recognized, thereby improving the response accuracy of the system.

[0061] 2. During the speech recognition process, the present invention takes into account the matching of empty text and expression keywords, which helps to reduce the misrecognition rate. It can not only identify the expression type, but also analyze the speech emotion, provide richer user emotion information, enhance the intelligence of human-computer interaction, and adjust the association between expressions and commands according to actual application needs. It has good flexibility and scalability and is suitable for a variety of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1A schematic flow chart of the steps of an overall method in an embodiment of the present invention;

[0063] Figure 2 A schematic diagram of the system architecture of an overall system in an embodiment of the present invention.

[0064] Figure numerals: 1. Data acquisition unit; 2. Sequence processing unit; 3. Expression recognition unit; 4. Comprehensive recognition unit; 5. Computer control unit. DETAILED DESCRIPTION

[0065] Embodiment 1, as Figure 1 As shown, the present invention proposes a tablet computer control method based on expression recognition, comprising:

[0066] S1, based on the camera and voice collection device integrated in the tablet computer, collect facial images of the target user according to a preset sampling period to obtain a facial image set and a voice set corresponding to the target user;

[0067] S2, performing image preprocessing on the face image set to obtain a standard face image set, sorting the standard face image set according to a time series to obtain a standard face image sequence, and sorting the voice set according to a time series to obtain a voice sequence;

[0068] S3, performing expression recognition on the standard face image sequence based on the trained expression recognition model to obtain a facial expression type sequence, and performing speech recognition on the speech sequence based on the trained speech recognition model to obtain a speech text sequence;

[0069] S4, obtaining the expression type of the target user based on the facial expression type sequence and the voice text sequence;

[0070] S5. Match the target user's expression type with a preset command association table to obtain a control command corresponding to the target user's expression type, and control the tablet computer based on the control command.

[0071] In the present invention, the voice acquisition device refers to a hardware device used to record and obtain user voice data; the expression recognition model is a machine learning or deep learning algorithm used to analyze and recognize facial expressions. These models judge the user's emotional state by processing facial images. Common expression recognition models include but are not limited to support vector machines, convolutional neural networks, recurrent neural networks, and long short-term memory networks; the speech recognition model is a machine learning or deep learning algorithm used to convert speech signals into text. These models recognize and understand the user's language by analyzing audio data. Common speech recognition models include but are not limited to hidden Markov models, Gaussian mixture models, convolutional neural networks, recurrent neural networks, long short-term memory networks, and sequence-to-sequence models; the command association table is a mapping relationship table that associates different expression types with specific control commands so that the system can perform corresponding operations according to the user's expression; the control command refers to an instruction generated based on the user's expression, which is used to control the function of the tablet computer.

[0072] Embodiment 2, a tablet computer control method based on expression recognition proposed by the present invention, compared with embodiment 1, this embodiment also includes: a facial expression type sequence corresponds to a voice text sequence, and a command association table includes the expression type of the target user and the control command corresponding to each expression type.

[0073] In an optional embodiment, acquiring the expression type of the target user based on the facial expression type sequence and the voice text sequence includes:

[0074] A1. Determine whether each voice text in the voice text sequence is an empty text. If the voice text is an empty text, take the facial expression type as the expression type of the target user;

[0075] A2. If the voice text is not an empty text, the voice text sequence is matched with a preset expression keyword table to obtain an expression keyword corresponding to each voice text in the voice text sequence;

[0076] A3, obtaining a facial expression image corresponding to the expression keyword corresponding to the voice text based on a facial expression image set pre-stored in the database;

[0077] A4. Calculate the similarity between the standard face image corresponding to the speech text and the facial expression image based on the similarity model. If the similarity is greater than a preset similarity threshold, use the expression keyword corresponding to the speech text or the facial expression type corresponding to the standard face image corresponding to the speech text as the expression type of the target user.

[0078] In an optional embodiment, acquiring the expression type of the target user based on the facial expression type sequence and the voice text sequence further includes:

[0079] A5. If the similarity is not greater than the preset similarity threshold, the speech text is classified based on the trained expression classification model to obtain the emotion label corresponding to the speech text, wherein the emotion classification model includes a vector embedding layer, an encoder, a hybrid attention mechanism and a decoder. The vector embedding layer adopts a BER model, the encoder includes a bidirectional gated recurrent unit and a long short-term memory network, the hybrid attention mechanism is used to fuse the encoder output results, and the decoder is used to map the output results of the fused encoder to the emotion label;

[0080] A6, matching the emotion label corresponding to the speech text with the facial expression type corresponding to the standard face image corresponding to the speech text to obtain a matching degree between the two. If the matching degree is less than a preset matching degree threshold, the emotion label corresponding to the speech text is used as the expression type of the target user;

[0081] A7. If the matching degree is not less than a preset matching degree threshold, the facial expression type corresponding to the standard facial image corresponding to the speech text is used as the expression type of the target user.

[0082] It should be noted that vector embedding refers to the conversion of discrete inputs (such as words, sentences or phrases) into continuous vector representations, which can capture the semantic relationship between inputs and are often used in natural language processing (NLP) tasks; the BER model (such as BERT) is a pre-trained model that can generate context-related embeddings; the bidirectional gated recurrent unit (Bi-GRU) is a special recurrent neural network (RNN) that improves the ability to capture contextual information by processing input sequences in two directions (from front to back and from back to front). The gating mechanism helps control the flow of information, thereby better learning the dependencies in the sequence.

[0083] In an optional embodiment, the empty text determination rule includes:

[0084] B1. Counting the number of words in the speech text to obtain the number of words in the text corresponding to the speech text. If the number of words in the text corresponding to the speech text is less than a preset word count threshold, the speech text is determined to be an empty text.

[0085] B2. If the number of words corresponding to the speech text is not less than the preset word count threshold, the sentiment score of each sentiment word in the speech text is calculated to obtain the sentiment score set corresponding to the speech text, the sentiment score set corresponding to the speech text is summed, and the sum result is normalized to obtain the comprehensive sentiment score corresponding to the speech text. If the comprehensive sentiment score corresponding to the speech text is less than the preset sentiment score threshold, the speech text is determined to be an empty text.

[0086] In an optional embodiment, the sentiment score of each sentiment word in the speech text is calculated to obtain a sentiment score set corresponding to the speech text, including:

[0087] C1. Pre-build a sentiment dictionary, where the sentiment dictionary includes sentiment words, sentiment polarity of sentiment words and sentiment intensity of sentiment words;

[0088] C2, matching the speech text with the sentiment dictionary to obtain the sentiment words in the speech text, and obtaining the word segments before the sentiment words in the sentiment words;

[0089] C3. Match the word segment before the sentiment word in the speech text with the preset degree adverb table. If the word segment matches the degree adverb table, the sentiment intensity of the word segment is obtained based on the sentiment dictionary, and the first sentiment score of the sentiment word is calculated based on the sentiment intensity of the word segment. Otherwise, the second sentiment score of the sentiment word is calculated based on the type of the word segment. The calculation formula of the first sentiment score is as follows:

[0090] S i =p wi ·s wi ·d i ;

[0091] Among them, S i represents the first sentiment score of the sentiment word, p wi Indicates the sentiment polarity of sentiment words, s wi Indicates the sentiment intensity of sentiment words, d i Indicates the emotional intensity of the word segment;

[0092] The calculation formula for the second sentiment score is as follows:

[0093]

[0094] in, Indicates the second sentiment score of the sentiment word. When the type of the word segment is a negative word, n i =1, when the type of the word segment is a negative word, n i =1, when the type of the word segment is not a negative word, n i =0;

[0095] C4. Acquire the sentiment score corresponding to the sentiment word based on the first sentiment score and the second sentiment score of each sentiment word in the speech text.

[0096] It should be noted that sentiment words refer to specific vocabulary, which are usually related to emotional states; sentiment polarity refers to the emotional tendency expressed by sentiment words; and sentiment intensity is used to quantify the intensity of the emotions expressed by sentiment words.

[0097] In an optional embodiment, image preprocessing is performed on the face image set to obtain a standard face image set, including:

[0098] D1. Set a sliding window, traverse the face image based on the sliding window, calculate the fluctuation coefficient within the sliding window, and characterize the clarity of all pixels in the sliding window based on the fluctuation coefficient. The calculation formula of the fluctuation coefficient is as follows:

[0099]

[0100] Among them, V ar (x,y) represents the fluctuation degree of the pixel with coordinates (x,y) in the face image, I(x,y) represents the gray value of the pixel with coordinates (x,y) in the face image, u(x,y) represents the mathematical expectation of the pixel, and

[0101] D2. Count the fluctuation coefficients of all sliding windows in the face image to obtain a fluctuation set, and traverse each fluctuation coefficient in the fluctuation set to obtain the maximum fluctuation coefficient in the fluctuation set;

[0102] D3. Assign the pixel value of the sliding window corresponding to the largest fluctuation coefficient as the pixel value of each pixel in the face image.

[0103] In an optional embodiment, the expression classification of the speech text is performed based on the trained expression classification model to obtain the emotion label corresponding to the speech text, including:

[0104] E1. Input the speech text into the BER model and convert the speech text into the target word vector through the BER model;

[0105] E2. Input the target word vector into the bidirectional gated recurrent unit and the long short-term memory network respectively, obtain the inter-sentence semantic features of the target word vector through the bidirectional gated recurrent unit, and convert the inter-sentence semantic features into an inter-sentence semantic feature vector;

[0106] E3, obtaining the intra-sentence semantic features of the target word vector through the long short-term memory network, and converting the intra-sentence semantic features into an intra-sentence semantic feature vector;

[0107] E4, fusing the inter-sentence semantic feature vector and the intra-sentence semantic feature vector through a hybrid attention mechanism to obtain a first fused vector, and fusing the inter-sentence semantic feature vector, the intra-sentence semantic feature vector, and the first fused vector to obtain a second fused vector;

[0108] E5. Input the second fusion vector into the decoder, and map the second fusion vector to the emotion type through the decoder to obtain the emotion label corresponding to the speech text.

[0109] In an optional embodiment, calculating the similarity between the standard face image corresponding to the speech text and the face expression image based on the similarity model includes:

[0110] F1. Extract the standard face image corresponding to the speech text based on the deep learning model to obtain the face image features corresponding to the standard face image;

[0111] F2. Extracting image features of the facial expression image based on the deep learning model to obtain expression image features corresponding to the facial expression image;

[0112] F3. Calculate the similarity between the facial image features corresponding to the standard facial image and the expression image features corresponding to the facial expression image based on the similarity model.

[0113] It should be noted that similarity models are usually used to measure the similarity between data, and similarity models include but are not limited to Euclidean distance, Manhattan distance and Chebyshev distance.

[0114] Embodiment three, as Figure 2 As shown, a tablet computer control system based on expression recognition is applicable to the above-mentioned tablet computer control method based on expression recognition, including:

[0115] The data collection unit 1 is used to collect facial images of a target user according to a preset sampling period based on a camera and a voice collection device integrated in a tablet computer, so as to obtain a facial image set and a voice set corresponding to the target user;

[0116] The sequence processing unit 2 is used to perform image preprocessing on the face image set to obtain a standard face image set, sort the standard face image set according to a time series to obtain a standard face image sequence, and sort the voice set according to a time series to obtain a voice sequence;

[0117] The expression recognition unit 3 is used to perform expression recognition on a standard face image sequence based on a trained expression recognition model to obtain a facial expression type sequence, and to perform speech recognition on a speech sequence based on a trained speech recognition model to obtain a speech text sequence;

[0118] A comprehensive recognition unit 4, which is used to obtain the expression type of the target user based on the facial expression type sequence and the voice text sequence;

[0119] The computer control unit 5 is used to match the expression type of the target user with a preset command association table to obtain the control command corresponding to the expression type of the target user, and control the tablet computer based on the control command.

[0120] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. A tablet computer control method based on expression recognition, It is characterized in that include: Based on the camera and voice collection device integrated in the tablet computer, facial images of the target user are collected according to a preset sampling period to obtain a facial image set and a voice set corresponding to the target user; Performing image preprocessing on the face image set to obtain a standard face image set, sorting the standard face image set according to a time series to obtain a standard face image sequence, and sorting the voice set according to a time series to obtain a voice sequence; Performing expression recognition on the standard face image sequence based on the trained expression recognition model to obtain a facial expression type sequence, and performing speech recognition on the speech sequence based on the trained speech recognition model to obtain a speech text sequence; Acquire the expression type of the target user based on the facial expression type sequence and the voice text sequence; The expression type of the target user is matched with a preset command association table to obtain a control command corresponding to the expression type of the target user, and the tablet computer is controlled based on the control command.

2. A tablet computer control method based on expression recognition according to claim 1, Features: The facial expression type sequence corresponds to the voice text sequence, and the command association table includes the expression type of the target user and the control command corresponding to each expression type.

3. The tablet computer control method based on expression recognition according to claim 1, Features: Acquiring the expression type of the target user based on the facial expression type sequence and the voice text sequence includes: Determine whether each voice text in the voice text sequence is an empty text, and if the voice text is an empty text, use the facial expression type as the expression type of the target user; If the speech text is not an empty text, the speech text sequence is matched with a preset expression keyword table to obtain an expression keyword corresponding to each speech text in the speech text sequence; Acquire a facial expression image corresponding to the expression keyword corresponding to the voice text based on a facial expression image set pre-stored in a database; The similarity between the standard face image corresponding to the speech text and the facial expression image is calculated based on a similarity model. If the similarity is greater than a preset similarity threshold, the expression keyword corresponding to the speech text or the facial expression type corresponding to the standard face image corresponding to the speech text is used as the expression type of the target user.

4. The tablet computer control method based on expression recognition according to claim 3, Features: Acquiring the expression type of the target user based on the facial expression type sequence and the voice text sequence also includes: If the similarity is not greater than a preset similarity threshold, the speech text is subjected to expression classification based on a trained expression classification model to obtain an emotion label corresponding to the speech text, wherein the emotion classification model includes a vector embedding layer, an encoder, a hybrid attention mechanism and a decoder, the vector embedding layer adopts a BER model, the encoder includes a bidirectional gated recurrent unit and a long short-term memory network, the hybrid attention mechanism is used to fuse the encoder output results, and the decoder is used to map the fused encoder output results to the emotion label; Matching the emotion label corresponding to the speech text with the facial expression type corresponding to the standard face image corresponding to the speech text to obtain a matching degree between the two. If the matching degree is less than a preset matching degree threshold, the emotion label corresponding to the speech text is used as the expression type of the target user; If the matching degree is not less than a preset matching degree threshold, the facial expression type corresponding to the standard facial image corresponding to the speech text is used as the expression type of the target user.

5. The tablet computer control method based on expression recognition according to claim 3, Features: The empty text determination rules include: Counting the number of words in the speech text to obtain the number of words in the text corresponding to the speech text, and if the number of words in the text corresponding to the speech text is less than a preset word count threshold, determining that the speech text is an empty text; If the number of text words corresponding to the speech text is not less than a preset word count threshold, the sentiment score of each sentiment word in the speech text is calculated to obtain a sentiment score set corresponding to the speech text, the sentiment score set corresponding to the speech text is summed, and the sum result is normalized to obtain a comprehensive sentiment score corresponding to the speech text. If the comprehensive sentiment score corresponding to the speech text is less than the preset sentiment score threshold, the speech text is determined to be an empty text.

6. The tablet computer control method based on expression recognition according to claim 5, Features: The sentiment score of each sentiment word in the speech text is calculated to obtain a sentiment score set corresponding to the speech text, including: Pre-constructing a sentiment dictionary, wherein the sentiment dictionary includes sentiment words, sentiment polarity of the sentiment words, and sentiment intensity of the sentiment words; Matching the speech text with the emotion dictionary to obtain the emotion words in the speech text, and obtaining the word segments before the emotion words in the emotion words; The word segment before the emotion word in the speech text is matched with the preset degree adverb table. If the word segment matches the degree adverb table, the emotion intensity of the word segment is obtained based on the emotion dictionary, and the first emotion score of the emotion word is calculated based on the emotion intensity of the word segment. Otherwise, the second emotion score of the emotion word is calculated based on the type of the word segment. The calculation formula of the first emotion score is as follows: S i =p wi ·s wi ·d i ; Among them, S i represents the first sentiment score of the sentiment word, p wi Indicates the sentiment polarity of sentiment words, s wi Indicates the sentiment intensity of sentiment words, d i Indicates the emotional intensity of the word segment; The calculation formula of the second sentiment score is as follows: in, Indicates the second sentiment score of the sentiment word. When the type of the word segment is a negative word, n i =1, when the type of the word segment is a negative word, n i =1, when the type of the word segment is not a negative word, n i =0; The sentiment score corresponding to the sentiment word is acquired based on the first sentiment score and the second sentiment score of each sentiment word in the speech text.

7. The tablet computer control method based on expression recognition according to claim 1, Features: Performing image preprocessing on the face image set to obtain a standard face image set, including: A sliding window is set, the face image is traversed based on the sliding window, the fluctuation coefficient in the sliding window is calculated, and the clarity of all pixels in the sliding window is characterized based on the fluctuation coefficient. The calculation formula of the fluctuation coefficient is as follows: Among them, V ar (x,y) represents the fluctuation degree of the pixel with coordinates (x,y) in the face image, I(x,y) represents the gray value of the pixel with coordinates (x,y) in the face image, u(x,y) represents the mathematical expectation of the pixel, and Counting the fluctuation coefficients of all sliding windows in the face image to obtain a fluctuation set, and traversing each fluctuation coefficient in the fluctuation set to obtain the maximum fluctuation coefficient in the fluctuation set; The pixel value of the sliding window corresponding to the maximum fluctuation coefficient is assigned as the pixel value of each pixel point in the face image.

8. The tablet computer control method based on expression recognition according to claim 4, Features: Performing expression classification on the speech text based on the trained expression classification model to obtain the emotion label corresponding to the speech text includes: Inputting the speech text into the BER model, and converting the speech text into a target word vector through the BER model; Inputting the target word vector into the bidirectional gated recurrent unit and the long short-term memory network respectively, obtaining the inter-sentence semantic features of the target word vector through the bidirectional gated recurrent unit, and converting the inter-sentence semantic features into an inter-sentence semantic feature vector; Acquire the intra-sentence semantic features of the target word vector through the long short-term memory network, and convert the intra-sentence semantic features into an intra-sentence semantic feature vector; The inter-sentence semantic feature vector and the intra-sentence semantic feature vector are fused by the hybrid attention mechanism to obtain a first fused vector, and the inter-sentence semantic feature vector, the intra-sentence semantic feature vector and the first fused vector are fused to obtain a second fused vector; The second fusion vector is input into the decoder, and the second fusion vector is mapped to the emotion type by the decoder to obtain the emotion label corresponding to the speech text.

9. The tablet computer control method based on expression recognition according to claim 3, Features: Calculating the similarity between the standard face image corresponding to the speech text and the face expression image based on the similarity model includes: Extracting the standard face image corresponding to the speech text based on the deep learning model to obtain the face image features corresponding to the standard face image; Extracting image features of the facial expression image based on a deep learning model to obtain expression image features corresponding to the facial expression image; The similarity between the facial image features corresponding to the standard facial image and the expression image features corresponding to the facial expression image is calculated based on the similarity model.

10. A tablet computer control system based on expression recognition, which is applicable to a tablet computer control method based on expression recognition as described in any one of claims 1 to 9, It is characterized in that include: A data collection unit (1), the data collection unit (1) being used to collect facial images of a target user according to a preset sampling period based on a camera and a voice collection device integrated in the tablet computer, so as to obtain a facial image set and a voice set corresponding to the target user; A sequence processing unit (2), the sequence processing unit (2) is used to perform image preprocessing on the face image set to obtain a standard face image set, sort the standard face image set according to a time series to obtain a standard face image sequence, and sort the voice set according to a time series to obtain a voice sequence; An expression recognition unit (3), the expression recognition unit (3) being used to perform expression recognition on the standard face image sequence based on a trained expression recognition model to obtain a facial expression type sequence, and to perform speech recognition on the speech sequence based on a trained speech recognition model to obtain a speech text sequence; A comprehensive recognition unit (4), the comprehensive recognition unit (4) is used to obtain the expression type of the target user based on the facial expression type sequence and the voice text sequence; A computer control unit (5), the computer control unit (5) is used to match the expression type of the target user with a preset command association table to obtain a control command corresponding to the expression type of the target user, and control the tablet computer based on the control command.