A packet collection method based on a multi-stage time domain convolution network

Abstract

The application discloses a message collection method based on a multi-stage time domain convolution network, and comprises the following steps: collecting lip movement video data of a user; detecting and extracting feature key point coordinates of a lip region from each image of the video data; performing standardization and normalization processing on the extracted lip key point coordinates; based on a calculated face space orientation vector, uniformly transforming the lip key point coordinates under different head postures into standard front-facing orientation coordinates; intelligently intercepting an effective lip movement action sequence segment from continuous video frames by analyzing the timing change of the lip opening amplitude and according to a preset start-stop determination rule; performing data enhancement processing on the lip key point sequence; training the processed lip key point sequence using a multi-stage time domain convolution network model to obtain a lip movement recognition model; and deploying the trained model on a server; and the application realizes message collection which is less affected by light and posture.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, specifically to a message acquisition method based on a multi-stage temporal convolutional network. Background Technology

[0002] Facial landmark detection refers to locating key points on a given face image, including points on the eyebrows, eyes, nose, lips, and facial contours. Due to factors such as pose and occlusion, facial landmark detection is a challenging task. It has wide applications in areas such as face pose alignment, face beautification and editing, and facial expression analysis.

[0003] Lip reading aims to predict what a person is saying based solely on visual information. With the development of neural networks, computers have largely surpassed human capabilities in this field. Lip reading can identify spoken content in noisy environments or with damaged audio, and can also extract the voice of a target speaker from overlapping audio files of multiple speakers. Traditional lip reading methods require training with user lip-reading video datasets. Because traditional lip reading methods use image sequences as input data for the model, they place high demands on the richness of the dataset. Ideally, for head pose, the image dataset would contain samples of different head poses to improve the model's adaptability and robustness. However, in real-world applications, building a large-scale lip reading dataset with rich data requires significant time and manpower. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a message acquisition method based on a multi-stage temporal convolutional network, which, while retaining the advantage of rich semantics in voice interaction, minimizes the sound interference of voice to the outside world and the sound interference of the outside world to voice, thereby solving the problems existing in the background technology.

[0005] Technical solution: The packet acquisition method based on a multi-stage temporal convolutional network described in this invention includes the following steps:

[0006] (1) Collect video data of the user's lip movements; detect and extract the coordinates of key feature points of the lip region from each frame of the video data;

[0007] (2) The extracted lip key point coordinates are standardized and normalized;

[0008] (3) Based on the calculated facial spatial orientation vector, the coordinates of key lip points under different head postures are uniformly transformed into standard frontal orientation coordinates; by analyzing the temporal changes of lip opening amplitude and according to the preset start and stop judgment rules, effective lip movement sequence segments are intelligently extracted from continuous video frames.

[0009] (4) Perform data augmentation on the lip key point sequence; use a multi-stage temporal convolutional network model to train the processed lip key point sequence to obtain a lip reading recognition model;

[0010] (5) Deploy the trained model on the server, and have the mobile device client collect lip video and send it to the server for recognition. The server will return the recognized text result to the client, and the client will automatically generate and execute the corresponding message operation based on the result.

[0011] Furthermore, in step (2), the standardization process is to convert the absolute coordinates of the key points of the lips into relative coordinates relative to the center point of the face; the normalization process refers to scaling the relative coordinates using the distance between the two eyes of the face as a reference.

[0012] Furthermore, in step (3), the coordinates of the key points of the lips under different head postures are uniformly transformed into standard frontal orientation coordinates. Specifically, the vertical vector representing the orientation of the face is calculated, and a rotation matrix is ​​calculated based on the vector. The rotation matrix is ​​used to rotate and transform the coordinates of all key points of the lips so that they correspond to the coordinates when facing the face.

[0013] Furthermore, in step (3), the intelligent extraction of effective lip-reading action sequence segments is as follows: calculate the temporal index reflecting the degree of lip opening; based on the magnitude, sum and comparison relationship of the index in multiple consecutive frames with the threshold, formulate composite rules for determining the start and end of lip-reading, and locate and cut out the complete lip-reading action sequence from the video stream according to the composite rules.

[0014] Furthermore, in step (4), the coordinate sequence of key points on the lips is used as the basic data unit to represent lip movements. For key points that are symmetrical on the left and right sides of the lips, the average value of their coordinates is taken as the representative point for storage.

[0015] Furthermore, in step (4), the data augmentation process includes one or more of temporal masking, adding random noise, and linear mixing between different sequences.

[0016] Furthermore, in step (4), the multi-stage temporal convolutional network model adopts a layer structure with multiple parallel branches, each branch using a convolutional kernel of a different size, in order to simultaneously capture feature information at different time scales in the lip action sequence.

[0017] The message acquisition system based on a multi-stage temporal convolutional network described in this invention includes:

[0018] Acquisition module: Used to acquire the user's lip movement data; detect and extract the coordinates of key feature points in the lip region from each frame of the data;

[0019] Preprocessing module: Used to standardize and normalize the extracted lip key point coordinates;

[0020] Lip reading module: Based on the calculated facial spatial orientation vector, it transforms the coordinates of key lip points under different head postures into standard frontal orientation coordinates; by analyzing the temporal changes in the lip opening amplitude and according to preset start and stop judgment rules, it intelligently extracts effective lip reading sequence segments from continuous video frames.

[0021] Lip reading module: used to perform data augmentation processing on the lip key point sequence; a multi-stage temporal convolutional network model is used to train the processed lip key point sequence to obtain the lip reading recognition model;

[0022] Message module: This module is used to deploy the trained model to the server. The mobile device client captures lip video and sends it to the server for recognition. The server returns the recognized text result to the client, and the client automatically generates and executes the corresponding message operation based on the result.

[0023] An electronic device according to the present invention includes a memory and a processor. The memory stores a computer program, and the processor executes the program to implement the steps of the method.

[0024] The present invention discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method.

[0025] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: Compared with touch control, lip reading avoids the difficulty of operating mobile devices with one hand. Compared with speech, lip reading avoids interference from external noise and does not disturb others. This invention uses transformed facial key points for lip reading recognition. During the extraction of facial key points, key points that cannot be recognized due to lighting and posture are inferred using a deep learning model. This allows for the acquisition of complete facial key point data based on images with poor head posture and lighting conditions, completing training and recognition, and thus enabling application in message acquisition scenarios in complex environments. Therefore, this method achieves message acquisition less affected by lighting and posture. Attached Figure Description

[0026] Figure 1This is a flowchart of the present invention;

[0027] Figure 2 This is a structural diagram of the MS-TCN of the present invention;

[0028] Figure 3 This is a diagram of the overall network structure of the present invention;

[0029] Figure 4 This is a schematic diagram of the system flow of the present invention. Detailed Implementation

[0030] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0031] like Figure 1 As shown, this embodiment of the invention provides a message acquisition method based on a multi-stage temporal convolutional network, including the following steps:

[0032] (1) Perform frame-by-frame detection of the user's lip movements. Open-source libraries such as dlib or MediaPipe can be used to extract facial key points. Taking MediaPipe as an example, the coordinates of 468 facial key points can be extracted from each frame of the video. Then, the focus is mainly on the coordinates of 86 lip key points, which are in three-dimensional form. shaft and The axis starts at the top-left corner of the camera's image and is normalized to 0 to 1 using both the image's width and depth. Coordinates The axis represents the depth of the key points, with the depth at the center of the head as the origin. The smaller the value, the closer the key points are to the camera.

[0033] (2) Standardize and normalize the obtained lip key point coordinates. Standardization involves transforming the coordinates of points at different spatial locations, converting their absolute coordinates to relative coordinates relative to the center point of the face, so that they can be compared. The standardization formula is:

[0034]

[0035] in, This represents the coordinates of the lower lip keypoints in the MediaPipe coordinate system. In the MediaPipe coordinate system, the top left corner of the image represents... shaft and The starting point of the axis, the average depth of the face coordinate points represents The starting point of the axis. This represents the coordinates of the center point of the face. The center point of the face is the intersection of the vector between the eyes and the vector from the center of the eyebrows to the bridge of the nose. This represents the standardized coordinates.

[0036] Normalization maps all keypoint coordinates to a fixed range, facilitating subsequent calculations. The normalization formula is:

[0037] in, This represents a vector that starts at the center point of the face and ends at a key point on the face. The distance between the eyes represents the width of a person's face. This represents the normalized vector. The specific calculation method for the face center point is as follows: Figure 1 As shown.

[0038] (3) After standardization and normalization, calculate the vertical vector of the face. First, calculate the cross product of the vector between the eyes and the vector from the center of the eyebrows to the bridge of the nose to obtain the facial orientation vector. The calculation formula is:

[0039]

[0040] in, This represents a vector originating from the left eye landmark and ending at the right eye landmark. It represents a vector that starts at the key point between the eyebrows and ends at the key point on the bridge of the nose. This represents the vertical vector of the face. Then, the Rhodrigus formula is used to calculate the rotation matrix, transforming the facial keypoint coordinates from the current coordinate system to those of a front-facing face. The formula for calculating the rotation matrix is:

[0041]

[0042] in, Represents a rotation matrix. for The identity matrix, represent In MediaPipe The included angle of the axis, Unit vector of rotation axis The cross product matrix. The expression is:

[0043] in Represents the unit vector of the rotation axis The coordinates of the axis of rotation. The formula for calculating the unit vector of the axis of rotation is:

[0044] in Represents the unit vector of the rotation axis. Represents the rotation axis vector. express The model. The formula for calculating the table is:

[0045]

[0046] in, Represents the rotation axis vector coordinates Represents the vector before rotation. This represents the rotated vector.

[0047] After obtaining the rotation matrix, the coordinates of each facial key point after rotation can be calculated using the following formula:

[0048] in, Represents the vector before rotation. This represents the rotated vector. Represents a rotation matrix.

[0049] Lip-reading sequences are extracted based on changes in lip opening amplitude. The formula for calculating lip opening amplitude is:

[0050]

[0051] Define the sliding variance of the lip opening amplitude o over k consecutive frames at time t. for

[0052]

[0053]

[0054] in, The value represents the sliding window size used to calculate the variance, and can be set to 20% of the total number of frames in the sequence. It is the average of the o values ​​of k frames within the window. If If the value exceeds the preset start threshold, it is determined that the user has started sending lip-reading messages; if it is below the preset end threshold, it is determined that the user has stopped sending lip-reading messages.

[0055] In addition, a series of rules were designed to prevent misjudgments. These mainly include the following:

[0056] (a) Determine when lip-reading begins, consecutive The extent to which the lips of the frame open There must be at least one frame that is greater than the threshold.

[0057] (b) Determine when lip-reading begins, continuously The extent to which the lips of the frame open The sum is greater than the threshold.

[0058] (c) When determining the end of lip-reading communication, continuously The extent to which the lips of the frame open All must be less than the threshold.

[0059] (d) When determining the end of lip-reading communication, continuously The extent to which the lips of the frame open The sum must be less than the threshold.

[0060] Based on the changes in the degree of lip opening, the above rules can be used to effectively cut out the user's lip-reading sequence.

[0061] The thresholds used to determine lip opening and closing need to be calibrated based on the specific application scenario. As an example, the steps are as follows:

[0062] (1) Collect video of the same user's complete messages multiple times in the application scenario, such as recording the user saying a specific command 10 times;

[0063] (2) Record the moment in the above sliding window from the time there is no lip reading to the moment when lip reading begins in these 10 videos. The peak values, calculate the average value V of these peak values. start_avg ;

[0064] (3) Record the sliding window during the non-lip-reading silent periods in these 10 videos. Calculate the mean V of the mean. sil_avg ;

[0065] (4) The activation threshold can be set to V sil_avg ×0.7, and this value must also be greater than V. sil_avg ×3, take the larger value of both to ensure robustness;

[0066] (5) The end threshold can be set to V sil_avg ×1.5;

[0067] (6) Similarly, for the amplitude threshold and sum threshold in the above composite rules, similar statistical methods can be used to set them by a certain multiple or quantile of the average value of multiple measurements in actual use.

[0068] It is understood that the specific numerical parameters mentioned above are for reference only. In actual deployment, they can be adjusted according to different data collection devices and usage environments to reduce false triggering and missed identification rates.

[0069] For each frame of a valid lip-reading sequence, the coordinates of the lip keypoints are saved. Furthermore, for symmetrical lip keypoints, the average value is calculated and saved. The formula for calculating the average value is:

[0070]

[0071] in, , , Indicates key points on the left side of the lip. , , This indicates the right lip keypoint corresponding to the keypoint on the left. , This represents the coordinates saved in the lip keypoint sequence. Then, 44 lip keypoints are saved in order of their spatial location on the lips. , , The three-axis coordinate system yields a sequence of key lip points. The spatial order follows a left-to-right, top-to-bottom principle.

[0072] (4) Data augmentation is required. When using images for recognition, flipping and cropping are commonly used for data augmentation. However, for coordinate point data at fixed locations, other data augmentation methods are needed. Temporal occlusion, adding Gaussian noise, and linear mixing are used to augment the key point coordinates.

[0073] Temporal masking is a common data augmentation method in lip-reading, enabling models to better recognize data in missing frames. This method masks N consecutive frames for each training sequence, where N is sampled using a uniform distribution between 0 and 15. Each masked frame is replaced by the average frame of its sequence. Each coordinate point in the average frame is the average of all identical coordinate points in the sequence.

[0074] Adding Gaussian noise is a widely used method for data augmentation of point cloud data. This method adds a normally distributed random number to one-third of the points in each training sequence. The numerical range of the added noise is (-1, 1).

[0075] Linear blending creates a new sequence by linearly blending frames from two different sequences. For each sequence, we randomly select 40% of the frames for linear blending.

[0076] (5) The MS-TCN (Multi-Stage Temporal Convolutional) network was used to train keypoint data and perform lip reading recognition. TCN is a one-dimensional convolutional network structure specifically designed for processing time series data. It directly models time series by stacking multiple one-dimensional convolutional layers. It can be considered that TCN extends the successful experience of convolutional neural networks in computer vision to time series data. TCN obtains a larger receptive field by changing the dilation factor, thereby acquiring more continuous temporal feature information. Compared with LSTM and Gate Recurrent Unit (GRU) models, TCN does not require a large amount of memory to store the partial results of its multi-unit gates, and it does not have the gradient explosion, gradient vanishing, and local overfitting problems common in LSTM and GRU. In general, TCN, which does not use a gate mechanism, has a longer temporal memory and is structurally simpler and clearer than recurrent networks such as LSTM and GRU. In the lip reading recognition task, the input data is a sequence of information about lip movements changing over time. Because TCN can effectively learn the characteristics of temporal information, it can handle complex temporal data such as lip reading very well.

[0077] In TCN, an important concept is causal convolution. Assuming... Given an input sequence for a time series forecasting model, the corresponding output sequence is: The training process of a time series prediction model involves analyzing and calculating the input sequence. The potential distribution. The prediction model finds the corresponding mapping function by training and adjusting parameters. So that the output satisfies At the same time, it minimizes the interpolation between the predicted and actual values, thereby optimizing the model's prediction performance.

[0078] For in Output in real time Predictive models can only utilize historical data. The input-output response relationship during training computation is called causal constraint. Causal convolution operates on data from the current time step and previous historical time steps. In this case, the convolution kernel can be regarded as a time-reversed filter. Causal convolution guarantees temporal causality and is better suited for modeling time-series data compared to non-causal convolution.

[0079] While causal convolutions in TCNs not only possess powerful temporal information processing capabilities but also enable parallel computation to improve network training efficiency, the number of hidden layers responsible for training increases with the length of the input time series, leading to a greater number of parameters involved in the network's computation. Simultaneously, to obtain a larger receptive field, the network depth also increases. With increasing network depth, the probability of gradient vanishing and other problems in neural networks also increases. Furthermore, overly complex training processes not only fail to improve the model's predictive performance but also reduce its operational efficiency, resulting in poor final fitting results. Therefore, to avoid the gradient vanishing problem caused by long input sequences, TCNs introduce dilated convolutions to further handle temporal sequence issues.

[0080] If the expansion factor is The kernel size is Assuming the filter is So for an input sequence The formula for calculating dilated convolution is:

[0081]

[0082] As can be seen from the formula, with the convolution kernel remaining constant, the larger the dilation factor, the larger the receptive field. When When, dilated convolution simplifies to regular convolution; when Furthermore, when the kernel size is 2, the receptive field is 5.

[0083] Therefore, combining causal convolution and dilated convolution can improve the parallelization capability and computational efficiency of the network. Causal dilated convolution layers can not only increase the network depth without increasing the number of model parameters, thus increasing the model's nonlinear fitting ability, but also prevent future information from being leaked during the model's training process, enabling it to better capture long-term dependencies in time-series data and further improve the model's predictive performance.

[0084] To further ensure model stability and training efficiency, and to avoid problems such as information loss, gradient explosion, and gradient vanishing that may occur as network depth increases, TCN designs corresponding residual modules. Assuming the causal dilated convolutional layer outputs after a series of convolution operations... The module input is Introducing cross-layer connections in each residual block can effectively avoid the above problems. The residual module is assembled from causal dilated convolutional layers, weight normalization layers (WeightNorm), rectified linear unit (ReLU) layers, and dropout layers.

[0085] The weight normalization layer trains the weights in the hidden layers between network layers using a weight normalization method, effectively reducing the gradient explosion problem caused by an excessive number of layers, which makes the model difficult to converge. ReLU is a common, simple, and effective neural network activation function, and its mathematical expression is:

[0086] ReLU layers suppress the activation of unimportant neurons during training, making the neurons in deep neural networks sparse and increasing the interpretability of the network. Compared to other common activation functions (such as sigmoid and tanh functions), the ReLU function can better extract relevant features to fit the training data. Furthermore, when parameters propagate between layers, the Dropout layer in the model randomly sets the output of some hidden layer nodes to zero with a certain probability, preventing some neurons from being triggered. This forces the neural network to learn more robust features, effectively reducing the complexity and number of parameters in the neural network.

[0087] Since the input and output have the same length, the receptive field of a TCN is defined by the kernel size and dilation factor. To provide the network with visibility across multiple time scales, allowing short-term and long-term information to be mixed during feature encoding, this paper employs MS-TCN. In MS-TCN, each temporal convolution consists of several branches, each with a different kernel size. Using three branches, the kernel sizes for each branch are 3, 5, and 7, respectively, and their outputs are concatenated as the final output. In this way, each convolutional layer mixes information across multiple time scales.

[0088] The TCN part consists of four temporal convolutional blocks. Each temporal convolutional block contains two MS-TCNs. The structure of the MS-TCN is as follows: Figure 2 As shown, each MS-TCN contains three branches: temporal convolutions with receptive fields of 3, 5, and 7. Each branch consists of a dilated casual convolutional layer, a normalization layer, and a ReLU activation function. The results of the three branches are concatenated, followed by a dropout layer. Applying a softmax layer after the TCN yields the final lip-reading prediction probability. The overall network structure is as follows. Figure 3 As shown.

[0089] During training, the epoch was set to 40, and Adam was used as the optimizer with an initial learning rate of 3e-4, weight decay of 1e-4, and a batch size of 16. Random horizontal flipping of coordinate points was used as data augmentation, consistently applied to all frames of the sequence. The commonly used cross-entropy loss function was employed. Specific settings can be adjusted depending on the dataset.

[0090] After obtaining the trained model, it can then be used to recognize the user's lip movements. During the recognition process, the data needs to be processed according to the steps described above.

[0091] (6) After the model is built, it is deployed on the server, and the mobile client is deployed for recognition. For the client, the common method of lip reading acquisition is to start lip reading when a face is detected. At the same time, a specific triggering mechanism is introduced to avoid misrecognition of messages, such as a specific trigger word such as "start". After the recognition starts, the client uses the algorithm to extract the lip reading sequence and returns it to the server. The server uses the model to recognize the lip reading and returns the result to the client. After receiving the message, the client performs the corresponding operation. Commonly used messages include task allocation and status report. The flowchart of the system operation is as follows. Figure 4 As shown.

Claims

1. A message acquisition method based on a multi-stage temporal convolutional network, characterized in that, Includes the following steps: (1) Collect video data of the user's lip movements; detect and extract the coordinates of key feature points of the lip region from each frame of the video data; The standardization process involves converting the absolute coordinates of key lip points into relative coordinates relative to the center of the face; the normalization process involves scaling the relative coordinates using the distance between the eyes as a reference. (2) The extracted lip key point coordinates are standardized and normalized; (3) Based on the calculated facial spatial orientation vector, the coordinates of key lip points under different head postures are uniformly transformed into standard frontal orientation coordinates; by analyzing the temporal changes in the lip opening amplitude and according to the preset start and stop judgment rules, effective lip-reading action sequence segments are intelligently extracted from continuous video frames; the coordinates of key lip points under different head postures are uniformly transformed into standard frontal orientation coordinates, specifically: by calculating the vertical vector representing the facial orientation, and calculating the rotation matrix based on the vector, the rotation matrix is ​​used to rotate and transform all key lip point coordinates so that they correspond to the coordinates when facing the face; the specific details of intelligently extracting effective lip-reading action sequence segments are as follows: calculate the temporal index reflecting the degree of lip opening; based on the magnitude, sum and comparison relationship of the index in multiple consecutive frames with the threshold, formulate a composite rule for determining the start and end of lip-reading, and locate and cut out the complete lip-reading action sequence from the video stream according to the composite rule; first, calculate the cross product of the vector between the eyes and the vector between the brow and the bridge of the nose to obtain the facial orientation vector; the calculation formula is: ； in, This represents a vector originating from the left eye landmark and ending at the right eye landmark. It represents a vector that starts at the key point between the eyebrows and ends at the key point on the bridge of the nose. This represents the vertical vector of the face; then, the rotation matrix is ​​calculated using the Rhodrigus formula to transform the coordinates of the facial key points in the current coordinate system into the coordinates of the facial key points facing forward; the formula for calculating the rotation matrix is: ;in, Represents a rotation matrix. for The identity matrix, represent In MediaPipe The included angle of the axis, Unit vector of rotation axis The cross product matrix; The expression is: ;in Represents the unit vector of the rotation axis The coordinates of the axis of rotation; the formula for calculating the unit vector of the axis of rotation is: ;in Represents the unit vector of the rotation axis. Represents the rotation axis vector. express The model; The formula for calculating the table is: ； in, Represents the rotation axis vector coordinates Represents the vector before rotation. Represents the rotated vector; After obtaining the rotation matrix, calculate the rotated coordinates of each facial key point using the following formula: ;in, Represents the vector before rotation. This represents the rotated vector. Represents a rotation matrix; extracts lip-reading sequence segments based on changes in lip opening amplitude; the formula for calculating lip opening amplitude is: ; Define the sliding variance of the lip opening amplitude o over k consecutive frames at time t. for: ； ； in, The value represents the size of the sliding window used to calculate the variance; It is the mean of the o values ​​of k frames within the window; if If the value exceeds the preset start threshold, it is determined that the user has started sending lip-reading messages; if it is below the preset end threshold, it is determined that the user has stopped sending lip-reading messages. The rules are as follows: (a) Determine when lip-reading begins, consecutive The extent to which the lips of the frame open There must be at least one frame that is greater than the threshold; (b) Determine when lip-reading begins, continuously The extent to which the lips of the frame open The sum is greater than the threshold; (c) When determining the end of lip-reading communication, continuously The extent to which the lips of the frame open All must be less than the threshold; (d) When determining the end of lip-reading communication, continuously The extent to which the lips of the frame open The sum must be less than the threshold; based on the changes in the degree of lip opening, the user's lip-reading sequence is ultimately effectively cropped out according to the rules. (4) Data augmentation processing of the lip keypoint sequence; training the processed lip keypoint sequence using a multi-stage temporal convolutional network model to obtain a lip reading recognition model; using the coordinate sequence of lip keypoints as the basic data unit representing lip movements, and storing the average coordinate of keypoints that are symmetrical about the left and right sides of the lips as representative points; data augmentation processing includes temporal occlusion, adding random noise, and linear mixing between different sequences; the multi-stage temporal convolutional network model adopts a layer structure with multiple parallel branches, each branch using a different size convolutional kernel to simultaneously capture feature information at different time scales in the lip movement sequence; in MS-TCN, each time The temporal convolution consists of several branches, each with a different kernel size. When using three branches, the kernel sizes of each branch are 3, 5, and 7, respectively. The outputs are concatenated together as the final output. The TCN part contains four temporal convolutional blocks in total. Each temporal convolutional block contains two MS-TCNs. Each MS-TCN contains three branches, which are temporal convolutions with receptive fields of 3, 5, and 7, respectively. Each branch consists of a causal dilated convolutional layer, a normalization layer, and a ReLU activation function. After concatenating the results of the three branches, there is a dropout layer. Applying a softmax layer after the TCN yields the final lip-reading prediction probability. (5) Deploy the trained model on the server, and have the mobile device client collect lip video and send it to the server for recognition. The server will return the recognized text results to the client, and the client will automatically generate and execute the corresponding message operation based on the results.

2. A message acquisition system based on a multi-stage temporal convolutional network, implemented using the method described in claim 1, characterized in that, include: Acquisition module: Used to acquire the user's lip movement data; detect and extract the coordinates of key feature points in the lip region from each frame of the data; Preprocessing module: used to standardize and normalize the extracted lip key point coordinates; Lip reading module: Based on the calculated facial spatial orientation vector, it transforms the coordinates of key lip points under different head postures into standard frontal orientation coordinates; by analyzing the temporal changes in the lip opening amplitude and according to preset start and stop judgment rules, it intelligently extracts effective lip reading sequence segments from continuous video frames. Lip reading module: used to perform data augmentation processing on the lip key point sequence; a multi-stage temporal convolutional network model is used to train the processed lip key point sequence to obtain the lip reading recognition model; Message module: Used to deploy the trained model on the server. The mobile device client captures lip video and sends it to the server for recognition. The server returns the recognized text results to the client, and the client automatically generates and executes the corresponding message operation based on the results.

Images