A method and system for recognizing patient lip reading based on improved big data algorithm

Abstract

The present invention relates to the field of machine learning technology, and discloses a method for recognizing patient lip readings based on an improved big data algorithm, comprising the following steps: step S1, collecting facial images of a person, and constructing the facial images into a data set using an improved PCA-SURF algorithm; step S2, constructing a feature recognition model based on an improved ResNeST module, expanding the data set using a data enhancement method, and then using the data set to train the feature recognition model; step S3, inputting the data set into the feature recognition model for training, wherein the feature recognition model includes an improved ResNeST model, and the improved ResNeST module is sequentially connected to an upstream module and a downstream branch module; step S4, collecting facial images of a person, and inputting them into the trained facial images to read the person's lip reading information. The present invention also provides a system for recognizing patient lip readings based on an improved big data algorithm. The present invention is used to improve the generalization ability of a machine learning model while solving the problem of information loss in PCA dimensionality reduction.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and more specifically, to a method and system for recognizing patient lip readings based on an improved big data algorithm. The method and system are used to solve the information loss caused by PCA dimensionality reduction while improving the generalization ability of the machine learning model and better recognizing patient lip readings. Background Art

[0002] In recent years, with the rapid development of deep learning in the field of computer recognition, deep learning and various pre-trained models have been widely used in many high-value application scenarios, such as medical scenarios. However, there are still challenges to be solved in improving the generalization ability of the model. Dataset construction is the key to the widespread application of deep learning pre-trained models. How to obtain and reasonably use the data set is the prerequisite for building an effective model.

[0003] PCA dimensionality reduction is an unsupervised algorithm because it can reduce data dimensionality without using labels. It is computationally simple, requiring only eigenvalue decomposition of a covariance matrix, and can retain most of the key information for dimensionality reduction. However, this algorithm has two major drawbacks in classification. One is that when the sample size is imbalanced, for example, if one class has a large sample size and the other classes have a small sample size, this can result in a new sample's K neighbors being dominated by samples from the large class. This algorithm only calculates the "nearest" neighbors. If a class has a large number of samples, then either that class is not close to the target sample or it is very close to the target sample. In either case, the number of samples does not affect the performance. This can be improved by using a weighting method, assigning larger weights to neighbors that are closer to the sample. Another drawback is the high computational complexity, as for each document to be classified, the distance from it to all known samples must be calculated to determine its K nearest neighbors. A common solution is to pre-clip the known samples to remove samples that are not very helpful for classification. This algorithm is more suitable for automatic classification of domains with large sample sizes.

[0004] Therefore, the present invention proposes a method for combining machine learning and an improved PCA-SURF algorithm to reduce the amount of information lost in dimensionality, combining the machine learning model with PCA dimensionality reduction, solving the problem of information loss in PCA dimensionality reduction while improving the generalization ability of the machine learning model. In addition, the present invention can be applied to medical scenarios, bringing more convenience to more and more troubled patients. For example, when a patient is unable to speak, such patients often cannot express themselves with their limbs when seeking medical treatment, so many problems will arise in communication. The use of the present invention can efficiently recognize the patient's lip language and conduct effective and fast communication with the patient. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for recognizing patient lip readings based on an improved big data algorithm, which is used to solve the problem of information loss caused by PCA dimensionality reduction while improving the generalization ability of the machine learning model.

[0006] The present invention is implemented through the following technical solution: a method for recognizing patient lip reading based on an improved big data algorithm, comprising the following steps:

[0007] Step S1, collecting facial images of people, and constructing the collected facial images into a data set using an improved PCA-SURF algorithm;

[0008] Step S2: constructing a feature recognition model based on the improved ResNeST module, expanding the data set using a data augmentation method, and then using the data set to train the feature recognition model;

[0009] Step S3: inputting the data set into a feature recognition model for training, wherein the feature recognition model includes an improved ResNeST model, wherein the improved ResNeST module is sequentially connected to an upstream module and a downstream branch module;

[0010] Step S4: collecting a human face image and inputting it into the trained human face image to read the human lip language information.

[0011] In order to better implement the present invention, further, step S1 includes:

[0012] The SURF algorithm extracts the feature points of the facial image through the Hessian algorithm and generates the descriptor feature vector of the SURF feature;

[0013] The PCA algorithm is decentralized and then combined with the sub-eigenvectors of the SURF feature to generate the covariance matrix;

[0014] The eigenvectors obtained from the covariance matrix during the PCA dimensionality reduction process are matched using the Euclidean distance, and the two feature points with a Euclidean distance of 0 are removed before constructing the data set.

[0015] In order to better implement the present invention, further, the improved ResNeST module in step 2 includes three channel layers, batch normalization layers and linear rectification unit layers connected in parallel, and finally performs softmax function calculation.

[0016] In order to better implement the present invention, further, the upstream module includes a multi-layer perception layer MLP and a first feature extraction layer and a second feature extraction layer connected in parallel, and the first feature extraction layer and the second feature extraction layer finally perform sigmoid function calculation.

[0017] In order to better implement the present invention, further, firstly, when the facial image of the human face is initially collected, the coordinates (x1~m, y1~m) of all samples m in the image are taken, and decentralized processing is performed in the PCA dimensionality reduction calculation to obtain the decentralized matrix H2;

[0018] The matrix H1x in the x-coordinate direction before decentralization is expressed as: H1x = [x1...xm] = [Tx1...Txm], where Tx1...Txm are eigenvectors;

[0019] The matrix H1y in the y-coordinate direction before decentralization is expressed as: H1y = [y1...ym] = [Ty1...Tym], where Ty1...Tym are eigenvectors;

[0020] The decentralized x-coordinate matrix H2x is expressed as: H2x = [T1-x01...Tm-x0m], where x01 is the mean of the x1 column and x0m is the mean of the xm column.

[0021] The decentralized y-coordinate matrix H2y is expressed as: H2y = [T1-y01...Tm-y0m] (4), where x01 is the mean of the x1 column, and x0m is the mean of the xm column.

[0022] Then, in the process of PCA dimensionality reduction, the eigenvector of the covariance matrix is ​​obtained. The definition formula is: Where m is all samples, j is the number of the jth sample, is a column vector, xi is the mean of x01 to x0m, and yi is the mean of y01 to y0m. The covariance matrix is ​​directly combined with the eigenvector of the SURF feature; the matching degree is determined according to the Euclidean distance formula.

[0023] In order to better implement the present invention, further, the downstream branch module includes a third feature extraction layer, and the third feature extraction layer includes a maximum pooling layer MAXPOOL3, a convolutional layer C6, an average pooling layer ACGPOOL3 and a convolutional layer C5 connected in sequence.

[0024] In order to better implement the present invention, further, the feature recognition model performs feature addition on the result calculated by the sigmoid function and the result output by the third feature extraction layer and then outputs the result.

[0025] In order to better implement the present invention, the present invention further provides a system for recognizing patient lip reading based on an improved big data algorithm, comprising an acquisition unit, a model building unit, a training unit, and a reading unit, wherein:

[0026] An acquisition unit, for acquiring facial images of human faces, and constructing the facial images into a data set using an improved PCA-SURF algorithm;

[0027] The model building unit is used to build a feature recognition model based on the improved ResNeST module, expand the data set using the data augmentation method, and use the data set to train the feature recognition model;

[0028] A training unit, configured to input a data set into a feature recognition model for training, wherein the feature recognition model comprises an improved ResNeST model, wherein the improved ResNeST module is sequentially connected to an upstream module and a downstream branch module;

[0029] The reading unit is used to collect human face images and input them into the trained human face images to read human lip language information.

[0030] In order to better implement the present invention, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the above method when executed by a processor.

[0031] In order to better implement the present invention, the present invention further provides an electronic device, comprising: a memory on which a computer program is stored; a processor for executing the computer program in the memory and the steps of the above method.

[0032] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0033] (1) The present invention uses an improved PCA-SURF algorithm, which uses the PCA algorithm to filter the feature points extracted by the SURF algorithm through the Hessian algorithm, removing redundant feature points while solving the problem of information loss in PCA dimensionality reduction. It is used to solve the problem of information loss in PCA dimensionality reduction while improving the generalization ability of the machine learning model;

[0034] (2) The present invention constructs a feature recognition model based on the improved ResNeST module. After expanding the data set using the data enhancement method, the data set is used to train the feature recognition model, which can improve the generalization ability of the model and better recognize the patient's lip language. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention is further described in conjunction with the following drawings and embodiments, and all concepts and innovations of the present invention should be regarded as disclosed contents and the protection scope of the present invention.

[0036] Figure 1 This is a structural diagram of the feature recognition model provided by the present invention.

[0037] Figure 2This is a schematic diagram of the structure of the improved ResNeST module provided by the present invention. DETAILED DESCRIPTION

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be understood that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, and therefore should not be regarded as limiting the scope of protection. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technical personnel in this field without making creative work are within the scope of protection of the present invention.

[0039] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "disposed," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections, electrical connections; direct connections, indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0040] Example 1:

[0041] This embodiment provides a method for recognizing patient lip reading based on an improved big data algorithm, such as Figure 1 As shown,.

[0042] Example 2:

[0043] This embodiment is further optimized based on Example 1. The present invention uses an improved PCA-SURF algorithm. The PCA algorithm is used to filter the feature points extracted by the SURF algorithm through the Hessian (Hessian matrix), removing redundant feature points while solving the problem of information loss in PCA dimensionality reduction.

[0044] The SURF algorithm is an improvement on the Sift algorithm proposed by David Lowe in 1999. It improves its execution efficiency and opens up the possibility of its application in real-time computer vision systems. Like the Sift algorithm, the basic process of the Surf algorithm can be divided into three parts: local feature point extraction, feature point description, and feature point matching. However, Surf achieves better execution efficiency than the Sift algorithm by using the integral graph on the Hessian matrix and using a dimensionality-reduced feature descriptor.

[0045] Specifically, the SURF feature point location process is to first construct a Hessian matrix, then use filters of the same size to construct a scale space to generate all the feature points for subsequent feature extraction. In this application, when the facial image is first collected, a Hessian matrix is ​​constructed for the image coordinates. The SURF algorithm extracts all the feature points of the facial image through the Hessian matrix. The specific operation is to perform a Gaussian filter on the Hessian matrix, and then calculate the decision value corresponding to each pixel and determinant in the image by discriminating the Hessian matrix. This decision value is then used to discriminate the local feature points of the image.

[0046] The specific process of discrimination is as follows: each pixel point processed by the Hessian matrix is ​​compared with all its adjacent points in the image domain and scale domain. After the feature points are preliminarily located, the erroneous feature points are removed through noise filtering to screen out the final feature points.

[0047] Like the Sift algorithm, the SURF algorithm determines the degree of match by calculating the Euclidean distance between the eigenvectors of two feature points. The shorter the Euclidean distance, the better the match between the two feature points. The Hessian matrix is ​​also added because the Hessian matrix is ​​a matrix composed of various coefficients of the neural network. Based on the positive and negative definite matrices, the positive and negative signs of the matrix traces of two feature points are determined. If they are the same, it means that the two features have contrast changes in the same direction. If they are different, it means that the contrast changes in the two feature points are in opposite directions. Even if the Euclidean distance is 0, the feature point is directly excluded.

[0048] After the SURF algorithm generates the descriptor eigenvector of the SURF feature, it takes 4×44×4 small rectangular areas in the neighborhood around the main direction of the feature point, counts the Haar features of each small area, and then obtains a 4-dimensional eigenvector for each area. A feature point has a total of 64-dimensional eigenvectors as the description of the SURF feature. When SURF is combined with PCA, the features described by the descriptor eigenvector of the SURF feature can be directly stored in the Hessian (Hessian matrix) to directly calculate the eigenvector of the covariance matrix. Because PCA dimensionality reduction is performed around each eigenvector, when PCA and SURF are combined, PCA will be decentralized and then combined with the eigenvector of the SURF feature. During the PCA dimensionality reduction process, the eigenvector of the covariance matrix is ​​obtained. The eigenvalue obtained from the covariance matrix is ​​the variance of the data projected on the corresponding new coordinate axis. It should be emphasized here that the PCA-SURF algorithm proposed in this application does not need to filter the SURF feature points first, because the sizes of the images between the image groups in SURF are the same, but different image groups will use filter templates of different sizes, and the same size but different coefficients will be used for the same image groups. Therefore, in the PCA-SURF algorithm, the features described by the descriptor eigenvector of the SURF feature can be directly stored in the Hessian (Hessian matrix) to directly calculate the eigenvector of the covariance matrix. The calculation process is as follows:

[0049] First, when extracting the facial image at the beginning, the coordinates (x1~m, y1~m) of all samples m in the image are taken, and decentralized processing is performed in the PCA dimensionality reduction calculation to obtain the decentralized matrix H2, as shown in formula (1), formula (2), formula (3) and formula (4):

[0050] The matrix H1x in the x-coordinate direction before decentralization is expressed as: H1x = [x1 ... xm] = [Tx1 ... Txm] (1) where Tx1 ... Txm are eigenvectors.

[0051] The matrix H1y in the y-coordinate direction before decentralization is expressed as: H1y = [y1...ym] = [Ty1...Tym] (2) where Ty1...Tym are eigenvectors.

[0052] The decentralized x-coordinate matrix H2x is expressed as: H2x = [T1-x01…Tm-x0m] (3) where x01 is the mean of the x1 column and x0m is the mean of the xm column.

[0053] The decentralized y-coordinate matrix H2y is expressed as: H2y = [T1-y01…Tm-y0m](4), where x01 is the mean of the x1 column and x0m is the mean of the xm column.

[0054] Then, the process of finding the eigenvector of the covariance matrix in the PCA dimensionality reduction process is shown in formula (5):

[0055] Covariance matrix The definition formula is: Among them, m is all samples, j is the number of j-th samples, is a column vector, xi is the mean of x01 to xom, and yi is the mean of y01 to y0m.

[0056] Formula (5) is directly combined with the feature vector of the SURF feature. Since the SURF algorithm takes a 4*4*4=64-dimensional vector as the descriptor of the Surf feature, the Euclidean distance between the two feature points is directly calculated when combining to determine the matching degree.

[0057] According to formula (6): Where ρ is the Euclidean distance between the point (x2, y2) and the point (x1, y1); |X| is the Euclidean distance from the point (x2, y2) to the origin.

[0058] Formula (7) can be obtained to calculate the Euclidean distance for a 4*4*4=64-dimensional vector Where d(x, y)P is the Euclidean distance between the point (xn, yn) and the point (x1, Luo1); |X| is the Euclidean distance from the point (xi, yi) to the origin.

[0059] The rest of this embodiment is the same as that of embodiment 1, so it will not be described again.

[0060] Example 3:

[0061] This embodiment is further optimized based on the above embodiment 1 or 2. Figure 1 and Figure 2As shown, the present invention constructs a feature recognition model based on an improved ResNeST module, the feature recognition model includes an improved ResNeST model, the improved ResNeST module is sequentially connected to an upstream module and a downstream branch module, the improved ResNeST module includes three channel layers, a batch normalization layer and a linear correction unit layer connected in parallel, and finally performs a softmax function calculation, the upstream module includes a multi-layer perception layer MLP and a first feature extraction layer and a second feature extraction layer connected in parallel, the first feature extraction layer and the second feature extraction layer finally perform a sigmoid function calculation, the downstream branch module includes a third feature extraction layer, the third feature extraction layer includes a maximum pooling layer MAXPOOL3, a convolutional layer C6, an average pooling layer ACGPOOL3 and a convolutional layer C5 connected in sequence, and the feature recognition model adds features of the result calculated by the sigmoid function and the result output by the third feature extraction layer and then outputs them.

[0062] The rest of this embodiment is the same as that of the above-mentioned embodiment 1 or 2, and thus will not be described in detail.

[0063] Example 4:

[0064] This embodiment is further optimized on the basis of any one of the above embodiments 1-3. The improved ResNeSt module includes three channel layers, batch normalization layers and linear correction unit layers connected in parallel. The first layer is the channel layer CLK1, the batch normalization layer BN1 and the linear correction unit layer R1, the second layer is the channel layer CLK2, the batch normalization layer BN2 and the linear correction unit layer R2, and the third layer is the channel layer CLK3, the batch normalization layer BN3 and the linear correction unit layer R3.

[0065] When data is input, it contains N samples, each with C channels, a height of H, and a width of W. After passing through the channel layer, the batch normalization layer calculates the mean and variance, operating on N, H, and W while retaining the dimension of channel C. The data then enters the linear rectification unit, so that different inputs activate different regions. The improved ResNeSt not only adapts to object recognition in complex and changing environments, but also has stronger feature extraction capabilities.

[0066] The rest of this embodiment is the same as any of the above embodiments 1-3, so it will not be repeated here.

[0067] Example 5:

[0068] This embodiment further optimizes any of the above embodiments 1-4. After the feature recognition model is built, relevant hyperparameters such as the number of iterations and learning rate are preset. The network parameter weights are optimized by continuously iterating and attenuating the loss value until the number of iterations equals the maximum number of iterations. Training of the training set is terminated, and the model performance is finally tested in a real-world scenario. The loss function is divided into a category loss function and a positioning regression loss function. The category loss function is used to calculate the loss value between the predicted category value and the predicted probability value, and the positioning regression loss function is used to calculate the loss value between the candidate region and the true bounding box.

[0069] The rest of this embodiment is the same as any of the above embodiments 1-4, so it will not be repeated here.

[0070] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0071] The present disclosure also provides an electronic device, including:

[0072] a memory having a computer program stored thereon;

[0073] A processor is used to execute the computer program in the memory to implement the steps of the method provided in the above method embodiment.

[0074] In another exemplary embodiment, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the above method embodiment. For example, the computer-readable storage medium may be the aforementioned memory including program instructions, which may be executed by a processor of an electronic device to perform the above method.

[0075] In another exemplary embodiment, a computer program product is also provided, which includes a computer program that can be executed by a programmable device, and has a code portion for executing the above method when executed by the programmable device.

[0076] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention falls within the scope of protection of the present invention.

Claims

1. A method for recognizing patient lip reading based on an improved big data algorithm, characterized in that: The following steps are involved: Step S1, collecting facial images of people, and constructing the collected facial images into a data set using an improved PCA-SURF algorithm; Step S2: constructing a feature recognition model based on the improved ResNeST module, expanding the data set using a data augmentation method, and then using the data set to train the feature recognition model; Step S3: inputting the data set into a feature recognition model for training, wherein the feature recognition model includes an improved ResNeST model, wherein the improved ResNeST module is sequentially connected to an upstream module and a downstream branch module; Step S4, collecting a human face image and inputting it into the trained human face image to read the human lip reading information; The step S1 comprises: The SURF algorithm extracts the feature points of the facial image through the Hessian algorithm and generates the descriptor feature vector of the SURF feature; The PCA algorithm is decentralized and then combined with the sub-eigenvectors of the SURF feature to generate the covariance matrix; The eigenvectors obtained from the covariance matrix during the PCA dimensionality reduction process are matched using the Euclidean distance, and the two feature points with a Euclidean distance of 0 are removed to construct the data set. The improved ResNeST module in step S2 includes three parallel-connected channel layers, batch normalization layers, and linear rectification unit layers, and finally performs softmax function calculation; The upstream module includes a multi-layer perception layer MLP and a first feature extraction layer and a second feature extraction layer connected in parallel, and the first feature extraction layer and the second feature extraction layer finally perform sigmoid function calculation; The downstream branch module includes a third feature extraction layer, and the third feature extraction layer includes a maximum pooling layer MAXPOOL3, a convolutional layer C6, an average pooling layer ACGPOOL3 and a convolutional layer C5 connected in sequence; The feature recognition model performs feature addition on the result calculated by the sigmoid function and the result output by the third feature extraction layer and then outputs the result.

2. The method for recognizing patient lip reading based on an improved big data algorithm according to claim 1, characterized in that: include: First, when extracting the facial image at the beginning, the coordinates (x1~m, y1~m) of all samples m in the image are taken, and decentralized processing is performed in the PCA dimensionality reduction calculation to obtain the decentralized matrix H2; The matrix H1x in the x-coordinate direction before decentralization is expressed as: H1x=[x1...xm]=[Tx1...Txm], where Tx1...Txm are eigenvectors; The matrix H1y in the y-coordinate direction before decentralization is expressed as: H1y=[y1...ym]=[Ty1...Tym], where Ty1...Tym are eigenvectors; The decentralized x-coordinate matrix H2x is expressed as: H2x=[T1-x01...Tm-x0m], where x01 is the mean of the x1 column and x0m is the mean of the xm column. The decentralized y-coordinate matrix H2y is expressed as: H2y=[T1-y01...Tm-y0m] (4), where x01 is the mean of the x1 column, and x0m is the mean of the xm column. Then, in the process of PCA dimensionality reduction, the eigenvector of the covariance matrix is ​​obtained. The definition formula is: = xi*yi, where m is the total number of samples, j is the number of the jth sample, is a column vector, xi is the mean of x01 to x0m, and yi is the mean of y01 to y0m. The covariance matrix is ​​directly combined with the eigenvector of the SURF feature; the matching degree is determined according to the Euclidean distance formula.

3. A system for recognizing patient lip reading based on an improved big data algorithm, used to implement the method for recognizing patient lip reading based on an improved big data algorithm as claimed in claim 1; characterized in that: It includes an acquisition unit, a model building unit, a training unit, and a reading unit, wherein: The acquisition unit is used to acquire facial images and construct a dataset from the facial images using the improved PCA-SURF algorithm; the model construction unit is used to construct a feature recognition model based on the improved ResNeST module, expand the dataset using the data enhancement method, and then use the dataset to train the feature recognition model; A training unit, configured to input a data set into a feature recognition model for training, wherein the feature recognition model comprises an improved ResNeST model, wherein the improved ResNeST module is sequentially connected to an upstream module and a downstream branch module; The reading unit is used to collect human face images and input them into the trained human face images to read human lip language information.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.

5. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 2.

Images