Gesture lip language recognition operation method applied to political data consultation

Through multi-camera calibration and natural language processing technology, the accuracy of gesture and lip recognition in government affairs halls is improved, and the problems of recognition accuracy and communication difficulties in the existing technology are solved, and convenient consultation and efficient recognition of special groups are achieved.

CN118939115APending Publication Date: 2024-11-12GUANGDONG SKYNET SMART CITY TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410969186.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing gesture and lip recognition technology has insufficient recognition accuracy and speed, especially in the government hall environment with concentrated services, which makes it difficult to communicate effectively, especially for citizens who are illiterate and can only speak sign or lip. Words lead to communication difficulties.

Method used

The political digital service consulting desk with multiple cameras is used to generate a three-dimensional model through camera calibration processing, combining gesture and lip recognition models, and using natural language processing technology to process data to improve recognition accuracy and robustness.

Benefits of technology

实现了对手势和唇语的高精度识别,方便特殊群体进行咨询,避免了市民办事难的困境,提高了识别精度和鲁棒性,能够在有遮挡情况下进行准确识别。

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939115A_ABST
    Figure CN118939115A_ABST
Patent Text Reader

Abstract

The invention discloses a gesture lip language recognition operation method applied to political data consultation. The gesture lip language recognition operation method comprises the following steps that S1, a camera module shoots videos of gestures and lip languages of a user and obtains a gesture image data set and a lip language image data set through preprocessing; s2, substituting the gesture image data set into a gesture recognition model to obtain a gesture recognition data set, and substituting the lip language image data set into a lip language recognition model to obtain a lip language recognition data set; s3, searching reply data preset in the control module according to the sign language recognition question data set and the lip language recognition question data set, and displaying the reply data through a display module; by the adoption of the operation method, the user can consult the consultation desk through sign language or lip language, convenience is brought to special groups, and the difficulty that citizens handle affairs is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an operating method for gesture and lip reading recognition applied to political and numerical consulting. Background Art

[0002] The government hall is part of the government, and is also a place for people to do business. It undertakes many public service functions. However, the service content of the government hall is diverse, including but not limited to taxation, household registration, social security, real estate and many other fields. Although this multi-business concentration model improves service efficiency, it also creates many problems. For citizens who come to the government hall to do business, they usually get lost in the face of complicated roads and numerous service windows. Therefore, citizens usually need to ask the information desk or guide, but there are currently some citizens who are illiterate and can only use sign language or lip reading. For illiterate citizens, they can solve their questions by consulting the guide, but for citizens who use sign language and lip reading, it is more difficult to communicate with the guide.

[0003] However, the text recognized by existing equipment and technology for sign language is rather stiff. Due to the rapid changes in recognition angle or gestures, sometimes the middle gesture cannot be correctly recognized, and even the concatenated sentences are very different from the intended meaning. It is necessary to express gestures at a slower speed to more accurately recognize all gestures. Similarly, when lip reading recognition cannot be correctly recognized, the concatenated sentences are very different from the intended meaning.

[0004] Therefore, a control method that can improve the accuracy of gesture and lip reading recognition is currently needed. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide an operating method for gesture and lip reading recognition applied to political and numerical consultation.

[0006] The technical solution adopted by the present invention to solve the technical problem is to provide an operation method for gesture and lip language recognition applied to political and digital consultation, including a political and digital service consultation desk, wherein the political and digital service consultation desk is provided with a camera module, a wake-up module, a control module and a display module;

[0007] The operation method includes the following steps:

[0008] S1. The user triggers the wake-up module to wake up the government digital service consultation desk. The user asks questions to the government digital service consultation desk through sign language or lip reading. The camera module captures the user's gesture and lip reading videos and obtains gesture image datasets and lip reading image datasets through preprocessing.

[0009] S2, the control module is preset with a gesture recognition model and a lip reading recognition model, the gesture image data set is substituted into the gesture recognition model to obtain a gesture recognition data set, and the lip reading image data set is substituted into the lip reading recognition model to obtain a lip reading recognition data set;

[0010] S3. Process the gesture recognition data set and the lip reading recognition data set through natural language processing technology to obtain a sign language recognition question data set and a lip reading recognition question data set, search for the answer data preset in the control module according to the sign language recognition question data set and the lip reading recognition question data set, and display them through the display module.

[0011] As a further improvement of the present invention, the camera module includes at least two cameras, and before step S1, it also includes the step of: performing camera calibration processing on all cameras simultaneously, and the camera calibration processing includes internal parameter calibration, external parameter calibration, stereo calibration verification, consistency check and occlusion check.

[0012] As a further improvement of the present invention, the specific steps of the intrinsic parameter calibration are: placing the calibration piece at different angles, brightness and positions in the common shooting area of ​​all cameras, the surface of the calibration piece is painted with a chessboard of a determined size, each camera shoots the calibration piece to obtain an intrinsic parameter calibration photo, locates the feature points in the intrinsic parameter calibration photo through a feature detection algorithm, substitutes the feature points and the coordinate system into the camera calibration algorithm to calculate the estimated intrinsic parameter matrix of each camera, calculates the theoretical feature point projection position according to the estimated intrinsic parameter matrix, calculates the actual feature point projection position according to the feature points and the coordinate system, calculates the projection error between the theoretical feature point projection position and the actual feature point projection position, and when the projection error is less than the standard error preset in the control module, the estimated intrinsic parameter matrix is ​​applied to the gesture recognition model and the lip reading recognition model, and when the projection error is greater than or equal to the standard error preset in the control module, continues to shoot the calibration piece at different angles, brightness and positions to calculate a new estimated intrinsic parameter matrix.

[0013] As a further improvement of the present invention, the specific steps of external parameter calibration are: placing the calibration piece at different angles, brightness and positions in the common shooting area of ​​all cameras, the surface of the calibration piece is painted with a chessboard of a certain size, each camera shoots the calibration piece to obtain an external parameter calibration photo, locates the feature points in the external parameter calibration photo through a feature detection algorithm, matches the feature points of the external parameter calibration photos of different cameras, calculates the rotation matrix and translation matrix between different cameras according to the matched feature points by the least squares method, calculates the position difference between the external parameter calibration photo of a certain camera and the external parameter calibration photo of another camera according to the rotation matrix and translation matrix to obtain disparity data, generates a disparity map according to the disparity data, converts the disparity map into point cloud data, and each pixel in the point cloud data has a unique three-dimensional coordinate so that when the gesture image data set is substituted into the gesture recognition model, a gesture three-dimensional model can be generated, and when the lip reading image data set is substituted into the lip reading recognition model, a lip reading three-dimensional model can be generated.

[0014] As a further improvement of the present invention, the specific steps of verifying the stereo calibration are: substituting the test gesture data set and the test lip reading data set to obtain the test gesture three-dimensional model and the test lip reading three-dimensional model, comparing the accuracy of the test gesture three-dimensional model and the actual gesture three-dimensional model, and comparing the accuracy of the test lip reading three-dimensional model and the actual lip reading three-dimensional model.

[0015] As a further improvement of the present invention, the specific steps of consistency check are: checking the consistency of each camera coordinate system;

[0016] The specific steps of occlusion check are: check the disparity map generated by each camera,

[0017] If there is a discontinuous area, it is determined that the area may be blocked;

[0018] If there is an unclear boundary, it is determined that the boundary may be blocked.

[0019] As a further improvement of the present invention, the specific steps of capturing the user's gesture and lip reading videos by the camera module in step S1 and obtaining the gesture image dataset and the lip reading image dataset through preprocessing respectively include:

[0020] S11, dividing the gesture video and the lip reading video into gesture images and lip reading images in sequence;

[0021] S12, comparing the differences between adjacent gesture images and adjacent lip reading images, and removing static images and images with little change;

[0022] S13, adjusting all gesture images and lip reading images to the same size and unifying the brightness of the images;

[0023] S14, converting all gesture images and lip reading images into HSV color space images, the control module is preset with a standard hand H hue range, a standard hand S saturation range, and a standard hand V brightness range to define the HSV range of hand skin color, the control module is preset with a standard lip H hue range, a standard lip S saturation range, and a standard lip V brightness range to define the HSV range of lip skin color, creating a binary mask, setting pixels that meet the standard hand H hue range, the standard hand S saturation range, and the standard hand V brightness range as white pixels, and setting pixels that do not meet the conditions as black pixels, wherein the largest and connected white pixel represents the hand area, deleting the black pixel area, and integrating all processed gesture images into a gesture image dataset;

[0024] The pixels that meet the standard lip H hue range, the standard lip S saturation range, and the standard lip V brightness range are set as white pixels, and the pixels that do not meet the conditions are set as black pixels. The largest and most connected white pixels represent the lip area. The black pixel area is deleted, and all the processed lip reading images are integrated into a lip reading image dataset.

[0025] As a further improvement of the present invention, the recognition model includes a plurality of standard three-dimensional posture data and vocabulary data corresponding to the standard three-dimensional posture data one by one. In step S2, the gesture image data set is substituted into the gesture recognition model to obtain the gesture recognition data set, and the lip reading image data set is substituted into the lip reading recognition model to obtain the lip reading recognition data set. The specific steps include:

[0026] S21, according to the estimated internal parameter matrix, rotation matrix and translation matrix in the gesture recognition model and the lip reading recognition model, identify each gesture image and lip reading image in the gesture image dataset and the lip reading image dataset to obtain disparity data, generate a disparity map according to the disparity data, convert the disparity map into point cloud data to generate point cloud data, and respectively form a three-dimensional model of the gesture to be recognized and a three-dimensional model of the lip reading to be recognized according to the point cloud data;

[0027] S22. Compare the three-dimensional model of the gesture to be recognized and the three-dimensional model of the lip reading to be recognized with the standard three-dimensional posture database one by one, select the standard three-dimensional posture database with the highest overlap, select the vocabulary data corresponding to the standard three-dimensional posture database, and obtain the gesture recognition data set and the lip reading recognition data set.

[0028] As a further improvement of the present invention, the specific steps of processing the gesture recognition data set and the lip reading recognition data set by natural language processing technology in step S3 to obtain the sign language recognition question data set and the lip reading recognition question data set include:

[0029] S31, using the SpaCy dependency parser to parse the vocabulary data in the gesture recognition dataset and the lip reading recognition dataset to obtain a dependency result database to determine the grammatical relationship between the vocabulary;

[0030] S32, using component syntax to analyze the vocabulary data in the gesture recognition dataset and the lip reading recognition dataset to obtain a component syntax result database, and construct a syntax tree to determine the component hierarchical relationship of the vocabulary in the sentence;

[0031] S33. Use SRL technology to analyze the vocabulary data in the gesture recognition dataset and the lip reading recognition dataset to obtain the SRL result database to identify the predicate and argument in the sentence, so as to determine the initiator, recipient and affected object of the action;

[0032] S34, using NER tools to identify specific entity names in the text to obtain a NER result database;

[0033] S35, integrating the dependency relationship result database, the component syntax result database, the SRL result database and the NER result database to obtain a preliminary sign language recognition question dataset and a preliminary lip reading recognition question dataset;

[0034] S36. Post-process the prepared sign language recognition question data set and the prepared lip reading recognition question data set to obtain a sign language recognition question data set and a lip reading recognition question data set.

[0035] As a further improvement of the present invention, the specific steps of post-processing are: converting the preliminary sign language recognition question data set and the preliminary lip reading recognition question data set into Unicode encoded text;

[0036] The Unicode encoded text is dereferenced through the stanza library, and all the modified places are integrated to form the first modified data set;

[0037] Correct the typos in the Unicode text through the corrector library, and integrate all the modified parts to form the second modified data set;

[0038] According to the first modified data set and the second modified data set, the corresponding codes in the Unicode encoded text are deleted or modified to form a sign language recognition question data set and a lip reading recognition question data set.

[0039] The beneficial effects of the present invention are at least: users can consult the information desk through sign language or lip reading, which is convenient for special groups and avoids the difficulties of citizens in handling affairs; recognition is performed through multiple cameras, and even if there are objects blocking the view, other cameras can be used to take pictures, which greatly improves the recognition accuracy, and simultaneous camera calibration processing for all cameras can effectively improve the recognition accuracy and improve robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0041] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0042] Reference Figure 1 , the present invention proposes an operation method for gesture and lip language recognition applied to political and digital consultation, including a political and digital service consultation desk, wherein the political and digital service consultation desk is provided with a camera module, a wake-up module, a control module and a display module;

[0043] The operation method includes the following steps:

[0044] S1. The user triggers the wake-up module to wake up the government digital service consultation desk. The user asks questions to the government digital service consultation desk through sign language or lip reading. The camera module captures the user's gesture and lip reading videos and obtains gesture image datasets and lip reading image datasets through preprocessing.

[0045] S2, the control module is preset with a gesture recognition model and a lip reading recognition model, the gesture image data set is substituted into the gesture recognition model to obtain a gesture recognition data set, and the lip reading image data set is substituted into the lip reading recognition model to obtain a lip reading recognition data set;

[0046] S3. Process the gesture recognition data set and the lip reading recognition data set through natural language processing technology to obtain a sign language recognition question data set and a lip reading recognition question data set, search for the answer data preset in the control module according to the sign language recognition question data set and the lip reading recognition question data set, and display them through the display module.

[0047] Users can consult the information desk through sign language or lip reading, which facilitates special groups and avoids the difficulties faced by citizens in getting things done.

[0048] As a further improvement of the present invention, the camera module includes at least two cameras, and before step S1, it also includes the step of: performing camera calibration processing on all cameras simultaneously, and the camera calibration processing includes internal parameter calibration, external parameter calibration, stereo calibration verification, consistency check and occlusion check.

[0049] By using multiple cameras for recognition, even if there are objects blocking the image, other cameras can be used to take pictures, which greatly improves the recognition accuracy. Moreover, calibrating all cameras at the same time can effectively improve the recognition accuracy and robustness.

[0050] As a further improvement of the present invention, the specific steps of the intrinsic parameter calibration are: placing the calibration piece at different angles, brightness and positions in the common shooting area of ​​all cameras, the surface of the calibration piece is painted with a chessboard of a determined size, each camera shoots the calibration piece to obtain an intrinsic parameter calibration photo, locates the feature points in the intrinsic parameter calibration photo through a feature detection algorithm, substitutes the feature points and the coordinate system into the camera calibration algorithm to calculate the estimated intrinsic parameter matrix of each camera, calculates the theoretical feature point projection position according to the estimated intrinsic parameter matrix, calculates the actual feature point projection position according to the feature points and the coordinate system, calculates the projection error between the theoretical feature point projection position and the actual feature point projection position, and when the projection error is less than the standard error preset in the control module, the estimated intrinsic parameter matrix is ​​applied to the gesture recognition model and the lip reading recognition model, and when the projection error is greater than or equal to the standard error preset in the control module, continues to shoot the calibration piece at different angles, brightness and positions to calculate a new estimated intrinsic parameter matrix.

[0051] Specifically, the feature points in the internal reference calibration photo are located through the feature detection algorithm. The edges and corners of the checkerboard are automatically identified through Harris corner detection. The positions of these feature points in the image are called image coordinates (uv). Subsequently, the world coordinates (XYZ) are calculated and the feature points and coordinate system are substituted into the pinhole camera model matrix equation for solution. The subsequent content has been clearly recorded in the above technical solution and will not be repeated here.

[0052] As a further improvement of the present invention, the specific steps of external parameter calibration are: placing the calibration piece at different angles, brightness and positions in the common shooting area of ​​all cameras, the surface of the calibration piece is painted with a chessboard of a certain size, each camera shoots the calibration piece to obtain an external parameter calibration photo, locates the feature points in the external parameter calibration photo through a feature detection algorithm, matches the feature points of the external parameter calibration photos of different cameras, calculates the rotation matrix and translation matrix between different cameras according to the matched feature points by the least squares method, calculates the position difference between the external parameter calibration photo of a certain camera and the external parameter calibration photo of another camera according to the rotation matrix and translation matrix to obtain disparity data, generates a disparity map according to the disparity data, converts the disparity map into point cloud data, and each pixel in the point cloud data has a unique three-dimensional coordinate so that when the gesture image data set is substituted into the gesture recognition model, a gesture three-dimensional model can be generated, and when the lip reading image data set is substituted into the lip reading recognition model, a lip reading three-dimensional model can be generated.

[0053] The steps of external parameter calibration are similar to those of internal parameter calibration. Specifically, the disparity map contains the lateral offset of each pixel relative to the baseline (the distance between the centers of the two cameras). This offset is inversely proportional to the depth of the scene point corresponding to the pixel (i.e., the distance from the camera). Through the disparity map, we can obtain the depth information of each pixel, which is the basis for building a three-dimensional model.

[0054] As a further improvement of the present invention, the specific steps of verifying the stereo calibration are: substituting the test gesture data set and the test lip reading data set to obtain the test gesture three-dimensional model and the test lip reading three-dimensional model, comparing the accuracy of the test gesture three-dimensional model and the actual gesture three-dimensional model, and comparing the accuracy of the test lip reading three-dimensional model and the actual lip reading three-dimensional model.

[0055] As a further improvement of the present invention, the specific steps of consistency check are: checking the consistency of each camera coordinate system;

[0056] The specific steps of occlusion check are: check the disparity map generated by each camera,

[0057] If there is a discontinuous area, it is determined that the area may be blocked;

[0058] If there is an unclear boundary, it is determined that the boundary may be blocked.

[0059] As a further improvement of the present invention, the specific steps of capturing the user's gesture and lip reading videos by the camera module in step S1 and obtaining the gesture image dataset and the lip reading image dataset through preprocessing respectively include:

[0060] S11, dividing the gesture video and the lip reading video into gesture images and lip reading images in sequence;

[0061] S12, comparing the differences between adjacent gesture images and adjacent lip reading images, and removing static images and images with little change;

[0062] S13, adjusting all gesture images and lip reading images to the same size and unifying the brightness of the images;

[0063] S14, converting all gesture images and lip reading images into HSV color space images, the control module is preset with a standard hand H hue range, a standard hand S saturation range, and a standard hand V brightness range to define the HSV range of hand skin color, the control module is preset with a standard lip H hue range, a standard lip S saturation range, and a standard lip V brightness range to define the HSV range of lip skin color, creating a binary mask, setting pixels that meet the standard hand H hue range, the standard hand S saturation range, and the standard hand V brightness range as white pixels, and setting pixels that do not meet the conditions as black pixels, wherein the largest and connected white pixel represents the hand area, deleting the black pixel area, and integrating all processed gesture images into a gesture image dataset;

[0064] The pixels that meet the standard lip H hue range, the standard lip S saturation range, and the standard lip V brightness range are set as white pixels, and the pixels that do not meet the conditions are set as black pixels. The largest and most connected white pixels represent the lip area. The black pixel area is deleted, and all the processed lip reading images are integrated into a lip reading image dataset.

[0065] Specifically, under normal circumstances, the H hue value of skin color is roughly between 0° and 35° (between red and yellow) and 330° and 360° (between orange and red), the S saturation threshold range may be between 30% and 75%, and the V brightness threshold range may be set between 20% and 95%. Since the light levels are different when shooting at different times, the V brightness judgment will be inaccurate, so the light and darkness of the photos will be uniformly adjusted according to the background environment during preprocessing.

[0066] As a further improvement of the present invention, the recognition model includes a plurality of standard three-dimensional posture data and vocabulary data corresponding to the standard three-dimensional posture data one by one. In step S2, the gesture image data set is substituted into the gesture recognition model to obtain the gesture recognition data set, and the lip reading image data set is substituted into the lip reading recognition model to obtain the lip reading recognition data set. The specific steps include:

[0067] S21, according to the estimated internal parameter matrix, rotation matrix and translation matrix in the gesture recognition model and the lip reading recognition model, identify each gesture image and lip reading image in the gesture image dataset and the lip reading image dataset to obtain disparity data, generate a disparity map according to the disparity data, convert the disparity map into point cloud data to generate point cloud data, and respectively form a three-dimensional model of the gesture to be recognized and a three-dimensional model of the lip reading to be recognized according to the point cloud data;

[0068] S22. Compare the three-dimensional model of the gesture to be recognized and the three-dimensional model of the lip reading to be recognized with the standard three-dimensional gesture database one by one, select the standard three-dimensional gesture database with the highest overlap, select the vocabulary data corresponding to the standard three-dimensional gesture database, and obtain the gesture recognition data set and the lip reading recognition data set.

[0069] The estimated intrinsic parameter matrix describes the internal properties of the camera, such as focal length, pixel size, principal point offset, etc. It defines the projection relationship from 3D space to the 2D image plane. The rotation matrix describes the rotation of the camera coordinate system relative to the world coordinate system. It is a 3x3 orthogonal matrix belonging to the SO(3) group, representing the direction change in three-dimensional space; the translation matrix describes the position offset of the camera coordinate system relative to the world coordinate system. It is usually a 3x1 vector, representing the displacement from the world origin to the camera center.

[0070] By estimating the intrinsic parameter matrix, rotation matrix and translation matrix, the points on the image can be back-projected into the three-dimensional space to find the corresponding points. The multi-view geometry principle is realized by more than two cameras. After the camera is calibrated, the positions of these feature points in the three-dimensional space can be determined by matching the feature points, thereby reconstructing the entire three-dimensional model.

[0071] As a further improvement of the present invention, the specific steps of processing the gesture recognition data set and the lip reading recognition data set by natural language processing technology in step S3 to obtain the sign language recognition question data set and the lip reading recognition question data set include:

[0072] S31, using the SpaCy dependency parser to parse the vocabulary data in the gesture recognition dataset and the lip reading recognition dataset to obtain a dependency result database to determine the grammatical relationship between the vocabulary;

[0073] S32, using component syntax to analyze the vocabulary data in the gesture recognition dataset and the lip reading recognition dataset to obtain a component syntax result database, and construct a syntax tree to determine the component hierarchical relationship of the vocabulary in the sentence;

[0074] S33. Use SRL technology to analyze the vocabulary data in the gesture recognition dataset and the lip reading recognition dataset to obtain the SRL result database to identify the predicate and argument in the sentence, so as to determine the initiator, recipient and affected object of the action;

[0075] S34, using NER tools to identify specific entity names in the text to obtain a NER result database;

[0076] S35, integrating the dependency relationship result database, the component syntax result database, the SRL result database and the NER result database to obtain a preliminary sign language recognition question dataset and a preliminary lip reading recognition question dataset;

[0077] S36. Post-process the prepared sign language recognition question data set and the prepared lip reading recognition question data set to obtain a sign language recognition question data set and a lip reading recognition question data set.

[0078] As a further improvement of the present invention, the specific steps of post-processing are: converting the preliminary sign language recognition question data set and the preliminary lip reading recognition question data set into Unicode encoded text;

[0079] The Unicode encoded text is dereferenced through the stanza library, and all the modified places are integrated to form the first modified data set;

[0080] Correct the typos in the Unicode text through the corrector library, and integrate all the modified parts to form the second modified data set;

[0081] According to the first modified data set and the second modified data set, the corresponding codes in the Unicode encoded text are deleted or modified to form a sign language recognition question data set and a lip reading recognition question data set.

Claims

1. An operating method for gesture and lip reading recognition applied to political and digital consultation, characterized in that: It includes a government digital service consultation desk, which is provided with a camera module, a wake-up module, a control module and a display module; The operation method includes the following steps: S1. The user triggers the wake-up module to wake up the government and digital service consultation desk. The user asks questions to the government and digital service consultation desk through sign language or lip reading. The camera module captures the user's gesture and lip reading videos and obtains gesture image datasets and lip reading image datasets through preprocessing. S2, the control module is preset with a gesture recognition model and a lip reading recognition model, the gesture image data set is substituted into the gesture recognition model to obtain a gesture recognition data set, and the lip reading image data set is substituted into the lip reading recognition model to obtain a lip reading recognition data set; S3. Process the gesture recognition data set and the lip reading recognition data set through natural language processing technology to obtain a sign language recognition question data set and a lip reading recognition question data set, search for the answer data preset in the control module according to the sign language recognition question data set and the lip reading recognition question data set, and display them through the display module.

2. According to claim 1, the method for gesture and lip reading recognition for political and digital consultation is characterized in that: The camera module includes at least two cameras, and before step S1, also includes the step of: performing camera calibration processing on all cameras simultaneously, and the camera calibration processing includes internal parameter calibration, external parameter calibration, stereo calibration verification, consistency check and occlusion check.

3. The method for gesture and lip reading recognition for political and digital consultation according to claim 2 is characterized in that: The specific steps of the intrinsic parameter calibration are: placing the calibration piece at different angles, brightness and positions in the common shooting area of ​​all cameras, the surface of the calibration piece is painted with a chessboard of a determined size, each camera shoots the calibration piece to obtain an intrinsic parameter calibration photo, locates the feature points in the intrinsic parameter calibration photo through a feature detection algorithm, substitutes the feature points and the coordinate system into the camera calibration algorithm to calculate the estimated intrinsic parameter matrix of each camera, calculates the theoretical feature point projection position according to the estimated intrinsic parameter matrix, calculates the actual feature point projection position according to the feature points and the coordinate system, calculates the projection error between the theoretical feature point projection position and the actual feature point projection position, and when the projection error is less than the standard error preset in the control module, applies the estimated intrinsic parameter matrix to the gesture recognition model and the lip reading recognition model, and when the projection error is greater than or equal to the standard error preset in the control module, continues to shoot the calibration piece at different angles, brightness and positions to calculate a new estimated intrinsic parameter matrix.

4. The method for gesture and lip reading recognition for political and digital consultation according to claim 2 is characterized in that: The specific steps of external parameter calibration are: placing the calibration piece at different angles, brightness and positions in the common shooting area of ​​all cameras, the surface of the calibration piece is painted with a chessboard of a certain size, each camera shoots the calibration piece to obtain an external parameter calibration photo, the feature points in the external parameter calibration photo are located by a feature detection algorithm, the feature points of the external parameter calibration photos of different cameras are matched, the rotation matrix and translation matrix between different cameras are calculated by the least squares method based on the matched feature points, the position difference between the external parameter calibration photo of a certain camera and the external parameter calibration photo of another camera is calculated based on the rotation matrix and translation matrix to obtain disparity data, a disparity map is generated based on the disparity data, and the disparity map is converted into point cloud data. Each pixel in the point cloud data has a unique three-dimensional coordinate so that when the gesture image data set is substituted into the gesture recognition model, a gesture three-dimensional model can be generated, and when the lip reading image data set is substituted into the lip reading recognition model, a lip reading three-dimensional model can be generated.

5. The method for gesture and lip reading recognition for political and digital consultation according to claim 3 is characterized in that: The specific steps for verifying stereo calibration are: substitute the test gesture data set and the test lip reading data set to obtain the test gesture 3D model and the test lip reading 3D model, compare and calculate the accuracy of the test gesture 3D model and the actual gesture 3D model, and compare and calculate the accuracy of the test lip reading 3D model and the actual lip reading 3D model.

6. The method for operating gesture and lip reading recognition for political and digital consultation according to claim 2 is characterized in that: The specific steps of consistency check are: check the consistency of each camera coordinate system; The specific steps of occlusion check are: check the disparity map generated by each camera, If there is a discontinuous area, it is determined that the area may be blocked; If there is an unclear boundary, it is determined that the boundary may be blocked.

7. The method for gesture and lip reading recognition for political and digital consultation according to claim 1 is characterized in that: In step S1, the camera module captures the user's gesture and lip reading videos and obtains the gesture image dataset and the lip reading image dataset through preprocessing, including: S11, dividing the gesture video and the lip reading video into gesture images and lip reading images in sequence; S12, comparing differences between adjacent gesture images and adjacent lip reading images, and removing static images and images with little change; S13, adjusting all gesture images and lip reading images to the same size and unifying the brightness of the images; S14, converting all gesture images and lip reading images into HSV color space images, the control module is preset with a standard hand H hue range, a standard hand S saturation range, and a standard hand V brightness range to define the HSV range of hand skin color, the control module is preset with a standard lip H hue range, a standard lip S saturation range, and a standard lip V brightness range to define the HSV range of lip skin color, creating a binary mask, setting pixels that meet the standard hand H hue range, the standard hand S saturation range, and the standard hand V brightness range as white pixels, and setting pixels that do not meet the conditions as black pixels, wherein the largest and connected white pixel represents the hand area, deleting the black pixel area, and integrating all processed gesture images into a gesture image dataset; The pixels that meet the standard lip H hue range, the standard lip S saturation range, and the standard lip V brightness range are set as white pixels, and the pixels that do not meet the conditions are set as black pixels. The largest and most connected white pixels represent the lip area. The black pixel area is deleted, and all the processed lip reading images are integrated into a lip reading image dataset.

8. The method for operating gesture and lip reading recognition for political and digital consultation according to claim 2 is characterized in that: The recognition model includes a plurality of standard three-dimensional posture data and vocabulary data corresponding to the standard three-dimensional posture data one by one. In step S2, the gesture image data set is substituted into the gesture recognition model to obtain the gesture recognition data set, and the lip reading image data set is substituted into the lip reading recognition model to obtain the lip reading recognition data set. The specific steps include: S21, according to the estimated internal parameter matrix, rotation matrix and translation matrix in the gesture recognition model and the lip reading recognition model, identify each gesture image and lip reading image in the gesture image dataset and the lip reading image dataset to obtain disparity data, generate a disparity map according to the disparity data, convert the disparity map into point cloud data to generate point cloud data, and respectively form a three-dimensional model of the gesture to be recognized and a three-dimensional model of the lip reading to be recognized according to the point cloud data; S22. Compare the three-dimensional model of the gesture to be recognized and the three-dimensional model of the lip reading to be recognized with the standard three-dimensional posture database one by one, select the standard three-dimensional posture database with the highest overlap, select the vocabulary data corresponding to the standard three-dimensional posture database, and obtain the gesture recognition data set and the lip reading recognition data set.

9. The method for operating gesture and lip reading recognition for political and digital consultation according to claim 1 is characterized in that: In step S3, the specific steps of processing the gesture recognition data set and the lip reading recognition data set by natural language processing technology to obtain the sign language recognition question data set and the lip reading recognition question data set include: S31, using the SpaCy dependency parser to parse the vocabulary data in the gesture recognition dataset and the lip reading recognition dataset to obtain a dependency result database to determine the grammatical relationship between the vocabulary; S32, using component syntax to analyze the vocabulary data in the gesture recognition dataset and the lip reading recognition dataset to obtain a component syntax result database, and construct a syntax tree to determine the component hierarchical relationship of the vocabulary in the sentence; S33. Use SRL technology to analyze the vocabulary data in the gesture recognition dataset and the lip reading recognition dataset to obtain the SRL result database to identify the predicate and argument in the sentence, so as to determine the initiator, recipient and affected object of the action; S34, using NER tools to identify specific entity names in the text to obtain a NER result database; S35, integrating the dependency relationship result database, the component syntax result database, the SRL result database and the NER result database to obtain a preliminary sign language recognition question dataset and a preliminary lip reading recognition question dataset; S36. Post-process the prepared sign language recognition question data set and the prepared lip reading recognition question data set to obtain a sign language recognition question data set and a lip reading recognition question data set.

10. The method for operating gesture and lip reading recognition for political and digital consultation according to claim 9 is characterized in that: The specific steps of post-processing are: converting the preliminary sign language recognition question data set and the preliminary lip reading recognition question data set into Unicode encoded text; The Unicode encoded text is dereferenced through the stanza library, and all the modified places are integrated to form the first modified data set; Correct the typos in the Unicode text through the corrector library, and integrate all the modified parts to form the second modified data set; According to the first modified data set and the second modified data set, the corresponding codes in the Unicode encoded text are deleted or modified to form a sign language recognition question data set and a lip reading recognition question data set.