Gesture voice recognition smart bracelet and recognition method
By combining IMU sensors and cameras with neural network models, smart bracelets can capture hand gestures and unlock interactive commands, predict voice and text, and solve the problem of limited interactive functions of smart bracelets, enabling wider application and accurate information transmission.
Patent Information
- Application Number
- CN202410943650.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing smart bracelets have limited interactive functions, resulting in low usage and a limited user base, making them unsuitable for flexible application in various real-world interactive scenarios.
The system captures user gesture images using an IMU sensor and camera, combines them with a neural network model for gesture and voice recognition, unlocks interactive commands, predicts the next interactive spatial action, and outputs interest-based voice signals to achieve interactive functions.
It improves the versatility of smart bracelets in various interaction scenarios, expands the user base, ensures timely and accurate information collection, and adapts to a variety of interactive environments.
Smart Images

Figure CN118963544A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of electronic information technology, and in particular to a smart bracelet for gesture and voice recognition and a recognition method. Background Art
[0002] With the rapid development of wearable devices, smart bracelets, as a technological product that people pay more attention to, are quietly penetrating and changing people's lives with their complete functions. Especially with the development of intelligent devices and related advanced technologies, the convenience of bracelets has led to a surge in demand for intelligent interactivity, and the interactive design and user experience of smart bracelets have also become the focus of research.
[0003] Existing smart bracelets with interactive function settings basically have gesture and voice recognition functions. Smart interactive bracelets can enhance the user's interactive experience, and portable bracelets can also provide users with corresponding physiological monitoring to improve the practicality of the bracelets.
[0004] Although the existing smart bracelets have basically realized some interactive functions, most of them remain at the control of signal detection and reception in wireless communication technology; and the control of feedback information on the display end of the smart bracelet. Since the display screen of the smart bracelet is very limited in most cases, and for some users with inconvenient vision range, it is more convenient to choose a portable mobile phone or broadcasting device that directly receives sound signals; and the intelligent dialogue inside the existing portable mobile phone can also realize the complete interactive process of user activity information, which is relatively more able to replace the versatility of the interactive functions of the smart bracelet, resulting in the current low general usage rate of smart bracelets, and they cannot be directly and flexibly applied to various real interactive space scenarios, resulting in the current smart bracelet application population being relatively limited. Summary of the invention
[0005] The purpose of the present invention is to provide a smart bracelet for gesture and speech recognition and a recognition method to solve the following technical problems:
[0006] How to improve the versatility of interactive scenarios of smart bracelets and expand the scope of application of smart bracelets.
[0007] The purpose of the present invention can be achieved through the following technical solutions:
[0008] A gesture speech recognition method, the method comprising:
[0009] Step 1: Capture the user's current gesture image through the IMU sensor and camera, obtain gesture information and unlock the interactive command; the gesture image includes the interactive gesture; the interactive command includes the start command, the execution command and the termination command;
[0010] Step 2: parse the current corresponding interactive space action information according to the unlocked interactive instruction and predict the next interactive space action, and obtain the voice text corresponding to the interactive space action information of two or more consecutive actions;
[0011] Step 3: Based on the user's historical interest voice database and voice timbre information, a neural network model is trained to build an interest voice output model, the voice text corresponding to the current interactive space action information is input into the model for synthesis training, and the corresponding interest voice and its text are output;
[0012] Step 4: The user receives the voice signal of interest through the voice recognition interface, and the interactive end executes the interactive command action corresponding to the voice signal of interest.
[0013] Preferably, the method of unlocking the interactive instruction in step 1 is:
[0014] Performing data collection on image frame signals of gesture actions;
[0015] Analyze the image frame signal and segment the gesture action images in the continuous frame state;
[0016] Analyze the gesture action image information through the preset gesture model to determine whether an interactive gesture is defined:
[0017] If yes, then perform interactive command unlocking;
[0018] If not, continue the current image analysis operation.
[0019] Preferably, the process of parsing the interactive space action information in step 2 is:
[0020] A simulated interactive space is constructed through an initial three-dimensional model, historical interactive gestures are input into the simulated interactive space for training, historical scene image information applicable to the simulated interactive space actions is determined, and a scene-interaction gesture list is output;
[0021] Extracting a specific interactive space scene image for the current unlocking interactive instruction action execution according to the scene-interaction gesture list;
[0022] Perform point cloud element recognition of the three-dimensional environment on the specific interactive space scene image to output point cloud data of object element features, and obtain key features corresponding to each object element based on the point cloud data of object element features and a preset recognition model;
[0023] The interactive space scene information model is obtained by inputting a combination of multiple key features into the three-dimensional model training;
[0024] According to the real-time interactive gesture input, the interactive space scene information model is used for prediction analysis, and the prediction parameters of the current interactive space action and the next interactive space action are output.
[0025] Preferably, step 2 also includes:
[0026] The variation curve E of the continuous frame time period of the prediction parameter of the preset continuous number of interactive space actions i ;
[0027] Collect the standard change curve E0 of the prediction parameter continuous frame time period of the same preset number of consecutive historical standard interactive space actions;
[0028] The curve E i Construct and obtain the absolute value ΔS of the area difference in the continuous frame time period between the two curves in the same coordinate system as curve E0; and analyze the absolute value ΔS of the area difference and the preset standard area difference threshold interval [ΔS 1 , ΔS 2 Size:
[0029] If ΔS∈[ΔS 1 , ΔS 2 ], it is judged that the dynamic change of the interactive space action conforms to the current interactive space scene information;
[0030] If ΔS<ΔS 1 , then it is judged that the dynamic change of the interactive space action is consistent with the current interactive space scene information;
[0031] If ΔS>ΔS 2 , it is judged that the dynamic change of the interactive space action does not conform to the current interactive space scene information.
[0032] Preferably, the process of acquiring key features includes:
[0033] Input the 3D point cloud data of the historical specific interactive space scene image into the PointNet model, and output the classification label corresponding to the 3D point cloud data;
[0034] Input the classification labels and the 3D point cloud data of the current historical specific interactive space scene image into the PointNet model for training to obtain the classification labels of the target object point cloud data;
[0035] The classification labels of the target object point cloud data are input into the preset recognition model for recognition to obtain the key feature information corresponding to each object element.
[0036] Preferably, it also includes receiving real-time voice information of the user through a sound sensor, labeling the real-time voice information data and inputting the real-time interactive gesture image into the interactive space scene information model for predictive analysis, and comprehensively obtaining the prediction results of continuous interactive space actions.
[0037] Preferably, in step 3, the method of inputting the current speech text into the model for synthesis training and outputting the corresponding speech signal of interest includes:
[0038] Extract text information from speech text and capture speech audio information and timbre information in the database;
[0039] Acquire the user's physiological data of the current voice text received by the user through the photoelectric sensor and the temperature sensor, the physiological data including heartbeat, muscle vibration parameters, and body temperature;
[0040] The physiological data set corresponding to the same speech text is counted and input into the neural network model for training to obtain the user interest coefficient corresponding to the current speech text;
[0041] The user interest coefficients corresponding to different voice texts and the voice texts corresponding to the interactive space action information are input into the trained neural network model for verification and comparison to determine the interest voice that best suits the current interactive scenario.
[0042] A smart bracelet for gesture and speech recognition, based on a gesture and speech recognition method, the smart bracelet comprises: an interface end, a transmission end, a display module, a microcontroller, a camera; and an interaction end that receives interaction information of the smart bracelet;
[0043] The interface end is used to receive the voice signal of interest, and the interface end includes a voice recognition interface, and the voice recognition interface is connected to a sensor module, and the sensor module includes an IMU sensor, a sound sensor, a photoelectric sensor, and a temperature sensor;
[0044] The IMU sensor is used to capture the user's current gesture image; the sound sensor is used to receive the user's real-time voice information; the photoelectric sensor and temperature sensor obtain the user's physiological data of the current voice text received;
[0045] The display module is used to display information;
[0046] The transmission end is used to transmit the interface end signal to the interaction end;
[0047] The microcontroller is used to unlock the interaction command according to the gesture action information to control the interaction space action of the interaction end;
[0048] The camera is used to obtain gesture image information;
[0049] The interactive terminal is used to execute the interactive command action corresponding to the voice signal of interest.
[0050] Beneficial effects of the present invention:
[0051] (1) The present invention ensures the timely and accurate collection of information on the current user interaction of the bracelet by designing image recognition and analysis of interactive gestures of gesture images, and unlocking according to the interactive instructions set by the system, and ensures that the gesture image information after unlocking can be accurately transmitted by setting a gesture of a specific unlocking instruction, and avoids the mistransmission of gesture image information that does not conform to the current unlocking instruction.
[0052] (2) The present invention parses the currently corresponding interactive space action information and predicts the next interactive space action according to the unlocked interactive instruction, and obtains the voice text corresponding to the interactive space action information of two or more consecutive actions; after the interactive instruction is unlocked, the current interactive environment is analyzed according to the interactive instruction situation, the action of the user interactive space in the interactive environment and the interactive space action information of the next consecutive action corresponding to the continuous action prediction background are determined; and the voice text corresponding to the interactive space action information of the multiple consecutive actions is obtained; and the voice text in a specific interactive environment is accurately obtained by using the prediction parameters in the analysis or other reference data that can directly reflect the prediction direction.
[0053] (3) The present invention also constructs a simulated interactive space through an initial three-dimensional model through the analysis process of interactive space action information, inputs the interactive gesture into the current simulated interactive space to obtain the interactive gesture space movement action, converts the movement feature corresponding to the interactive gesture into indication information, determines the scene image information pre-matched with the gesture through the indication information, and obtains the scene image corresponding to each gesture; the scene image is obtained based on the historical scene image determined by the big database, and a scene-interaction gesture list is formed based on the scene and gesture pairing; the scene-interaction gesture list can be used to determine the specific interactive space scene image of the current unlocking interaction instruction action based on the recognition and extraction of the geometric structure; and the current specific interactive space scene image can also better reflect the real state of the user's hand movement in terms of form and action.
[0054] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0056] Figure 1 This is a step diagram of a gesture speech recognition method of the present invention. DETAILED DESCRIPTION
[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0058] Although the existing smart bracelets have basically realized some interactive functions, most of them remain at the control of signal detection and reception in wireless communication technology; and the control of feedback information on the display end of the smart bracelet. Since the display screen of the smart bracelet is very limited in most cases, and for some users with inconvenient vision range, it is more convenient to choose a portable mobile phone or broadcasting device that directly receives sound signals; and the intelligent dialogue inside the existing portable mobile phone can also realize the complete interactive process of user activity information, which is relatively more able to replace the versatility of the interactive functions of the smart bracelet, resulting in the current low general usage rate of smart bracelets, and they cannot be directly and flexibly applied to various real interactive space scenarios, resulting in the current smart bracelet application population being relatively limited.
[0059] To solve the above technical problems, please refer to Figure 1 As shown, the present invention is a gesture speech recognition method, the method comprising:
[0060] Step 1: Capture the user's current gesture image through the IMU sensor and camera, obtain gesture information and unlock the interactive command; the gesture image includes the interactive gesture; the interactive command includes the start command, the execution command and the termination command;
[0061] Step 2: parse the current corresponding interactive space action information according to the unlocked interactive instruction and predict the next interactive space action, and obtain the voice text corresponding to the interactive space action information of two or more consecutive actions;
[0062] Step 3: Based on the user's historical interest voice database and voice timbre information, a neural network model is trained to build an interest voice output model, the voice text corresponding to the current interactive space action information is input into the model for synthesis training, and the corresponding interest voice and its text are output;
[0063] Step 4: The user receives the voice signal of interest through the voice recognition interface, and the interactive end executes the interactive command action corresponding to the voice signal of interest.
[0064] In the above technical solution, the present invention designs a gesture speech recognition method, which is specifically:
[0065] First, the user's current gesture image is captured through the IMU sensor and camera, and the gesture information is obtained to unlock the interaction command; the gesture image includes the interaction gesture; the interaction command includes the start command, the execution command and the termination command; because the bracelet worn on the wrist usually has a certain environmental perception ability, especially by obtaining the user's gesture information and the collection of wrist movement conditions, the IMU sensor can be set to measure the acceleration and angle based on the wrist contact; it mainly includes two parts: the three-axis accelerometer and the gyroscope; and the main function of the IMU sensor set in the bracelet is to provide accurate position, speed and posture information for the moving object; at the same time, through the camera The head acquires the dynamic gesture image to comprehensively determine the gesture information; then the image of the gesture action is analyzed through image processing technology to obtain the information of the unlocking interaction instruction. Due to the different gesture indication information of the user, his different interaction instruction indication information can be realized. This design basically guarantees the acquisition of the gesture information image and the communication and operation of the overall gesture information, and sets three basic instructions in combination with conventional gesture indication instructions, namely the start instruction, the execution instruction and the termination instruction; ensure the acquisition of the unlocking instruction that conforms to the gesture action; of course, different application scenarios can be paired according to the needs of the actual production process to further select and expand the design of gesture action unlocking instructions.
[0066] Most of the current image processing technologies for analyzing gesture images use the number of frames to collect images. The more frames collected, the clearer the continuity information of the image. The specific gesture image analysis method of this embodiment is: by collecting data from the image frame signal of the gesture action; then by analyzing the image frame signal, segmenting the gesture action image in the continuous frame state; and analyzing the gesture action image information through a preset gesture model to determine whether an interactive gesture is defined: if so, perform an interactive command to unlock; if not, continue the current image analysis operation; the above image analysis and parsing process further realizes the unlocking interactive command.
[0067] In the above technical solution, by designing image recognition and analysis of interactive gestures of gesture images, and unlocking according to the interactive instructions set by the system, timely and accurate collection of information used by the current user of the bracelet is guaranteed, and by setting gestures of specific unlocking instructions, it is ensured that the gesture image information after unlocking can be accurately transmitted, and the mistransmission of gesture image information that does not conform to the current unlocking instructions is avoided.
[0068] Secondly, the currently corresponding interactive space action information is analyzed according to the unlocked interactive instruction and the next interactive space action is predicted, and the voice text corresponding to the interactive space action information of two or more consecutive actions is obtained; after the interactive instruction is unlocked, the current interactive environment is analyzed according to the interactive instruction situation, and the action of the user interactive space in the interactive environment and the interactive space action information of the next continuous action corresponding to the continuous action prediction background are determined; and the voice text corresponding to the interactive space action information of multiple continuous actions is obtained; the prediction parameters in the analytical analysis or other reference data that can directly reflect the prediction direction are used to accurately obtain the voice text in a specific interactive environment.
[0069] The process of parsing the interactive space action information is as follows: first, a simulated interactive space is constructed through an initial three-dimensional model, and historical interactive gestures are input into the simulated interactive space for training, the historical scene image information applicable to the simulated interactive space action is determined, and a scene-interaction gesture list is output; then, the specific interactive space scene image for the current unlocking interactive instruction action is extracted according to the scene-interaction gesture list; then, the point cloud element recognition of the three-dimensional environment of the specific interactive space scene image is performed to output the point cloud data of the object element characteristics, and the key features corresponding to each object element are obtained based on the point cloud data of the object element characteristics and the preset recognition model; further, according to a combination of multiple key features, a three-dimensional model is input for training to obtain an interactive space scene information model; finally, according to the real-time interactive gesture input, a predictive analysis is performed in the interactive space scene information model, and the prediction parameters of the current interactive space action and the next interactive space action are output.
[0070] In the above technical scheme, unlike the prior art which uses gesture surface features of two-dimensional images to obtain parameters corresponding to image attributes, image motion and image edge contours, this scheme adopts three-dimensional modeling, which is more suitable for realizing human-computer interaction application process by using interactive gesture detection.
[0071] Specifically, the simulated interactive space is constructed through the initial three-dimensional model, and the interactive gesture is input into the current simulated interactive space to obtain the interactive gesture space motion action. The motion characteristics corresponding to the interactive gesture are converted into indication information, and the scene image information pre-matched with the gesture is determined through the indication information, and then the scene image corresponding to each gesture is obtained. The scene image is obtained based on the historical scene image determined by the big database, and the scene-interaction gesture list is composed according to the scene and gesture pairing; the scene-interaction gesture list can be used to determine the specific interactive space scene image of the current unlocking interaction instruction action based on the recognition and extraction of the geometric structure; and the current specific interactive space scene image can also better reflect the real state of the user's hand movement in terms of form and action.
[0072] Furthermore, point cloud data of object element features in the specific interactive space scene image is output by performing point cloud element recognition of the three-dimensional environment on the acquired specific interactive space scene image, and key features corresponding to each object element are obtained based on the point cloud data of the object element features and the preset recognition model. Since the point cloud is a collection of a series of discrete geometric points in space; compared with the data information of two-dimensional plane points, the point cloud contains rich information, including three-dimensional coordinates X, Y, Z, color, classification value, intensity value, etc., and the three-dimensional point cloud has higher spatial resolution and point position accuracy, so the emergence of point clouds is of great significance to the expression of the appearance of spatial scenes and objects; the three-dimensional point cloud depends on its spatial coordinates, and usually uses three-dimensional geometric parameters and stereo vision systems to construct point cloud data for measuring the corresponding object element features; and according to the transformation of the point cloud data of the current object element features into a point cloud capable of feature recognition, in addition to using the three-dimensional object recognition method of machine learning, such as the PointNet model for point cloud object classification, the matching result can also be determined by calculating the matching error between the original point cloud and the target point cloud through the existing point cloud template to obtain the relevant information of the original point cloud, which will not be described in detail here.
[0073] The machine learning method is used to input multiple key features into the three-dimensional model training to obtain the interactive space scene information model; and the real-time interactive gesture is input into the interactive space scene information model for prediction analysis, and the prediction parameters of the current interactive space action and the next interactive space action are output; the scene of continuous interactive space action is accurately matched; the prediction of multiple consecutive interactive space actions can also reflect whether the current interactive space action matches the current interactive space scene information. The further analysis method is:
[0074] First, the variation curve E of the prediction parameters of the interactive space action of the preset continuous number of consecutive frames is collected. i ; Then similarly collect the same preset number of consecutive times of the historical standard interactive space action prediction parameter continuous frame time period standard change curve E0; then curve E i Construct and obtain the absolute value ΔS of the area difference in the continuous frame time period between the two curves in the same coordinate system as curve E0; finally, analyze the absolute value ΔS of the area difference and the preset standard area difference threshold interval [ΔS 1 , ΔS 2 ], judge if ΔS∈[ΔS 1 , ΔS 2 ], it is judged that the dynamic change of the interactive space action conforms to the current interactive space scene information; if ΔS<ΔS 1 , then it is judged that the dynamic change of the interactive space action is consistent with the current interactive space scene information; if ΔS>ΔS 2 , it is judged that the dynamic change of the interactive space action does not conform to the current interactive space scene information.
[0075] As an embodiment of the present invention, a specific implementation process for acquiring key features is: first, input the three-dimensional point cloud data of the historical specific interactive space scene image into the PointNet model, and output the classification label corresponding to the three-dimensional point cloud data; then, input the classification label and the three-dimensional point cloud data of the current historical specific interactive space scene image into the PointNet model for training, and obtain the classification label of the target object point cloud data; finally, input the classification label of the target object point cloud data into the preset recognition model for recognition, and obtain the key feature information corresponding to each object element.
[0076] In the above technical solution, since the PointNet model is a deep learning model for point cloud data processing, its input is a three-channel unordered point cloud, and it outputs the classification label of the entire point cloud or the category of each point; in the point cloud data processing, the network learning included in the point cloud position ensures the use of the maximum pooling operation to extract the global features of the point cloud and also avoids the influence of the point cloud rotation and translation on the classification or segmentation, solves the point cloud disorder problem, and ensures the acquisition of more accurate classification labels and point class distinction. In order to realize the extraction of target object feature information and obtain key features, the key features here are determined based on the preset recognition model: the preset recognition model is a kind of model built based on machine learning method, and its role is to pre-label the classification label of the target object point cloud data according to the task requirements and design a specific classifier for labeling, and realize the recognition and extraction of key feature information corresponding to each object element.
[0077] As an embodiment of the present invention, it also includes receiving real-time voice information of the user through a sound sensor, labeling the real-time voice information data and inputting the real-time interactive gesture image into the interactive space scene information model for predictive analysis, and comprehensively obtaining the prediction results of continuous interactive space actions.
[0078] In the above technical solution, by building based on three-dimensional vision, the input voice information data in the current interactive space scene is matched and classified with the signal corresponding to the current interactive gesture to predict the interactive space action corresponding to the next interactive gesture, and the prediction results of the interactive space action in a continuous state are obtained through multiple trainings.
[0079] As an implementation mode of the present invention, in step 3, the method of inputting the current speech text into the model for synthesis training and outputting the corresponding speech signal of interest includes:
[0080] Extract text information from speech text and capture speech audio information and timbre information in the database;
[0081] Acquire the user's physiological data of the current voice text received by the user through the photoelectric sensor and the temperature sensor, the physiological data including heartbeat, muscle vibration parameters, and body temperature;
[0082] The physiological data set corresponding to the same speech text is counted and input into the neural network model for training to obtain the user interest coefficient corresponding to the current speech text;
[0083] The user interest coefficients corresponding to different voice texts and the voice texts corresponding to the interactive space action information are input into the trained neural network model for verification and comparison to determine the interest voice that best suits the current interactive scenario.
[0084] In the above technical scheme, since individual users have different interest preferences for the timbre in the interactive command voice information, further screening can be performed based on their physiological characteristics such as preferences for different sound timbre, so as to obtain the user's personalized settings for the sound. Specifically, by first obtaining voice feature information: such as audio, timbre, etc.; by obtaining the physiological data of the user's current received voice text (generally normal hearing and the default receiving voice means hearing), the physiological data includes heartbeat, muscle vibration parameters, and body temperature; then by designing the standard value of the physiological data under the preferred timbre state, the user interest coefficient corresponding to the current voice text is judged according to the comparison size, and then a trained model is generated; finally, the user interest coefficients corresponding to different voice texts and the voice text corresponding to the interactive space action information are input into the trained neural network model for verification and comparison, and the voice of interest that best suits the current interactive scenario is determined and confirmed.
[0085] The present invention also provides a smart bracelet for gesture and speech recognition. Based on the gesture and speech recognition method, the smart bracelet includes: an interface end, a transmission end, a display module, a microcontroller, a camera; and an interaction end that receives interaction information of the smart bracelet;
[0086] The interface end is used to receive the voice signal of interest, and the interface end includes a voice recognition interface, and the voice recognition interface is connected to a sensor module, and the sensor module includes an IMU sensor, a sound sensor, a photoelectric sensor, and a temperature sensor;
[0087] The IMU sensor is used to capture the user's current gesture image; the sound sensor is used to receive the user's real-time voice information; the photoelectric sensor and temperature sensor obtain the user's physiological data of the current voice text received;
[0088] The display module is used to display information;
[0089] The transmission end is used to transmit the interface end signal to the interaction end;
[0090] The microcontroller is used to unlock the interaction command according to the gesture action information to control the interaction space action of the interaction end;
[0091] The camera is used to obtain gesture image information;
[0092] The interactive terminal is used to execute the interactive command action corresponding to the voice signal of interest.
[0093] The above contents are merely examples and explanations of the concept of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.
Claims
1. A gesture speech recognition method, characterized in that: The method comprises: Step 1: Capture the user's current gesture action image through the IMU sensor and the camera, and obtain gesture action information to unlock the interactive instruction; the gesture action image includes the interactive gesture; the interactive instruction includes the start instruction, the execution instruction and the termination instruction; Step 2: parse the current corresponding interactive space action information according to the unlocked interactive instruction and predict the next interactive space action, and obtain the voice text corresponding to the interactive space action information of two or more consecutive actions; Step 3: Based on the user's historical interest voice database and voice timbre information, a neural network model is trained to build an interest voice output model, the voice text corresponding to the current interactive space action information is input into the model for synthesis training, and the corresponding interest voice and its text are output; Step 4: The user receives the voice signal of interest through the voice recognition interface, and the interactive end executes the interactive command action corresponding to the voice signal of interest.
2. A gesture speech recognition method according to claim 1, characterized in that: The method of unlocking the interactive instruction in step 1 is: Performing data collection on image frame signals of gesture actions; Analyze the image frame signal and segment the gesture action images in the continuous frame state; Analyze the gesture action image information through the preset gesture model to determine whether an interactive gesture is defined: If yes, then perform interactive command unlocking; If not, continue the current image analysis operation.
3. A gesture speech recognition method according to claim 1, characterized in that: The process of parsing the interactive space action information in step 2 is as follows: A simulated interactive space is constructed through an initial three-dimensional model, historical interactive gestures are input into the simulated interactive space for training, historical scene image information applicable to the simulated interactive space actions is determined, and a scene-interaction gesture list is output; Extracting a specific interactive space scene image for the current unlocking interactive instruction action execution according to the scene-interaction gesture list; Perform point cloud element recognition of the three-dimensional environment on the specific interactive space scene image to output point cloud data of object element features, and obtain key features corresponding to each object element based on the point cloud data of object element features and a preset recognition model; The interactive space scene information model is obtained by inputting a combination of multiple key features into the three-dimensional model training; According to the real-time interactive gesture input, the interactive space scene information model is used for prediction analysis, and the prediction parameters of the current interactive space action and the next interactive space action are output.
4. A gesture speech recognition method according to claim 3, characterized in that: The step 2 also includes: The variation curve E of the continuous frame time period of the prediction parameter of the preset continuous number of interactive space actions i ; Collect the standard change curve E0 of the prediction parameter continuous frame time period of the same preset number of consecutive historical standard interactive space actions; The curve E i Construct and obtain the absolute value ΔS of the area difference in the continuous frame time period between the two curves in the same coordinate system as curve E0; and analyze the absolute value ΔS of the area difference and the preset standard area difference threshold interval [ΔS 1 , ΔS 2 Size: If ΔS∈[ΔS 1 , ΔS 2 ], it is judged that the dynamic change of the interactive space action conforms to the current interactive space scene information; If ΔS<ΔS 1 , then it is judged that the dynamic change of the interactive space action is consistent with the current interactive space scene information; If ΔS>ΔS 2 , it is judged that the dynamic change of the interactive space action does not conform to the current interactive space scene information.
5. A gesture speech recognition method according to claim 3, characterized in that: The process of obtaining the key features includes: Input the 3D point cloud data of the historical specific interactive space scene image into the PointNet model, and output the classification label corresponding to the 3D point cloud data; Input the classification labels and the 3D point cloud data of the current historical specific interactive space scene image into the PointNet model for training to obtain the classification labels of the target object point cloud data; The classification labels of the target object point cloud data are input into the preset recognition model for recognition to obtain the key feature information corresponding to each object element.
6. A gesture speech recognition method according to claim 3, characterized in that: It also includes receiving real-time voice information of the user through a sound sensor, labeling the real-time voice information data and inputting the real-time interactive gesture image into the interactive space scene information model for predictive analysis, and comprehensively obtaining the prediction results of continuous interactive space actions.
7. A gesture speech recognition method according to claim 1, characterized in that: In the step 3, the current speech text is input into the model for synthesis training, and the method of outputting the corresponding speech signal of interest includes: Extract text information from speech text and capture speech audio information and timbre information in the database; Acquire the physiological data of the user currently receiving the voice text through the photoelectric sensor and the temperature sensor, wherein the physiological data includes heartbeat, muscle vibration parameters, and body temperature; The physiological data set corresponding to the same speech text is counted and input into the neural network model for training to obtain the user interest coefficient corresponding to the current speech text; The user interest coefficients corresponding to different voice texts and the voice texts corresponding to the interactive space action information are input into the trained neural network model for verification and comparison to determine the interest voice that best suits the current interactive scenario.
8. The smart bracelet with gesture and voice recognition according to claim 1, characterized in that: Based on the gesture and speech recognition method according to any one of claims 1 to 7, the smart bracelet comprises: an interface end, a transmission end, a display module, a microcontroller, a camera; and an interaction end that receives interaction information of the smart bracelet; The interface end is used to receive the voice signal of interest, and the interface end includes a voice recognition interface, and the voice recognition interface is connected to a sensor module, and the sensor module includes an IMU sensor, a sound sensor, a photoelectric sensor, and a temperature sensor; The IMU sensor is used to capture the user's current gesture action image; the sound sensor is used to receive the user's real-time voice information; the photoelectric sensor and temperature sensor obtain the user's physiological data of the current voice text received; The display module is used to display information; The transmission end is used to transmit the interface end signal to the interaction end; The microcontroller is used to unlock the interaction instruction according to the gesture action information to control the interaction space action of the interaction end; The camera is used to obtain gesture image information; The interactive terminal is used to execute the interactive instruction action corresponding to the voice signal of interest.
