Visual interaction multifunctional robot based on AI large model

Abstract

The invention discloses a visual interaction multifunctional robot based on an AI large model. The product has double functions of independent operation and computer accessory. In the independent mode, the robot activates voice dialogue in three modes of key awakening, awakening word appointing or real-time dialogue monitoring, voice is converted into characters, the characters are sent to the large model after being processed through a Hash algorithm, and answers of the large model are converted into voice feedback in real time. The method is characterized in that emotion recognition and expression synchronization ability is achieved, emotion features in user voice are analyzed through an emotion artificial intelligence algorithm, expressions are adjusted in real time to achieve emotion resonance, and user experience is improved. When the robot is connected to a computer, the robot can control mouse movement, click operation, computer unlocking through face recognition, application opening control through gestures and keyboard input simulation, meanwhile, operations such as film playing, fast forward and full screen control through voice are supported, and the working efficiency and the interestingness of user interaction are remarkably improved.

Description

(I) Technical field

[0001] The present invention belongs to the field of intelligent robots, and specifically relates to a multifunctional desktop robot capable of large-scale AI model dialogue and visual interaction. (II) Background technology

[0002] There are many types of desktop robots on the market, but they have certain limitations in functionality and practicality. Although some high-end products have certain practicality, their high prices make them difficult to popularize. Most desktop robots can only perform basic interactions and have not yet reached the actual utility of being a productivity assistant to improve work efficiency. Therefore, the challenge we face is how to develop a desktop robot that is both affordable and highly practical to meet the market's demand for improving work efficiency. (III) Summary of the invention

[0003] The present invention is mainly aimed at the above-mentioned problems to make the desktop robot more companionable and practical. The present invention adopts the following technical solutions:

[0004] In terms of speech-to-text, I used an innovative algorithm that can interact with large AI models through a unique text splicing method during speech recognition to ensure the coherence and integrity of the transcribed content. In addition, the algorithm supports real-time speech transcription and realizes the function of transmitting while speaking, which significantly improves the fluency and response speed of conversations with AI.

[0005] Emotion recognition technology aims to identify the emotional state of users by analyzing their voice, facial expressions, etc. This technology can enable robots or computer systems to understand the user's emotions and adjust their responses accordingly to achieve more natural human-computer interaction. The expression synchronization mechanism means that the robot automatically adjusts its facial expressions based on the recognized emotions to match the user's emotions and enhance emotional resonance.

[0006] By using machine learning to learn multiple gestures for interaction, we can read various information about the finger, including position, direction, speed, trajectory, etc., so that the finger can convey richer information and make the interaction more intelligent: Finger position: coordinate information of the finger in space. Finger direction: direction vector of the finger. Finger speed: speed vector of the finger in space. Finger posture: degree of bending of the finger. The trajectory of the finger moving in the air. (IV) Description of the drawings

[0007] Figure 1 Flowchart for speech processing

[0008] Figure 2 Flowchart for gesture recognition

[0009] Figure 3Flowchart for interaction between robot and computer (V) Specific implementation methods

[0010] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0011] The specific implementation method of realizing the dialogue with the AI ​​model is as follows: First, use an efficient ASR system to convert the voice signal into text. Based on the search results, use a model such as iFlytek, which supports multiple languages ​​and can process audio signals and convert them into text. This model can be deployed as an API service to facilitate real-time processing of voice input.

[0012] During the ASR conversion process, a special concatenation algorithm is used to process the recognized text to ensure coherence and completeness. This algorithm may need to consider semantic and paralinguistic information in the speech, such as pitch, timbre, intonation, etc., to reduce information loss.

[0013] The converted text is fed into a large AI model that can understand and process natural language and generate appropriate responses. Based on the search results, the framework of "Automatic Speech Recognition (ASR) + LLM + Text-to-Speech (TTS)" is adopted. LLM (Large Language Model) can generate responses based on the transcribed text.

[0014] In order to achieve transmission while speaking, a streaming processing method is adopted. The ASR system can process the voice stream in real time and send the recognized text fragments to the AI ​​big model for processing immediately. This enables both the ASR system and the AI ​​big model to support fast response and processing.

[0015] To ensure that the conversation with AI is more coherent and responsive, a deep learning model BERT is used, which can parse sentences quickly and accurately through the attention mechanism.

[0016] Model deployment: Finally, the ASR model and AI big model are deployed as API services using the FastAPI framework so that clients can easily interact with them and achieve real-time speech recognition and conversation.

[0017] Large model output speed evaluation formula: Time To First Token (TTFT) and Time PerOutputToken (TPOT):

[0018] Latency=(TTFT)+(TPOT)×(the number of tokens to be generated)

[0019]

[0020] The specific implementation process of gesture motion recognition is as follows: collect gesture data and use Kinect device to capture gesture video. Pre-process the video, crop, rotate, scale, etc. for subsequent processing.

[0021] Image processing: Perform edge detection on the video frame to extract the edge information of the gesture. Perform shape recognition on the edge information to extract the shape information of the gesture. Perform color segmentation on the color information of the gesture.

[0022] Feature extraction: Extract color features, RGB and HSV color space features. Extract shape features, contour length, area, convex hull and other features. Extract direction features, gradient direction, Hough transform features. Extract speed features, finger speed and acceleration features.

[0023] Model training: Divide the dataset into training and test sets. Select supervised learning. Train the gesture recognition model and adjust the model parameters to optimize the model performance.

[0024] Recognition: Use the trained model to recognize new gestures. Determine the user's gesture based on the model's prediction results.

[0025] Shape characteristics: Shape characteristics can be calculated by the following formula:

[0026] A=∫ C dA

[0027] Among them, A represents the area of ​​the shape, L represents the length of the shape, and C represents the contour of the shape.

[0028] Directional characteristics: Directional characteristics can be calculated by the following formula:

[0029] Among them, θ represents the angle of the direction vector, and x and y represent the coordinates of the gradient direction respectively.

[0030] Speed ​​characteristics: Speed ​​characteristics can be calculated by the following formula:

[0031] Where v represents the velocity of the finger, x(t) and x(t-△t) represent the positions of the finger at time t and t-△t, respectively.

[0032] Emotional analysis is mainly achieved by using Emotion AI technology, which uses this technology to infer the user's emotional state by analyzing multiple signals such as facial expressions, voice, body language, text, heart rate, etc. Computer vision technology is used to analyze subtle changes in the user's facial muscles to infer emotions. Machine learning algorithms are used to train large amounts of emotional data to automatically identify new emotional patterns. By combining multiple sensory signals (voice, text, facial expressions), the user's emotional state can be judged more accurately. Voice-based emotion recognition mainly uses Hume AI EVI technology, which can judge the speaker's emotions through changes in tone, volume, speed, etc.

[0037] Formula for speech sentiment analysis in Hume AI EVI technology: Hume AI's EVI technology provides a model for streaming measurement of the pitch, rhythm, and timbre (prosody) of user voice, and is integrated with eLLM to respond to the user's emotional expression: EVI response = eLLM (user voice prosody, user language) EVI response = eLLM (user voice prosody, user language) EVI generates responses based on the user's voice prosody and language, matching the user's "atmosphere" (such as calm, interest, excitement, etc.), and responds to different emotional expressions (such as frustration, sadness) with corresponding tones.

[0038] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A visual interactive multifunctional robot based on an AI large model, characterized in that: include: An AI dialogue function, which allows users to communicate with AI using natural language; A sentiment analysis function, which is used to analyze the user's facial expressions and tone when talking to the robot, and judge the user's emotional state; A gesture trajectory recognition function that can identify the trajectory of fingers moving in the air to complete air handwriting input; An emotional response emotion, used to adjust the content and tone of responses to users based on their emotional state; A mood recording function, used to record and analyze the user's mood changes over a certain period of time; A user habit learning feature that records the user's computer usage habits and automatically adjusts the computer settings when the user connects.

2. The robot according to claim 1, characterized in that: The sentiment analysis module further includes: a facial recognition sub-function for capturing the user's facial expressions; A speech analysis sub-function for analyzing the user's speech characteristics; An emotion judgment algorithm is used to combine the results of facial recognition and voice analysis to determine the user's emotional state. Infer the user's emotional state by analyzing facial expressions, voice, body language, text, and multiple signals. Use computer vision technology to analyze subtle changes in the user's facial muscles and infer emotions. Voice-based emotion recognition can judge the speaker's emotions through changes in tone, volume, and speaking speed. Machine learning and deep learning: Emotion AI uses machine learning algorithms to train large amounts of emotional data to automatically recognize new emotional patterns. Multimodal perception: By combining multiple perception signals (voice, text, facial expressions), Emotion AI can more accurately judge the user's emotional state.

3. The robot according to claim 1, characterized in that: The user habit learning unit includes: a habit recorder for recording the user's computer usage behavior within a certain period of time; a habit analyzer to analyze recorded behavior and identify user preferences; An automatic settings adjuster that automatically adjusts the computer's volume, brightness, and other related settings based on the user's preferences.

4. The robot according to claim 1, characterized in that: The gesture recognition unit is capable of learning and performing the following operations: Recognizes a variety of complex gesture commands and records the trajectory of gestures in the air to achieve air handwriting input; Map gesture commands to corresponding computer operations, such as clicking, sliding, and tapping the keyboard; Optimize the accuracy and response speed of gesture recognition through machine learning algorithms. Through the multi-target detection method of deep learning, we are able to recognize a variety of complex gesture commands, including but not limited to static and dynamic gestures. To improve the accuracy of recognition, we combined ReXNet (Residual eXceptionNetwork) to detect the key positions of the hand and ResNet-50 to classify the gestures. The recognized gestures are mapped to predefined computer operations, such as clicking, sliding, and tapping the keyboard. This process involves gesture modeling and analysis. By determining gesture samples and using geometric invariance, combined with algorithms to implement gesture classification processing, accurate mapping of gestures to computer operations is achieved. We optimized the feature set, selected the features most relevant to gesture recognition, and reduced irrelevant or redundant features to reduce computational complexity and improve recognition accuracy. We used multi-scale feature fusion technology to combine feature information at different scales to better capture the global and local features of gestures, thereby optimizing the accuracy of gesture recognition.

5. A method for using the visually interactive multifunctional robot based on an AI large model as described in any one of claims 1 to 4, characterized in that: The following steps are involved: Start the AI ​​dialogue interaction function to communicate with users; Capture the user’s emotional state through sentiment analysis capabilities; Adjust the content and tone of the conversation based on the user’s emotional state; Record the user's emotional changes; Learning user's gesture commands through a gesture recognition unit; Simulate the user's computer operation through the computer control unit; Receive user's voice commands and perform corresponding computer operations; Records user's computer usage habits and automatically adjusts computer settings when the user connects.

Images