Desktop type electronic pet for monitoring working state of user

Abstract

The invention relates to a desktop type electronic pet for monitoring the working state of a user. The desktop type electronic pet comprises a shell, a main control board, a touch display unit, a sensing unit and a motion unit. The microprocessor unit is responsible for processing user interaction information from the touch display unit; acquiring shooting data of the high-definition camera and acquisition data of the pressure sensor; a voice command of a user is captured and processed, conversion from voice to text is completed or the voice command is directly recognized into a control command, voice recognition and voice wake-up functions are achieved, and voice response is fed back; data interaction with an IMU inertial measurement unit sensor is carried out through a Bluetooth technology, a user state is monitored in real time, user behavior data is uploaded to a cloud end, and user experience is optimized by utilizing a self-learning and emotion recognition technology; and the control unit also controls the motion unit to receive face tracking data of the yov5 model and output a PWM signal to the steering engine to ensure that the face is kept in the visual field of the camera so as to complete mechanical interaction.

Description

Technical Field

[0001] The invention relates to an electronic pet, in particular to a desktop electronic pet used for monitoring the working status of a user. Background Art

[0002] With the development of science and technology and the needs of society, smart desktop pets have gradually attracted attention as an electronic product that can provide companionship for users. For example, the robot Emopet is an AI desktop children's electronic pet with intelligent emotional voice interaction. It can interact with users through voice and has functions such as capturing sound, language communication, touch sensing and face recognition.

[0003] Existing desktop electronic pets do not have the ability to monitor the user's work and study status, cannot solve the user's mental health problems in a high-pressure working environment, and only have voice interaction but cannot provide more personalized support and interaction. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a desktop electronic pet for monitoring the user's working status, which can accurately judge the user's working and learning status through multi-modal posture detection and face recognition technology, and provide real-time feedback and suggestions.

[0005] The present invention provides a desktop electronic pet for monitoring the working status of a user, comprising:

[0006] The shell is the main structure of the electronic pet robot;

[0007] The main control board is embedded in the housing and includes a microprocessor unit, a serial communication interface, and an expansion interface;

[0008] A touch display unit is provided on the housing and includes a touch screen, and collects and processes data through an onboard analog-to-digital conversion module;

[0009] A perception unit, comprising a high-definition camera, a pressure sensor, a microphone array, an analog-to-digital conversion module, an audio codec chip, an audio power amplifier and a speaker; wherein the high-definition camera, the pressure sensor and the speaker are arranged on the housing, and the microphone array, the analog-to-digital conversion module, the audio codec chip and the audio power amplifier are arranged on the main control board; the high-definition camera and the pressure sensor are connected to the microprocessor unit through the analog-to-digital conversion module; the microphone array is connected to the microprocessor through the analog-to-digital conversion module and the audio codec chip, the microprocessor is connected to the speaker through the audio codec chip and the audio power amplifier, the microphone array, the analog-to-digital conversion module and the audio codec chip are used to capture voice commands and perform preliminary processing; the audio power amplifier and the speaker achieve high-quality audio output; the perception unit also exchanges data with the IMU inertial measurement unit in the user-worn device through the onboard Bluetooth, the user-worn device includes the IMU inertial measurement unit, which can collect one or more data of acceleration, angular velocity, magnetic field, Euler angle, number of rotations of the x and z axes, temperature, and air pressure;

[0010] The motion unit includes a horizontal servo and a vertical servo arranged between the shell and the base. It tracks the face through the yolov5 model and outputs a PWM with a corresponding duty cycle to control the horizontal servo and the vertical servo to rotate to a specified position at the same time, and controls the angle change of the electronic pet, thereby ensuring that the face is within the imaging range of the high-definition camera and completing the mechanical interaction; the base is used to fix the electronic pet on the desktop;

[0011] The microprocessor unit is responsible for processing user interaction information from the touch display unit; obtaining shooting data from the high-definition camera and data collected by the pressure sensor; capturing and processing user voice commands, completing voice-to-text conversion or directly recognizing control commands, realizing voice recognition and voice wake-up functions, and feeding back voice responses; interacting with the IMU inertial measurement unit sensor through Bluetooth technology, monitoring user status in real time, and uploading user behavior data to the cloud, optimizing user experience by using emotion recognition and emotion feedback technology, as well as data fusion, adaptive learning, and user customization; and controlling the motion unit, receiving face tracking data from the yolov5 model, and outputting PWM signals to the servo to ensure that the face remains in the camera field of view and complete mechanical interaction;

[0012] The entire system architecture is divided into perception layer, transmission layer, reasoning layer and user layer;

[0013] The perception layer is composed of an IMU inertial measurement unit, a microphone array, a high-definition camera and a pressure sensor, and is responsible for collecting multimodal data such as the user's posture, voice, expression and environment. These data are transmitted to the transmission layer through ADC analog-to-digital conversion and data preprocessing. The transmission layer uses communication protocols such as Bluetooth, MQTT and interfaces such as GPIO to aggregate the data to the microprocessor unit of the main control board and upload it to the cloud;

[0014] Neural network models such as WakeNet, ResNet, and Yolov5 in the cloud perform inference and decision-making on data, forming the inference layer, processing data streams from the perception layer, responsible for model training, data processing, and keyword data set management, working in collaboration with large models deployed locally to achieve deeper inference and decision-making, conduct status monitoring and analysis, and implement functions such as voice wake-up, chat, and face tracking, providing users with more personalized and intelligent services;

[0015] Ultimately, the user layer interacts emotionally with users through voice interaction, display interaction, and mechanical interaction, providing expression and gesture feedback while connecting to community culture to enhance user experience.

[0016] Furthermore, the data collected by the MU inertial measurement unit include:

[0017] (1) Three-axis angular velocity

[0018] The angular velocity of a device along the x, y, and z axes as measured by a gyroscope, usually in degrees per second or radians per second.

[0019] (2) Three-axis magnetic field (eliminating external magnetic field interference)

[0020] The magnetic field strength of a device along the x, y, and z axes as measured by a magnetometer, usually in Gauss or Tesla.

[0021] (3) Three-axis acceleration including gravity gX, gY, gZ

[0022] The acceleration of the device on the x, y, and z axes measured by the accelerometer, including the gravity component; when stationary, the acceleration on the z axis is usually 1g (9.81m / s 2 ).

[0023] (4) Quaternions

[0024] Quaternion is a mathematical tool used to represent three-dimensional rotation. Quaternion (w, x, y, z) can describe the posture of the device and avoid the universal joint deadlock problem of Euler angles.

[0025] (5) Euler angles

[0026] Euler angles (roll, pitch, yaw) represent the rotation angle of the device on three axes, usually in degrees.

[0027] (6) Measurement of the number of rotations of the x and z axes

[0028] Measures the number of rotations of the device on the x-axis and z-axis to detect the number of complete rotations of the device.

[0029] (7) Triaxial acceleration aX, aY, aZ after removing the gravity component

[0030] The acceleration of the device on the x, y, and z axes after removing the gravity component, as measured by the accelerometer. The acceleration is 0g when the device is stationary.

[0031] (8) Navigation system acceleration asX, asY, asZ (gravity removed and aligned to the reference coordinate system)

[0032] The navigation frame acceleration is the acceleration in the reference coordinate system, with the gravity component removed and aligned to the reference coordinate system for ease of navigation and motion analysis.

[0033] (9) Three-dimensional spatial position

[0034] The position of the device in three-dimensional space is calculated by quadratically integrating the acceleration and angular velocity data. These data are used to capture the motion trajectory in three-dimensional space.

[0035] After the microprocessor unit establishes a connection with the IMU inertial measurement unit, it receives the data packet sent by the IMU server and sends it to the cloud. The cloud transmits and receives the image information collected by the high-definition camera and the attitude information collected by the IMU through the MQTT protocol. The image information is in JPEG hexadecimal encoding format, which is converted into a binary image file using a Python script for the next step. Regarding the analysis of the IMU attitude information, first set a series of conversion ratios as follows to convert the raw data into actual physical quantities:

[0036] (1) The conversion ratio of acceleration data is 0.00478515625, which is used to convert the raw data into acceleration values ​​in g units;

[0037] (2) The conversion ratio of quaternion data is 0.000030517578125, which is used to convert the original data into the component value of the quaternion;

[0038] (3) The conversion ratio of the angle data is 0.0054931640625, which is used to convert the original data into angle values;

[0039] (4) The conversion ratio of angular velocity data is 0.06103515625, which is used to convert the raw data into angular velocity values;

[0040] (5) The conversion ratio of magnetic field intensity data is 0.15106201171875, which is used to convert the original data into magnetic field intensity values;

[0041] (6) The conversion ratio of temperature data is 0.01, which is used to convert the original data into temperature values;

[0042] (7) The conversion ratio of air pressure data is 0.0002384185791, which is used to convert the raw data into air pressure values;

[0043] (8) The conversion ratio of height data is 0.0010728836, which is used to convert the original data into height values.

[0044] After parsing the data, the cloud deployment part is divided into two processes:

[0045] First, the face detection deployment part: by inputting the image into yolov5 to capture the user's face, it can be determined whether the user is within the range that the electronic pet can shoot;

[0046] Once the user's face is detected, the microprocessor unit responds immediately and controls the horizontal and vertical servos to adjust the direction of the electronic pet, ensuring that the face is always within the imaging range of the high-definition camera, thereby achieving dynamic interaction and line of sight calibration, and enhancing the natural interactive experience between the user and the electronic pet.

[0047] Second, IMU data analysis and deployment: After detecting the user's face, the user's behavior status is judged according to the IMU data features. An improved ResNeXt network (deep residual X network) is used as the neural network for IMU data feature extraction. The network structure is as follows:

[0048] (1) Input layer:

[0049] The input data is time series data from a three-axis accelerometer, a three-axis gyroscope, a three-axis magnetometer, and a temperature sensor. The data dimension is (n, 10), where n represents the length of the time series and 10 represents the sensor data dimension. The specific input data vector is shown in the following table:

[0050]

[0051] The corresponding input matrix is ​​X∈R n×10 , the input after normalization is expressed as:

[0052]

[0053] Among them, X is the input feature, R is a real number set, X' is the standardized input feature, μ and σ represent the mean and standard deviation of sensor data respectively;

[0054] (2) Pre-treatment layer:

[0055] The input sensor data is standardized to eliminate the scale differences between different sensors; then, more training samples are generated through data enhancement technology to improve the robustness of the model.

[0056] The data augmentation operation is expressed as:

[0057] X′′=X′+∈

[0058] Among them, ∈ is the simulated noise, which obeys Gaussian distribution

[0059] (3) ResNeXt convolution module:

[0060] In this layer, the multi-path convolution design of ResNeXt is used, with 3x3 and 1x1 convolution kernels and 32 paths to capture diverse features;

[0061] The output of each residual block is expressed as:

[0062]

[0063] in, represents the convolution operation on the i-th path, X is the input feature, and k is the number of parallel paths;

[0064] (4) Improved ResNeXt module:

[0065] In each ResNeXt residual block, an attention mechanism, such as a SE module or a CBAM module, is embedded to adaptively adjust the weights of each channel;

[0066] The calculation formula of the attention mechanism is:

[0067] Z=σ(W2δ(W1S))

[0068] Where Z is the weight vector, S is the global pooling result of the input feature map, W1 and W2 are the weights of the fully connected layer, δ(·) is the ReLU activation function, and σ(·) is the Sigmoid activation function;

[0069] (5) Recurrent Neural Network Layer:

[0070] After the convolution module, a bidirectional LSTM layer is added to capture the long-term dependencies in the time series data. The output of each LSTM unit is:

[0071] h t =LSTM(x t ,h t-1 )

[0072] Among them, h t is the hidden state at time t, x t is the input sequence, h t-1 is the hidden state at the previous moment;

[0073] (6) Fully connected layer:

[0074] The output of the LSTM layer passes through the fully connected layer to map the high-dimensional features to the output space; Dropout is used for regularization to prevent overfitting; the output of the fully connected layer is:

[0075] o=W f h+b f

[0076] Among them, o is the output of the fully connected layer, W f is the weight matrix of the fully connected layer, h is the hidden state vector of the LSTM layer, and b f is the bias vector.

[0077] (7) Output layer:

[0078] The output layer uses a multi-task learning framework, and each task predicts the data of different IMU sensors; the output layer of each task uses a linear activation function:

[0079] y i =W i o+b i

[0080] Among them, y i is the output of the ith task, W i and b i are the weight and bias of the task respectively.

[0081] Use the improved ResNeXt network as the training base model and train it on the PAMAP2 dataset:

[0082] According to the PAMAP2 dataset, there are 19 types of label classifications, including one type of "uncertain" type. Under this type, the user may be in the remaining 18 states, or may be out of these 18 states; according to the needs, these 19 state labels are mapped to 3 state labels: working state, non-working state, and uncertain state. The corresponding mapping rules are as follows:

[0083] The label states of the original dataset include: 1 lying down, 2 sitting, 3 standing, 4 walking, 5 running, 6 cycling, 7 Nordic walking, 9 watching TV, 10 working on the computer, 11 driving, 12 going up stairs, 13 going down stairs, 16 vacuuming, 17 ironing, 18 folding clothes, 19 cleaning the room, 20 playing football, 24 skipping rope, 0 other (temporary activities);

[0084] Mapping rules:

[0085] 0 represents an uncertain state; 1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 13, 16, 17, 18, 19, 20, 24 represent non-working states; 10 represents working state;

[0086] After the two processes deployed in the cloud obtain the face detection results of Yolov5 and the classification results of the IMU input ResNeXt network respectively, when the user's face can be detected and the user is in a working state, it is considered that the user is in the working state recorded by the robot.

[0087] The microprocessor unit pre-sets the working time reminder threshold and upper limit threshold. When the user's working time reaches the reminder threshold, the microprocessor unit issues a command to control the touch display unit and the sensing unit to issue an image or sound reminder to remind the user of the continuous working time; when the user's working time reaches or exceeds the upper limit threshold, the microprocessor unit issues a command to control the touch display unit and the sensing unit to issue an image or sound reminder to remind the user of the continuous working time and to remind the user to take a break, or to play music, or to play a communication voice for relieving stress. The user can interact with the electronic pet to relieve stress by touching the touch screen or inputting commands by voice, or by touching the pressure sensor or communicating by voice.

[0088] Furthermore, the emotion recognition comprises the following steps:

[0089] 1.1 Facial Expression Recognition

[0090] HD camera acquisition: Use a HD camera to capture the user's facial image in real time;

[0091] Face detection: Input the captured images into the Haar Cascades model for face detection and locate key facial areas;

[0092] Feature extraction: extract the position and shape changes of facial key points;

[0093] Emotion classification: Through a trained model such as ResNet-50, the extracted facial features are mapped to predefined emotional states including happiness, sadness, surprise, and anger.

[0094] 1.2 Speech Emotion Recognition

[0095] Voice collection: Capture the user's voice through a microphone array;

[0096] Speech preprocessing: including denoising, segmentation, and feature extraction;

[0097] Emotion classification: Use the model network to classify the emotions of speech features, including happiness, sadness, surprise, and anger.

[0098] The emotional feedback comprises the following steps:

[0099] 2.1 Emotion-driven interfaces

[0100] Touch display unit feedback: According to the emotion recognition results, the touch display unit displays the corresponding expression or animation;

[0101] Voice feedback: Play voice or music that matches the recognized emotion through an audio power amplifier and speaker.

[0102] 2.2 Mechanical motion feedback

[0103] Servo control: According to the emotional state, control the horizontal and vertical servos to adjust the angle of the electronic pet to express the corresponding emotions, including tilting the head to express curiosity or attention, looking up to express happiness, and looking down to express sadness.

[0104] The data fusion, adaptive learning and user customization include the following steps:

[0105] 3.1 Multimodal data fusion: Combining facial expression and voice emotion recognition results to improve the accuracy of emotion recognition through the model;

[0106] 3.2 Adaptive learning: Analyze the interaction data between users and electronic pets through models, and continuously optimize the emotion recognition model and feedback strategy to provide a more personalized interactive experience;

[0107] 3.3 User customization: Allow users to customize the sensitivity and feedback method of emotion recognition through the touch display unit; users can train personalized emotion recognition models through interaction with electronic pets.

[0108] Beneficial effects of the present invention:

[0109] The present invention aims to propose an electronic pet deployed on a desktop, which combines face recognition and IMU posture data features to extract the user's working status, monitors the user's tiny movements through the IMU inertial measurement unit and the high-definition camera, and identifies them through image processing and motion tracking algorithms, so as to accurately judge the user's work, study and other life status. The captured motion information is converted into an instant response of the electronic pet, providing users with a more natural and smooth interactive experience, and by recording the time when the user is in different states, it solves the mental health problems of modern young people caused by the fast-paced life and high-pressure working environment.

[0110] The device can not only help users manage and optimize their work and study status, but also provide emotional support and relieve stress. The main technical obstacles overcome include accurately judging the user's work and study status through multimodal gesture detection and face recognition technology, and providing real-time feedback and suggestions. At the same time, through voice interaction and touch interaction functions, users can interact with the device as if they were communicating with a real person, relieving stress in work and study.

[0111] The device has a self-learning function. It continuously collects behavioral data of users interacting with electronic pets and uses machine learning algorithms for in-depth analysis to intelligently identify users' usage habits and preferences and continuously optimize its own interaction mode. This adaptive learning mechanism enables the device to better understand users' habits and preferences, thereby providing personalized services, which is an advantage that existing technologies do not have.

[0112] The device is not just a simple electronic pet, but also an intelligent companion that integrates multiple advanced technologies to help users establish a healthy and regular lifestyle. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] Figure 1 This is a schematic diagram of the structure of the electronic pet of the present invention;

[0114] Figure 2 This is a schematic diagram of the electronic pet system architecture of the present invention;

[0115] 1. Shell, 2. Touch screen, 3. HD camera, 4. Pressure sensor, 5. Base, 6. Servo. DETAILED DESCRIPTION

[0116] like Figure 1 As shown, the present invention provides a desktop electronic pet for monitoring the working status of a user, comprising:

[0117] Shell 1, serving as the main structure of the electronic pet robot;

[0118] The main control board, embedded in the shell, is ESP32-S3-LCD-EV-Board, which is the main core board of the electronic pet. It uses the ESP32-S3-WROOM-1 module as the microprocessor unit. It has designed a variety of power management solutions, including 5V to 3.3V DC-DC conversion, to ensure that peripherals with different voltage requirements can work stably. The onboard USB-to-UART module CP2102N provides a serial communication interface with the PC, and the USB-to-USB port supports device program download and debugging. It has an XH2.54 speaker interface. It is also equipped with a TCA9554 I / O expansion chip to enhance the system's IO expansion capability. In order to provide a richer interactive experience, it also integrates a WS2812C LED expansion interface and a variety of GPIO expansion connectors to support a variety of LCD display connection methods.

[0119] The touch display unit is arranged on the housing and includes a 3.95-inch LCD touch screen 2 (touch driver chip FT5x06) with a 480×480 pixel size and an RGB interface. It collects and processes the data of the RP-C18.3-ST+ linear voltage conversion module through the onboard ADC module (ES7210), thereby providing a user interaction interface and responding to touch operations;

[0120] A perception unit, comprising a high-definition camera 3, a pressure sensor 4, a microphone array, an analog-to-digital conversion module, an audio codec chip, an audio power amplifier and a speaker; wherein the high-definition camera 3, the pressure sensor 4 and the speaker are arranged on the shell, and the microphone array, the analog-to-digital conversion module, the audio codec chip and the audio power amplifier are arranged on the main control board; the high-definition camera 3 and the pressure sensor 4 are connected to the microprocessor unit through the analog-to-digital conversion module; the microphone array is connected to the microprocessor through the analog-to-digital conversion module and the audio codec chip, the microprocessor is connected to the speaker through the audio codec chip and the audio power amplifier, the microphone array, the analog-to-digital conversion module and the audio codec chip are used to capture voice commands and perform preliminary processing; the audio power amplifier and the speaker realize high-quality audio output; the perception unit also exchanges data with the IMU inertial measurement unit in the user-worn device through the onboard Bluetooth, and the user-worn device includes the IMU inertial measurement unit, which can collect one or more data of acceleration, angular velocity, magnetic field, Euler angle, number of rotations of the x and z axes, temperature and air pressure;

[0121] In this embodiment, the audio codec chip model is ES8311 audio codec chip, which is a low-power mono codec with single-channel ADC and DAC, low-noise preamplifier, headphone driver, digital sound effects and analog mixing functions. ES8311 is connected to the ESP32-S3-WROOM-1 module via I2S and I2C buses, providing dedicated hardware audio processing capabilities. The microphone arrays are respectively arranged on the left and right sides of the main control board, and the microphone arrays are respectively connected to the ADC module audio analog-to-digital converter ES7210, which is a high-performance, low-power 4-channel audio analog-to-digital converter, particularly suitable for microphone array applications, and has an acoustic echo cancellation (AEC) function, which is very suitable for music and voice applications, thereby optimizing the quality of audio input. The AEC function is implemented by the AEC circuit, and the echo reference signal is collected through the specific pins of the ADC module to provide input for the AEC algorithm, thereby improving the quality of voice communication. The audio components and the main control board ESP32-S3 module of this embodiment adopt an independent power supply design, which effectively reduces the interference of digital noise on the audio signal and improves the overall audio performance. The processed digital audio signal passes through the ES8311 codec chip and is then amplified by the NS4150 audio power amplifier, which is a low EMI, 3W mono Class D amplifier suitable for driving speakers (4Ω / 3W). Finally, the amplified audio signal is played through the speaker, achieving high-fidelity audio output.

[0122] In this embodiment, the data collected by the MU inertial measurement unit includes:

[0123] 1. Three-axis angular velocity

[0124] The angular velocity of a device along the x, y, and z axes as measured by a gyroscope, usually in degrees per second or radians per second.

[0125] 2. Three-axis magnetic field (removing external magnetic field interference)

[0126] The magnetic field strength of a device along the x, y, and z axes as measured by a magnetometer, usually in Gauss or Tesla.

[0127] 3. Three-axis acceleration including gravity gX, gY, gZ

[0128] The acceleration of the device on the x, y, and z axes measured by the accelerometer, including the gravity component; when stationary, the acceleration on the z axis is usually 1g (9.81m / s 2 ).

[0129] 4. Quaternions

[0130] Quaternion is a mathematical tool used to represent three-dimensional rotation. Quaternion (w, x, y, z) can describe the posture of the device and avoid the universal joint deadlock problem of Euler angles.

[0131] 5. Euler Angles

[0132] Euler angles (roll, pitch, yaw) represent the rotation angle of the device on three axes, usually in degrees.

[0133] 6. Measurement of x and z axis rotation times

[0134] Measures the number of rotations of the device on the x-axis and z-axis to detect the number of complete rotations of the device.

[0135] 7. Three-axis acceleration aX, aY, aZ after removing the gravity component

[0136] The acceleration of the device on the x, y, and z axes after removing the gravity component, as measured by the accelerometer. The acceleration is 0g when the device is stationary.

[0137] 8. Navigation system acceleration asX, asY, asZ (gravity removed and aligned to the reference coordinate system)

[0138] The navigation frame acceleration is the acceleration in the reference coordinate system, with the gravity component removed and aligned to the reference coordinate system for ease of navigation and motion analysis.

[0139] 9. Three-dimensional spatial position

[0140] The position of the device in three-dimensional space is calculated by quadratically integrating the acceleration and angular velocity data. These data are used to capture the motion trajectory in three-dimensional space.

[0141] The measurement data, range and accuracy of the IMU inertial measurement unit in this embodiment are shown in the following table:

[0142]

[0143] The motion unit includes a servo 6 disposed between the housing 1 and the base 5, including a horizontal servo and a vertical servo, which tracks the face through the YOLOv5 model, and outputs a PWM with a corresponding duty cycle, controls the horizontal servo and the vertical servo to rotate to a specified position at the same time, controls the angle change of the electronic pet, thereby ensuring that the face is within the imaging range of the high-definition camera, and completes the mechanical interaction;

[0144] The microprocessor unit is responsible for processing user interaction information from the touch display unit; the microprocessor unit obtains the shooting data of the high-definition camera and the collected data of the pressure sensor; the microprocessor unit captures and processes the user's voice commands, completes the conversion of voice to text or directly recognizes it into control commands, realizes functions such as voice recognition and voice wake-up, and feeds back voice response; the microprocessor unit interacts with the IMU inertial measurement unit sensor through Bluetooth technology, monitors the user status in real time, and uploads the user behavior data to the cloud, using self-learning and emotion recognition technology to optimize the user experience; the microprocessor unit also controls the motion unit, receives the face tracking data of the YOLOv5 model, and outputs a PWM signal to the servo to ensure that the face remains in the camera's field of view and completes the mechanical interaction.

[0145] like Figure 2 As shown, the whole system architecture is divided into perception layer, transmission layer, reasoning layer and user layer;

[0146] The perception layer is composed of an IMU inertial measurement unit, a microphone array, a high-definition camera and a pressure sensor, and is responsible for collecting multimodal data such as the user's posture, voice, expression and environment. These data are transmitted to the transmission layer through ADC analog-to-digital conversion and data preprocessing. The transmission layer uses communication protocols such as Bluetooth, MQTT and interfaces such as GPIO to aggregate the data to the microprocessor unit of the main control board and upload it to the cloud;

[0147] Neural network models such as WakeNet, ResNet, and Yolov5 in the cloud perform inference and decision-making on data, forming the inference layer, processing data streams from the perception layer, responsible for model training, data processing, and keyword data set management, working in collaboration with large models deployed locally to achieve deeper inference and decision-making, conduct status monitoring and analysis, and implement functions such as voice wake-up, chat, and face tracking, providing users with more personalized and intelligent services;

[0148] Ultimately, the user layer interacts emotionally with users through voice interaction, display interaction, and mechanical interaction, providing expression and gesture feedback while connecting to community culture to enhance user experience.

[0149] Furthermore, after the microprocessor unit establishes a connection with the IMU inertial measurement unit through Bluetooth GATT (Generic Attribute Profile), it receives the data packet sent by the IMU server and sends it to the cloud; the cloud transmits and receives the image information collected by the high-definition camera and the attitude information collected by the IMU through the MQTT protocol. Among them, the image information is in JPEG hexadecimal encoding format, and is converted into a binary image file using a Python script for the next step. Regarding the analysis of the IMU attitude information, first set a series of conversion ratios as follows to convert the raw data into actual physical quantities:

[0150] (1) The conversion ratio of acceleration data is 0.00478515625, which is used to convert the raw data into acceleration values ​​in g units;

[0151] (2) The conversion ratio of quaternion data is 0.000030517578125, which is used to convert the original data into the component value of the quaternion;

[0152] (3) The conversion ratio of the angle data is 0.0054931640625, which is used to convert the original data into angle values;

[0153] (4) The conversion ratio of angular velocity data is 0.06103515625, which is used to convert the raw data into angular velocity values;

[0154] (5) The conversion ratio of magnetic field intensity data is 0.15106201171875, which is used to convert the original data into magnetic field intensity values;

[0155] (6) The conversion ratio of temperature data is 0.01, which is used to convert the original data into temperature values;

[0156] (7) The conversion ratio of air pressure data is 0.0002384185791, which is used to convert the raw data into air pressure values;

[0157] (8) The conversion ratio of height data is 0.0010728836, which is used to convert the original data into height values.

[0158] After parsing the data, the cloud deployment part is divided into two processes:

[0159] First, the face detection deployment part: by inputting the image into yolov5 to capture the user's face, it can be determined whether the user is within the range that the robot can "see";

[0160] Using the yolov5 version of the model, the image from MQTT is directly input into the yolov5 model to obtain the detection results. The default hyperparameters of the YOLOV5 model are as follows:

[0161] The initial learning rate is set to 0.01, which is suitable for the SGD optimizer, and combined with the OneCycleLR scheduling strategy, so that the final learning rate is 20% of the initial learning rate; the momentum is set to 0.937, which is mainly used to accelerate gradient updates; at the same time, we introduced weight decay with a value of 0.0005 to reduce the risk of overfitting; the learning rate in the warm-up phase gradually increases within 3 epochs, the momentum starts from 0.8, and the bias learning rate is initially 0.1.

[0162] In the loss function part, the weight of bounding box regression loss is set to 0.05, the weight of classification loss is set to 0.5, and the weight of positive samples is set to 1.0. At the same time, the weight of target detection loss is set to 1.0, and the weight of positive samples is also set to 1.0. The intersection-over-union ratio threshold is 0.20, and prediction boxes below this value will be ignored. The anchor box threshold is set to 4.0 to judge the matching degree between the anchor box and the target box. The gamma value of the focus loss is kept at 0, indicating that this mechanism is not used.

[0163] In terms of image enhancement, the hue, saturation and brightness were adjusted by 1.5%, 70% and 40% respectively; at the same time, the image can be translated within a range of 10%, scaled within a range of 50%, and there is a 50% probability of flipping left and right; other transformations such as rotation, shearing, perspective transformation, and upside down flipping were not used; in addition, Mosaic data enhancement technology was fully applied, while Mixup data enhancement was not enabled.

[0164] Once the user's face is detected, the microprocessor unit responds immediately and controls the horizontal and vertical servos to adjust the direction of the electronic pet, ensuring that the face is always within the imaging range of the high-definition camera, thereby achieving dynamic interaction and line of sight calibration, and enhancing the natural interactive experience between the user and the electronic pet.

[0165] Second, IMU data analysis and deployment: After detecting the user's face, the user's behavior status is judged according to the IMU data features. An improved ResNeXt network (deep residual X network) is used as the neural network for IMU data feature extraction. The network structure is as follows:

[0166] (1) Input layer:

[0167] The input data is time series data from a three-axis accelerometer, a three-axis gyroscope, a three-axis magnetometer, and a temperature sensor. The data dimension is (n, 10), where n represents the length of the time series and 10 represents the sensor data dimension. The specific input data vector is shown in the following table:

[0168]

[0169] The corresponding input matrix is ​​X∈R n×10 , the input after normalization is expressed as:

[0170]

[0171] Among them, X is the input feature, R is a real number set, X' is the standardized input feature, μ and σ represent the mean and standard deviation of sensor data respectively;

[0172] (2) Pre-treatment layer:

[0173] The input sensor data is standardized to eliminate the scale differences between different sensors; then, more training samples are generated through data enhancement technology to improve the robustness of the model.

[0174] The data augmentation operation is expressed as:

[0175] X′′=X′+∈

[0176] Among them, ∈ is the simulated noise, which obeys Gaussian distribution

[0177] (3) ResNeXt convolution module:

[0178] In this layer, the multi-path convolution design of ResNeXt is used, with 3x3 and 1x1 convolution kernels and 32 paths to capture diverse features;

[0179] The output of each residual block is expressed as:

[0180]

[0181] in, represents the convolution operation on the i-th path, X is the input feature, and k is the number of parallel paths;

[0182] (4) Improved ResNeXt module:

[0183] In each ResNeXt residual block, an attention mechanism, such as a SE module or a CBAM module, is embedded to adaptively adjust the weights of each channel;

[0184] The calculation formula of the attention mechanism is:

[0185] Z=σ(W2δ(W1S))

[0186] Where Z is the weight vector, S is the global pooling result of the input feature map, W1 and W2 are the weights of the fully connected layer, δ(·) is the ReLU activation function, and σ(·) is the Sigmoid activation function;

[0187] (5) Recurrent Neural Network Layer:

[0188] After the convolution module, a bidirectional LSTM layer is added to capture the long-term dependencies in the time series data. The output of each LSTM unit is:

[0189] h t =LSTM(x t ,h t-1 )

[0190] Among them, h t is the hidden state at time t, x t is the input sequence, h t-1 is the hidden state at the previous moment;

[0191] (6) Fully connected layer:

[0192] The output of the LSTM layer passes through the fully connected layer to map the high-dimensional features to the output space; Dropout is used for regularization to prevent overfitting; the output of the fully connected layer is:

[0193] o=W f h+b f

[0194] Among them, o is the output of the fully connected layer, W f is the weight matrix of the fully connected layer, h is the hidden state vector of the LSTM layer, and b f is the bias vector.

[0195] (7) Output layer:

[0196] The output layer uses a multi-task learning framework, and each task predicts the data of different IMU sensors; the output layer of each task uses a linear activation function:

[0197] y i =W i o+b i

[0198] Among them, y i is the output of the ith task, W i and b i are the weight and bias of the task respectively.

[0199] Use the improved ResNeXt network as the basic training model and use the following hyperparameters to train on the PAMAP2 dataset:

[0200] batch_size == 128;

[0201] epoch == 300;

[0202] lr==0.0005;

[0203] Weight decay is set to 0.001 in the AdamW optimizer;

[0204] The learning rate scheduler uses StrpLR, the cycle is 1 / 3 of the training cycle, and the learning rate decay factor is 0.5;

[0205] Loss_Function is set to Cross Entropy Loss;

[0206] Mixed precision training: using GradScaler;

[0207] According to the PAMAP2 dataset, there are 19 types of label classifications, including one type of "uncertain" type. Under this type, the user may be in the remaining 18 states, or may be out of these 18 states; according to the needs, these 19 state labels are mapped to 3 state labels: working state, non-working state, and uncertain state. The corresponding mapping rules are as follows:

[0208] The label states of the original dataset include: 1 lying down, 2 sitting, 3 standing, 4 walking, 5 running, 6 cycling, 7 Nordic walking, 9 watching TV, 10 working on the computer, 11 driving, 12 going up stairs, 13 going down stairs, 16 vacuuming, 17 ironing, 18 folding clothes, 19 cleaning the room, 20 playing football, 24 skipping rope, 0 other (temporary activities);

[0209] Mapping rules:

[0210] 0 represents an uncertain state; 1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 13, 16, 17, 18, 19, 20, 24 represent non-working states; 10 represents working state;

[0211] After the two processes deployed in the cloud obtain the face detection results of Yolov5 and the classification results of the IMU input ResNeXt network respectively, when the user's face can be detected and the user is in a working state, it is considered that the user is in the working state recorded by the robot.

[0212] The microprocessor unit pre-sets the working time reminder threshold and upper limit threshold. When the user's working time reaches the reminder threshold, the microprocessor unit issues a command to control the touch display unit and the sensing unit to issue an image or sound reminder to remind the user of the continuous working time; when the user's working time reaches or exceeds the upper limit threshold, the microprocessor unit issues a command to control the touch display unit and the sensing unit to issue an image or sound reminder to remind the user of the continuous working time and to remind the user to take a break, or to play music, or to play a communication voice for relieving stress. The user can interact with the electronic pet to relieve stress by touching the touch screen or inputting commands by voice, or by touching the pressure sensor or communicating by voice, or set other personalized interaction commands for human-computer interaction, thereby enhancing the user's immersion and emotional connection.

[0213] The emotion recognition comprises the following steps:

[0214] 1.1 Facial Expression Recognition

[0215] HD camera acquisition: Use a HD camera to capture the user's facial image in real time;

[0216] Face detection: Input the captured images into the Haar Cascades model for face detection and locate key facial areas;

[0217] Feature extraction: extract the position and shape changes of facial key points (such as eyes, eyebrows, corners of mouth, etc.);

[0218] Emotion classification: The extracted facial features are mapped to predefined emotional states (such as happiness, sadness, surprise, anger, etc.) through a trained model such as ResNet-50.

[0219] 1.2 Speech Emotion Recognition

[0220] Voice collection: Capture the user's voice through a microphone array;

[0221] Speech preprocessing: including denoising, segmentation, and feature extraction (such as Mel-frequency cepstral coefficients MFCC);

[0222] Emotion classification: Use models such as LSTM networks to classify the emotions of speech features (such as happiness, sadness, surprise, anger, etc.).

[0223] The emotional feedback comprises the following steps:

[0224] 2.1 Emotion-driven interfaces

[0225] Touch display unit feedback: According to the emotion recognition results, the touch display unit displays corresponding expressions or animations, such as smiling faces, sad faces, etc.

[0226] Voice feedback: Through the audio power amplifier and speaker, play the voice or music that matches the recognized emotion, such as relaxing and cheerful music corresponding to happy emotions.

[0227] 2.2 Mechanical motion feedback

[0228] Servo control: According to the emotional state, control the horizontal and vertical servos to adjust the angle of the electronic pet to express the corresponding emotions, such as tilting the head to express curiosity or attention, looking up to express happiness, and looking down to express sadness.

[0229] The data fusion, adaptive learning and user customization include the following steps:

[0230] 3.1 Multimodal data fusion: Combining facial expression and speech emotion recognition results, and improving the accuracy of emotion recognition through models such as LateFusion;

[0231] 3.2 Adaptive learning: Analyze the interaction data between users and electronic pets through models such as AdaBoost, and continuously optimize the emotion recognition model and feedback strategy to provide a more personalized interaction experience;

[0232] 3.3 User customization: Allow users to customize the sensitivity and feedback method of emotion recognition through the touch display unit; users can train personalized emotion recognition models such as support vector machines (SVM) through interaction with electronic pets.

Claims

1. A desktop electronic pet for monitoring the user's working status, characterized by: include: The shell is the main structure of the electronic pet robot; The main control board is embedded in the housing and includes a microprocessor unit, a serial communication interface, and an expansion interface; A touch display unit is provided on the housing and includes a touch screen, and collects and processes data through an onboard analog-to-digital conversion module; A perception unit, comprising a high-definition camera, a pressure sensor, a microphone array, an analog-to-digital conversion module, an audio codec chip, an audio power amplifier and a speaker; wherein the high-definition camera, the pressure sensor and the speaker are arranged on the housing, and the microphone array, the analog-to-digital conversion module, the audio codec chip and the audio power amplifier are arranged on the main control board; the high-definition camera and the pressure sensor are connected to the microprocessor unit through the analog-to-digital conversion module; the microphone array is connected to the microprocessor through the analog-to-digital conversion module and the audio codec chip, the microprocessor is connected to the speaker through the audio codec chip and the audio power amplifier, the microphone array, the analog-to-digital conversion module and the audio codec chip are used to capture voice commands and perform preliminary processing; the audio power amplifier and the speaker achieve high-quality audio output; the perception unit also exchanges data with the IMU inertial measurement unit in the user-worn device through the onboard Bluetooth, the user-worn device includes the IMU inertial measurement unit, which can collect one or more data of acceleration, angular velocity, magnetic field, Euler angle, number of rotations of the x and z axes, temperature, and air pressure; The motion unit includes a horizontal servo and a vertical servo arranged between the shell and the base. It tracks the face through the yolov5 model and outputs a PWM with a corresponding duty cycle to control the horizontal servo and the vertical servo to rotate to a specified position at the same time, and controls the angle change of the electronic pet, thereby ensuring that the face is within the imaging range of the high-definition camera and completing the mechanical interaction; the base is used to fix the electronic pet on the desktop; The microprocessor unit is responsible for processing user interaction information from the touch display unit; obtaining shooting data from the high-definition camera and data collected by the pressure sensor; capturing and processing user voice commands, completing voice-to-text conversion or directly recognizing control commands, realizing voice recognition and voice wake-up functions, and feeding back voice responses; interacting with the IMU inertial measurement unit sensor through Bluetooth technology, monitoring user status in real time, and uploading user behavior data to the cloud, optimizing user experience using self-learning and emotion recognition technology; and controlling the motion unit, receiving face tracking data from the yolov5 model, and outputting PWM signals to the servo to ensure that the face remains in the camera's field of view and complete mechanical interaction; After the microprocessor unit establishes a connection with the IMU inertial measurement unit, it receives the data packet sent by the IMU server and sends it to the cloud. The cloud transmits and receives the image information collected by the high-definition camera and the attitude information collected by the IMU through the MQTT protocol. The image information is in JPEG hexadecimal encoding format and is converted into a binary image file using a Python script for the next step. The IMU attitude information is parsed, and a series of conversion ratios are set to convert the raw data into actual physical quantities. After parsing the data, the cloud deployment part is divided into two processes: First, the face detection deployment part: by inputting the image into yolov5 to capture the user's face, it can be determined whether the user is within the range that the electronic pet can shoot; Once the user's face is detected, the microprocessor unit responds immediately and controls the horizontal and vertical servos to adjust the direction of the electronic pet to ensure that the face is always within the imaging range of the high-definition camera, thereby achieving dynamic interaction and line of sight calibration; Second, IMU data analysis and deployment: After detecting the user's face, the user's behavior status is judged according to the IMU data features. An improved ResNeXt network is used as the neural network for IMU data feature extraction. The network structure is as follows: (1) Input layer: The input data is time series data from a three-axis accelerometer, a three-axis gyroscope, a three-axis magnetometer, and a temperature sensor. The data dimension is (n, 10), where n represents the length of the time series and 10 represents the sensor data dimension. The specific input data vector is as follows: The corresponding input matrix is ​​X∈R n×10 , the input after normalization is expressed as: Among them, X is the input feature, R is a real number set, X' is the standardized input feature, μ and σ represent the mean and standard deviation of sensor data respectively; (2) Pre-treatment layer: The input sensor data is standardized to eliminate the scale differences between different sensors. Then, more training samples are generated through data enhancement technology to improve the robustness of the model. The data augmentation operation is expressed as: X′′=X′+∈ Among them, ∈ is the simulated noise, which obeys Gaussian distribution (3) ResNeXt convolution module: In this layer, the multi-path convolution design of ResNeXt is used, with 3x3 and 1x1 convolution kernels and 32 paths to capture diverse features; The output of each residual block is expressed as: in, represents the convolution operation on the i-th path, X is the input feature, and k is the number of parallel paths; (4) Improved ResNeXt module: In each ResNeXt residual block, an attention mechanism, such as a SE module or a CBAM module, is embedded to adaptively adjust the weights of each channel; The calculation formula of the attention mechanism is: Z=σ(W2δ(W1S)) Where Z is the weight vector, S is the global pooling result of the input feature map, W1 and W2 are the weights of the fully connected layer, δ(·) is the ReLU activation function, and σ(·) is the Sigmoid activation function; (5) Recurrent Neural Network Layer: After the convolution module, a bidirectional LSTM layer is added to capture the long-term dependencies in the time series data. The output of each LSTM unit is: h t =:STM(x t ,h t-1 ) Among them, h t is the hidden state at time t, x t is the input sequence, h t-1 is the hidden state at the previous moment; (6) Fully connected layer: The output of the LSTM layer passes through the fully connected layer to map the high-dimensional features to the output space; Dropout is used for regularization to prevent overfitting; the output of the fully connected layer is: o=W f h+b f Among them, o is the output of the fully connected layer, W f is the weight matrix of the fully connected layer, h is the hidden state vector of the LSTM layer, and b f is the bias vector; (7) Output layer: The output layer uses a multi-task learning framework, and each task predicts the data of different IMU sensors; the output layer of each task uses a linear activation function: y i =W i o+b i Among them, y i is the output of the ith task, W i and b i are the weight and bias of the task respectively; Use the improved ResNeXt network as the training base model and train it on the PAMAP2 dataset: According to the PAMAP2 dataset, there are 19 types of label classifications, including one type of "uncertain" type. Under this type, the user may be in the remaining 18 states, or may be out of these 18 states; according to the needs, these 19 state labels are mapped to 3 state labels: working state, non-working state, and uncertain state. The corresponding mapping rules are as follows: The label states of the original dataset include: 1 lying down, 2 sitting, 3 standing, 4 walking, 5 running, 6 cycling, 7 Nordic walking, 9 watching TV, 10 working on the computer, 11 driving, 12 going up the stairs, 13 going down the stairs, 16 vacuuming, 17 ironing, 18 folding clothes, 19 cleaning the room, 20 playing football, 24 skipping rope, 0 other temporary activities; Mapping rules: 0 represents an uncertain state; 1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 13, 16, 17, 18, 19, 20, 24 represent non-working states; 10 represents working state; After the two processes deployed in the cloud obtain the face detection results of Yolov5 and the classification results of the IMU input ResNeXt network respectively, when the user's face can be detected and the user is in a working state, it is considered that the user is in the working state recorded by the robot.

2. A desktop electronic pet for monitoring user working status according to claim 1, characterized in that: The data collected by the IMU inertial measurement unit includes: (1) Three-axis angular velocity The angular velocity of the device on the x, y, and z axes as measured by the gyroscope, usually in degrees per second or radians per second; (2) Three-axis magnetic field The magnetic field strength of the device along the x, y, and z axes as measured by a magnetometer, usually in Gauss or Tesla; (3) Three-axis acceleration including gravity gX, gY, gZ The acceleration of the device on the x, y, and z axes measured by the accelerometer, including the gravity component; when stationary, the acceleration on the z axis is usually 1g (9.81m / s 2 ); (4) Quaternions Quaternions are mathematical tools used to represent three-dimensional rotations. Quaternions (w, x, y, z) can describe the posture of a device and avoid the gimbal deadlock problem of Euler angles. (5) Euler angles Euler angles include roll angle, pitch angle, and yaw angle, which represent the rotation angle of the device on three axes, usually in degrees; (6) Measurement of the number of rotations of the x and z axes Measure the number of rotations of the device on the x-axis and z-axis to detect the number of complete rotations of the device; (7) Triaxial acceleration aX, aY, aZ after removing the gravity component The acceleration of the device on the x, y, and z axes after removing the gravity component, as measured by the accelerometer. The acceleration is 0g when stationary. (8) Navigation system acceleration asX, asY, asZ The navigation frame acceleration is the acceleration in the reference coordinate system, with the gravity component removed and aligned to the reference coordinate system for ease of navigation and motion analysis; (9) Three-dimensional spatial position The position of the device in three-dimensional space is calculated by quadratically integrating the acceleration and angular velocity data. These data are used to capture the motion trajectory in three-dimensional space.

3. A desktop electronic pet for monitoring user working status according to claim 1, characterized in that: Parse the IMU attitude information and set a series of conversion ratios as follows: (1) The conversion ratio of acceleration data is 0.00478515625, which is used to convert the raw data into acceleration values ​​in g units; (2) The conversion ratio of quaternion data is 0.000030517578125, which is used to convert the original data into the component value of the quaternion; (3) The conversion ratio of the angle data is 0.0054931640625, which is used to convert the original data into angle values; (4) The conversion ratio of angular velocity data is 0.06103515625, which is used to convert the raw data into angular velocity values; (5) The conversion ratio of magnetic field intensity data is 0.15106201171875, which is used to convert the original data into magnetic field intensity values; (6) The conversion ratio of temperature data is 0.01, which is used to convert the original data into temperature values; (7) The conversion ratio of air pressure data is 0.0002384185791, which is used to convert the raw data into air pressure values; (8) The conversion ratio of height data is 0.0010728836, which is used to convert the original data into height values.

4. A desktop electronic pet for monitoring user working status according to claim 1, characterized in that: The working time reminder threshold and upper limit threshold are pre-set in the microprocessor unit. When the user's working time reaches the reminder threshold, the microprocessor unit issues an instruction to control the touch display unit and the perception unit to issue an image or sound reminder to remind the user of the continuous working time; when the user's working time reaches or exceeds the upper limit threshold, the microprocessor unit issues an instruction to control the touch display unit and the perception unit to issue an image or sound reminder to remind the user of the continuous working time, and remind the user to take a break, or play music, or play communication voice for relieving stress; the user interacts with the electronic pet to relieve stress by touching the touch screen or inputting instructions by voice, or by touching the pressure sensor or communicating by voice.

5. A desktop electronic pet for monitoring user working status according to claim 1, characterized in that: The emotion recognition comprises the following steps: (1.1) Facial Expression Recognition HD camera acquisition: Use a HD camera to capture the user's facial image in real time; Face detection: Input the captured images into the Haar Cascades model for face detection and locate key facial areas; Feature extraction: extract the position and shape changes of facial key points; Emotion classification: Through trained models such as ResNet-50, the extracted facial features are mapped to predefined emotional states including happiness, sadness, surprise, and anger; (1.2) Speech Emotion Recognition Voice collection: Capture the user's voice through a microphone array; Speech preprocessing: including denoising, segmentation, and feature extraction; Emotion classification: Use the model network to classify the emotions of speech features, including happiness, sadness, surprise, and anger; The emotional feedback comprises the following steps: (2.1) Emotion-driven interactive interface Touch display unit feedback: According to the emotion recognition results, the touch display unit displays the corresponding expression or animation; Voice feedback: Play voice or music that matches the recognized emotion through an audio power amplifier and speaker; (2.2) Mechanical motion feedback Servo control: According to the emotional state, control the horizontal and vertical servos to adjust the angle of the electronic pet to express the corresponding emotions, including tilting the head to express curiosity or attention, looking up to express happiness, and looking down to express sadness; The data fusion, adaptive learning and user customization include the following steps: (3.1) Multimodal data fusion: Combining facial expression and voice emotion recognition results to improve the accuracy of emotion recognition through the model; (3.2) Adaptive learning: Analyze the interaction data between users and electronic pets through models, and continuously optimize the emotion recognition model and feedback strategy to provide a more personalized interaction experience; (3.3) User customization: Allows users to customize the sensitivity and feedback method of emotion recognition through the touch display unit; users can train personalized emotion recognition models through interaction with electronic pets.

Images