A family companion spherical unmanned aerial vehicle family accompanying method and system

Abstract

The application discloses a family companion ball-shaped unmanned aerial vehicle family accompanying method and system, comprising adopting a laser radar sensor, scanning and point cloud registration of the surrounding environment based on a Cartographer laser SLAM algorithm, identifying a safe flight area and obstacle distribution, establishing family environment data, and continuously monitoring obstacle distance to calculate a collision risk value. The application combines a spherical shell protection design, uses multi-modal emotion fusion for emotion recognition and converts into emotion expression instructions, reduces safety risks, enables users to obtain safer interaction, has strong spatial expression, more intelligent emotional companion experience, improves interaction effect, enhances the immersion and friendliness of accompanying service, has the advantages of accurate emotion recognition, rich interaction expression, perfect safety protection and the like, and is suitable for a family emotional accompanying scene.

Description

Technical Field

[0001] This invention relates to the field of drone emotional interaction technology, specifically to a spherical drone-based family companionship method and system. Background Technology

[0002] With the rapid development of artificial intelligence and drone technology, smart home devices are gradually evolving towards emotional and personalized features. Traditional home companionship devices typically rely on fixed interactive interfaces, lacking spatial mobility and emotional expression capabilities, making it difficult to meet users' growing needs for emotional companionship. Existing home service robots are mainly divided into two categories: ground-based mobile robots and static interactive devices.

[0003] While ground-based mobile robots possess a certain degree of mobility, their range of movement is limited by ground obstacles and spatial layout. Furthermore, they are prone to collisions in complex home environments, making safety difficult to guarantee. Static interactive devices, while relatively mature in terms of voice recognition and content services, lack initiative and spatial expressiveness, and are unable to achieve three-dimensional emotional interaction.

[0004] Existing technologies also have the following problems: they lack effective emotion recognition and expression mechanisms, making it impossible to accurately perceive users' emotional states and provide corresponding feedback; they lack personalized learning capabilities, making it impossible to adjust interaction modes according to user preferences; and they have insufficient security protection capabilities, which can easily cause personal injury or equipment damage in the home environment. Summary of the Invention

[0005] The purpose of this invention is to provide a home companionship method using a spherical drone, in order to solve the technical problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] Step 1: Using a LiDAR sensor, the surrounding environment is scanned and point cloud registered based on the Cartographer LiDAR SLAM algorithm to identify safe flight areas and obstacle distribution, establish home environment data, and continuously monitor obstacle distances to calculate collision risk values. The home environment data includes a 3D grid map of the home environment and the distribution of safe flight areas and obstacles in the 3D grid map.

[0008] Step 2: Collect user facial images through the emotion perception unit. The embodied intelligent processor identifies the current interactive object based on the face recognition algorithm of the FaceNet deep learning model and obtains the personal emotional profile of the interactive object. The emotion perception unit collects facial images, voice signals, physiological data and current actual environmental parameters of the interactive object in real time to form multimodal emotional data.

[0009] Step 3: The embodied intelligent processor first preprocesses the multimodal emotion data, and then extracts features from the preprocessed data;

[0010] Step 4: Employ an attention-based weighted fusion strategy to weight and fuse the extracted features, and perform temporal smoothing on the fusion result. Input the smoothed fusion result into a Softmax layer, outputting the probability distribution of sentiment. The sentiment with the highest probability is taken as the current sentiment state. Quantize the personal sentiment profile into a user preference vector. The emotion recognition result is obtained by fine-tuning the current emotional state.

[0011] Step 5: Based on the emotion recognition results, use them together with family environment data, actual environment parameters and user preference vector as the current state vector. Input the current state vector into the personalized interaction strategy network of deep reinforcement learning to generate a comprehensive behavior plan, and record it in the personal emotion profile as a personalized interaction parameter. The comprehensive behavior plan includes LED display HSV parameter value, voice response emotion style ID and flight trajectory number parameter.

[0012] Step 6: Based on the real-time response of the interactive object during the interaction process, construct the feedback vector F. Based on the feedback vector, use the incremental learning algorithm to update the user preference vector in real time. At the same time, use the feedback vector as the optimization parameter to optimize the interaction strategy network through the Q-learning algorithm based on reinforcement learning.

[0013] Step 7: The embodied intelligent processor converts the flight trajectory number parameter into flight control commands and executes them. It converts the voice response emotion style ID into voice with emotional rhythm and plays it through the speaker. Based on the HSV parameter value of the LED display, it adjusts the PWM duty cycle and frequency of the LED driver circuit to display the corresponding emotion color.

[0014] Furthermore, the calculation of the collision risk value in step 1 specifically involves: continuously fusing LiDAR sensor data to assess the collision risk in real time. The formula for calculating the collision risk value is:

[0015] ,

[0016] In the above formula, This is the actual measured distance to the obstacle; As a safety threshold, this embodiment sets it to 0.5 meters; It is the relative velocity;

[0017] when If the current action is interrupted, an emergency obstacle avoidance mechanism is initiated to ensure safety. An obstacle avoidance trajectory is generated through a fast path planning algorithm, and the optimal deceleration is calculated based on the current speed and available braking distance using an emergency braking algorithm.

[0018] Furthermore, the specific steps of step 3 are as follows:

[0019] Step 31: For facial images, Gaussian filtering algorithm is used for noise reduction; for speech signals, DC component is removed first, then pre-emphasis filter is applied, and frame segmentation is performed using sliding window method. Then, Z-score normalization method is used to normalize all data.

[0020] Step 32: Use a deep learning network model to extract the deep features of each modality of sentiment data after normalization, specifically as follows:

[0021] Facial expression feature extraction: First, 68 facial key points are detected, and a facial action coding system is introduced to calculate the intensity of facial action units. Then, the image data containing the intensity of facial key points and facial action units is input into a convolutional neural network to analyze facial expressions and extract a 136-dimensional facial feature vector containing AU intensity and texture information. The convolutional neural network contains 4 convolutional layers, 2 pooling layers and 3 fully connected layers, and uses the ReLU activation function.

[0022] Speech emotion feature extraction: The Mel frequency cepstral coefficient algorithm is used to extract 13-dimensional static coefficients and their first and second order difference coefficients, totaling 39-dimensional feature vectors. The extracted sequence features are then input into a bidirectional long short-term memory network for temporal modeling to capture long-term emotional dependence in speech intonation and obtain speech emotion features.

[0023] Physiological and emotional feature extraction: The facial temperature distribution is analyzed using thermal imaging, and the temperature difference between the forehead and nose is calculated to assess physiological activation. The temperature difference and facial temperature distribution features are input into a support vector machine classifier to generate physiological and emotional feature vectors.

[0024] Furthermore, the specific steps of step 4 are as follows:

[0025] Step S41: Employ an attention-based weighted fusion strategy to dynamically fuse the extracted facial, speech, and physiological features. First, calculate the attention weights for each modality using the Softmax function.

[0026] ,

[0027] In the above formula, express Attention weights assigned to each modality; This refers to the learnable weight matrix obtained through joint training via multi-task learning; for The eigenvectors of a mode; This represents the eigenvector of mode j used for normalization calculation. For bias terms;

[0028] Step S42, Final fusion result It is represented as a weighted sum of the features of each modality, specifically as follows:

[0029] ,

[0030] In the above formula, These represent feature vectors for facial, voice, and physiological modalities, respectively. , , Attention weights are assigned to facial, speech, and physiological modalities, respectively, and the weights satisfy the following conditions: ;

[0031] Step S43: Introduce a temporal smoothing mechanism to output the fusion result instantaneously. Perform exponential moving average smoothing to obtain the smoothed fusion result. The data is fed into a Softmax layer, which outputs a probability distribution of several emotions. The emotion with the highest probability is taken as the current emotional state. Smoothed fusion result The calculation formula is:

[0032] ,

[0033] In the above formula, For smoothing coefficients; This represents the smoothed fusion result at time t. This represents the fusion result at the current time t. This represents the fusion result after smoothing at the previous time t−1;

[0034] Step S44: Quantify personal emotional profiles into user preference vectors Record the historical preference weights of the interacting objects and combine them with their current emotional state. Fine-tuning was performed to obtain the emotion recognition results. The calculation formula is:

[0035] ,

[0036] In the above formula, For personalized weighting coefficients.

[0037] Furthermore, the calculation of the LED display HSV parameter value in the comprehensive behavior scheme in step 5 specifically involves: defining the mapping relationship between emotion and RGB reference color value, and converting the mapped RGB reference color value into the HSV color space using a general algorithm;

[0038] The specific steps are as follows: First, normalize the RGB values ​​to... Interval and calculate extreme values , and difference Then, the components are calculated using the following logic: Brightness V is directly taken. ; Saturation S in Time calculation is Otherwise, it is 0; Hue The calculation is based on The corresponding color channels R, G, and B are represented by a piecewise function formula:

[0039] when As the main force, ;

[0040] when As the main force, ;

[0041] when As the main force, .

[0042] Furthermore, the construction of the feedback vector F in step 6 is specifically as follows: obtain the emotion recognition results during the interaction process, compare the changes in the emotion recognition results before and after the interaction, use them as the emotion feedback index, obtain the time when the interaction object's gaze stays on the device as the interaction duration, match the speech during the interaction process by calling the built-in emotion polarity dictionary, score 1 point for a positive emotion word, -1 point for a negative emotion word, and 0 points for no match, and calculate the total score, using the emotion feedback index, interaction duration, and total score as the feedback vector.

[0043] Furthermore, the emotion perception unit includes a color camera, an infrared thermal imaging camera, a ring array of several microphones, a temperature and humidity sensor, and a light intensity sensor.

[0044] Furthermore, in step 7, the flight control command conversion and execution specifically involves: defining several basic flight trajectories and corresponding to different emotional states; converting the corresponding flight trajectory into flight control commands based on the flight trajectory parameters; and executing the flight trajectory based on the flight control commands, which include speed, amplitude, and frequency.

[0045] The speech conversion and playback process is as follows: an end-to-end emotional speech synthesis network based on variational autoencoder is adopted. This network has a built-in prosody prediction submodule. The speech text sequence and the speech response emotional style ID are used as conditional inputs. The three key acoustic parameters of the audio are predicted by a multilayer perceptron to form speech with emotional prosody. Finally, the speech with emotional prosody is played through a speaker. The key acoustic parameters include fundamental frequency profile, energy envelope and duration.

[0046] The present invention also provides a spherical drone companion system for family use, including a spherical protective shell, an emotion sensing unit integrated within the shell, an embodied intelligent processor, a wireless communication module, and a drone flight module. The wireless communication module includes Wi-Fi 6, Bluetooth 5.0, and 4G / 5G mobile network connectivity for data transmission and remote connection.

[0047] The spherical protective shell features a double-layer composite material design. The inner layer is a multi-layered airbag made of flexible polyurethane, while the outer layer is made of rigid polycarbonate. Several RGB LED chips are evenly distributed on the surface of the spherical protective shell to form an LED matrix. An electromagnetically controlled opening and closing maintenance port is located at the bottom of the spherical protective shell.

[0048] The emotion perception unit includes a color camera, an infrared thermal imaging camera, a ring array of microphones, a temperature and humidity sensor, and a light intensity sensor. It acquires facial images through the high-resolution color camera, monitors facial temperature distribution through the infrared thermal imaging camera to obtain physiological data, acquires voice signals through the ring microphone array and uses its ring geometry to locate the sound source, guiding the drone to automatically turn towards the user. It also collects actual environmental parameters through the temperature and humidity sensor and the light intensity sensor to analyze the impact of environmental factors on emotions and form multimodal emotion data.

[0049] The embodied intelligent processor has a built-in deep neural network acceleration chip; the embodied intelligent processor establishes communication connections with the emotion perception unit and the drone flight module, and establishes a connection with the LED matrix through a bus; it includes a data processing module, a personalized interaction and update module, and an emotion expression module;

[0050] The data processing module is used to preprocess multimodal emotion data, extract features from the preprocessed data, fuse the extracted features, perform temporal smoothing on the fusion results, and generate emotion recognition results; it is also used to identify the current interaction object, obtain the interaction object's personal emotion profile and quantify it into a user preference vector; it is also used to convert flight trajectory number parameters into flight control commands and send them to the UAV flight module for execution, and convert the voice response emotion style ID into speech with emotional rhythm;

[0051] The personalized interaction and update module is used to generate a comprehensive behavior plan based on emotion recognition results, family environment data, actual environment parameters and user preference vectors. It constructs a feedback vector based on the real-time response of the interactive object during the interaction process, updates the user preference vector in real time based on the feedback vector, and optimizes the interaction strategy network using the feedback vector. The comprehensive behavior plan includes LED display HSV parameter values, voice response emotion style ID and flight trajectory number parameters.

[0052] The emotion expression module drives the LED matrix on the surface of the spherical shell to display the corresponding emotion color based on the HSV parameter value of the LED display, and plays voice with emotional rhythm through the speaker;

[0053] The drone flight module includes four brushless DC motor-driven propellers, an integrated IMU (Inertial Measurement Unit) consisting of a three-axis gyroscope, a three-axis accelerometer, and a barometer, a lidar sensor, and a modular lithium polymer battery pack. Using the lidar sensor and Cartographer's lidar SLAM algorithm, it identifies safe flight areas and obstacle distributions, establishes home environment data, receives flight control commands, and coordinates the four brushless DC motors and the IMU to execute the flight trajectory. During flight, it continuously monitors obstacle distances and calculates collision risks for safety monitoring.

[0054] Furthermore, the multi-layer airbag includes, from the inside out, a soft TPU airbag layer, a medium-hardness EVA foam layer, and an outer shock-absorbing adhesive layer.

[0055] Beneficial effects: This invention combines a spherical shell protection design with multimodal emotion fusion for emotion recognition and conversion into emotion expression commands, reducing security risks and enabling users to have a safer interaction. It has strong spatial expressiveness, a more intelligent emotional companionship experience, improves interaction effects, and enhances the immersion and friendliness of companionship services. It has the advantages of accurate emotion recognition, rich interactive expression, and comprehensive security protection, and is suitable for family emotional companionship scenarios. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Fig. 1 This is a flowchart of the family companionship spherical drone companionship method of the present invention;

[0058] Fig. 2 This is a schematic diagram of steps 2 to 4 of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] like Figs. 1-2 The present invention provides a method for family companionship using a spherical drone, which specifically includes the following steps:

[0061] Step 1: Using a LiDAR sensor, the surrounding environment is scanned and point cloud registered based on the Cartographer LiDAR SLAM algorithm to identify the safe flight area and obstacle distribution, establish home environment data, and continuously monitor obstacle distances. If the calculated collision risk value exceeds 0.7, the current action is immediately interrupted and emergency obstacle avoidance is initiated to ensure safety. The home environment data includes a 3D grid map of the home environment, the safe flight area and obstacle distribution in the 3D grid map.

[0062] Specifically, it involves continuously fusing LiDAR sensor data to assess collision risk in real time. The formula for calculating the collision risk value is as follows:

[0063] ,

[0064] In the above formula, This is the actual measured distance to the obstacle; As a safety threshold, this embodiment sets it to 0.5 meters; It is the relative velocity;

[0065] when When an obstacle avoidance event is triggered, the obstacle avoidance trajectory is generated using a fast path planning algorithm, ensuring safety is prioritized while maintaining interactive effects. The emergency braking algorithm calculates the optimal deceleration based on the current speed and available braking distance, with a maximum deceleration limit of 3 m / s², providing a smooth user experience while ensuring device safety.

[0066] Step 2: Collect user facial images through the emotion perception unit. The embodied intelligent processor uses the FaceNet deep learning model-based face recognition algorithm to extract 128-dimensional facial feature vectors and compares them with the local database to identify the current interaction object and obtain the interaction object's personal emotion profile to prepare for subsequent personalized interaction. In this step, the user's personal emotion profile is loaded from the storage space. If it is a new user, the personal emotion profile will be empty.

[0067] Subsequently, the system continuously collects the user's facial images, voice signals, physiological and emotional data, and current environmental parameters in real time through the emotion perception unit. The current environmental parameters are used as auxiliary analysis data to form multimodal emotion data.

[0068] Before officially commencing the companionship service, it is necessary to establish a personal emotional profile for the individuals to be interacted with. Through a 3-5 day observation and learning process, the emotional expression patterns of users in different time periods and situations are recorded. This includes individualized characteristics such as the user's facial expression habits, vocal emotional characteristics, and physiological response patterns.

[0069] During the establishment of the emotional baseline, the system adopts a non-intrusive observation mode, collecting data without active interaction to ensure the acquisition of the user's most natural emotional expression data. Through statistical analysis, the system calculates the average and variability of various emotional states of the user, providing an accurate reference standard for subsequent personalized services.

[0070] Step 3: The embodied intelligent processor receives multimodal emotion data, first preprocesses it, and then extracts features from the preprocessed data;

[0071] This embodiment further explains the specific implementation of step 3, which includes the following steps:

[0072] Step 31: To eliminate environmental interference and unify data volume, Gaussian filtering is used for denoising facial images, with a filter kernel standard deviation of 1.2 to effectively eliminate interference from sensor noise and uneven lighting. For speech signals, the DC component is first removed, followed by a pre-emphasis filter to improve high-frequency resolution, and a sliding window method is used for frame segmentation (frame length 25ms, frame shift 10ms). Subsequently, to ensure comparability of data from different modalities during subsequent fusion, all data... The Z-score normalization method is used for normalization, and its calculation formula is as follows: In the formula, This represents the normalized data value. This is the historical average of this type of data. The standard deviation is used to ensure that data from different modalities are within the same order of magnitude, laying the foundation for subsequent fusion algorithms;

[0073] Step 32: Use a deep learning network model to extract the deep features of each modality of sentiment data after normalization, specifically as follows:

[0074] Facial expression feature extraction: A cascaded detection architecture is adopted. First, the face region is quickly located by a Haar feature detector, and 68 facial key points are detected. Then, a facial action coding system (FACS) is introduced to calculate the intensity of facial action units to further quantify the subtle changes of local facial muscles. Subsequently, the image data containing the intensity of facial key points and facial action units is input into a convolutional neural network to analyze facial expressions and extract a 136-dimensional facial feature vector containing AU intensity and texture information. The convolutional neural network contains 4 convolutional layers, 2 pooling layers and 3 fully connected layers, using the ReLU activation function. The network parameters are trained using the cross-entropy loss function and the Adam optimizer.

[0075] FACS breaks down facial muscle movements into several independent action units (AUs), for example This represents "inner brow lift" (corresponding to frontalis muscle movement). Represents "corner of the mouth stretch" (corresponding to the movement of the zygomaticus major muscle), the first Intensity of facial motion units (That is, the degree of contraction of this muscle group) is calculated by weighted average of key point displacements, using the following formula:

[0076] ,

[0077] In the formula, Let be the pre-trained weight parameters, representing the th . The key point for the first The influence weight of each action unit; For the first The geometric displacement of each key point relative to a neutral expression.

[0078] Speech emotion feature extraction: The Mel-frequency cepstral coefficients (MFCC) algorithm is used to extract 13-dimensional static coefficients and their first- and second-order difference coefficients, resulting in a total of 39-dimensional feature vectors. The core calculation formula for MFCC is as follows:

[0079] ,

[0080] In the formula, For the first One cepstral coefficient; The total number of filters in the Mel filter bank; For the first The logarithmic energy value of the filter output; This is the cepstral coefficient index. The extracted sequence features are then input into a bidirectional long short-term memory (Bi-LSTM) network for temporal modeling. This network has 128 hidden layer dimensions, which can effectively capture long-term emotional dependence in speech intonation and obtain speech emotional features.

[0081] Physiological and emotional feature extraction: Thermographic imaging is used to analyze facial temperature distribution and calculate the temperature difference between the forehead and nose to assess physiological arousal. The temperature difference and facial temperature distribution features are input into a support vector machine (SVM) classifier to generate physiological and emotional feature vectors, which are used to help determine the user's physiological arousal level.

[0082] Specifically, since infrared images lack texture and are difficult to locate directly, facial key points are first detected on synchronously acquired RGB color images. Using a pre-calibrated binocular camera extrinsic matrix and perspective transformation algorithm, the coordinates of key points on the RGB images are accurately mapped to the coordinate system of the infrared thermal imaging image, thereby accurately segmenting the forehead and nose regions on the thermal image. Then, the average temperature of these two regions is calculated respectively. and Finally, the temperature difference between the two is calculated. It defines an emotion-temperature reference scale to assess the level of physiological stress or agitation (nasal temperature typically decreases under stress, leading to a greater temperature difference) and can identify subtle physiological and emotional changes. .

[0083] Step 4: Employ an attention-based weighted fusion strategy to weight and fuse the extracted features, and perform temporal smoothing on the fusion result. Input the smoothed fusion result into a Softmax layer, outputting the probability distribution of sentiment. The sentiment with the highest probability is taken as the current sentiment state. Quantize the personal sentiment profile into a user preference vector. The present invention uses a multimodal deep learning-based architecture to achieve high-precision emotion recognition by combining the current emotional state with fine-tuning.

[0084] This embodiment further explains the specific implementation of step 4, which includes the following steps:

[0085] Step S41: To comprehensively utilize information from different modalities and highlight key features, an attention-based weighted fusion strategy is adopted to dynamically fuse the extracted facial, speech, and physiological features. This strategy can adaptively adjust the importance of each modality according to the current context. First, the attention weights of each modality are calculated using the Softmax function:

[0086] ,

[0087] In the above formula, express Attention weights assigned to modalities (such as facial, voice, or physiological). This refers to the learnable weight matrix obtained through joint training via multi-task learning; for The eigenvectors of a mode; This represents the eigenvector of mode j used for normalization calculation. This is a bias term.

[0088] Step S42, Final fusion result This is represented as a weighted sum of the features of each modality, which can comprehensively reflect the user's overall emotional state. Specifically, it is expressed as:

[0089] ,

[0090] In the formula, These represent feature vectors for facial features, voice features, and physiological features, respectively. , , Attention weights are assigned to facial, speech, and physiological modalities, respectively, and the weights satisfy the following conditions: .

[0091] Step S43: Since the single-frame recognition result may have jitter, in order to reduce the volatility of the emotion recognition result, a temporal smoothing mechanism is introduced to output the fusion result instantaneously. Perform exponential moving average smoothing to obtain the smoothed fusion result. The data is fed into a Softmax layer, which outputs a probability distribution of several emotions. The emotion with the highest probability is taken as the current emotional state, and the result is a smoothed fusion. The calculation formula is:

[0092] ,

[0093] in, The smoothing factor is set to 0.7 in this embodiment to balance the requirements of real-time performance and stability. This represents the smoothed fusion result at time t. This represents the fusion result at the current time t. This represents the fusion result after smoothing at the previous time t−1. Emotions include happiness, excitement, sadness, anger, etc., and can be customized according to the actual situation.

[0094] Step S44: Quantify personal emotional profiles into user preference vectors Record the historical preference weights of the interacting objects and combine them with their current emotional state. Fine-tuning was performed to obtain the emotion recognition results. The calculation formula is:

[0095] ,

[0096] In the formula, The personalized weight coefficient (set to 0.6) is dynamically adjusted through this step, which can generate interactive commands that best match the user's habits based on emotional state.

[0097] Based on the emotion recognition results, the user's emotional state is determined, and the interaction decision-making and execution phase begins. This phase generates the optimal strategy using deep reinforcement learning algorithms and provides embodied emotional feedback through visual, auditory, and spatial motion sensing.

[0098] Step 5: Based on the emotion recognition results, the current state vector is obtained by combining it with family environment data, actual environmental parameters, and user preference vectors. This current state vector is then input into a deep reinforcement learning-based personalized interaction strategy network to generate a comprehensive behavioral plan. This plan is recorded in the individual's emotional profile as a personalized interaction parameter. The comprehensive behavioral plan includes LED display HSV parameter values, voice response emotional style ID, and flight trajectory number parameters. Family environment data and actual environmental parameters are fused into environmental context components, such as "bedroom area + low light + low noise." The deep reinforcement learning-based personalized interaction strategy network uses a DQN architecture to calculate the value function of each action and output a comprehensive behavioral plan. For example, when faced with a "sad" and "night" state input, the strategy network will output a combined strategy of "blue light mode + low, comforting tone + hovering companionship."

[0099] Specifically, the calculation of HSV parameter values ​​for LED displays involves defining the mapping relationship between emotion and RGB base color values. In order to achieve more delicate control over brightness and saturation, the mapped RGB base color values ​​are converted into the HSV color space using a general algorithm.

[0100] The specific steps are as follows: First, normalize the RGB values ​​to... Interval and calculate extreme values , and difference Then, the components are calculated using the following logic: Brightness V is directly taken. ; Saturation S in Time calculation is Otherwise, it is 0; Hue The calculation is based on The corresponding color channel (R, G, or B) uses a piecewise function formula:

[0101] when As the main force, ;

[0102] when As the main force, ;

[0103] when As the main force, ;

[0104] For example, "happiness" corresponds to a warm yellow color scheme (HSV: 45°, 0.8, 0.9), "sadness" corresponds to a calm blue color scheme (HSV: 240°, 0.6, 0.5), and "anger" corresponds to a red color scheme (HSV: 0°, 0.9, 0.8). The intensity of emotion is represented by color saturation, and the flashing frequency of 1-10Hz represents the level of emotional activity. The color mapping is user-defined.

[0105] Step 6: After the interaction ends, based on the immediate reactions of the interactive object during the interaction, a feedback vector F is constructed. Based on this feedback vector, an incremental learning algorithm is used to update the user preference vector in real time. Simultaneously, the feedback vector is used as an optimization parameter to optimize the interaction policy network using a reinforcement learning-based Q-learning algorithm. This gradually aligns the interaction policy network with the user's personalized emotional needs. The update formula is:

[0106] ,

[0107] In the above formula, For the updated preference vector, The current preference vector controls the speed and stability of model updates. =0.1 is the learning rate to control the update speed. The two-layer online learning and updating is used to achieve continuous evolution and personalized adaptation.

[0108] The construction of the feedback vector F is as follows: The emotion recognition results during the interaction process are obtained; the changes in emotion recognition results before and after the interaction are compared and used as the emotion feedback index; the time the interaction object's gaze lingers on the device is obtained as the interaction duration; the built-in emotion polarity dictionary is used to match the speech during the interaction process, assigning 1 point for a positive emotion word, -1 point for a negative emotion word, and 0 points for no match; and the total score is calculated. The emotion feedback index, interaction duration, and total score are used as the feedback vector. For example, a positive emotion word (such as "like") is recorded as +1, a negative emotion word (such as "dislike") as -1, and no match as 0.

[0109] Step 7: The embodied intelligent processor converts the flight trajectory number parameter into flight control commands and executes them. It converts the voice response emotion style ID into voice with emotional rhythm and plays it through the speaker. It adjusts the PWM duty cycle and frequency of the LED driver circuit according to the HSV parameter value of the LED display and displays the corresponding emotion color to reproduce the static color and dynamic light effect required by the command.

[0110] The specific steps of flight control command conversion and execution are as follows:

[0111] Several basic flight trajectories are defined and correspond to different emotional states. The corresponding flight trajectory is generated based on the flight trajectory parameters and converted into flight control commands, which include speed, amplitude, and frequency. Based on the flight control commands, four brushless DC motors and an IMU inertial measurement unit are coordinated to execute the flight trajectory. Emotional feedback is executed synchronously from three dimensions: visual, auditory, and spatial motion.

[0112] The basic flight trajectory is defined as follows: tilt angle ±30 degrees represents emotional intensity, rotation speed 10-60 rpm represents emotional activity, hovering height 0.5-2m represents intimacy level. The emotional state is expressed through changes in flight attitude angle, and the emotional type is expressed through changes in flight trajectory. Personalized adjustment is achieved through three parameters: speed, amplitude, and frequency.

[0113] For example, "Happy" uses an ascending spiral flight, "Sad" uses a slow descending flight, and "Excited" uses a rapid swinging flight. Each mode can be personalized by adjusting three parameters: speed, amplitude, and frequency, and supports users to customize the trajectory of emotional expression.

[0114] Specifically: by superimposing the sine and cosine velocity components in the horizontal direction. With vertical velocity To synthesize a spiral trajectory, where the amplitude angular frequency and vertical velocity All are dynamically assigned values ​​based on the emotional intensity parameter, thereby achieving diverse expressions ranging from gentle swaying to violent circling;

[0115] For example: when the flight trajectory parameters correspond to the "happy" state, an ascending spiral flight trajectory is executed (amplitude 0.5m, frequency 0.8Hz); when they correspond to the "calm" state, a gentle oscillation is executed (amplitude 0.1m, frequency 0.3Hz).

[0116] The voice conversion and playback are as follows:

[0117] An end-to-end emotional speech synthesis network based on variational autoencoder (VAE) is adopted. This network has a built-in prosody prediction submodule. The speech text sequence and the emotional style ID of the speech response are used as conditional inputs. The three key acoustic parameters of the audio are predicted by multilayer perceptron to form speech with emotional prosody. Finally, the speech with emotional prosody is played through a speaker. The key acoustic parameters include fundamental frequency profile, energy envelope and duration.

[0118] The present invention also provides a spherical drone companion system for family use, including a spherical protective shell, an emotion sensing unit integrated within the shell, an embodied intelligent processor, a wireless communication module, and a drone flight module. The wireless communication module includes Wi-Fi 6, Bluetooth 5.0, and 4G / 5G mobile network connectivity for data transmission and remote connection.

[0119] The spherical protective shell is designed with a double-layer composite material. The inner layer is a multi-layer airbag made of flexible polyurethane material for cushioning, and the outer layer is made of rigid polycarbonate material for strength protection. The sphere diameter is 200-400mm, the wall thickness is 2-5mm, and it is made by injection molding. The surface roughness Ra≤0.8μm and the impact strength is ≥15J / cm².

[0120] Specifically, the multi-layer airbag includes a soft TPU airbag layer, a medium-hardness EVA foam layer and an outer shock-absorbing rubber layer arranged sequentially from the inside to the outside, with a total thickness of 15-25mm, which can withstand the impact of a drop from a height of 2m without damaging the internal components.

[0121] The spherical protective shell has 768 RGB LED chips evenly distributed on its surface, arranged in a geodesic pattern to form an emotional expression matrix. Each light-emitting unit has a power of 0.2W, supports 256 levels of brightness adjustment and 16.7 million colors, and has a refresh rate of ≥60Hz.

[0122] The bottom of the spherical protective shell is equipped with an electromagnetically controlled opening and closing maintenance port. It adopts a locking device driven by a 12V DC electromagnet, has a dual unlocking mechanism to prevent misoperation, a sealed dustproof design, and an opening and closing response time of ≤0.5s.

[0123] The spherical protective shell protects the internal flight components, providing safety and enabling rich emotional expression through an LED matrix; the multi-layered buffer structure provides shock resistance, and the LED emotional expression matrix enables intuitive emotional display.

[0124] The emotion perception unit collects external data through multimodal sensors, including a color camera, an infrared thermal imaging camera, a ring array of several microphones, as well as a temperature and humidity sensor and a light intensity sensor.

[0125] The system uses a high-resolution color camera to capture facial images in real time, an infrared thermal imaging camera to monitor facial temperature distribution and obtain physiological and emotional data, a ring microphone array to collect voice signals and use its ring geometry to locate the sound source, guiding the drone to automatically turn towards the user. Temperature and humidity sensors and light intensity sensors collect actual environmental parameters to analyze the impact of environmental factors on emotions, forming multimodal emotional data. The emotion perception unit continuously monitors the user's state, providing comprehensive data support for emotion recognition. The multimodal emotional data includes facial expressions, physiological emotions, sound source localization, voice emotions, and actual environmental parameters.

[0126] The color camera captures facial video streams at a resolution of no less than 1080P at a frame rate of 30fps, while the infrared thermal imaging camera captures facial thermal distribution maps in the 8-14μm wavelength range. This band is most sensitive to human body temperature radiation and can capture minute temperature changes on the face.

[0127] A circular array of eight microphones records speech at a sampling frequency of 44.1 kHz and uses its circular geometry to locate the sound source, guiding the drone to automatically turn towards the user.

[0128] The specific method for sound source localization is as follows: calculate the time difference of arrival (TDOA) of the sound signal at different microphones, combine it with the geometric radius parameters of the circular array, and calculate the azimuth angle of the sound source relative to the UAV coordinate system using a generalized cross-correlation algorithm. and pitch angle The positioning accuracy can reach ±5 degrees.

[0129] The temperature sensor has a detection range of -10℃ to +60℃ and an accuracy of ±0.5℃. The humidity sensor has a detection range of 0-100%RH and an accuracy of ±3%RH. The light sensor has a detection range of 0.01-100000 lux. All of the above sensors use existing technology and only require that the necessary parameters be met. They will not be described in detail here.

[0130] The embody intelligent processor adopts the ARM Cortex-A78 architecture or an equivalent processor, with no less than 8GB of RAM and 128GB of storage space. It has a built-in deep neural network acceleration chip. The high-performance processor ensures real-time performance, and the dedicated neural network acceleration chip improves the efficiency of AI algorithms. The embody intelligent processor establishes communication connections with the emotion perception unit and the drone flight module. It includes a data processing module, a personalized interaction and update module, and an emotion expression module.

[0131] The data processing module is used to preprocess multimodal sentiment data, extract features from the preprocessed data, fuse the extracted features, and perform temporal smoothing to generate sentiment recognition results.

[0132] The data processing module is used to preprocess multimodal emotion data, extract features from the preprocessed data, fuse the extracted features, perform temporal smoothing on the fusion results, and generate emotion recognition results; it is also used to identify the current interaction object, obtain the interaction object's personal emotion profile and quantify it into a user preference vector; it is also used to convert flight trajectory number parameters into flight control commands and send them to the UAV flight module for execution, and convert the voice response emotion style ID into speech with emotional rhythm;

[0133] The personalized interaction and update module is used to generate a comprehensive behavior plan based on emotion recognition results, family environment data, actual environment parameters and user preference vectors. It constructs a feedback vector based on the real-time response of the interactive object during the interaction process, updates the user preference vector in real time based on the feedback vector, and optimizes the interaction strategy network using the feedback vector. The comprehensive behavior plan includes LED display HSV parameter values, voice response emotion style ID and flight trajectory number parameters.

[0134] The emotion expression module drives the LED matrix on the surface of the spherical shell to display the corresponding emotion color based on the HSV parameter value of the LED display, and plays voice with emotional rhythm through the speaker;

[0135] The drone flight module includes four brushless DC motor-driven propellers, an integrated IMU (Inertial Measurement Unit) consisting of a three-axis gyroscope, a three-axis accelerometer, and a barometer, a lidar sensor for environmental perception, and a modular lithium polymer battery pack with a capacity of 2000-5000mAh. Using the lidar sensor and the Cartographer-based laser SLAM algorithm, it identifies safe flight areas and obstacle distributions, establishes environmental data, receives flight control commands, and coordinates the four brushless DC motors and the IMU to execute the flight trajectory. During flight, it continuously monitors obstacle distances, calculates collision risks, and performs safety monitoring. The IMU enables attitude perception and multi-sensor fusion, sensing the drone's current flight status to ensure flight safety. The modular battery pack supports extended operation.

[0136] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0137] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for family companionship using a spherical drone, characterized in that, Includes the following steps: Step 1: Using a LiDAR sensor, the surrounding environment is scanned and point cloud registered based on the Cartographer LiDAR SLAM algorithm to identify safe flight areas and obstacle distribution, establish home environment data, and continuously monitor obstacle distances to calculate collision risk values. The home environment data includes a 3D grid map of the home environment and the distribution of safe flight areas and obstacles in the 3D grid map. Step 2: Collect user facial images through the emotion perception unit. The embodied intelligent processor identifies the current interactive object based on the face recognition algorithm of the FaceNet deep learning model and obtains the personal emotional profile of the interactive object. The emotion perception unit collects facial images, voice signals, physiological data and current actual environmental parameters of the interactive object in real time to form multimodal emotional data. Step 3: The embodied intelligent processor first preprocesses the multimodal emotion data, and then extracts features from the preprocessed data; Step 4: Employ an attention-based weighted fusion strategy to weight and fuse the extracted features, and perform temporal smoothing on the fusion result. Input the smoothed fusion result into a Softmax layer, outputting the probability distribution of sentiment. The sentiment with the highest probability is taken as the current sentiment state. Quantize the personal sentiment profile into a user preference vector. The emotion recognition result is obtained by fine-tuning the current emotional state. Step 5: Based on the emotion recognition results, use them together with family environment data, actual environment parameters and user preference vector as the current state vector. Input the current state vector into the personalized interaction strategy network of deep reinforcement learning to generate a comprehensive behavior plan, and record it in the personal emotion profile as a personalized interaction parameter. The comprehensive behavior plan includes LED display HSV parameter value, voice response emotion style ID and flight trajectory number parameter. Step 6: Based on the real-time response of the interactive object during the interaction process, construct the feedback vector F. Based on the feedback vector, use the incremental learning algorithm to update the user preference vector in real time. At the same time, use the feedback vector as the optimization parameter to optimize the interaction strategy network through the Q-learning algorithm based on reinforcement learning. Step 7: The embodied intelligent processor converts the flight trajectory number parameter into flight control commands and executes them. It converts the voice response emotion style ID into voice with emotional rhythm and plays it through the speaker. Based on the HSV parameter value of the LED display, it adjusts the PWM duty cycle and frequency of the LED driver circuit to display the corresponding emotion color.

2. The caregiving method according to claim 1, characterized in that, The calculation of the collision risk value in step 1 specifically involves: continuously fusing LiDAR sensor data to assess the collision risk in real time. The formula for calculating the collision risk value is as follows: In the above formula, This is the actual measured distance to the obstacle; This is a safety threshold; It is the relative velocity; Maximum flight speed; when If the current action is interrupted, an emergency obstacle avoidance mechanism is initiated to ensure safety. An obstacle avoidance trajectory is generated through a fast path planning algorithm, and the optimal deceleration is calculated based on the current speed and available braking distance using an emergency braking algorithm.

3. The caregiving method according to claim 1, characterized in that: The specific steps of step 3 are as follows: Step 31: For facial images, Gaussian filtering algorithm is used for noise reduction; for speech signals, DC component is removed first, then pre-emphasis filter is applied, and frame segmentation is performed using sliding window method. Then, Z-score normalization method is used to normalize all data. Step 32: Use a deep learning network model to extract the deep features of each modality of sentiment data after normalization, specifically as follows: Facial expression feature extraction: First, 68 facial key points are detected, and a facial action coding system is introduced to calculate the intensity of facial action units. Then, the image data containing the intensity of facial key points and facial action units is input into a convolutional neural network to analyze facial expressions and extract a 136-dimensional facial feature vector containing AU intensity and texture information. The convolutional neural network contains 4 convolutional layers, 2 pooling layers and 3 fully connected layers, and uses the ReLU activation function. Speech emotion feature extraction: The Mel frequency cepstral coefficient algorithm is used to extract 13-dimensional static coefficients and their first and second order difference coefficients, totaling 39-dimensional feature vectors. The extracted sequence features are then input into a bidirectional long short-term memory network for temporal modeling to capture long-term emotional dependence in speech intonation and obtain speech emotion features. Physiological and emotional feature extraction: The facial temperature distribution is analyzed using thermal imaging, and the temperature difference between the forehead and nose is calculated to assess physiological activation. The temperature difference and facial temperature distribution features are input into a support vector machine classifier to generate physiological and emotional feature vectors.

4. The caregiving method according to claim 1, characterized in that: The specific steps of step 4 are as follows: Step S41: Employ an attention-based weighted fusion strategy to dynamically fuse the extracted facial, speech, and physiological features. First, calculate the attention weights for each modality using the Softmax function. In the above formula, express Attention weights assigned to each modality; This refers to the learnable weight matrix obtained through joint training via multi-task learning; for The eigenvectors of a mode; This represents the eigenvector of mode j used for normalization calculation. For bias terms; Step S42, Final fusion result It is represented as a weighted sum of the features of each modality, specifically as follows: In the above formula, These represent feature vectors for facial, voice, and physiological modalities, respectively. , , Attention weights are assigned to facial, speech, and physiological modalities, respectively, and the weights satisfy the following conditions: ; Step S43: Introduce a temporal smoothing mechanism to output the fusion result instantaneously. Perform exponential moving average smoothing to obtain the smoothed fusion result. The data is fed into a Softmax layer, which outputs a probability distribution of several emotions. The emotion with the highest probability is taken as the current emotional state. Smoothed fusion result The calculation formula is: In the above formula, For smoothing coefficients; This represents the smoothed fusion result at time t. This represents the fusion result at the current time t. This represents the fusion result after smoothing at the previous time t−1; Step S44: Quantify personal emotional profiles into user preference vectors Record the historical preference weights of the interacting objects and combine them with their current emotional state. Fine-tuning was performed to obtain the emotion recognition results. The calculation formula is: In the above formula, For personalized weighting coefficients.

5. The caregiving method according to claim 1, characterized in that: The calculation of the HSV parameter value of the LED display in the comprehensive behavior scheme in step 5 is specifically as follows: define the mapping relationship between emotion and RGB reference color value, and convert the mapped RGB reference color value into HSV color space using a general algorithm; The specific steps are as follows: First, normalize the RGB values ​​to... Interval and calculate extreme values , and difference Then, the components are calculated using the following logic: Brightness V is directly taken. ; Saturation S in Time calculation is Otherwise, it is 0; Hue The calculation is based on The corresponding color channels R, G, and B are represented by a piecewise function formula: when As the main force, ; when As the main force, ; when As the main force, .

6. The caregiving method according to claim 1, characterized in that: The construction of the feedback vector F in step 6 is specifically as follows: obtain the emotion recognition results during the interaction process, compare the changes in the emotion recognition results before and after the interaction, use them as the emotion feedback index, obtain the time when the interaction object's gaze stays on the device as the interaction duration, match the speech during the interaction process by calling the built-in emotion polarity dictionary, score 1 point for a positive emotion word, -1 point for a negative emotion word, and 0 points for no match, and calculate the total score, using the emotion feedback index, interaction duration, and total score as the feedback vector.

7. The caregiving method according to claim 1, characterized in that: The emotion perception unit includes a color camera, an infrared thermal imaging camera, a ring array of several microphones, a temperature and humidity sensor, and a light intensity sensor.

8. The caregiving method according to claim 1, characterized in that: The flight control command conversion and execution in step 7 specifically involves: defining several basic flight trajectories and corresponding to different emotional states; converting the corresponding flight trajectory into flight control commands based on the flight trajectory parameters; and executing the flight trajectory based on the flight control commands, which include speed, amplitude, and frequency. The speech conversion and playback process is as follows: an end-to-end emotional speech synthesis network based on variational autoencoder is adopted. This network has a built-in prosody prediction submodule. The speech text sequence and the speech response emotional style ID are used as conditional inputs. The three key acoustic parameters of the audio are predicted by a multilayer perceptron to form speech with emotional prosody. Finally, the speech with emotional prosody is played through a speaker. The key acoustic parameters include fundamental frequency profile, energy envelope and duration.

9. A spherical drone-based family companionship system based on the companionship method according to any one of claims 1-8, characterized in that: It includes a spherical protective shell, an emotion-sensing unit integrated within the shell, an embodied intelligent processor, a wireless communication module, and a drone flight module. The wireless communication module includes Wi-Fi 6, Bluetooth 5.0, and 4G / 5G mobile network connectivity for data transmission and remote connection. The spherical protective shell features a double-layer composite material design. The inner layer is a multi-layered airbag made of flexible polyurethane, while the outer layer is made of rigid polycarbonate. Several RGB LED chips are evenly distributed on the surface of the spherical protective shell to form an LED matrix. An electromagnetically controlled opening and closing maintenance port is located at the bottom of the spherical protective shell. The emotion perception unit includes a color camera, an infrared thermal imaging camera, a ring array of microphones, a temperature and humidity sensor, and a light intensity sensor. It acquires facial images through the high-resolution color camera, monitors facial temperature distribution through the infrared thermal imaging camera to obtain physiological data, acquires voice signals through the ring microphone array and uses its ring geometry to locate the sound source, guiding the drone to automatically turn towards the user. It also collects actual environmental parameters through the temperature and humidity sensor and the light intensity sensor to analyze the impact of environmental factors on emotions and form multimodal emotion data. The embodied intelligent processor has a built-in deep neural network acceleration chip; the embodied intelligent processor establishes communication connections with the emotion perception unit and the drone flight module, and establishes a connection with the LED matrix through a bus; it includes a data processing module, a personalized interaction and update module, and an emotion expression module; The data processing module is used to preprocess multimodal emotion data, extract features from the preprocessed data, fuse the extracted features, perform temporal smoothing on the fusion results, and generate emotion recognition results; it is also used to identify the current interaction object, obtain the interaction object's personal emotion profile and quantify it into a user preference vector; it is also used to convert flight trajectory number parameters into flight control commands and send them to the UAV flight module for execution, and convert the voice response emotion style ID into speech with emotional rhythm; The personalized interaction and update module is used to generate a comprehensive behavior plan based on emotion recognition results, family environment data, actual environment parameters and user preference vectors. It constructs a feedback vector based on the real-time response of the interactive object during the interaction process, updates the user preference vector in real time based on the feedback vector, and optimizes the interaction strategy network using the feedback vector. The comprehensive behavior plan includes LED display HSV parameter values, voice response emotion style ID and flight trajectory number parameters. The emotion expression module drives the LED matrix on the surface of the spherical shell to display the corresponding emotion color based on the HSV parameter value of the LED display, and plays voice with emotional rhythm through the speaker; The drone flight module includes four brushless DC motor-driven propellers, an integrated IMU (Inertial Measurement Unit) consisting of a three-axis gyroscope, a three-axis accelerometer, and a barometer, a lidar sensor, and a modular lithium polymer battery pack. Using the lidar sensor and Cartographer's lidar SLAM algorithm, it identifies safe flight areas and obstacle distributions, establishes home environment data, receives flight control commands, and coordinates the four brushless DC motors and the IMU to execute the flight trajectory. During flight, it continuously monitors obstacle distances and calculates collision risks for safety monitoring.

10. The caregiving system according to claim 9, characterized in that: The multi-layer airbag includes, from the inside out, a soft TPU airbag layer, a medium-hardness EVA foam layer, and an outer shock-absorbing adhesive layer.

Images