Humanoid robot somatosensory control system and method based on deep learning

Abstract

The invention discloses a humanoid robot somatosensory control system and method based on deep learning. The method comprises the steps that physiological data and environment information of a user are collected through a wearable device and an environment sensor, and full-modal synchronous collection is achieved in combination with audio and visual data. The system carries out noise reduction processing on the collected sound, extracts a voice instruction and recognizes non-language sound features to judge the physiological state of the user; and analyzing the posture and gesture intention of the user through a human body detection and posture recognition algorithm. And multi-modal data is further fused, the intention of a user is accurately judged in combination with scene context and a security verification loop, a multi-dimensional state space and an atomic action library are constructed, and a composite reward function is designed to realize intelligent decision and execution of robot actions. The system comprises a multi-modal data acquisition module, a user intention understanding module, a dynamic action planning module, a safety monitoring and exception handling module and a continuous learning and collaborative optimization module, and can effectively improve the intention recognition precision, environmental adaptability and task execution efficiency of the robot in a complex home scene. And intelligent and personalized family service support is provided for the elderly.

Description

Technical Field

[0001] The present invention relates to the fields of robotics and computer technology, and particularly to a somatosensory control system and method for a humanoid robot based on deep learning. Background Art

[0002] In an aging society, humanoid robots need to have precise somatosensory control capabilities to assist the elderly in completing daily tasks (such as delivering items). The existing technologies have the following problems:

[0003] Poor environmental adaptability: Complex lighting, ground state changes, or clothing interference can easily lead to visual misjudgment.

[0004] Fuzzy intention recognition: It is difficult to accurately analyze the dialects, unclear pronunciations, or non-verbal behaviors (such as coughing) of the elderly.

[0005] Low task decomposition efficiency: Traditional planning methods for complex instructions (such as "hand the cup to the person in the living room") lack the ability of dynamic optimization.

[0006] The present invention significantly improves the intention recognition accuracy, environmental adaptability, and task execution efficiency by integrating multi-modal data (voice, vision, environmental perception, and wearable devices), designing a multi-modal decision arbitration mechanism, and combining task decomposition with the dynamic strategy planning of the robot. Summary of the Invention

[0007] Aiming at the deficiencies of the existing detection methods, a somatosensory control method for a humanoid robot based on deep learning is proposed, and a somatosensory control system for a humanoid robot based on deep learning is built.

[0008] The somatosensory control method for a humanoid robot based on deep learning is characterized in that:

[0009] Step S1: Dynamically collect the user's heart rate, body temperature, and gait pressure through wearable devices, and adaptively adjust the sampling frequency based on the action complexity (low frequency for static / dynamic high frequency). At the same time, deploy environmental sensors to obtain standardized temperature and humidity, ground friction coefficient, and light intensity data in real time. Combine the audio synchronization collection of sounds by the wearable device and the robot microphone array, and the multi-view image capture of the fixed camera and the robot body vision to achieve the full-modal synchronous collection of the user's physiological state, environmental information, and behavior data;

[0010] Step S2: Denoise the collected sound to extract pure user speech, convert it into text through speech recognition technology, and then perform dialect adaptation and semantic correction in combination with a multi-modal large model to accurately understand the user's instructions. At the same time, use speech separation technology to extract non-verbal sound features such as coughing, rapid wheezing, and falling from the audio, and perform physiological state recognition based on a classification model constructed by a speech vectorization model and a support vector machine. Finally, achieve comprehensive judgment and provide accurate multi-dimensional speech analysis results for subsequent interactions.

[0011] Step S3: Perform real-time human detection on the pictures taken by the fixed camera and the robot camera, screen out the valid images containing the user, and then extract the key frames per second within 5 seconds and input them into the multi-modal large model to accurately identify the user's posture and gestures and convert them into text descriptions. At the same time, combine with a pre-built gesture database for the elderly group for matching verification. If no known gesture is matched, it is marked as none. Finally, achieve a comprehensive analysis of the user's posture and gesture intentions.

[0012] Step S4: Integrate multi-modal data to accurately judge the user's intention. First, construct an environmental awareness by combining the scene context (time, location, and user's historical behavior). Then, standardize the physiological data (heart rate, body temperature, gait pressure) collected by the wearable device, environmental sensor information (temperature and humidity, ground friction coefficient, light intensity), voice commands, physiological states, user postures, specific gestures, and spatio-temporal behavior characteristics, and calculate the intention probability through an algorithm. Finally, the safety verification loop performs a secondary confirmation (such as voice or gesture verification) on high-risk commands to ensure reliable understanding and safe response to the user's true intention in a complex home scenario.

[0013] Step S5: Implement intelligent decision-making and execution of the robot's actions. First, construct a multi-dimensional state space including the robot state, item state, user state, and environmental state, and define an atomic action library composed of basic movement, hand operation, and macro interaction actions. At the same time, design a composite reward function that combines task completion reward, distance penalty, time efficiency penalty, grasping stability reward, and user comfort reward. Finally, through the collaborative optimization of the action model (outputting action probabilities) and the judgment model (evaluating action values), achieve dynamic task decomposition and highly adaptable execution under safety constraints.

[0014] Step S6: Dynamically update the robot's action policy learning model and establish a continuous learning mechanism. The model update is triggered in the following situations:

[0015] When the task fails or the user gives negative feedback, trigger local parameter update and only adjust the parameters of the last layer of the network to maintain stability.

[0016] Multiple robots share non-private experiences, protect user identities through differential privacy, and have a global model aggregation period of 7 days to synchronize the knowledge evolution of each robot;

[0017] The present invention also relates to a humanoid robot somatosensory control system based on deep learning. The system is used to execute the humanoid robot somatosensory control method based on deep learning as described above. The system includes:

[0018] Multimodal data acquisition module: Real-time collect user physiological, environmental, voice, and visual data, construct a multi-source heterogeneous data pool, and dynamically adjust the sampling frequency to balance accuracy and energy consumption;

[0019] User intention understanding module: Comprehensively analyze the user's true intention based on voice, vision, and physiological data, including the following sub-modules:

[0020] Voice parsing sub-module: Process voice commands using advanced noise reduction algorithms and dialect adaptation technologies, and simultaneously identify non-verbal sounds;

[0021] Visual recognition sub-module: Combine human body detection and pose recognition algorithms to match a preset elderly gesture library;

[0022] Multimodal decision-making sub-module: Make a final intention judgment by synthesizing the weights of each modal data;

[0023] Dynamic action planning module: Generate actions based on task decomposition and action strategy learning under safety constraints;

[0024] Safety monitoring and exception handling module: Real-time detect execution risks and trigger safety protocols, including joint limit detection, collision prediction, fall prediction, emergency braking, and a three-level response mechanism (voice warning → deceleration and stop → forced withdrawal);

[0025] Continuous learning and collaborative optimization module: Support online system updates and multi-robot collaborative learning, and achieve knowledge sharing through a federated learning framework;

[0026] Each module works in coordination to provide intelligent and personalized home service support. The system pays special attention to safety and adaptability, and demonstrates excellent reliability and interaction experience in complex home environments.

[0027] The present invention also discloses a non-volatile storage medium, characterized in that the non-volatile storage medium includes a stored program, wherein the program, when running, controls the device where the non-volatile storage medium is located to execute the above method.

[0028] The present invention also discloses an electronic device, which is characterized by comprising a processor and a memory; computer-readable instructions are stored in the memory, and the processor is used to run the computer-readable instructions, wherein the computer-readable instructions, when running, execute the above-mentioned method.

[0029] Beneficial effects

[0030] Accurate intention understanding: Through multi-modal data fusion and deep learning algorithms, effectively solve the recognition problems of the dialects, unclear pronunciation and non-verbal behaviors of the elderly, and improve the accuracy of intention judgment.

[0031] Dynamic environment adaptation: The motion planning system can respond to complex home environment changes in real time.

[0032] Continuous evolution ability: The federated learning framework supports multi-robot collaborative optimization, and the model is iteratively updated every 7 days, enabling the system to continuously improve its performance.

[0033] Elderly-friendly design: Exclusive elderly gesture libraries, dialect adaptation, and buffered action strategies and other user-friendly designs significantly improve the acceptance and usage experience of elderly users. Brief description of the drawings

[0034] Figure 1 It is the system architecture diagram of the present invention. Detailed implementation manners

[0035] Embodiment 1

[0036] A humanoid robot somatosensory control method based on deep learning includes the following steps:

[0037] Step S1: Dynamically collect the user's heart rate, body temperature and gait pressure through a wearable device, and adaptively adjust the sampling frequency based on the action complexity (low frequency in static state / high frequency in dynamic state). At the same time, deploy environmental sensors to continuously obtain standardized temperature and humidity, ground friction coefficient and light intensity data, combine the audio synchronous collection of sound by the wearable device and the robot microphone array, and the multi-view image capture of the fixed camera and the robot body vision to achieve the full-modal synchronous collection of the user's physiological state, environmental information and behavior data;

[0038] Step S11: Collect the user's heart rate, body temperature and gait pressure through a wearable device, and the sampling frequency is dynamically adjusted according to the action complexity, with a low sampling frequency when stationary and a high sampling frequency when moving;

[0039] Step S12: Deploy a multi-parameter environmental sensor array to continuously monitor temperature and humidity, ground friction coefficient and light intensity information, and monitor the environmental state in real time. The data format is standardized to FP16 floating-point numbers from 0 to 1;

[0040] Step S13: Collect sound through the microphone modules of the wearable device and the robot;

[0041] Step S14: Capture user pictures in real time through the scene-fixed camera and the robot body camera;

[0042] Step S2: Perform noise reduction processing on the collected sound to extract pure user speech, convert it into text through speech recognition technology, and then combine with a multi-modal large model for dialect adaptation and semantic correction to accurately understand the user's instructions; at the same time, use speech separation technology to extract non-verbal sound features such as coughing, rapid wheezing, and falling from the audio, and perform physiological state recognition based on the classification model constructed by the speech vectorization model and the support vector machine. Finally, comprehensive judgment is achieved to provide accurate multi-dimensional speech analysis results for subsequent interactions;

[0043] Step S21: Process the collected sound, remove the environmental background noise through noise reduction technology, and retain the user speech;

[0044] Step S22: Convert the user's voice into text using speech recognition technology;

[0045] Step S23: Correct the text based on the multi-modal large model to achieve dialect adaptation for the elderly and obtain the user's instructions;

[0046] The multi-modal large model can achieve the goal based on multi-source heterogeneous data through text prompts;

[0047] Step S24: Use speech separation technology to extract the user's non-verbal sounds, where the user's non-verbal sounds refer to coughing, rapid wheezing, falling, and painful groaning features;

[0048] Step S25: Train a speech classification model to judge the user's non-verbal sound features and identify the user's physiological state, such as healthy, weak, or emergency state;

[0049] The input of the speech classification model is the user's non-verbal sound, and the output is the user's physiological state. The model architecture adopts a speech vectorization model plus a support vector machine classification. 2000 pieces of manually annotated data are prepared for the training dataset, covering all states, and trained for 50 rounds, which can achieve accurate judgment of the user's physiological state;

[0050] Step S3: Perform real-time human detection on the pictures taken by the fixed camera and the robot camera, screen out the valid images containing the user, and then extract the key frames per second within 5 seconds and input them into the multi-modal large model to accurately identify the user's posture and gestures and convert them into text descriptions. At the same time, combine with the pre-built specific gesture database for the elderly group for matching verification. If no known gesture is matched, it is marked as none. Finally, a comprehensive analysis of the user's posture and gesture intentions is achieved;

[0051] Step S31: Extract the pictures from the fixed camera and the robot camera. First, perform object detection on the human body through the YOLO model, and delete the pictures without a human body.

[0052] Step S32: When there is a person in the camera, extract the pictures collected by the camera every second within 5 seconds, input them into the multi-modal large model to detect the user's posture and gestures, and the output is the text version of the user's posture and gestures.

[0053] Step S33: Establish a database of specific gestures for the elderly group. Based on the output of Step S31, determine whether the user's gesture belongs to a specific gesture. If not, the specific gesture is recorded as none.

[0054] Step S4: Integrate multi-modal data to achieve accurate judgment of the user's intention. First, construct environmental awareness by combining the scene context (time, location, and user's historical behavior). Then, standardize the physiological data (heart rate, body temperature, gait pressure) collected by the wearable device, environmental sensor information (temperature and humidity, ground friction coefficient, light intensity), voice commands, physiological state, user's posture, specific gestures, and spatio-temporal behavior characteristics, and calculate the intention probability through an algorithm. Finally, the safety verification loop performs a secondary confirmation on high-risk commands (such as voice or gesture verification) to ensure reliable understanding and safe response to the user's true intention in a complex home scenario.

[0055] Step S41: Enhance the understanding of the scene context. Collect the current time (morning / noon / night), current location (bedroom / living room / kitchen), and the user's historical behavior (often getting a water cup) to assist in judging the user's intention.

[0056] Step S42: Judge the user's true intention through multi-modal data. Standardize or vectorize all the data collected in Step S11 (user's heart rate, body temperature, gait pressure), Step S12 (temperature and humidity, ground friction coefficient, light intensity information), Step S23 (user's command), Step S25 (user's physiological state), Step S32 (user's posture), Step S33 (specific gesture), and Step S41 (current time, current location, and user's historical behavior) into a floating-point feature vector of 0-1, and perform confidence-weighted fusion. The formula is as follows:

[0057] Arrange all the data in this step in order and denote it as

[0058] X = [x1, x2,...., x13], where x1 represents the user's heart rate feature vector and x13 represents the user's historical behavior feature vector.

[0059] Calculation method of the weight coefficient

[0060]

[0061] where i represents the i-th feature vector, ai represents the weight of the i-th feature, and the larger the weight, the higher the importance. Wq, Wv, and Wk are learnable parameters, d is the dimension of each feature, exp represents the exponential function, and tanh represents the hyperbolic tangent function;

[0062] Intention probability judgment formula

[0063]

[0064] The feature mapping is implemented using a three-layer MLP encoder (256-128-64), denoted as f MLP (), softmax is the activation function, P(y|X) represents the probability that the intention is y under the X feature, y represents the intention number, and the y with the highest probability is the user's true intention;

[0065] Step S43, Safety verification loop:

[0066] Initiate secondary confirmation for high-risk instructions (such as "Emergency stop"), and verify through voice interaction or gestures (such as giving a thumbs up);

[0067] Step S5, Implement intelligent decision-making and execution of robot actions. First, construct a multi-dimensional state space including robot state, item state, user state, and environmental state, define an atomic action library consisting of basic movement, hand operations, and macro interaction actions, and design a composite reward function that combines task completion reward, distance penalty, time efficiency penalty, grasping stability reward, and user comfort reward. Finally, through the collaborative optimization of the action model (outputting action probabilities) and the judgment model (evaluating action values), dynamic task decomposition and highly adaptable execution are achieved under safety constraints;

[0068] Step S51, Define the state space, including the following:

[0069] Robot state S1 = [xr, yr, zr, θ, v, g1...gn], where xr, yr, zr represent the spatial position of the robot, θ represents the orientation of the robot, v represents the forward speed of the robot, and g1...gn represent the states of each joint of the robot, and n represents the number of joints;

[0070] Item state S2 = [xt, yt, zt, volume, material], where xt, yt, zt represent the spatial position of the item, volume represents the estimated volume of the item, and material represents the estimated surface material of the object;

[0071] User state S3 = [xu, yu, zu, usereye, userp, userps, userin], where xu, yu, and zu represent the user's spatial position, usereye represents the user's line of sight direction, userp represents the user's posture, userps represents the user's physiological state, and userin represents the user's intention;

[0072] Environmental state S4 = [light, sto, fri] represents the lighting condition, obstacle distribution, and ground friction coefficient in sequence;

[0073] The total state space is the combination of all states. At time step t, it is denoted as St = [S1t, S2t, S3t, S4t];

[0074] Step S52: Define the atomic action library, decompose complex tasks into standard action units, and the action set includes:

[0075] Robot position actions: forward, backward, left, right, turn left, turn right;

[0076] Hand actions: move the left hand up, move the left hand down, move the right hand up, move the right hand down, grab;

[0077] Macro actions: wait for confirmation, voice reminder, emergency stop, call for rescue;

[0078] Constraints: Each orientation angle adjustment is ±1°, the gait control step length is 0.1 m, and the grasping force increases by 10 N;

[0079] Number all actions, and denote the formulaic representation of the action space as A = [a1, a2,..., am], where a1 represents a certain action and m is the total number of actions;

[0080] Step S53: Design a reward function, give rewards for positive behaviors and punishments for negative behaviors;

[0081] The content included in the reward function R is as follows:

[0082] Index Score Explanation Task Completion Reward +100 Reward for Successful Task Completion Distance Penalty -0.1×‖p_robot -p_object‖² Square difference of the distance between the robot p_robot and the current target p_object Time Penalty -0.01 / step step represents the number of action executions, encouraging fewer actions to complete the task Grasping Task Stability Reward -0.1×|f_desired -f_actual| f_desired represents the desired force for grasping an item, f_actual represents the actual force for grasping an item, and the absolute value difference is used to encourage grasping the item with an appropriate force User Comfort Reward +1.0 (if the rate of change of the robot's speed v < 0.1 m / s³) Smoothness of Movements

[0083] Step S54: Robot action policy learning model

[0084] Action model design:

[0085] Input the state St, and output the predicted action probability P(at|St) and the action at with the highest probability;

[0086] The architecture of the action model is as follows:

[0087] h1 = LayerNorm(ReLU(W1·St + b1))

[0088] h2 = g ⊙ h1+(1 - g)⊙ St

[0089] Where St is the input state, W1 is the mapping weight matrix, b1 is the mapping bias vector, g is the residual connection weight, ⊙ is the Hadamard product, h1 represents the mapped feature, h2 represents the feature after the residual connection, ReLU is the activation function, and LayerNorm layer normalization is used;

[0090] logits = Wp·h2 + bp

[0091] P(at|St) = Gumbel-Softmax(logits, τ = 0.7)

[0092] Where Wp is the prediction weight matrix, bp is the prediction bias vector, logits represents the unnormalized action probability distribution, Gumbel-Softmax represents the activation function with added noise, which is more robust, τ is the temperature parameter, and P(at|St) represents the action probability. The at with the maximum probability is taken as the candidate action;

[0093] Judgment model design:

[0094] Input the state St and output the value Qt of the current action;

[0095] The judgment model architecture is as follows:

[0096] h3 = Swish(W3·St + b3)

[0097] Where h3 represents the shared feature, W3 is the shared feature weight matrix, b2 is the shared feature bias vector, and Swish is the activation function;

[0098] Vv = Wvv·h3 + bv

[0099] Aa = Waa·h3 + ba

[0100] Qt = Vv+(Aa - ∑Aa / m)

[0101] Where Vv represents the value feature, evaluating the quality of the current action, Aa represents the advantage feature, ∑Aa / m represents the average advantage, Wvv and Waa are the corresponding weight matrices respectively, bv and ba are the corresponding bias vectors, and the value function Qt is composed of Vv and Aa;

[0102] Loss function design:

[0103] L =

[0104] Where y and α are weight parameters, which are 0.99 and 0.9 respectively

[0105] Step S55. The training process of the robot motion strategy learning model is as follows: Initialize the parameters of the motion model and the judgment model. Execute the current action at in a virtual / real environment, collect the state-action-reward sequence-new state (St, At, Rt, St+1), and preferentially retain high-risk states (such as fall intervention, item slipping).

[0106] Calculate the Q value using the judgment model every 48 steps, calculate the loss function and update the judgment model, and update the motion model:

[0107] Perform physical feasibility verification on the output action (such as joint limit, collision detection), and reset the illegal action to the safety preset value; Stop training the model when the average reward in 10 consecutive rounds ≥ the threshold or reach the maximum number of training rounds (50);

[0108] Step S6. Dynamically update the robot motion strategy learning model and establish a continuous learning mechanism. The model update is triggered in the following situations:

[0109] When the task fails or the user gives negative feedback, trigger local parameter update, and only adjust the parameters of the last layer network to maintain stability;

[0110] Multiple robots share non-private experiences, protect user identities through differential privacy, the global model aggregation period is 7 days, and synchronize the knowledge evolution of each robot;

[0111] Embodiment 2

[0112] A humanoid robot somatosensory control system based on deep learning. The system is used to execute the humanoid robot somatosensory control method based on deep learning as described above, and is characterized in that:

[0113] Multimodal data acquisition module: Real-time collect user physiological, environmental, voice and visual data, construct a multi-source heterogeneous data pool, and dynamically adjust the sampling frequency to balance accuracy and energy consumption, including the following sub-modules:

[0114] Wearable device sub-module: Collect heart rate, body temperature, gait pressure;

[0115] Environmental perception sub-module: Monitor temperature and humidity, ground friction coefficient, light intensity;

[0116] Acoustic acquisition sub-module: Integrate wearable microphones and robot array microphones;

[0117] Visual perception sub-module: Fixed camera + robot binocular vision (local close-up)

[0118] User intention understanding module: Comprehensively analyze the real intention of the user based on voice, vision, and physiological data, including the following sub-modules:

[0119] Voice parsing sub-module: Processes voice commands using advanced noise reduction algorithms and dialect adaptation techniques, and simultaneously identifies non-verbal sounds;

[0120] Visual recognition sub-module: Combines human detection and pose recognition algorithms to match a preset elderly gesture library;

[0121] Multi-modal decision-making sub-module: Makes a final intention judgment by integrating the weights of various modal data;

[0122] Dynamic action planning module: Generates actions based on task decomposition and action strategy learning under safety constraints;

[0123] Safety monitoring and exception handling module: Real-time detects execution risks and triggers safety protocols, including joint limit detection, collision prediction, fall prediction, emergency braking, and a three-level response mechanism (voice warning → deceleration stop → forced retraction);

[0124] Continuous learning and collaborative optimization module: Supports online system updates and multi-robot collaborative learning, and realizes knowledge sharing through a federated learning framework;

[0125] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification is only the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A humanoid robot somatosensory control method based on deep learning, characterized in that: Step S1. Dynamically collect the user's heart rate, body temperature, and gait pressure through wearable devices, and adaptively adjust the sampling frequency based on the action complexity (low frequency for static / dynamic high frequency). At the same time, deploy environmental sensors to obtain standardized temperature and humidity, ground friction coefficient, and light intensity data in real time. Combine the audio synchronous collection of sound by the wearable device and the robot microphone array, and the multi-view image capture of the fixed camera and the robot body vision to achieve the full-modal synchronous collection of the user's physiological state, environmental information, and behavior data; Step S2. Denoise the collected sound to extract pure user speech, convert it into text through speech recognition technology, and then perform dialect adaptation and semantic correction through a multi-modal large model to accurately understand the user's instructions. At the same time, use speech separation technology to extract non-verbal sound features such as coughing, rapid breathing, and falling from the audio, and perform physiological state recognition based on the classification model constructed by the speech vectorization model and the support vector machine. Finally, achieve comprehensive judgment and provide accurate multi-dimensional speech analysis results for subsequent interactions; Step S3. Perform real-time human detection on the pictures taken by the fixed camera and the robot camera, screen out the effective images containing the user, and then extract the key frames per second within 5 seconds and input them into the multi-modal large model to accurately identify the user's posture and gestures and convert them into text descriptions. At the same time, combine the pre-built specific gesture database for the elderly group for matching verification. If no known gesture is matched, it is marked as none. Finally, achieve a comprehensive analysis of the user's posture and gesture intentions; Step S4. Integrate multi-modal data to accurately judge the user's intention. First, construct an environmental perception by combining the scene context (time, location, and user's historical behavior). Then, standardize the physiological data (heart rate, body temperature, gait pressure) collected by the wearable device, environmental sensor information (temperature and humidity, ground friction coefficient, light intensity), voice commands, physiological state, user's posture, specific gestures, and spatio-temporal behavior characteristics, and calculate the intention probability through an algorithm. Finally, the safety verification loop performs a secondary confirmation on high-risk commands (such as voice or gesture verification) to ensure reliable understanding and safe response to the user's true intention in a complex home scenario; Step S5. Implement intelligent decision-making and execution of the robot's actions. First, construct a multi-dimensional state space including the robot state, item state, user state, and environmental state, and define an atomic action library composed of basic movement, hand operation, and macro interaction actions. At the same time, design a composite reward function that combines task completion reward, distance penalty, time efficiency penalty, grasping stability reward, and user comfort reward. Finally, through the collaborative optimization of the action model (outputting action probability) and the judgment model (evaluating action value), achieve dynamic task decomposition and highly adaptable execution under safety constraints; Step S6. Dynamically update the robot's action strategy learning model and establish a continuous learning mechanism. The model update is triggered in the following situations: When the task fails or the user gives negative feedback, trigger local parameter update and only adjust the parameters of the last layer network to maintain stability; Multiple robots share non-private experiences, protect user identities through differential privacy, and have a global model aggregation period of 7 days to synchronize the knowledge evolution of each robot.

2. The method for controlling the humanoid robot's body sensation based on deep learning according to claim 1 is characterized in that: The step S1 further includes the following contents: Step S11: Collect the user's heart rate, body temperature, and gait pressure through a wearable device. The sampling frequency is dynamically adjusted according to the action complexity, with a low sampling frequency when stationary and a high sampling frequency when moving. Step S12: Deploy a multi-parameter environmental sensor array to continuously monitor temperature, humidity, ground friction coefficient, and light intensity information, and monitor the environmental status in real time. The data format is standardized to FP16 floating-point numbers from 0 to 1. Step S13: Collect sound through the microphone modules of the wearable device and the robot. Step S14: Capture the user's pictures in real time through the fixed camera in the scene and the camera on the robot body.

3. The humanoid robot somatosensory control method based on deep learning according to claim 1, characterized in that: The step S2 further includes the following contents: Step S21: Process the collected sound, remove the environmental background sound through noise reduction technology, and retain the user's voice. Step S22: Convert the user's voice into text using speech recognition technology. Step S23: Correct the text based on a multi-modal large model to adapt to the elderly's dialect and obtain the user's instructions. The multi-modal large model can achieve the goal based on multi-source heterogeneous data through text prompts. Step S24: Use voice separation technology to extract the user's non-verbal sounds, which refer to the characteristics of coughing, rapid wheezing, falling, and groaning in pain. Step S25: Train a voice classification model to judge the characteristics of the user's non-verbal sounds and identify the user's physiological state, such as healthy, weak, or emergency state. The input of the voice classification model is the user's non-verbal sound, and the output is the user's physiological state. The model architecture uses a voice vectorization model plus a support vector machine classification. 2000 pieces of manually annotated data covering all states are prepared for the training dataset, and it is trained for 50 rounds to achieve accurate judgment of the user's physiological state.

4. The method for controlling the humanoid robot's body sensation based on deep learning according to claim 1, characterized in that: the step S3 further includes the following contents: Step S31: Extract the pictures from the fixed camera and the robot camera. First, perform object detection tasks on the human body through the yolo model and delete the pictures without a human body. Step S32: When there is a person in the camera, extract the pictures collected by the camera every second within 5 seconds, input them into the multi-modal large model to detect the user's posture and gestures, and the output is the text version of the user's posture and gestures. Step S33: Establish a database of specific gestures for the elderly group, and judge whether the user's gesture belongs to a specific gesture based on the output of step S31. If not, the specific gesture is recorded as none.

5. The method for controlling the humanoid robot's somatosensory based on deep learning according to claim 1, characterized in that: The step S4 further includes the following contents: Step S41: Increase the understanding of the scene context, collect the current time (morning / noon / night), the current location (bedroom / living room / kitchen), and the user's historical behavior (often taking a water cup) to assist in judging the user's intention. Step S42: Determine the user's true intention through multimodal data. Standardize or vectorize all the data collected in step S11 (user heart rate, body temperature, gait pressure), step S12 (temperature and humidity, ground friction coefficient, light intensity information), the user instructions in step S23, the user's physiological state in step S25, the user's posture in step S32, the specific gestures in step S33, and the current time, current location, and user's historical behavior in step S41 into a floating-point feature vector of 0-1, and perform confidence-weighted fusion. The formula is as follows: Arrange all the data in this step in order and denote it as X = [x1, x2,...., x13], where x1 represents the user's heart rate feature vector and x13 represents the user's historical behavior feature vector; Calculation method of weight coefficient ； where i represents the i-th feature vector, ai represents the weight of the i-th feature, the greater the weight, the higher the importance, Wq, Wv, Wk are learnable parameters, d is the dimension of each feature, exp represents the exponential function, and tanh is the tangent function; Formula for judging intention probability ； The feature mapping is implemented by a three-layer MLP encoder (256-128-64), denoted as f MLP (), with softmax as the activation function. P(y|X) represents the probability that the intention is y under the X feature. y represents the intention number, and the y with the highest probability is the true intention of the user; Step S43: Safety verification loop: Initiate secondary confirmation for high-risk instructions (such as "emergency stop") and verify through voice interaction or gestures (such as giving a thumbs up).

6. The humanoid robot somatosensory control method based on deep learning according to claim 1, characterized in that: Step S5 further includes the following content: Step S51: Define the state space, including the following: Robot state S1 = [xr, yr, zr, θ, v, g1...gn], where xr, yr, zr represent the spatial position of the robot, θ represents the orientation of the robot, v represents the forward speed of the robot, and g1...gn represent the states of each joint of the robot, and n represents the number of joints; Item state S2 = [xt, yt, zt,, volume, material], where xt, yt, zt represent the spatial position of the item, volume represents the estimated volume of the item, and material represents the estimated surface material of the object; User state S3 = [xu, yu, zu, usereye, userp, userps, userin], where xu, yu, zu represent the spatial position of the user, usereye represents the direction of the user's line of sight, userp represents the user's posture, userps represents the user's physiological state, and userin represents the user's intention; Environmental state S4 = [light, sto, fri], representing the light condition, obstacle distribution, and ground friction coefficient in sequence; The total state space is the combination of all states. At time step t, it is denoted as St = [S1t, S2t, S3t, S4t]; Step S52: Define the action atom library, decompose complex tasks into standard action units, and the action set includes: Robot position actions: forward, backward, left, right, turn left, turn right; Hand actions: move the left hand up, move the left hand down, move the right hand up, move the right hand down, grab; Macro actions: wait for confirmation, voice reminder, emergency stop, call for rescue; Constraints: Each adjustment of the orientation angle is ±1°, the gait control step length is 0.1m, and the grasping force increases by 10N; Number all actions. Denote the action space as A = [a1, a2,..., am] in a formulated manner, where a1 represents a certain action and m is the total number of actions. Step S53: Design a reward function to give rewards for positive behaviors and punishments for negative behaviors. The content included in the reward function R is as follows: Step S54: Robot action policy learning model Action model design: Input the state St, and output the predicted action probability P(at|St) and the action at with the highest probability. The architecture of the action model is as follows: h1 = LayerNorm(ReLU(W1·St + b1)) h2 = g⊙h1 + (1−g)⊙St Where St is the input state, W1 is the mapping weight matrix, b1 is the mapping bias vector, g is the residual connection weight, ⊙ is the Hadamard product, h1 represents the mapped features, h2 represents the features after the residual connection, ReLU is the activation function, and LayerNorm layer normalization is used. logits = Wp·h2 + bp P(at|St) = Gumbel-Softmax(logits, τ = 0.7) Where Wp is the prediction weight matrix, bp is the prediction bias vector, logits represents the unnormalized action probability distribution, Gumbel-Softmax represents the activation function with noise added, which is more robust, τ is the temperature parameter, P(at|St) represents the action probability, and at with the maximum probability is taken as the candidate action. Judgment model design: Input the state St, and output the value Qt of the current action. The architecture of the judgment model is as follows: h3 = Swish(W3·St + b3) Where h3 represents the shared features, W3 is the shared feature weight matrix, b2 is the shared feature bias vector, and Swish is the activation function. Vv = Wvv·h3 + bv Aa = Waa·h3 + ba Qt = Vv + (Aa − ∑Aa / m) Where, Vv represents the value feature, evaluating the quality of the current action, Aa represents the advantage feature, represents the average advantage, Wvv and Waa are the corresponding weight matrices respectively, bv and ba are the corresponding bias vectors, and the value function Qt is composed of Vv and Aa. Loss function design: L= ； Where y and α are weight parameters, which are 0.99 and 0.9 respectively Step S55: The training process of the robot action policy learning model is as follows: Initialize the parameters of the action model and the judgment model, execute the current action at in the virtual / real environment, collect the state-action-reward sequence-new state (St, At, Rt, St+1), and preferentially retain high-risk states (such as fall intervention, item slipping). Calculate the Q value using the judgment model every 48 steps, calculate the loss function and update the judgment model, and update the action model: Perform physical feasibility verification on the output action (such as joint limit, collision detection), and reset the illegal action to the safety preset value; Stop training the model when the average reward in 10 consecutive rounds ≥ the threshold or reach the maximum number of training rounds (50).

7. A somatosensory control system for a humanoid robot based on deep learning, which is used to execute the method for somatosensory control of a humanoid robot based on deep learning according to any one of claims 1-6, characterized in that: Multimodal data acquisition module: Real-time collect user physiological, environmental, voice and visual data, construct a multi-source heterogeneous data pool, and dynamically adjust the sampling frequency to balance accuracy and energy consumption, including the following sub-modules: Wearable device sub-module: Collect heart rate, body temperature, and gait pressure; Environmental perception sub-module: Monitor temperature and humidity, ground friction coefficient, and light intensity; Acoustic acquisition sub-module: Integrate wearable microphones and robot array microphones; Visual perception sub-module: Fixed camera + robot binocular vision (local close-up); User intention understanding module: Analyze the user's true intention by integrating voice, visual, and physiological data, including the following sub-modules: Voice parsing sub-module: Use advanced noise reduction algorithms and dialect adaptation technologies to process voice commands, and at the same time recognize non-verbal sounds; Visual recognition sub-module: Combine human body detection and pose recognition algorithms to match a preset elderly gesture library; Multimodal decision-making sub-module: Make a final intention judgment by integrating the weights of each modal data; Dynamic action planning module: Realize action generation under safety constraints based on task decomposition and action strategy learning; Safety monitoring and exception handling module: Real-time detect execution risks and trigger safety protocols, including joint limit detection, collision prediction, fall prediction, emergency braking, and a three-level response mechanism (voice warning → deceleration stop → forced withdrawal); Continuous learning and collaborative optimization module: Support online system updates and multi-robot collaborative learning, and achieve knowledge sharing through a federated learning framework.

8. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein the program, when running, controls the device where the non-volatile storage medium is located to execute the method according to claim 1.

9. An electronic device, characterized in that, Comprising a processor and a memory; computer-readable instructions are stored in the memory, and the processor is used to run the computer-readable instructions, wherein the computer-readable instructions, when running, execute the method according to claim 1.

Images