Humanoid robot somatosensory control system and method based on deep learning

Abstract

The application discloses a kind of somatosensory control system and method of humanoid robot based on deep learning.Method includes: through wearable device and environmental sensor, user physiological data and environmental information are collected, realize full mode synchronous acquisition in combination with audio and visual data.System carries out noise reduction processing to the sound collected, extracts voice instruction and identifies non-verbal sound characteristics to judge user physiological state;Through human body detection and posture recognition algorithm, user posture and gesture intention are analyzed.Further fusion multi-modal data, in combination with scene context and safety verification loop, accurately judge user intention, and construct multi-dimensional state space and atomic action library, design composite reward function to realize the intelligent decision and execution of robot action.System includes multi-modal data acquisition module, user intention understanding module, dynamic action planning module, safety monitoring and abnormal processing module and continuous learning and collaborative optimization module, can effectively improve the intention recognition accuracy, environmental adaptability and task execution efficiency of robot in complex home scene, provide intelligent, personalized home service support for the elderly.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robots and computer technology, in particular to a somatosensory control system and method for humanoid robots based on deep learning. BACKGROUND

[0002] In an aging society, humanoid robots need to have precise somatosensory control capabilities to assist the elderly in completing daily tasks (such as delivering items). The existing technology has the following problems:

[0003] Poor environmental adaptability: complex lighting, ground state changes or clothing interference can easily lead to visual misjudgment.

[0004] Intention recognition is ambiguous: the elderly dialect, ambiguous pronunciation or non-verbal behavior (such as coughing) is difficult to accurately analyze.

[0005] Low task decomposition efficiency: complex instructions (such as "deliver the cup to the person in the living room") lack dynamic optimization capabilities in traditional planning methods.

[0006] The present application significantly improves the intention recognition accuracy, environmental adaptability and task execution efficiency by fusing multi-modal data (voice, vision, environmental perception and wearable devices), designing a multi-modal decision arbitration mechanism, and combining task decomposition and robot dynamic strategy planning. SUMMARY

[0007] In view of the shortcomings of the existing detection method, a somatosensory control method for humanoid robots based on deep learning is proposed, and a somatosensory control system for humanoid robots based on deep learning is built.

[0008] The somatosensory control method for humanoid robots based on deep learning has the following characteristics:

[0009] Step S1, dynamically collect the user's heart rate, body temperature and gait pressure through wearable devices, and adaptively adjust the sampling frequency (low frequency for static / high frequency for dynamic) based on the action complexity, at the same time deploy environmental sensors to obtain standardized temperature and humidity, ground friction coefficient and light intensity data in real time, combine the audio synchronous collection of sound by wearable devices and robot microphone array, and multi-view image capture by fixed camera and robot body vision, realize the full-modal synchronous collection of user physiological state, environmental information and behavior data;

[0010] Step S2, the collected sound is denoised to extract pure user voice, which is converted into text through voice recognition technology, and then combined with a multi-modal large model for dialect adaptation and semantic correction to accurately understand user instructions; meanwhile, voice separation technology is used to extract non-verbal sound features such as coughing, rapid panting, and falling from the audio, and a classification model based on a voice vectorization model and a support vector machine is used for physiological state recognition, and finally a comprehensive judgment is made to provide accurate multi-dimensional voice analysis results for subsequent interaction;

[0011] Step S3, real-time human body detection is performed on the pictures taken by the fixed camera and the robot camera, and the effective images containing the user are screened out, then the key frames of each second within 5 seconds are extracted and input into the multi-modal large model, the user's posture and gesture are accurately recognized and converted into text description, and at the same time, the pre-built specific gesture database of the elderly group is matched and verified, if no known gesture is matched, it is marked as none, and finally the user's posture and gesture intention are comprehensively analyzed;

[0012] Step S4, fusion of multi-modal data to accurately judge user intention, first, the environment cognition is constructed by combining the scene context (time, place and user historical behavior), then the physiological data (heart rate, body temperature, gait pressure) collected by the wearable device, environmental sensor information (temperature and humidity, ground friction coefficient, light intensity), voice instructions, physiological state, user posture, specific gestures and spatio-temporal behavior characteristics are standardized, and the intention probability is calculated through algorithm, finally the high-risk instructions are confirmed again by the safety verification loop (such as voice or gesture verification), to ensure reliable understanding and safe response to user's true intention in complex home scene;

[0013] Step S5, intelligent decision and execution of robot action, first, a multi-dimensional state space including robot state, object state, user state and environment state is constructed, and an atomic action library composed of basic movement, hand operation and macro interaction action is defined, and a compound reward function integrating task completion reward, distance penalty, time efficiency penalty, grabbing stability reward and user comfort reward is designed, finally through the cooperative optimization of action model (output action probability) and judgment model (evaluate action value), dynamic task decomposition and high adaptability execution are realized under safety constraints;

[0014] Step S6, dynamically updating the robot action strategy learning model, establishing a continuous learning mechanism, the model is updated under the following conditions:

[0015] When the task fails or the user feedback is negative, trigger local parameter update, only adjust the last layer network parameters to maintain stability;

[0016] Multiple robots share non-private experience, protect user identity through differential privacy, global model aggregation cycle is 7 days, and knowledge evolution of each robot is synchronized;

[0017] The application also relates to a somatosensory control system for a humanoid robot based on deep learning, which is used to execute the somatosensory control method for a humanoid robot based on deep learning as described above, and comprises the following components:

[0018] A multi-modal data acquisition module: real-time acquisition of physiological, environmental, speech and visual data of a user, construction of a multi-source heterogeneous data pool, and dynamic adjustment of a sampling frequency to balance accuracy and energy consumption;

[0019] A user intention understanding module: comprehensive analysis of the real intention of a user based on speech, visual and physiological data, comprising the following sub-modules:

[0020] A speech analysis sub-module: advanced noise reduction algorithms and dialect adaptation techniques are used to process speech instructions, and non-verbal sounds are identified at the same time;

[0021] A visual recognition sub-module: a pre-set gesture library for the elderly is matched by combining human body detection and posture recognition algorithms;

[0022] A multi-modal decision sub-module: comprehensive final intention judgment based on the weight of each modal data;

[0023] A dynamic action planning module: action generation under safety constraints based on task decomposition and action strategy learning;

[0024] A safety monitoring and abnormality processing module: real-time detection of execution risks and triggering of safety protocols, including joint limit detection, collision prediction, fall prediction, emergency braking and a three-level response mechanism (speech warning -> speed reduction -> forced retreat);

[0025] A continuous learning and collaborative optimization module: support for online updating of the system and collaborative learning of multiple robots, and realization of knowledge sharing through a federated learning framework;

[0026] The modules work collaboratively to provide intelligent and personalized home service support. The system pays special attention to safety and adaptability, and exhibits excellent reliability and interactive experience in a complex home environment.

[0027] The application also discloses a non-volatile storage medium, characterized by comprising a stored program, wherein the program controls a device where the non-volatile storage medium is located to execute the method when running.

[0028] The application further discloses an electronic device, characterized in that the electronic device comprises a processor and a memory; the memory stores computer readable instructions, and the processor is configured to execute the computer readable instructions, wherein the computer readable instructions execute the method described above.

[0029] Advantages

[0030] Precise intention understanding: through multi-modal data fusion and deep learning algorithms, effectively solve the recognition problems of old dialects, ambiguous pronunciation and non-verbal behaviors, and improve the accuracy of intention judgment.

[0031] Dynamic environment adaptation: the motion planning system can respond to complex home environment changes in real time.

[0032] Continuous evolution capability: the federated learning framework supports multi-robot collaborative optimization, and the model is updated every 7 days, so that the system maintains continuous performance improvement.

[0033] Old people-friendly design: special old gesture library, dialect adaptation and buffer action strategy and other humanized design, significantly improve the acceptance and experience of old users. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The system architecture diagram of the application. DETAILED DESCRIPTION

[0035] Embodiment 1

[0036] The somatosensory control method of the humanoid robot based on deep learning comprises the following steps:

[0037] Step S1, dynamically collect the heart rate, body temperature and gait pressure of the user through the wearable device, and adaptively adjust the sampling frequency (low frequency in static state / high frequency in dynamic state) based on the motion complexity, and deploy the environmental sensor to obtain the standardized temperature and humidity, ground friction coefficient and light intensity data in real time, combine the audio synchronous collection of the wearable device and the microphone array of the robot, and the multi-angle image capture of the fixed camera and the vision of the robot body, realize the full-modal synchronous collection of the user's physiological state, environmental information and behavior data;

[0038] Step S11: collect the heart rate, body temperature and gait pressure of the user through the wearable device, and dynamically adjust the sampling frequency according to the motion complexity, the sampling frequency is low when static, and the sampling frequency is high when moving;

[0039] Step S12: deploy a multi-parameter environmental sensor array to continuously monitor the temperature and humidity, ground friction coefficient and light intensity information, and monitor the environmental state in real time, and the data format is standardized to FP16 floating point number of 0-1;

[0040] Step S13: Collect sound through the microphone module of the wearable device and the robot;

[0041] Step S14: Capture user pictures in real time through the scene fixed camera and the robot body camera;

[0042] Step S2, denoise the collected sound to extract pure user voice, convert it to text through voice recognition technology, and then combine a multi-modal large model for dialect adaptation and semantic correction to accurately understand user instructions; At the same time, use voice separation technology to extract non-verbal sound features such as coughing, rapid panting, and falling from the audio, and build a classification model based on a voice vectorization model and a support vector machine to recognize physiological states, and finally realize comprehensive judgment to provide accurate multi-dimensional voice analysis results for subsequent interaction;

[0043] Step S21, process the collected sound and remove environmental background noise through noise reduction technology to retain user voice;

[0044] Step S22, convert user voice into text using voice recognition technology;

[0045] Step S23, correct the text based on a multi-modal large model to achieve old dialect adaptation and obtain user instructions;

[0046] The multi-modal large model can achieve the target based on multi-source heterogeneous data through text prompts;

[0047] Step S24, extract user non-verbal sounds using voice separation technology, said user non-verbal sounds refer to coughing, rapid panting, falling, and painful moaning features;

[0048] Step S25, train a voice classification model to judge user non-verbal sound features and identify user physiological states such as health, weakness, or emergency state;

[0049] The input of the voice classification model is user non-verbal sound, and the output is user physiological state. The model architecture uses a voice vectorization model plus a support vector machine classification. The training data set is prepared with 2000 artificially annotated data covering all states, and the model is trained for 50 rounds to achieve accurate user physiological state judgment;

[0050] Step S3, perform real-time human body detection on the pictures taken by the fixed camera and the robot camera, filter out the effective images containing the user, then extract the key frames of each second within 5 seconds and input them into the multi-modal large model to accurately identify the user's posture and gestures and convert them into text descriptions, and at the same time, match and verify them with the pre-built specific gesture database of the elderly group. If no known gesture is matched, it is marked as none, and finally the user's posture and gesture intention are fully analyzed;

[0051] Step S31: Extract images from the fixed camera and the robot camera. First, use the YOLO model to perform object detection to detect human bodies and delete images that do not contain human bodies.

[0052] Step S32: When there is a person in the camera, extract the images captured by the camera every second within 5 seconds, input them into the multimodal large model to detect user posture and user gesture, and output them as text version of user posture and user gesture;

[0053] Step S33: Establish a database of specific gestures for the elderly. Based on the output of step S31, determine whether the user's gesture belongs to a specific gesture. If not, record the specific gesture as none.

[0054] Step S4: Integrating multimodal data to achieve accurate judgment of user intent. First, environmental cognition is constructed by combining scene context (time, location and user's historical behavior). Then, physiological data (heart rate, body temperature, gait pressure), environmental sensor information (temperature and humidity, ground friction coefficient, light intensity), voice commands, physiological state, user posture, specific gestures and spatiotemporal behavioral characteristics collected by wearable devices are standardized and processed. The probability of intent is calculated by algorithm. Finally, a security verification loop performs secondary confirmation on high-risk commands (such as voice or gesture verification) to ensure reliable understanding and safe response to the user's true intent in complex home scenarios.

[0055] Step S41: Enhance scene context understanding by collecting current time (morning / noon / night), current location (bedroom / living room / kitchen), and user history behavior (frequently picking up a water glass) to help determine user intent;

[0056] Step S42: Determine the user's true intent through multimodal data. The user's heart rate, body temperature, and gait stress collected in Step S11; temperature and humidity, ground friction coefficient, and light intensity information collected in Step S12; user commands in Step S23; user physiological state in Step S25; user posture in Step S32; specific gestures in Step S33; and current time, current location, and user historical behavior data from Step S41, all standardized or vectorized into 0-1 floating-point feature vectors, are fused using a confidence-weighted fusion method. The formula is as follows:

[0057] Arrange all the data from this step in order, and denote it as follows:

[0058] X = [x1, x2, ..., x13], where x1 represents the user's heart rate feature vector and x13 represents the user's historical behavior feature vector;

[0059] Weighting coefficient calculation method

[0060]

[0061] Where i represents the ith eigenvector, ai represents the weight of the ith feature, the greater the weight represents the higher the importance, Wq, Wv, and Wk are learnable parameters, d is the dimension of each feature, exp represents the exponential function, and tanh is the tangent function.

[0062] Intention probability judgment formula

[0063]

[0064] A three-layer MLP encoder (256-128-64) is used to realize feature mapping, denoted as f MLP (), softmax is the activation function, P(y|X) represents the probability of the intention y under the feature X, y represents the intention number, and the y with the maximum probability is the real intention of the user.

[0065] Step S43, safety verification loop:

[0066] Secondary confirmation is started for high-risk instructions (such as “emergency stop”), and voice interaction or gesture verification (such as raising the thumb) is used;

[0067] Step S5, intelligent decision and execution of robot action, first construct a multi-dimensional state space containing robot state, object state, user state and environment state, and define an atomic action library composed of basic movement, hand operation and macro interaction action, and design a compound reward function integrating task completion reward, distance penalty, time efficiency penalty, grasping stability reward and user comfort reward, finally realize dynamic task decomposition and high adaptability execution under safety constraints through the cooperative optimization of action model (output action probability) and judgment model (evaluate action value);

[0068] Step S51, define the state space, including the following contents:

[0069] Robot state S1=[xr,yr,zr,θ,v,g1...gn], wherein xr, yr, and zr represent the spatial position of the robot, θ represents the orientation of the robot, v represents the forward speed of the robot, g1...gn represents the state of each joint of the robot, and n represents the number of joints.

[0070] Object state S2=[xt,yt,zt,volume,material], wherein xt, yt, and zt represent the spatial position of the object, volume represents the estimated volume of the object, and material represents the estimated surface material of the object.

[0071] User state S3=[xu,yu,zu,usereye,userp,userps,userin], wherein xu,yu,zu represent user spatial position, usereye represents user line of sight direction, userp represents user posture, userps represents user physiological state, and userin represents user intention;

[0072] Environment state S4=[light,sto,fri] represents light condition, obstacle distribution, and ground friction coefficient in sequence;

[0073] The total state space is a combination of all states, and at time step t, it is denoted as St=[S1t,S2t,S3t,S4t];

[0074] Step S52, define the action atom library, decompose the complex task into standard action units, and the action set includes:

[0075] Robot position action: forward, backward, left, right, left turn, right turn;

[0076] Hand action: left hand up, left hand down, right hand up, right hand down, grab;

[0077] Macro action: wait for confirmation, voice reminder, emergency stop, call for help;

[0078] Constraints: adjust the angle of each direction ±1°, control the step length of gait 0.1m, and increase the grip strength 10N;

[0079] Number all actions, and denote the action space formula as A=[a1,a2,...,am], wherein a1 represents a certain action, and m is the total number of actions;

[0080] Step S53, design the reward function, give rewards for positive behaviors and penalties for negative behaviors;

[0081] The reward function R includes the following contents:

[0082] Metrics Score Explanation Task completion reward +100 Reward for successfully completing a task Distance penalty -0.1 x ||p_robot - p_object||2 Squared difference in distance between robot p_robot and current target p_object Time penalty -0.01 / step step represents the number of action steps, encouraging fewer actions to complete a task Grasp task stability reward -0.1 x |f_desired - f_actual| f_desired represents the desired force to grasp an object, f_actual represents the actual force to grasp an object, and the absolute value difference is used to encourage grasping an object with the appropriate force User comfort reward +1.0 (if robot velocity v rate of change < 0.1 m / s³) Action smoothness

[0083] Step S54, robot action strategy learning model

[0084] Action model design:

[0085] Input state St, output predicted action probability P(at|St) and action at with maximum probability;

[0086] The action model architecture is as follows:

[0087] h1=LayerNorm(ReLU(W1·St+b1)

[0088] h2 = g o h1 + (1 - g) o St

[0089] Where St is the input state, W1 is the mapping weight matrix, b1 is the mapping bias vector, g is the residual connection weight, o is the Hadamard product, h1 represents the mapped features, h2 represents the residual linked features, ReLU is the activation function, and the LayerNorm layer is used for normalization;

[0090] logits = Wp h2 + bp

[0091] P(at|St) = Gumbel-Softmax(logits, τ = 0.7)

[0092] Where Wp is the prediction weight matrix, bp is the prediction bias vector, logits represents the unnormalized action probability distribution, Gumbel-Softmax represents the noise-added activation function, which is more robust, τ is the temperature parameter, and P(at|St) represents the action probability, taking the at with the maximum probability as the candidate action;

[0093] Judgment model design:

[0094] Input state St, output the value Qt of the current action;

[0095] The judgment model architecture is as follows:

[0096] h3 = Swish(W3 St + b3)

[0097] Where h3 represents the shared features, W3 is the shared feature weight matrix, b2 is the shared feature bias vector, and Swish is the activation function;

[0098] Vv = Wvv h3 + bv

[0099] Aa = Waa h3 + ba

[0100] Qt = Vv + (Aa - ∑Aa / m)

[0101] Where Vv represents the value feature, evaluating the goodness of the current action, Aa represents the advantage feature, representing the average advantage, Wvv and Waa are the corresponding weight matrices, and bv and ba are the corresponding bias vectors, and the value function Qt is composed of Vv and Aa;

[0102] Loss function design:

[0103] L =

[0104] Where y and α are weight parameters, which are 0.99 and 0.9 respectively

[0105] Step S55, the robot action strategy learning model training process is: initializing the action model and the judgment model parameters, executing the current action at in the virtual / real environment, collecting the state-action-reward sequence-new state (St, At, Rt, St+1), and preferentially retaining high-risk states (such as fall intervention and article sliding);

[0106] Every 48 steps, the judgment model is used to calculate the Q value, the loss function is calculated, and the judgment model is updated, and the action model is updated:

[0107] The output action is physically checked for feasibility (such as joint limit and collision detection), and the illegal action is reset to a safe preset value; the model training is stopped when the average reward of 10 consecutive rounds is greater than or equal to a threshold or the maximum training round (50) is reached;

[0108] Step S6, dynamically updating the robot action strategy learning model, and establishing a continuous learning mechanism, the model update is triggered under the following conditions:

[0109] When the task fails or the user provides negative feedback, local parameter update is triggered, and only the last layer of network parameters is adjusted to maintain stability;

[0110] Multiple robots share non-private experience, protect user identity through differential privacy, and the global model aggregation period is 7 days, synchronizing the knowledge evolution of each robot;

[0111] Embodiment 2

[0112] The somatosensory control system of the humanoid robot based on deep learning, the system is used for executing the somatosensory control method of the humanoid robot based on deep learning as described above, and the system is characterized by:

[0113] Multi-modal data acquisition module: real-time acquisition of user physiological, environmental, voice and visual data, construction of multi-source heterogeneous data pool, dynamic adjustment of sampling frequency to balance accuracy and energy consumption, including the following sub-modules:

[0114] Wearable device sub-module: acquisition of heart rate, body temperature, gait pressure;

[0115] Environment perception sub-module: monitoring of temperature and humidity, ground friction coefficient, and light intensity;

[0116] Acoustic acquisition sub-module: integration of wearable microphones and robot array microphones;

[0117] Visual perception sub-module: fixed camera + robot binocular vision (local close-up)

[0118] User intention understanding module: comprehensive analysis of user real intention based on voice, visual and physiological data, including the following sub-modules:

[0119] Voice analysis submodule: advanced noise reduction algorithm and dialect adaptation technology are used to process voice commands, and non-verbal sounds are also recognized;

[0120] Visual recognition submodule: combined with human body detection and posture recognition algorithm, matching the preset old gesture library;

[0121] Multi-modal decision-making submodule: comprehensive weight of each modal data for final intention judgment;

[0122] Dynamic action planning module: based on task decomposition and action strategy learning, action generation is realized under the safety constraint;

[0123] Safety monitoring and abnormal handling module: real-time detection of execution risk and triggering of safety protocol, including joint limit detection, collision prediction, fall prediction, emergency braking, three-level response mechanism (voice warning → slow down and stop → forced retreat);

[0124] Continuous learning and collaborative optimization module: support online updating of system and collaborative learning of multiple robots, realize knowledge sharing through federated learning framework;

[0125] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection required by the present application is defined by the appended claims and their equivalents.

Claims

1. A humanoid robot somatosensory control method based on deep learning, characterized by: Step S1, dynamically collect user heart rate, body temperature and gait pressure through wearable devices, and adaptively adjust the sampling frequency based on the complexity of the action, at the same time, deploy environmental sensors to obtain standardized temperature, humidity, ground friction coefficient and light intensity data in real time, combine the audio synchronous collection of wearable devices and robot microphone array to capture sound, and the multi-view image capture of fixed camera and robot body vision to realize the full modal synchronous collection of user physiological state, environmental information and behavior data; Step S2, denoise the collected sound to extract pure user voice, convert it to text through voice recognition technology, and then combine a multi-modal large model for dialect adaptation and semantic correction to accurately understand user instructions; At the same time, use voice separation technology to extract cough, rapid panting and fall non-verbal sound features from the audio, and use a classification model based on a voice vectorization model and a support vector machine to recognize physiological state, and finally realize comprehensive judgment to provide accurate multi-dimensional voice analysis results for subsequent interaction; Step S3, real-time human body detection is performed on the pictures taken by the fixed camera and the robot camera, and the effective images containing the user are screened out, then the key frames of each second within 5 seconds are extracted and input into the multi-modal large model to accurately identify the user's posture and gestures and convert them into text description, and at the same time, a pre-built specific gesture database for the elderly group is matched and verified, if no known gesture is matched, it is marked as none, and finally the user's posture and gesture intention are comprehensively analyzed; Step S4, fuse multi-modal data to accurately judge user intention, first build environment cognition combined with scene context, then standardize physiological data such as heart rate, body temperature, gait pressure collected by wearable devices, environmental sensor information including temperature, humidity, ground friction coefficient, light intensity, voice instructions, physiological state, user posture, specific gestures and spatio-temporal behavior characteristics, and calculate the intention probability through algorithm, finally the high-risk instructions are confirmed again by the safety verification loop to ensure reliable understanding and safe response to user's true intention in complex home scenarios; Step S5, realize intelligent decision and execution of robot action, first build a multi-dimensional state space containing robot state, object state, user state and environment state, define an atomic action library consisting of basic movement, hand operation and macro interaction action, and design a compound reward function integrating task completion reward, distance penalty, time efficiency penalty, grasping stability reward and user comfort reward, finally realize dynamic task decomposition and high adaptability execution under safety constraints through the cooperative optimization of action model and judgment model; Step S6, dynamically update the robot action strategy learning model, establish a continuous learning mechanism, the model is updated under the following conditions: When the task fails or the user feedback is negative, trigger local parameter update, only adjust the last layer network parameters to maintain stability; Multiple robots share non-private experience, protect user identity through differential privacy, global model aggregation period is 7 days, synchronize the knowledge evolution of each robot.

2. The somatic control method for humanoid robots based on deep learning according to claim 1, characterized in that: The step S1 further includes the following contents: Step S11: Collect the user's heart rate, body temperature, and gait pressure through the wearable device. The sampling frequency is dynamically adjusted according to the complexity of the action. When stationary, the sampling frequency is low, and when moving, the sampling frequency is high. Step S12: Deploy a multi-parameter environmental sensor array to continuously monitor temperature, humidity, ground friction coefficient, and light intensity information. Real-time monitoring of environmental conditions, data format standardized to 0-1 FP16 floating point numbers. Step S13: Collect sound through the microphone module of the wearable device and the robot. Step S14: Capture user pictures in real time through the scene fixed camera and the robot body camera. 3.The deep learning-based somatosensory control method for humanoid robots according to claim 1, characterized in that: The step S2 further includes the following contents: Step S21: Process the collected sound and remove environmental background noise using noise reduction technology to retain user voice. Step S22: Convert user voice into text using voice recognition technology. Step S23: Correct the text based on a multi-modal large model to achieve old dialect adaptation and obtain user instructions. The multi-modal large model can provide text prompts and achieve the target based on multi-source heterogeneous data. Step S24: Extract user non-verbal sounds using voice separation technology, such as coughing, rapid panting, falling, and painful moaning characteristics. Step S25: Train a voice classification model to judge user non-verbal sound characteristics and identify user health, weakness, or emergency physiological states. The input of the voice classification model is user non-verbal sound, and the output is user physiological state. The model architecture uses a voice vectorization model and a support vector machine classification. The training data set is prepared with 2000 artificially annotated data covering all states. After 50 rounds of training, it can accurately judge the user's physiological state.

4. The deep learning-based humanoid robot somatosensory control method according to claim 1, characterized in that the step S3 further includes the following contents: Step S31: Extract pictures from fixed cameras and robot cameras. First, use the YOLO model to perform target detection tasks to detect human bodies and delete pictures without human bodies. Step S32: When there is a person in the camera, extract pictures collected by the camera every second within 5 seconds, input them into the multi-modal large model to detect user posture and user gestures, and output them as text version of user posture and user gestures. Step S33: Establish a specific gesture database for the elderly population. Based on the output of step S31, determine whether the user's gesture belongs to a specific gesture. If not, the specific gesture is recorded as none.

5. The deep learning-based somatosensory control method for humanoid robots according to claim 1, characterized in that: The step S5 further includes the following contents: Step S51: Define the state space, including the following contents: Robot state S1 = [xr, yr, zr, θ, v, g1...gn], where xr, yr, and zr represent the spatial position of the robot, θ represents the orientation of the robot, v represents the forward speed of the robot, and g1...gn represents the state of each joint of the robot. n represents the number of joints. The object state S2=[xt,yt,zt,volume,material], wherein xt,yt,zt represent the spatial position of the object, volume represents the estimated volume of the object, and material represents the estimated surface material of the object; The user state S3=[xu,yu,zu,usereye,userp,userps,userin], wherein xu,yu,zu represent the spatial position of the user, usereye represents the line of sight direction of the user, userp represents the posture of the user, userps represents the physiological state of the user, and userin represents the intention of the user; The environment state S4=[light,sto,fri] represents the lighting condition, obstacle distribution, and ground friction coefficient in sequence; The total state space is the combination of all states, denoted as St=[S1t,S2t,S3t,S4t] at time step t; Step S52, define the action atom library, decompose the complex task into standard action units, and the action set includes: Robot position actions: forward, backward, left, right, left turn, right turn; Hand actions: left hand up, left hand down, right hand up, right hand down, grab; Macro actions: wait for confirmation, voice reminder, emergency stop, call for help; Constraints: adjust the angle of each direction by ±1°, control the step length of gait by 0.1m, and increase the grip strength by 10N; Number all actions, and represent the action space as A=[a1,a2,...,am], wherein a1 represents a certain action, and m is the total number of actions; Step S53, design the reward function, give rewards for positive behaviors and penalties for negative behaviors; The reward function R includes the following contents: Step S54, robot action strategy learning model Action model design: Input state St, output predicted action probability P(at|St) and the action at with the maximum probability; The action model architecture is as follows: h1=LayerNorm(ReLU(W1·St+b1)); h2=g⊙h1+(1-g)⊙St; Wherein St is the input state, W1 is the mapping weight matrix, b1 is the mapping bias vector, g is the residual connection weight, ⊙ is the Hadamard product, h1 represents the mapped feature, h2 represents the residual linked feature, ReLU is the activation function, and LayerNorm layer is used for normalization processing; logits=Wp·h2+bp; P(at|St)=Gumbel-Softmax(logits,τ=0.7); Wherein Wp is the prediction weight matrix, bp is the prediction bias vector, logits represents the unnormalized action probability distribution, Gumbel-Softmax represents the noise added activation function, which is more robust, τ is the temperature parameter, P(at|St) represents the action probability, and the at with the maximum probability is taken as the candidate action; Judgment model design: Input state St, output the value Qt of the current action; The judgment model architecture is as follows: h3=Swish(W3·St+b3); Wherein h3 represents a shared feature, W3 is a shared feature weight matrix, b2 is a shared feature bias vector, and Swish is an activation function. Vv = Wvv·h3 + bv; Aa = Waa·h3 + ba; Qt = Vv + (Aa - ∑Aa / m); Wherein Vv represents a value feature, evaluating the pros and cons of the current action, Aa represents an advantage feature, representing the average advantage, Wvv and Waa are the corresponding weight matrices, and bv and ba are the corresponding bias vectors, and the value function Qt is composed of Vv and Aa. Loss function design: L= ； Wherein y and a are weight parameters, which are 0.99 and 0.9 respectively. Step S55, the robot action strategy learning model training process is: initializing the action model and the judgment model parameters, executing the current action at in the virtual / real environment, collecting the state-action-reward sequence-new state (St, At, Rt, St+1), and preferentially retaining high-risk states; Every 48 steps, use the judgment model to calculate the Q value, calculate the loss function and update the judgment model, and update the action model: Physically check the output action for feasibility, and reset the illegal action to a safe preset value; stop training the model when the average reward of 10 consecutive rounds is greater than or equal to the threshold or the maximum training rounds are reached.

6. A somatosensory control system for a humanoid robot based on deep learning, the system being used to perform the somatosensory control method for a humanoid robot based on deep learning as claimed in any one of claims 1-5, characterized in that: A multi-modal data acquisition module: real-time acquisition of user physiological, environmental, voice and visual data, construction of a multi-source heterogeneous data pool, dynamic adjustment of the sampling frequency to balance accuracy and energy consumption, including the following sub-modules: Wearable device sub-module: acquisition of heart rate, body temperature, gait pressure; Environmental perception sub-module: monitoring of temperature and humidity, ground friction coefficient, and light intensity; Acoustic acquisition sub-module: integration of wearable microphones and robot array microphones; Visual perception sub-module: fixed camera + robot binocular vision; User intention understanding module: comprehensive analysis of user's real intention based on voice, visual and physiological data, including the following sub-modules: Voice analysis sub-module: advanced noise reduction algorithm and dialect adaptation technology are used to process voice instructions, and non-verbal sounds are also identified; Visual recognition sub-module: combined with human body detection and posture recognition algorithm, matching the preset old gesture library; Multi-modal decision sub-module: comprehensive judgment of the final intention based on the weight of each modal data; Dynamic action planning module: action generation based on task decomposition and action strategy learning under safety constraints; Safety monitoring and abnormal handling module: real-time detection of execution risks and triggering of safety protocols, including joint limit detection, collision prediction, fall prediction, emergency braking, and three-level response mechanism: voice warning -> slow down -> forced retreat; Continuous learning and collaborative optimization module: support for online system updates and multi-robot collaborative learning, and knowledge sharing through a federated learning framework.

7. A non-volatile storage medium, characterized by The non-volatile storage medium comprises a stored program, wherein the program controls the device in which the non-volatile storage medium is located to perform the method of claim 1 when executed.

8. An electronic device, comprising: A computer readable storage medium storing computer readable instructions, wherein the computer readable instructions, when executed by a processor, perform the method of claim 1. A computer readable storage medium storing computer readable instructions, wherein the computer readable instructions, when executed by a processor, perform the method of claim 1.

Images