Robot control method based on multi-behavior fusion intelligent perception

Abstract

The invention discloses a robot control method based on multi-behavior fusion intelligent perception, and the method comprises the following steps: A, respectively constructing a facial expression intelligent recognition model, a body movement intelligent recognition model and a voice recognition control protocol set, and respectively establishing control instruction mapping rules; b, tracking and acquiring a facial expression image, a limb action image and a voice audio of a service object in real time; c, outputting a facial expression prediction result, a body language of the state and a voice instruction of the service object; d, robot action control instructions based on facial expressions, limb actions and voice instructions are obtained and fused, and a robot action control instruction after multi-behavior fusion is obtained; and E, controlling the robot to execute a corresponding action according to the robot action control instruction after multi-behavior fusion. According to the method, the cognitive ability and the context awareness ability of the robot can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of robot control, and in particular to a robot control method based on multi-behavior fusion intelligent perception. Background Art

[0002] With the acceleration of modern life and the increase of work pressure, more and more people are in sub-health status and need proper companionship and care. In hospitals, due to the limited number of medical staff and the increase in workload, the companionship and care of patients also face a series of problems. The traditional manual accompanying method has insufficient manpower, low efficiency and uneven service quality.

[0003] The invention patent with application number 202410073810.3 and titled "Household Robot Control System and Method Based on Visual and Auditory Fusion" discloses a household robot control system and method based on visual and auditory fusion, which uses the robot's camera and recording equipment to collect gesture monitoring video and voice signals of user control instructions, and introduces data processing and analysis algorithms at the back end to realize the recognition of user gestures and the understanding of voice semantics, so that the robot can understand and respond to the user's natural language instructions and gestures, and improve the intelligence and friendliness of the robot. In this way, it is possible to realize the behavior control of household robots based on video-voice multimodal recognition of user control instructions, and provide more comprehensive command perception capabilities through visual and auditory fusion technology, so that the robot can adapt to different environments and user needs, improve the flexibility and robustness of the robot, and improve the performance and efficiency of the robot.

[0004] The above patent only focuses on the recognition of user gestures and the understanding of voice semantics, and the visual and auditory fusion effect is limited. It lacks more sufficient and comprehensive multi-behavior fusion intelligent perception, resulting in the need to further improve the command perception ability. Summary of the invention

[0005] The purpose of the present invention is to provide a robot control method based on multi-behavior fusion intelligent perception, which can form robot motion control instructions after multi-behavior fusion by fusing the user's facial expressions, body movements and voice commands, thereby effectively improving the robot's cognitive ability and situational perception ability, enabling it to provide more accurate, personalized and humane services in complex environments.

[0006] The present invention adopts the following technical solutions:

[0007] A robot control method based on multi-behavior fusion intelligent perception includes the following steps:

[0008] A: Construct facial expression intelligent recognition model, body movement intelligent recognition model and voice recognition control protocol set respectively, and establish expression-control command mapping rules between facial expression and robot behavior control command, body movement and service object behavior state body-control command mapping rules, and voice command and robot behavior control command voice-control command mapping rules respectively;

[0009] B: Use the robot's built-in image acquisition module and voice acquisition module to track and obtain the service object's facial expression images, body movement images and voice audio in real time;

[0010] C: Based on the collected continuous facial expression images of the service object, the facial expression prediction results are output through the trained facial expression intelligent recognition model;

[0011] Based on the collected continuous body movement images of the service object, the body language of the service object's behavior status is output through the body movement intelligent recognition model;

[0012] Based on the collected voice audio of the service object, the voice command of the service object is output through the voice recognition control protocol set;

[0013] D: Using the obtained facial expression prediction results, body language and voice commands, respectively, the expression-control command mapping rules, body-control command mapping rules and semantic-control command mapping rules are combined to obtain the robot action control commands based on facial expressions, the robot action control commands based on body movements and the robot action control commands based on voice commands; then the robot action control commands based on facial expressions, body movements and voice commands are fused to obtain the robot action control commands after multi-behavior fusion;

[0014] E: Based on the robot motion control instructions obtained after multi-behavior fusion, the robot control module uses the robot's own execution module to perform the corresponding action.

[0015] In the step A, a facial expression intelligent recognition model is constructed based on a wavelet neural network and an improved residual convolutional neural network, the facial expression intelligent recognition model is trained using a facial expression state data set to obtain a trained facial expression intelligent recognition model; and an expression-control instruction mapping rule between facial expressions and robot behavior control instructions is established.

[0016] In step A, an intelligent limb motion recognition model is constructed based on a posture estimation algorithm; then, based on a video frame sequence of human limb motion, the coordinates of limb nodes are located through an image processing algorithm, the human limb motion space is estimated in real time, and a limb-control instruction mapping rule between limb motion and the behavior state of the service object is established.

[0017] In the step A, various voice commands are encoded and corresponding robot control commands are established, and voice-control command mapping rules between voice commands and robot behavior control commands are constructed.

[0018] The facial expression intelligent recognition model adopts a Python-based network model structure including sub-network a and sub-network b; sub-network a is a shallow feature extraction part, and the structure of sub-network a is, from top to bottom: sub-network a first feature extraction layer, sub-network a first batch normalization layer, sub-network a second feature extraction layer, sub-network a second batch normalization layer, sub-network a third feature extraction layer, sub-network a third batch normalization layer, sub-network a third random inactivation layer; sub-network b is a residual attention enhancement part, and the structure of sub-network b is, from top to bottom: sub-network b first residual attention layer, sub-network b second residual attention layer, sub-network b third residual attention layer, sub-network b fourth residual attention layer.

[0019] The facial expression intelligent recognition model combines the Softmax loss function and the center loss function in parallel, uses the Softmax loss function for inter-class classification, and the center loss function for intra-class aggregation, and sets the weights of the Softmax loss function and the center loss function to α and β, respectively, α+β=1; the total loss L total It is the weighted sum of two loss functions:

[0020] L total =αL softmax +βL center ;

[0021] Among them, L softmax is the cross entropy loss of the Softmax branch, L center is the loss of the center loss branch.

[0022] In the step D, the robot motion control instructions based on facial expressions, body movements and voice instructions are fused, and the fused information is used to comprehensively analyze the current user's emotional state, body movements and voice instructions, and form robot motion control instructions after multi-behavior fusion with different priority levels.

[0023] In the step D, an intelligent body movement recognition model is used to recognize three actions, namely, falling, covering the chest, and crawling due to illness; a convolutional neural network model is used to recognize seven expressions, namely, anger, neutrality, disgust, fear, happiness, sadness, and surprise, and 1 is used to indicate that the current action is detected, and 0 is used to indicate that the current action is not detected; a weighted voting mechanism is set to assign weights to each posture and expression category; finally, the posture recognition and expression recognition results are weighted and summed to calculate a comprehensive score, and the priority of the robot action control instructions after multi-behavior fusion is determined based on the comprehensive score and the threshold.

[0024] The calculation method of the comprehensive score is:

[0025]

[0026] Among them, Final_score is the comprehensive score, Posture i is the result of the i-th posture recognition, The corresponding i-th posture weight, Expression j is the result of the jth expression recognition, is the corresponding j-th expression weight; if Final_score>threshold, it is judged as the highest priority; if Final_score=threshold, it is judged as the second priority; if Final_score<threshold, it is judged as a normal operation.

[0027] When training the facial expression intelligent recognition model, the following two strategies are used for monitoring:

[0028] (1) Overfitting monitoring: Early stopping strategy, continuously monitor the loss change on the validation set during training. If the loss function does not decrease within 15 consecutive epochs, stop training and restore the model to the weight corresponding to the optimal validation loss;

[0029] (2) Learning rate monitoring: Dynamically adjust the learning rate, with an initial learning rate of 0.01. If the accuracy on the validation set does not improve for 5 consecutive epochs, reduce the learning rate.

[0030] The present invention constructs a facial expression intelligent recognition model, a body movement intelligent recognition model and a voice recognition control protocol set, and respectively establishes expression-control instruction mapping rules between facial expressions and robot behavior control instructions, body movement-control instruction mapping rules between service object behavior states and voice-control instruction mapping rules between voice instructions and robot behavior control instructions; then based on the fusion of the user's facial expressions, body movements and voice instructions, a multi-behavior fused robot action control instruction is formed to control the robot's own execution module to perform the corresponding action. The present invention effectively improves the robot's cognitive ability and situational perception ability, enabling it to provide more accurate, personalized and humane services in complex environments.

[0031] The present invention can also form robot motion control instructions after fusion of multiple behaviors with different priority levels, and use the recognition results of behaviors and facial expression characteristics that endanger the user's life as the robot motion control instructions with the highest priority, immediately start the emergency procedure without waiting for other input confirmation, capture on-site photos, and attach geographic location information, which are quickly sent to preset emergency contacts via the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the process of the present invention;

[0033] Figure 2 Schematic diagram of the residual attention module in the facial expression intelligent recognition model of the present invention;

[0034] Figure 3 This is a network structure diagram of the facial expression intelligent recognition model of the present invention. DETAILED DESCRIPTION

[0035] The present invention is described in detail below with reference to the accompanying drawings and embodiments:

[0036] like Figures 1 to 3 As shown, the robot control method based on multi-behavior fusion intelligent perception of the present invention comprises the following steps in sequence:

[0037] A: Construct facial expression intelligent recognition model, body movement intelligent recognition model and voice recognition control protocol set respectively, and establish expression-control command mapping rules between facial expression and robot behavior control command, body movement and service object behavior state body-control command mapping rules, and voice command and robot behavior control command voice-control command mapping rules respectively;

[0038] In the present invention, a facial expression intelligent recognition model is constructed based on a wavelet neural network and an improved residual convolutional neural network, and the facial expression intelligent recognition model is trained using a facial expression state data set to obtain a trained facial expression intelligent recognition model; and an expression-control instruction mapping rule between facial expressions and robot behavior control instructions is established;

[0039] Based on the posture estimation algorithm, an intelligent limb motion recognition model is constructed; then based on the video frame sequence of human limb motion, the coordinates of limb nodes are located through image processing algorithms, the human limb motion space is estimated in real time, and the limb-control instruction mapping rules between limb motion and the behavior state of the service object are established;

[0040] Construct a speech recognition control protocol set, that is, encode various voice commands and establish corresponding robot control commands, and construct speech-control command mapping rules between voice commands and robot behavior control commands;

[0041] In step A, the established expression-control instruction mapping rules are used to associate corresponding robot actions for various set expression categories. For example, when "smile" is detected, the associated behavior is "greeting or telling a joke", and when "pain" is detected, the associated behavior is "asking the user if he needs help";

[0042] In step A, the posture estimation algorithm uses the pre-trained posture estimation algorithm provided by MediaPipe; human limb movements include waving, covering the chest, falling, crawling due to illness, etc.;

[0043] In step A, the voice commands include "forward", "backward", "turn left", "turn right", etc.;

[0044] B: Use the robot's built-in image acquisition module and voice acquisition module to track and obtain the service object's facial expression images, body movement images and voice audio in real time;

[0045] C: Based on the collected continuous facial expression images of the service object, the facial expression prediction results are output through the trained facial expression intelligent recognition model;

[0046] Based on the collected continuous body movement images of the service object, the body language of the service object's behavior status is output through the body movement intelligent recognition model;

[0047] Based on the collected voice audio of the service object, the voice command of the service object is output through the voice recognition control protocol set;

[0048] D: Using the facial expression prediction results obtained in step C and combining the expression-control instruction mapping rules, obtain the robot action control instructions based on facial expressions;

[0049] Using the body language obtained in step C and combining the body-control instruction mapping rules, obtain the robot motion control instructions based on the body movements;

[0050] Using the voice command obtained in step C and combining the semantic-control command mapping rule, obtain the robot action control command based on the voice command;

[0051] Fuse the robot motion control instructions based on facial expressions, body movements and voice commands to obtain the robot motion control instructions after multi-behavior fusion;

[0052] E: Based on the robot motion control instructions obtained after multi-behavior fusion, the robot control module uses the robot's own execution module to perform the corresponding action.

[0053] In step A of the present invention, when constructing a facial expression state data set, the facial expression state data set is divided into a training set, a validation set and a test set, and data preprocessing and data enhancement are performed; the specific steps are as follows:

[0054] Step A1: Obtain the public facial expression datasets Fer2013 and CK+ datasets. Both datasets contain 7 categories of expression: angry, neutral, disgusted, afraid, happy, sad, and surprised.

[0055] Among them, 80% of the data in the Fer2013 dataset is used as the training dataset, 10% of the data is used as the validation dataset, and 10% of the data is used as the test dataset; the CK+ dataset is divided into training set and test set according to 8:2, and K-fold cross validation is used;

[0056] Step A2: Preprocess the data; extract the face part in the image, resize the image to 48*48 pixels, and perform data augmentation operations such as random scaling, horizontal / vertical translation, random rotation angle, and horizontal mirroring on the image.

[0057] In the step A, the facial expression intelligent recognition model constructed based on wavelet neural network and improved residual convolutional neural network adopts a network model structure based on Python including sub-network a and sub-network b;

[0058] Among them, sub-network a is the shallow feature extraction part, and the structure of sub-network a from top to bottom is: sub-network a first feature extraction layer, sub-network a first batch normalization layer, sub-network a second feature extraction layer, sub-network a second batch normalization layer, sub-network a third feature extraction layer, sub-network a third batch normalization layer, sub-network a third random loss layer;

[0059] The convolution kernel size from the first feature extraction layer of sub-network a to the third feature extraction layer of sub-network a is 3x3, the convolution step is 1, the filling method is the same size filling, and the number of convolution kernel channels is 64.

[0060] Sub-network b is the residual attention enhancement part. The structure of sub-network b from top to bottom is: sub-network b first residual attention layer, sub-network b second residual attention layer, sub-network b third residual attention layer, sub-network b fourth residual attention layer;

[0061] The first residual attention layer of subnetwork b to the fourth residual attention layer of subnetwork b are all composed of residual attention blocks, each residual attention block is composed of two residual attention units, and the trunk path of its residual attention unit structure is a feature extraction layer with a convolution kernel size of 3x3, a feature extraction layer with a convolution kernel size of 3x3, and a CBAM attention mechanism layer; the mask branch is an average pooling layer with a convolution kernel size of 2x2 and a convolution layer with a convolution kernel size of 1x1 for adjusting the number of channels; the output features of the trunk path are added to the features of the mask branch to form a residual connection;

[0062] The residual attention blocks from the first residual attention layer of subnetwork b to the fourth residual attention layer of subnetwork b are 3, 4, 6, and 3 respectively. The number of channels of the convolution kernel is 64, 128, 256, and 512 respectively. The convolution step size is 2 when performing downsampling operations, and the other convolution step sizes are 1.

[0063] The input-output relationship of the residual connection of subnetwork b is:

[0064]

[0065] Among them, h, w and c represent the feature map height, width and number of channels respectively, x a The feature map representing the backbone path, x b Represents the feature map of the mask branch, y sum Represents the feature map after residual connection.

[0066] The intelligent facial expression recognition model uses the wavelet function in the wavelet neural network as the activation function, which can enhance the multi-level and multi-scale understanding of expression features and improve the accuracy and robustness of recognition.

[0067] The input of sub-network a is the face image processed by step A. Sub-network a can deepen the network depth to achieve network capacity and complexity. The input of sub-network b is the output of sub-network a. The CBAM of the backbone path consists of a channel attention module CAM and a spatial attention module SAM, which focus on channel information and spatial information respectively.

[0068] Among them, the channel attention module CAM applies global average pooling and global maximum pooling to the input single feature layer, and then obtains the channel attention weight through a shared multi-layer perceptron. Finally, the sigmoid function is used to limit it between 0 and 1 to obtain the weight of each channel, and multiply it channel by channel with the original feature map; the spatial attention module SAM applies average pooling and maximum pooling to the input single feature layer on the width and height dimensions of the feature map, stacks the results, uses a convolutional layer to adjust the number of channels, and finally uses the sigmoid function to limit it between 0 and 1 to obtain the weight of each feature point, and multiplies it element by element with the original feature map, so as to enhance the network's attention to important features and guide the network to focus more on key features and spatial areas, thereby improving the overall model performance.

[0069] Dropout is added to the output layer of the facial expression intelligent recognition model to suppress overfitting and the activation function of the hidden layer.

[0070] In the present invention, the Softmax loss function and the central loss function are combined in parallel, and the Softmax loss function is used for inter-class classification, and the central loss function is used for intra-class aggregation to achieve excellent results. The weights α and β of the two loss functions are set, where α+β=1.

[0071] Calculate the total loss L total It is the weighted sum of two loss functions:

[0072] Ltotal =αL softmax +βL center

[0073] Among them, L softmax is the cross entropy loss of the Softmax branch, L center is the loss of the Center Loss branch.

[0074] When training the facial expression intelligent recognition model, the SGD method based on Nesterov momentum is used to better update the weights and biases, and the momentum is set to 0.9.

[0075] When training the facial expression intelligent recognition model, the following two strategies are used for monitoring:

[0076] 1. Overfitting monitoring: Early stopping strategy continuously monitors the loss changes on the validation set during training. If the loss function does not decrease within 15 consecutive epochs, training is stopped and the model is restored to the weight corresponding to the optimal validation loss to prevent overfitting and save computing resources.

[0077] 2. Learning rate monitoring: Dynamically adjust the learning rate, with an initial learning rate of 0.01. If the accuracy on the validation set does not improve for 5 consecutive epochs, reduce the learning rate to try to break through the training bottleneck and improve the generalization ability of the model.

[0078] In the step C, when the facial expression prediction result is output by the trained facial expression intelligent recognition model, for each facial expression image of the service object collected, a face detector is first used to detect the face area, then the face area is grayed, and the size of the facial expression image is scaled to 48x48, and finally sent to the trained facial expression intelligent recognition model for prediction, and the trained facial expression intelligent recognition model outputs the predicted expression category.

[0079] In the step C, when the body language of the service object's behavior status is output through the body movement intelligent recognition model, based on the collected continuous body movement images of the service object, the body movement intelligent recognition model uses a 3D human posture estimation algorithm to calculate the angles of human joints, spatial relationships of various body parts and other features based on the extracted human key points, identify specific action patterns and sequences, such as waving to call, covering the chest, falling, and crawling due to illness, and finally outputs the body language of the service object's behavior status.

[0080] In this embodiment, when the intelligent limb movement recognition model identifies emergency action intentions such as covering the chest, falling down, or crawling due to illness, the robot control module immediately executes the emergency response command and transmits the on-site photos with geographic location information to the preset emergency contacts to achieve instant alarm and intervention to ensure the safety of the service recipients.

[0081] In the step D, the robot motion control instructions based on facial expressions, body movements and voice instructions are fused, and the fused information is used to comprehensively analyze the current user's emotional state, body movements and voice instructions, and form robot motion control instructions after multi-behavior fusion with different priority levels.

[0082] Highest priority: When an emergency action intention of covering the chest or crawling due to illness is detected, the emergency procedure is immediately initiated without waiting for other input confirmations, and a photo of the scene is captured with geographic location information, which is quickly sent to the preset emergency contact via the network;

[0083] Second priority: When a fall is detected or a distress expression is recognized, ask the user if they need help to assess whether escalation is required;

[0084] Routine operation: Execution of voice commands (such as "forward", "backward", "turn left", "turn right"), and rapid response and execution of corresponding navigation actions when confirming that the current environment is safe and no emergency has occurred; when a waving action is detected, the robot approaches and enters standby mode, ready to receive further voice commands or conduct other interactions.

[0085] In the step E, according to the robot motion control instructions obtained after the fusion of multiple behaviors, the robot control module uses the robot's own execution module to perform corresponding actions, such as driving the drive motors on the left and right sides to move the robot forward, or adjusting the speed difference of the motors on both sides to make the robot turn left or right.

[0086] In the present invention, the body movement intelligent recognition model is used to identify three key actions: falling, covering the chest and crawling due to illness; the convolutional neural network model is used to identify seven expressions (such as anger, neutral, disgust, fear, happiness, sadness and surprise), and 1 is used to indicate that the current action is detected, and 0 is used to indicate that the current action is not detected. By setting a weighted voting mechanism, weights are assigned to each posture and expression category, and adjusted according to their importance to the final decision.

[0087] In this embodiment, the posture weight is expressed as w_posture, and the expression weight is expressed as w_expression, that is, falling: w_fall=0.5, covering chest: w_chest=0.7, crawling due to illness: w_crawl=0.7, sad expression: w_pain=0.5, and other expressions are all set to 0.2.

[0088] Perform weighted summation on the results of gesture recognition and expression recognition, and the comprehensive score calculation formula is:

[0089]

[0090] Among them, Final_score is the comprehensive score, Posture i is the result of the i-th posture recognition, The corresponding i-th posture weight, Expression j is the result of the jth expression recognition, is the corresponding j-th expression weight; if Final_score>threshold, it is judged as the highest priority; if Final_score=threshold, it is judged as the second priority; if Final_score<threshold, it is judged as a normal operation.

[0091] If the chest is covered (posture_fall=1) and the expression of pain (expression_pain=1) is detected, the final score is: Final_score=(0.7x1)+(0.5x1)=1.2. Then, according to the size of Final_score, a threshold P is set to determine the priority level. If Final_score>P, it is judged as a serious situation and the highest priority. The emergency procedure is immediately initiated without waiting for other input confirmations. The scene photo is captured and the geographic location information is attached, which is quickly sent to the preset emergency contact through the network; if Final_score=P, it is a secondary priority; ask the user if he needs help to evaluate whether it needs to be upgraded; if Final_score<P, it is a routine operation. The value of P can be 1.0.

Claims

1. A robot control method based on multi-behavior fusion intelligent perception, characterized in that: The following steps are involved: A: Construct facial expression intelligent recognition model, body movement intelligent recognition model and voice recognition control protocol set respectively, and establish expression-control command mapping rules between facial expression and robot behavior control command, body movement and service object behavior state body-control command mapping rules, and voice command and robot behavior control command voice-control command mapping rules respectively; B: Use the robot's built-in image acquisition module and voice acquisition module to track and obtain the service object's facial expression images, body movement images and voice audio in real time; C: Based on the collected continuous facial expression images of the service object, the facial expression prediction results are output through the trained facial expression intelligent recognition model; Based on the collected continuous body movement images of the service object, the body language of the service object's behavior status is output through the body movement intelligent recognition model; Based on the collected voice audio of the service object, the voice command of the service object is output through the voice recognition control protocol set; D: Using the obtained facial expression prediction results, body language and voice commands, respectively, the expression-control command mapping rules, body-control command mapping rules and semantic-control command mapping rules are combined to obtain the robot action control commands based on facial expressions, the robot action control commands based on body movements and the robot action control commands based on voice commands; then the robot action control commands based on facial expressions, body movements and voice commands are fused to obtain the robot action control commands after multi-behavior fusion; E: Based on the robot motion control instructions obtained after multi-behavior fusion, the robot control module uses the robot's own execution module to perform the corresponding action.

2. The robot control method based on multi-behavior fusion intelligent perception according to claim 1 is characterized in that: In the step A, a facial expression intelligent recognition model is constructed based on a wavelet neural network and an improved residual convolutional neural network, and the facial expression intelligent recognition model is trained using a facial expression state data set to obtain a trained facial expression intelligent recognition model; And establish expression-control instruction mapping rules between facial expressions and robot behavior control instructions.

3. The robot control method based on multi-behavior fusion intelligent perception according to claim 1 is characterized in that: In step A, an intelligent limb motion recognition model is constructed based on a posture estimation algorithm; then, based on a video frame sequence of human limb motion, the coordinates of limb nodes are located through an image processing algorithm, the human limb motion space is estimated in real time, and a limb-control instruction mapping rule between limb motion and the behavior state of the service object is established.

4. The robot control method based on multi-behavior fusion intelligent perception according to claim 1 is characterized in that: In the step A, various voice commands are encoded and corresponding robot control commands are established, and voice-control command mapping rules between voice commands and robot behavior control commands are constructed.

5. The robot control method based on multi-behavior fusion intelligent perception according to claim 2 is characterized in that: The facial expression intelligent recognition model adopts a Python-based network model structure including sub-network a and sub-network b; sub-network a is a shallow feature extraction part, and the structure of sub-network a is, from top to bottom: sub-network a first feature extraction layer, sub-network a first batch normalization layer, sub-network a second feature extraction layer, sub-network a second batch normalization layer, sub-network a third feature extraction layer, sub-network a third batch normalization layer, sub-network a third random inactivation layer; sub-network b is a residual attention enhancement part, and the structure of sub-network b is, from top to bottom: sub-network b first residual attention layer, sub-network b second residual attention layer, sub-network b third residual attention layer, sub-network b fourth residual attention layer.

6. The robot control method based on multi-behavior fusion intelligent perception according to claim 2 is characterized by: The facial expression intelligent recognition model combines the Softmax loss function and the center loss function in parallel, uses the Softmax loss function for inter-class classification, and the center loss function for intra-class aggregation, and sets the weights of the Softmax loss function and the center loss function to α and β, respectively, α+β=1; the total loss L total It is the weighted sum of two loss functions: L total =αL softmax +βL center ; Among them, L softmax is the cross entropy loss of the Softmax branch, L center is the loss of the center loss branch.

7. The robot control method based on multi-behavior fusion intelligent perception according to claim 1 is characterized by: In the step D, the robot motion control instructions based on facial expressions, body movements and voice instructions are fused, and the fused information is used to comprehensively analyze the current user's emotional state, body movements and voice instructions, and form robot motion control instructions after multi-behavior fusion with different priority levels.

8. The robot control method based on multi-behavior fusion intelligent perception according to claim 7 is characterized in that: In the step D, an intelligent body movement recognition model is used to recognize three actions, namely, falling, covering the chest, and crawling due to illness; a convolutional neural network model is used to recognize seven expressions, namely, anger, neutrality, disgust, fear, happiness, sadness, and surprise, and 1 is used to indicate that the current action is detected, and 0 is used to indicate that the current action is not detected; a weighted voting mechanism is set to assign weights to each posture and expression category; finally, the posture recognition and expression recognition results are weighted and summed to calculate a comprehensive score, and the priority of the robot action control instructions after multi-behavior fusion is determined based on the comprehensive score and the threshold.

9. The robot control method based on multi-behavior fusion intelligent perception according to claim 7 is characterized in that: The calculation method of the comprehensive score is: Among them, Final_score is the comprehensive score, Posture i is the result of the i-th posture recognition, The corresponding i-th posture weight, Expression j is the result of the jth expression recognition, is the corresponding j-th expression weight; if Final_score>threshold, it is judged as the highest priority; if Final_score=threshold, it is judged as the second priority; if Final_score<threshold, it is judged as a normal operation. .

10. The robot control method based on multi-behavior fusion intelligent perception according to claim 2 is characterized in that: When training the facial expression intelligent recognition model, the following two strategies are used for monitoring: (1) Overfitting monitoring: Early stopping strategy, continuously monitor the loss change on the validation set during training. If the loss function does not decrease within 15 consecutive epochs, stop training and restore the model to the weight corresponding to the optimal validation loss; (2) Learning rate monitoring: Dynamically adjust the learning rate. The initial learning rate is 0.

01. If the accuracy on the validation set does not improve for five consecutive epochs, the learning rate will be reduced.

Images