A control method, system, device and medium of a humanoid robot

Abstract

The application discloses a kind of humanoid robot control method, system, equipment and medium, wherein, the method obtains control decision model and the multi-source environment perception data of humanoid robot in task scene, and the electroencephalogram data of target user in task scene;The multi-source environment perception data and the electroencephalogram data are input into trained state analysis model and are analyzed, obtain the state evaluation result output by the trained state analysis model, and the state evaluation result is used to indicate the emotion category and fatigue grade of the target user currently;The state evaluation result and the multi-source environment perception data are input into the control decision model and are decided to generate, obtain the action control instruction of the humanoid robot;According to the action control instruction, the collaborative action control of the humanoid robot is carried out.The control method can effectively improve the safety and efficiency of man-machine cooperation.The application relates to the technical field of man-machine cooperation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer collaboration, and in particular to a control method, system, device and medium for a humanoid robot. BACKGROUND

[0002] With the increasing application of humanoid robots in industrial production lines, warehouse logistics, high-risk operations and other task scenarios, the collaborative control of humanoid robots and users has become one of the focuses of attention.

[0003] Currently, the related technology usually makes decision control on the actions to be performed by the humanoid robot in the human-robot collaboration process based on a single environmental parameter (such as a distance parameter) collected by the humanoid robot. This method often cannot accurately respond to and collaborate with the actions of the staff, resulting in certain safety risks for the staff, and the safety and efficiency of human-machine collaboration are not satisfactory.

[0004] Therefore, the problems of the related technology still need to be solved and optimized. SUMMARY

[0005] The present application aims to at least partially solve one of the technical problems in the related art.

[0006] To this end, an object of an embodiment of the present application is to provide a control method, system, device and medium for a humanoid robot, wherein the control method can effectively improve the safety and efficiency of human-machine collaboration.

[0007] In order to achieve the above technical purpose, the technical solutions adopted by the embodiments of the present application include:

[0008] In a first aspect, the present application provides a control method for a humanoid robot, comprising:

[0009] obtaining a control decision model and multi-source environmental perception data of a humanoid robot in a task scenario, and electroencephalogram data of a target user in the task scenario;

[0010] inputting the multi-source environmental perception data and the electroencephalogram data into a trained state analysis model for analysis to obtain a state evaluation result output by the trained state analysis model, the state evaluation result being used to indicate the current emotional category and fatigue level of the target user;

[0011] inputting the state evaluation result and the multi-source environmental perception data into the control decision model for decision generation to obtain an action control instruction for the humanoid robot;

[0012] collaboratively controlling the humanoid robot according to the action control instruction.

[0013] In addition, the method according to the above-mentioned embodiment of the present application can further have the following additional technical features:

[0014] Further, in an embodiment of the present application, the step of inputting the multi-source environment perception data and the electroencephalogram data into the trained state analysis model for analysis to obtain the state evaluation result output by the trained state analysis model comprises:

[0015] obtaining an environmental noise feature matrix;

[0016] performing collaborative filtering on the electroencephalogram data according to the environmental noise feature matrix to obtain electroencephalogram modality data;

[0017] performing data standardization on the multi-source environment perception data to obtain environment modality data;

[0018] performing time alignment on the electroencephalogram modality data and the environment modality data to obtain multi-modality data;

[0019] inputting the multi-modality data into the trained state analysis model for analysis to obtain the state evaluation result.

[0020] Further, in an embodiment of the present application, the step of inputting the multi-modality data into the trained state analysis model for analysis to obtain the state evaluation result comprises:

[0021] performing spatial feature extraction on the multi-modality data to obtain electroencephalogram spatial topology features and environment spatial topology features;

[0022] performing time series feature extraction on the electroencephalogram spatial topology features to obtain electroencephalogram space-time features, and performing time series feature extraction on the environment spatial topology features to obtain environment space-time features;

[0023] performing feature attention fusion on the electroencephalogram space-time features and the environment space-time features to obtain fusion features;

[0024] performing double-branch feature analysis on the fusion features to obtain the state evaluation result.

[0025] Further, in an embodiment of the present application, the step of performing time series feature extraction on the target spatial topology features to obtain target space-time features comprises:

[0026] obtaining a first task weight matrix;

[0027] performing dynamic scheduling on the target spatial topology features according to the first task weight matrix to obtain a feature scheduling strategy corresponding to the target spatial topology features;

[0028] According to the feature scheduling strategy, the target space topology feature is subjected to policy feature extraction to obtain the target space-time feature.

[0029] If the target space topology feature is an electroencephalogram space topology feature, the target space-time feature is an electroencephalogram space-time feature; or if the target space topology feature is an environment space topology feature, the target space-time feature is an environment space-time feature.

[0030] Further, in an embodiment of the present application, the obtaining of the first task weight matrix comprises:

[0031] obtaining a second task weight matrix, and a task completion degree and a standard completion degree of an execution task corresponding to the second task weight matrix, the second task weight matrix being a previous task weight matrix of the first task weight matrix;

[0032] According to the task completion degree, the standard completion degree is subjected to completion degree loss analysis to obtain task loss data of the execution task;

[0033] According to the task loss data, the second task weight matrix is subjected to dynamic weight updating to obtain the first task weight matrix.

[0034] Further, in an embodiment of the present application, the trained state analysis model is obtained through the following steps:

[0035] obtaining multi-modal training data and multi-source labels of the multi-modal training data, the multi-modal training data comprising environment training data and electroencephalogram training data, and the multi-source labels comprising emotion labels and fatigue labels;

[0036] inputting the multi-modal training data into a current state analysis model for analysis to obtain training data analysis results;

[0037] According to the training data analysis results and the multi-source labels, the current state analysis model is subjected to parameter updating to obtain the trained state analysis model.

[0038] Further, in an embodiment of the present application, the inputting of the state evaluation result and the multi-source environment perception data into the control decision model for decision generation to obtain the action control instruction of the humanoid robot comprises:

[0039] obtaining an action execution rule of the humanoid robot;

[0040] inputting the state evaluation result and the multi-source environment perception data into the control decision model for coefficient analysis to obtain a decision coefficient;

[0041] According to the decision coefficient, the action execution rule is matched with the coefficient to obtain the action control instruction.

[0042] In a second aspect, the embodiments of the present application provide a control system of a humanoid robot, comprising:

[0043] A first processing unit is configured to acquire a control decision model and multi-source environment perception data of the humanoid robot in a task scene and electroencephalogram data of a target user in the task scene.

[0044] A second processing unit is configured to input the multi-source environment perception data and the electroencephalogram data into a trained state analysis model for analysis to obtain a state evaluation result output by the trained state analysis model, the state evaluation result being used to indicate an emotional category and a fatigue level of the target user at present.

[0045] A third processing unit is configured to input the state evaluation result and the multi-source environment perception data into the control decision model for decision generation to obtain an action control instruction of the humanoid robot.

[0046] A fourth processing unit is configured to perform collaborative action control on the humanoid robot according to the action control instruction.

[0047] In a third aspect, the embodiments of the present application further provide an electronic device, comprising:

[0048] At least one processor;

[0049] At least one memory configured to store at least one program;

[0050] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0051] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, wherein a processor executable program is stored, and the processor executable program is used to implement the method described above when executed by the processor.

[0052] The advantages and beneficial effects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be known by the practice of the present application:

[0053] A control method, system, device and medium of a humanoid robot are disclosed, wherein the method obtains a control decision model and multi-source environment perception data of the humanoid robot in a task scene, and electroencephalogram data of a target user in the task scene; inputs the multi-source environment perception data and the electroencephalogram data into a trained state analysis model for analysis to obtain a state evaluation result output by the trained state analysis model, the state evaluation result being used to indicate an emotion category and a fatigue level of the target user currently; inputs the state evaluation result and the multi-source environment perception data into the control decision model for decision generation to obtain an action control instruction of the humanoid robot; and performs cooperative action control on the humanoid robot according to the action control instruction. The method analyzes the multi-source environment perception data and the electroencephalogram data through the state analysis model, and generates a decision based on the state evaluation result and the multi-source environment perception data, which can realize multi-modal fusion decision of the target user state and environmental factors, so that the obtained action control instruction can better adapt to the action of different workers, which is beneficial to improve the efficiency and effect of the humanoid robot in responding to the action of the cooperative worker, and thus effectively improves the safety and efficiency of human-machine cooperation. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of expressing part of the embodiments of the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0055] Figure 1 A flowchart of a control method of a humanoid robot is provided for the embodiments of the present application.

[0056] Figure 2 A frame diagram of a control system of a humanoid robot is provided for the embodiments of the present application.

[0057] Figure 3 A structural diagram of an electronic device is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0058] Embodiments of the present application are described below in detail, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary only, and are for the purpose of explanation only, and are not to be understood as limiting the present application. For the step numbers in the following embodiments, they are only set for the convenience of setting out the description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application is for the purpose of describing embodiments of this application only, and is not intended to be limiting of this application.

[0060] At present, the related technology is usually based on a single environmental parameter (such as a distance parameter) collected by a humanoid robot to make decision control on the action to be performed by the humanoid robot in the human-robot collaboration process. This kind of way often cannot accurately respond and cooperate with the action of the worker, and cannot perceive the physiological and psychological state of the worker, so that when the worker appears in a state of restlessness, fatigue, etc., the humanoid robot still maintains the original work rhythm, which is easy to lead to a decrease in collaboration efficiency and even a safety accident, and the worker has a certain safety risk, and the safety and efficiency of human-robot collaboration are not satisfactory.

[0061] In addition, the related technology usually applies electroencephalogram data to the medical rehabilitation field, and in an industrial strong electromagnetic interference environment, the signal stability of the electroencephalogram data is poor (such as the signal stability of the electroencephalogram data collected by a certain existing brain-computer interface device in a dependent scene can reach 90%, while the signal stability of the electroencephalogram data collected in an industrial strong electromagnetic interference environment is less than 60%), and a closed loop is not formed with robot action control.

[0062] It should be noted that the above-mentioned related technology is only used to assist in understanding the technical solutions of the present application, and does not mean that it belongs to the prior art disclosed.

[0063] Therefore, the embodiments of the present application provide a control method, system, device and medium for a humanoid robot, wherein the method analyzes multi-source environmental perception data and electroencephalogram data through a state analysis model, and generates a decision based on a state evaluation result and the multi-source environmental perception data, which can realize multi-modal fusion decision of the target user state and environmental factors, so that the obtained action control instruction can better adapt to the action of different workers, which is beneficial to improve the efficiency and effect of the humanoid robot in responding to the action of the worker, and thus effectively improve the safety and efficiency of human-robot collaboration.

[0064] Further, the method performs collaborative filtering on the electroencephalogram data through an environmental noise feature matrix, and introduces a task weight matrix to schedule electroencephalogram space-time features or environmental space-time features, which can improve the anti-interference ability and feature extraction efficiency of the electroencephalogram signal in an industrial environment, and then realize accurate recognition of the personnel state, and effectively improve the effect of human-machine cooperation.

[0065] With reference to Figure 1 In an embodiment of the present application, a control method of a humanoid robot comprises:

[0066] In step 110, a control decision model and multi-source environment perception data of the humanoid robot in a task scene, and electroencephalogram data of a target user in the task scene are acquired.

[0067] In an embodiment of the present application, the control decision model is a model for generating decision control parameters; the task scene can be an industrial scene requiring close human-machine cooperation, such as intelligent manufacturing (such as automobile assembly and electronic component quality inspection), dangerous operation assistance (such as chemical equipment maintenance and high-altitude operation cooperation), etc.

[0068] It can be understood that the multi-source environment perception data of the humanoid robot can include relative distance data acquired by a laser radar carried by the humanoid robot, worker body temperature data collected by an infrared sensor, and work environment decibel value data recorded by a sound sensor, etc., wherein the relative distance data is used to indicate the relative distance between the worker and the humanoid robot. The target user can be a worker in the task scene, and the electroencephalogram data can be obtained by collecting the electroencephalogram signal of the worker through an electroencephalogram helmet.

[0069] In step 120, the multi-source environment perception data and the electroencephalogram data are input into a trained state analysis model for analysis, to obtain a state evaluation result output by the trained state analysis model, wherein the state evaluation result is used to indicate the current emotional category and fatigue level of the target user.

[0070] In an embodiment of the present application, the multi-source environment perception data and the electroencephalogram data can be input into a pre-trained state analysis model together, to learn the multi-modal data through the state analysis model and predict the psychological and physical state of the target user, to obtain the state evaluation result.

[0071] In some embodiments, the inputting of the multi-source environment perception data and the electroencephalogram data into the trained state analysis model for analysis to obtain the state evaluation result output by the trained state analysis model comprises:

[0072] An environmental noise feature matrix is acquired;

[0073] The electroencephalogram data is collaboratively filtered according to the environmental noise feature matrix, to obtain electroencephalogram modality data.

[0074] data standardization is performed on the multi-source environment perception data to obtain environment modality data;

[0075] time alignment is performed on the electroencephalogram modality data and the environment modality data to obtain multi-modality data;

[0076] In the embodiments of the present application, the environmental noise feature matrix can be a noise signal of a typical electromagnetic interference source (such as a 220V device) in a task scene, which is constructed after Fourier transform feature extraction. Collaborative filtering can first be to use a 50Hz notch filter to eliminate power frequency interference in the electroencephalogram data, and use a 2-50Hz band-pass filter to retain the effective frequency band in the electroencephalogram data. Then, the improved second-order blind identification algorithm is used to perform matched filtering on the electroencephalogram data. The improved second-order blind identification algorithm uses the environmental noise feature matrix in the whitening process to perform matched filtering on the noise components in the electroencephalogram data, and then obtains the electroencephalogram modality data through signal reconstruction and feature extraction. Specifically, the improved second-order blind identification algorithm in the whitening process can be represented as:

[0077]

[0078] wherein, is the matched filtered electroencephalogram data; is a parameter variable; is the electroencephalogram data before matched filtering; is the environmental noise feature matrix; is the noise signal of the electroencephalogram data before matched filtering.

[0079] It can be understood that data standardization can be feature normalization of relative distance data, text data, and decibel value data in multi-source environment perception data, thereby obtaining multi-source environment modality data. Then, according to the time stamp of the electroencephalogram modality data and the time stamp of the environment modality data, time alignment is performed on the electroencephalogram modality data and the environment modality data, and the aligned electroencephalogram modality data and the environment modality data are determined as multi-modality data.

[0080] The multi-modality data is input into the trained state analysis model for analysis to obtain the state evaluation result.

[0081] In some embodiments, the inputting the multi-modality data into the trained state analysis model for analysis to obtain the state evaluation result comprises:

[0082] spatial feature extraction is performed on the multi-modality data to obtain electroencephalogram spatial topology features and environment spatial topology features;

[0083] extracting time sequence features from the brain electrical spatial topology features to obtain brain electrical space-time features, and extracting time sequence features from the environment spatial topology features to obtain environment space-time features;

[0084] performing feature attention fusion on the brain electrical space-time features and the environment space-time features to obtain fusion features;

[0085] performing double-branch feature analysis on the fusion features to obtain the state evaluation result.

[0086] In the embodiments of the present application, the spatial feature extraction can be inputting the multi-modal data into a graph neural network (GCN), and extracting the dependency relationship and topology structure features between the nodes of the aligned brain electrical modal data and environment modal data through the graph convolution kernel in the graph neural network, so as to obtain the brain electrical spatial topology features and the environment spatial topology features. The time sequence feature extraction can be inputting the brain electrical spatial topology features and the environment spatial topology features into an improved neural gateway control unit (NGCU) respectively, so as to obtain the brain electrical space-time features and the environment space-time features.

[0087] It can be understood that the feature attention fusion can be based on attention records, and the attention weights of the brain electrical space-time features and the environment space-time features are calculated respectively, and the brain electrical space-time features and the environment space-time features are fused by weighted summation or the like based on all the calculated attention weights, so as to obtain the fusion features. The double-branch feature analysis can be using a double-branch output layer to classify and evaluate the fusion features, wherein the double-branch output layer can output emotion categories representing the psychological state of the target user and fatigue levels representing the physical state of the target user through Softmax activation function and Sigmoid activation function respectively.

[0088] In some embodiments, the time sequence feature extraction on the target spatial topology features to obtain target space-time features comprises:

[0089] obtaining a first task weight matrix;

[0090] Further, the obtaining the first task weight matrix comprises:

[0091] obtaining a second task weight matrix, and a task completion degree and a standard completion degree of a task performed corresponding to the second task weight matrix, the second task weight matrix being a previous task weight matrix of the first task weight matrix;

[0092] performing completion degree loss analysis on the standard completion degree according to the task completion degree to obtain task loss data of the task performed;

[0093] According to the task loss data, the second task weight matrix is dynamically updated to obtain the first task weight matrix.

[0094] In the embodiments of the present application, the current task type weight matrix corresponding to the target space topology feature can be denoted as a first task weight matrix, which can be dynamically updated from a previous task weight matrix (i.e., a second task weight matrix). The task completion degree is used to represent the actual completion of the humanoid robot in executing the task, which can be quantified by the ratio of the actual completion time to the standard completion time. The completion degree is used to represent the theoretical completion of the humanoid robot in executing the task.

[0095] It can be understood that the completion degree loss analysis can input the task completion degree and the standard completion degree into a task loss function for calculation and analysis, which can be a mean square error loss function, so as to obtain task loss data. Then, the dynamic weight update can update the second task weight matrix based on the task loss data to obtain the first task weight matrix, which can be represented as:

[0096]

[0097] wherein, is the first task weight matrix, which can be a task type weight matrix at a time point ; is the second task weight matrix, which can be a task type weight matrix at a time point ; is a learning rate parameter; is the gradient of the task loss data at the time point .

[0098] According to the first task weight matrix, the target space topology feature is dynamically scheduled to obtain a feature scheduling strategy corresponding to the target space topology feature;

[0099] According to the feature scheduling strategy, the target space topology feature is extracted to obtain the target space-time feature;

[0100] If the target space topology feature is an electroencephalogram space topology feature, the target space-time feature is an electroencephalogram space-time feature. Or, if the target space topology feature is an environment space topology feature, the target space-time feature is an environment space-time feature.

[0101] In this embodiment, dynamic scheduling can involve inputting the target space topological features into the improved neural gateway control unit, which then learns the importance and correlation of these features and balances the allocation of computational resources to ensure that key features are prioritized. Specifically, the improved neural gateway control unit determines a feature scheduling strategy corresponding to the target space topological features based on a first task weight matrix and a forgetting gate mechanism. This feature scheduling strategy records the scheduling order and computational resources of each feature in the target space topological features. The forgetting gate factor corresponding to this feature scheduling strategy can be expressed as:

[0102]

[0103] in, Forget gate factor; Use the Sigmoid activation function; This is the EEG feature-forget gate weight matrix, used to hide the previous time step's state. The computational space mapped to the forgetting gate determines the strength of the influence of historical EEG states on the current forgetting operation; This refers to the network's hidden state at the previous moment, including the timeframe up to [the previous moment]. At any given moment, the feature representation is synthesized after processing by the graph neural network (GCN) and the neural gateway control unit (NGCU); This is the current input-forget gate weight matrix, used to weight the input at the current time step. Mapped to the forget gate, this determines the strength of the current input's influence on the forget operation; For is Multi-source environmental perception data and / or EEG data collected in real time; The encoded representation of the currently executing task, specifically at a point in time. The encoded representation of the task being executed; This is the forget gate bias term, used to adjust the activation threshold of the forget gate, providing a basic offset for the network, making forget gate calculation more flexible, and avoiding output limitations caused by linear combination of weight matrices.

[0104] It can be understood that the policy feature extraction can be based on the feature scheduling policy of the improved neural gateway control unit to perform feature extraction on the target space topology features, so as to obtain target space-time features. There are various specific feature extraction methods, which will not be repeated here. In addition, after determining the forgetting gate factor, there are various ways to determine the feature scheduling policy, such as performing table lookup mapping on the policy rule table based on the forgetting gate factor to obtain the feature scheduling policy based on the preset policy rule table; or, the relative size of each feature data in the target space topology features and the required computing resources can be used to generate the feature scheduling policy, which will not be repeated here.

[0105] In some embodiments, the trained state analysis model is obtained by the following steps:

[0106] Obtain multi-modal training data and multi-source labels of the multi-modal training data, the multi-modal training data including environmental training data and electroencephalogram training data, and the multi-source labels including emotion labels and fatigue labels;

[0107] Input the multi-modal training data into a current state analysis model for analysis to obtain training data analysis results;

[0108] According to the training data analysis results and the multi-source labels, update the parameters of the current state analysis model to obtain the trained state analysis model.

[0109] In the embodiments of the present application, the historical electroencephalogram data and multi-source environmental perception data of the target user can be obtained and recorded as multi-modal training data. Then, the multi-modal training data is input into the current state analysis model, and the emotion category and fatigue level of the target user are predicted by the current state analysis model, which is recorded as the training data analysis result. Then, by measuring the consistency between the training data analysis result and the multi-source label, the target loss value is calculated by the loss function. The specific loss function used can be 0-1 loss function, square loss function, absolute loss function, logarithmic loss function, cross-entropy loss function, etc., which is not limited by the present application. Based on the trained target loss value, the parameters of the model are updated using the back propagation algorithm, and after several iterations, the trained state analysis model can be obtained. The specific number of iterations can be pre-set, or the training can be considered complete when the test set reaches the accuracy requirement.

[0110] Step 130, input the state evaluation result and the multi-source environmental perception data into the control decision model for decision generation to obtain the motion control instruction of the humanoid robot;

[0111] In the embodiments of the present application, the state evaluation result and the multi-source environment perception data can be input into the control decision model together, and the action control instruction for the humanoid robot to execute is generated through the control decision model.

[0112] In some embodiments, the state evaluation result and the multi-source environment perception data are input into the control decision model for decision generation, and the action control instruction for the humanoid robot is obtained, including:

[0113] Obtaining an action execution rule of the humanoid robot;

[0114] Inputting the state evaluation result and the multi-source environment perception data into the control decision model for coefficient analysis to obtain a decision coefficient;

[0115] According to the decision coefficient, the action execution rule is matched with the coefficient to obtain the action control instruction.

[0116] In the embodiments of the present application, the action execution rule can be a rule set for adjusting the action executed by the humanoid robot. For example, the action execution rule includes a first execution rule, a second execution rule and a third execution rule. The first execution rule can be a rule for controlling the humanoid robot to execute the action at a normal speed; the second execution rule can be a rule for controlling the humanoid robot to execute the action at a speed reduced by 30% and a force reduced by 20%; and the third execution rule can be a rule for controlling the humanoid robot to stop the action.

[0117] It can be understood that the coefficient analysis can be inputting the state evaluation result and the multi-source environment perception data into the control decision model, and the decision coefficient is obtained through the analysis and calculation of the control decision model. The decision coefficient of the first embodiment can be represented as:

[0118]

[0119] wherein, is the decision coefficient; , and is an adjustable parameter; is an emotion category code in the state evaluation result; is a maximum emotion category code value; is a fatigue level code in the state evaluation result; is a maximum fatigue level code value; is relative distance data in the multi-source environment perception data; is a maximum value of the relative distance data.

[0120] It should be noted that the decision coefficient in the embodiments of the present application can be various, for example, the decision coefficient in the second embodiment can be expressed as:

[0121]

[0122] wherein, and are adjustable parameters; is the decibel value data in the multi-source environmental perception data; is the maximum value of the decibel value data; is the body temperature data in the multi-source environmental perception data; is the maximum value of the body temperature data.

[0123] It is worth mentioning that the coefficient matching can be based on the obtained decision coefficient, and the threshold range corresponding to each specific rule in the action execution rule is matched, so as to obtain the action control instruction. Specifically, if the decision coefficient is less than the first threshold value, the action control instruction can be generated based on the first execution rule; or, if the decision coefficient is greater than or equal to the first threshold value and less than the second threshold value, the action control instruction can be generated based on the second execution rule; or, if the decision coefficient is greater than or equal to the second threshold value, the action control instruction can be generated based on the third execution rule.

[0124] It should be noted that the specific values of the first threshold value and the second threshold value can be set according to actual conditions, for example, the first threshold value can be 0.3, and the second threshold value can be 0.8, which will not be repeated here. In addition, in actual application, the number of specific execution rules protected by the action execution rule can also be set according to actual conditions. The first execution rule, the second execution rule and the third execution rule shown in the examples of the present application are simple examples, which are only for the convenience of understanding, and do not limit the specific execution rules and the corresponding threshold range.

[0125] Step 140, according to the action control instruction, the collaborative action control of the humanoid robot is performed.

[0126] In the embodiments of the present application, after obtaining the action control instruction, the execution action of the humanoid robot in executing the task can be adjusted and controlled to realize the safe and efficient human-machine collaborative work of the humanoid robot and the target user.

[0127] A control system of a humanoid robot according to an embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0128] Referring to Figure 2 , the control system of the humanoid robot according to an embodiment of the present application comprises:

[0129] The first processing unit 101 is configured to acquire a control decision model and multi-source environment perception data of a humanoid robot in a task scene and electroencephalogram data of a target user in the task scene.

[0130] The second processing unit 102 is configured to input the multi-source environment perception data and the electroencephalogram data into a trained state analysis model for analysis to obtain a state evaluation result output by the trained state analysis model, the state evaluation result being used to indicate an emotional category and a fatigue level of the target user at present.

[0131] The third processing unit 103 is configured to input the state evaluation result and the multi-source environment perception data into the control decision model for decision generation to obtain a motion control instruction of the humanoid robot.

[0132] The fourth processing unit 104 is configured to perform cooperative motion control on the humanoid robot according to the motion control instruction.

[0133] It can be understood that the contents in the above method embodiments are all applicable to the present system embodiment, the present system embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0134] With reference to Figure 3 The present application embodiment further provides an electronic device, comprising:

[0135] at least one processor 201;

[0136] at least one memory 202 configured to store at least one program;

[0137] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the above method embodiments.

[0138] Similarly, it can be understood that the contents in the above method embodiments are all applicable to the present device embodiment, the present device embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0139] The present application embodiment further provides a computer readable storage medium, wherein a program executable by a processor 201 is stored, and the program executable by the processor 201 is used to implement the above method embodiments when executed by the processor 201.

[0140] Similarly, the contents in the method embodiments described above are applicable to the computer readable storage medium embodiments, the computer readable storage medium embodiments specifically implement the functions of the method embodiments described above, and achieve the same beneficial effects as the method embodiments described above.

[0141] The embodiments of the present application further provide a computer program product, comprising a computer program which, when executed by a processor, implements the steps in the method embodiments described above.

[0142] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the computer program can include the processes of the above-mentioned embodiments.

[0143] In some alternative embodiments, the functions / operations mentioned in the block diagram can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially simultaneously or the blocks can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flowcharts of the present application are provided by way of example, and the purpose is to provide a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical processes presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are independently executed.

[0144] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It can also be understood that a detailed discussion of the actual implementation of each module is unnecessary for an understanding of the present application. Rather, given the properties, functions and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be within the routine skill of an engineer, given the properties, functions and internal relationships of the various functional modules in the devices disclosed herein. Therefore, those skilled in the art can implement the present application as set forth in the claims without undue experimentation using ordinary skill. It can also be understood that the disclosed specific concepts are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the appended claims and their equivalents.

[0145] If the functions are implemented in software, the functions can be stored in or implemented as one or more instructions or code on a computer-readable medium. Computer-readable media include both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage medium can be any available medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, or twisted pair, then the coaxial cable, fiber optic cable, or twisted pair are included in the definition of medium. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), and Blu-Ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0146] In other words, like a human driver of a vehicle, the autonomous vehicle 100 can be programmed to follow traffic laws and rules of the road, and to make decisions based on its programming and the information it receives from its sensors and other sources. The autonomous vehicle 100 can also be programmed to make decisions based on its programming and the information it receives from its sensors and other sources, even if those decisions are not in accordance with traffic laws and rules of the road. For example, the autonomous vehicle 100 can be programmed to avoid a collision with another vehicle, even if doing so would violate a traffic law or rule of the road.

[0147] In other words, like a human driver of a vehicle, the autonomous vehicle 100 can be programmed to follow traffic laws and rules of the road, and to make decisions based on its programming and the information it receives from its sensors and other sources. The autonomous vehicle 100 can also be programmed to make decisions based on its programming and the information it receives from its sensors and other sources, even if those decisions are not in accordance with traffic laws and rules of the road. For example, the autonomous vehicle 100 can be programmed to avoid a collision with another vehicle, even if doing so would violate a traffic law or rule of the road.

[0148] It should be understood that portions of the application can be realized with hardware, software, firmware or a combination thereof. In the foregoing description, multiple steps or methods can be realized as software or firmware to be executed by a suitable instruction executing system. For example, if realized with hardware, and as in another embodiment, any one or a combination of the following technologies known in the art can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0149] In the above description of the present specification, the description of the terms "one embodiment", "another embodiment", or "certain embodiments" or the like means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0150] Although the embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made thereto without departing from the principles and spirit of the present application, the scope of which is defined by the claims and their equivalents.

[0151] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present application, and these equivalent modifications or substitutions are included in the scope defined by the claims of the present application.

Claims

1. A control method of a humanoid robot, characterized by, The method comprises the following steps: obtaining a control decision model and multi-source environment perception data of a humanoid robot in a task scene, and electroencephalogram data of a target user in the task scene; inputting the multi-source environment perception data and the electroencephalogram data into a trained state analysis model for analysis to obtain a state evaluation result output by the trained state analysis model, the state evaluation result being used to indicate an emotional category and a fatigue level of the target user at present; inputting the state evaluation result and the multi-source environment perception data into the control decision model for decision generation to obtain a motion control instruction of the humanoid robot; controlling the motion of the humanoid robot according to the motion control instruction; the step of inputting the multi-source environment perception data and the electroencephalogram data into the trained state analysis model for analysis to obtain the state evaluation result output by the trained state analysis model comprises the following steps: obtaining an environmental noise feature matrix; performing collaborative filtering on the electroencephalogram data according to the environmental noise feature matrix to obtain electroencephalogram modality data; performing data standardization on the multi-source environment perception data to obtain environmental modality data; time-aligning the electroencephalogram modality data and the environmental modality data to obtain multi-modal data; inputting the multi-modal data into the trained state analysis model for analysis to obtain the state evaluation result; the step of inputting the multi-modal data into the trained state analysis model for analysis to obtain the state evaluation result comprises the following steps: performing spatial feature extraction on the multi-modal data to obtain electroencephalogram spatial topology features and environmental spatial topology features; performing time sequence feature extraction on the electroencephalogram spatial topology features to obtain electroencephalogram space-time features, and performing time sequence feature extraction on the environmental spatial topology features to obtain environmental space-time features; performing feature attention fusion on the electroencephalogram space-time features and the environmental space-time features to obtain fusion features; performing double-branch feature analysis on the fusion features to obtain the state evaluation result; performing time sequence feature extraction on target spatial topology features to obtain target space-time features, comprising the following steps: obtaining a first task weight matrix; performing dynamic scheduling on the target spatial topology features according to the first task weight matrix to obtain a feature scheduling strategy corresponding to the target spatial topology features; performing strategy feature extraction on the target spatial topology features according to the feature scheduling strategy to obtain the target space-time features; wherein, if the target spatial topology features are electroencephalogram spatial topology features, the target space-time features are electroencephalogram space-time features; or, if the target spatial topology features are environmental spatial topology features, the target space-time features are environmental space-time features.

2. The method of claim 1, wherein, the step of obtaining a first task weight matrix comprises the following steps: obtaining a second task weight matrix, a task completion degree of a task performed corresponding to the second task weight matrix, and a standard completion degree, the second task weight matrix being a previous task weight matrix of the first task weight matrix; performing completion degree loss analysis on the standard completion degree according to the task completion degree to obtain task loss data of the task performed; According to the task loss data, the second task weight matrix is dynamically updated to obtain the first task weight matrix.

3. The method according to any of claims 1-2, characterized in that, The trained state analysis model is obtained by the following steps: Obtain multi-modal training data and multi-source labels of the multi-modal training data, the multi-modal training data including environmental training data and electroencephalogram training data, and the multi-source labels including emotion labels and fatigue labels; The multi-modal training data is input into the current state analysis model for analysis to obtain training data analysis results; According to the training data analysis results and the multi-source labels, the current state analysis model is updated to obtain the trained state analysis model.

4. The method of claim 1, wherein, The state evaluation results and the multi-source environmental perception data are input into the control decision model for decision generation to obtain the action control instruction of the humanoid robot, including: Obtain the action execution rule of the humanoid robot; The state evaluation results and the multi-source environmental perception data are input into the control decision model for coefficient analysis to obtain a decision coefficient; According to the decision coefficient, the action execution rule is matched to obtain the action control instruction.

5. A control system of a humanoid robot, characterized by, Including: The first processing unit is configured to obtain a control decision model and multi-source environmental perception data of a humanoid robot in a task scene, and electroencephalogram data of a target user in the task scene; The second processing unit is configured to input the multi-source environmental perception data and the electroencephalogram data into a trained state analysis model for analysis to obtain a state evaluation result output by the trained state analysis model, the state evaluation result being used to indicate an emotion category and a fatigue level of the target user at present; The third processing unit is configured to input the state evaluation result and the multi-source environmental perception data into the control decision model for decision generation to obtain an action control instruction of the humanoid robot; The fourth processing unit is configured to perform cooperative action control on the humanoid robot according to the action control instruction; The multi-source environmental perception data and the electroencephalogram data are input into the trained state analysis model for analysis to obtain a state evaluation result output by the trained state analysis model, including: Obtain an environmental noise feature matrix; According to the environmental noise feature matrix, the electroencephalogram data is cooperatively filtered to obtain electroencephalogram modal data; The multi-source environmental perception data is standardized to obtain environmental modal data; The electroencephalogram modal data and the environmental modal data are time-aligned to obtain multi-modal data; The multi-modal data is input into the trained state analysis model for analysis to obtain the state evaluation result; The multi-modal data is input into the trained state analysis model for analysis to obtain the state evaluation result, including: The multi-modal data is spatially feature-extracted to obtain electroencephalogram spatial topology features and environmental spatial topology features; The electroencephalogram spatial topology feature is subjected to time sequence feature extraction to obtain an electroencephalogram space-time feature, and the environmental spatial topology feature is subjected to time sequence feature extraction to obtain an environmental space-time feature; The electroencephalogram space-time feature and the environmental space-time feature are subjected to feature attention fusion to obtain a fusion feature; The fusion feature is subjected to double-branch feature analysis to obtain the state evaluation result; The target spatial topology feature is subjected to time sequence feature extraction to obtain a target space-time feature, including: A first task weight matrix is obtained; According to the first task weight matrix, the target spatial topology feature is subjected to dynamic scheduling to obtain a feature scheduling strategy corresponding to the target spatial topology feature; According to the feature scheduling strategy, the target spatial topology feature is subjected to strategy feature extraction to obtain the target space-time feature; If the target spatial topology feature is an electroencephalogram spatial topology feature, the target space-time feature is an electroencephalogram space-time feature; or if the target spatial topology feature is an environmental spatial topology feature, the target space-time feature is an environmental space-time feature.

6. An electronic device, comprising: including: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-4.

7. A computer readable storage medium having stored therein a program that is executable by a processor, characterized in that, The program executable by the processor, when executed by the processor, is used to implement the method of any one of claims 1-4.

Images