Humanoid robot two-arm cooperation method and device, electronic equipment and storage medium

Abstract

The invention discloses a humanoid robot double-arm cooperation method and device, electronic equipment and a storage medium, and relates to the technical field of robots, and the method comprises the steps: obtaining a target feature vector; the target feature vector comprises surrounding environment information and two-arm joint state information of the humanoid robot; a neural network model is constructed, and then a motion branch model and a track branch model are constructed based on the neural network model; respectively inputting the target feature vector into a motion branch model and a trajectory branch model to obtain a control instruction of the two arms of the humanoid robot output by the motion branch model and predicted trajectory points of the two arms of the humanoid robot output by the trajectory branch model; generating a control strategy according to the control instruction and the prediction track point; and controlling the two arms of the humanoid robot to move according to the control strategy. According to the method, the environment and the states of the two arms are sensed more comprehensively and accurately by fusing the surrounding environment information and the joint state information of the two arms, so that the humanoid robot can better adapt to complex environments and tasks.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a method, device, electronic device and storage medium for dual-arm coordination of a humanoid robot. Background Art

[0002] With the development of humanoid robot technology, dual-arm collaborative operation has become an important research direction. At present, there are some problems with dual-arm operation of humanoid robots. On the one hand, in terms of information acquisition, it is difficult to obtain comprehensive status information of humanoid robots. On the other hand, in terms of control strategy, traditional solutions often find it difficult to achieve efficient dual-arm collaboration, and lack an effective end-to-end solution from information acquisition to control command generation. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a humanoid robot dual-arm collaboration method, device, electronic device and storage medium to achieve efficient and accurate collaborative operation of the humanoid robot's dual arms for different tasks in a complex environment, thereby improving the intelligence and adaptability of the humanoid robot.

[0004] To achieve the above object, an embodiment of the present application provides a humanoid robot dual-arm collaboration method, the method comprising the following steps:

[0005] Acquire a target feature vector; wherein the target feature vector includes a feature vector corresponding to the surrounding environment information of the humanoid robot and a feature vector corresponding to the joint state information of both arms of the humanoid robot;

[0006] Constructing a neural network model and then constructing a motion branch model and a trajectory branch model based on the neural network model;

[0007] Inputting the target feature vector into the motion branch model and the trajectory branch model respectively, obtaining control instructions of the two arms of the humanoid robot output by the motion branch model and predicted trajectory points of the two arms of the humanoid robot output by the trajectory branch model;

[0008] Generate a control strategy according to the control instruction and the predicted trajectory point;

[0009] The arms of the humanoid robot are controlled to move according to the control strategy.

[0010] In some embodiments, obtaining the target feature vector comprises the following steps:

[0011] Acquire a time-series image of the environment information around the humanoid robot through a sensor disposed on the head of the humanoid robot; convert the time-series image into a first branch feature vector through a deep convolutional neural network encoder based on visual perception;

[0012] Acquire joint state information of the left arm and the right arm of the humanoid robot through sensors arranged on the arms of the humanoid robot; convert the joint state information into a second branch feature vector corresponding to the left arm and a second branch feature vector corresponding to the right arm through a multi-layer perceptron network encoder based on joint state perception;

[0013] The first branch feature vector and the two second branch feature vectors are merged to obtain the target feature vector.

[0014] In some embodiments, constructing a neural network model comprises the following steps:

[0015] Constructing multiple network layers and multiple different types of neurons; wherein the types of neurons include perception neurons, internal neurons, command neurons and motor neurons;

[0016] Allocating each of the neurons to each of the network layers;

[0017] In any two consecutive network layers, synapses are inserted into any source neurons, and the synapses are connected to target neurons to obtain the neural network model; wherein the source neuron is the neuron at the starting point of information transmission, and the target neuron is the neuron at the end point of information transmission.

[0018] In some embodiments, inserting synapses into any source neurons in any two consecutive network layers and connecting the synapses to target neurons to obtain the neural network model comprises the following steps:

[0019] In any two consecutive network layers, the synapses whose quantity and polarity satisfy the Bernoulli distribution are inserted into any source neurons, and the synapses are randomly connected to the target neurons according to the binomial distribution to obtain the neural network model.

[0020] In some embodiments, inserting synapses into any source neurons in any two consecutive network layers and connecting the synapses to target neurons to obtain the neural network model comprises the following steps:

[0021] In any two consecutive network layers, each of the instruction neurons is set to be cyclically connected and a preset number of synapses are inserted to obtain the neural network model.

[0022] In some embodiments, the step of constructing a motion branch model and a trajectory branch model based on the neural network model comprises the following steps:

[0023] Determine the motion branch loss function according to the difference between the predicted control distribution and the expert distribution and the feature loss between the supervisory signal and the prediction signal of the predicted control distribution and the expert data at the current moment; construct the motion branch model according to the neural network model and the motion branch loss function;

[0024] The trajectory branch loss function is determined according to the difference between the predicted trajectory point and the true trajectory point and the feature loss between the predicted trajectory point and the supervisory signal and the predicted signal of the expert data at the current moment; and the trajectory branch model is constructed according to the neural network model and the trajectory branch loss function.

[0025] In some embodiments, generating a control strategy according to the control instruction and the predicted trajectory point comprises the following steps:

[0026] The control command and the predicted trajectory point are weightedly summed to generate the control strategy.

[0027] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a humanoid robot dual-arm coordination device, the device comprising:

[0028] A feature vector acquisition unit, used to acquire a target feature vector; wherein the target feature vector includes a feature vector corresponding to the surrounding environment information of the humanoid robot and a feature vector corresponding to the joint state information of both arms of the humanoid robot;

[0029] A model building unit, used to build a neural network model and then build a motion branch model and a trajectory branch model based on the neural network model;

[0030] an output prediction unit, used to input the target feature vector into the motion branch model and the trajectory branch model respectively, to obtain control instructions of the two arms of the humanoid robot output by the motion branch model and predicted trajectory points of the two arms of the humanoid robot output by the trajectory branch model;

[0031] A control strategy generating unit, used for generating a control strategy according to the control instruction and the predicted trajectory point;

[0032] The dual-arm driving unit is used to control the dual arms of the humanoid robot to move according to the control strategy.

[0033] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0034] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0035] The embodiments of the present application include at least the following beneficial effects:

[0036] The present application can obtain a target feature vector; wherein the target feature vector includes a feature vector corresponding to the surrounding environment information of the humanoid robot and a feature vector corresponding to the joint state information of the two arms of the humanoid robot; construct a neural network model and then construct a motion branch model and a trajectory branch model based on the neural network model; input the target feature vector into the motion branch model and the trajectory branch model respectively, and obtain the control instructions of the two arms of the humanoid robot output by the motion branch model and the predicted trajectory points of the two arms of the humanoid robot output by the trajectory branch model; generate a control strategy according to the control instructions and the predicted trajectory points; and control the two arms of the humanoid robot to move according to the control strategy. This application overcomes the limitations of a single sensor by integrating the surrounding environment information and the joint status information of both arms, and perceives the environment and the status of both arms more comprehensively and accurately, providing a reliable basis for subsequent operations; the motion branch model and trajectory branch model constructed based on the neural network model can effectively realize the coordinated operation of both arms, improve the efficiency and accuracy of the dual-arm operation of the humanoid robot, and enable the humanoid robot to better adapt to complex environments and tasks; through the end-to-end solution from the target feature vector to the control strategy, combined with the optimization of the humanoid robot's own characteristics, the humanoid robot has better generalization ability and can operate stably in different environments and task scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A schematic diagram of the process of the humanoid robot dual-arm collaboration method provided in an embodiment of the present application;

[0039] Figure 2 An example flow chart of a humanoid robot dual-arm collaboration method provided in an embodiment of the present application;

[0040] Figure 3 An example flow chart of a more specific humanoid robot dual-arm collaboration method provided in an embodiment of the present application;

[0041] Figure 4An example architecture flow chart of a humanoid robot dual-arm collaboration method provided in an embodiment of the present application;

[0042] Figure 5 An example flow chart of information fusion provided in an embodiment of the present application;

[0043] Figure 6 An example diagram of the connection of each neuron in the neural network model provided in the embodiment of the present application;

[0044] Figure 7 A motion branch loss function calculation flow chart provided in an embodiment of the present application;

[0045] Figure 8 A flow chart of the calculation of the trajectory branch loss function provided in the embodiment of the present application;

[0046] Fig. 9 An example flow chart of the collaborative operation of two arms of a humanoid robot provided in an embodiment of the present application;

[0047] Fig.10 A schematic diagram of the structure of a humanoid robot dual-arm collaborative device provided in an embodiment of the present application;

[0048] Fig.11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0050] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0051] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0053] Reference Figure 1 The embodiment of the present application provides a humanoid robot dual-arm collaboration method, which may include but is not limited to S100 to S140, as follows:

[0054] S100: Obtain a target feature vector; wherein the target feature vector includes a feature vector corresponding to the surrounding environment information of the humanoid robot and a feature vector corresponding to the joint state information of both arms of the humanoid robot.

[0055] Furthermore, S100 may include S101 to S103:

[0056] S101: acquiring a time-series image of the environment information around the humanoid robot through a sensor disposed on the head of the humanoid robot; converting the time-series image into a first branch feature vector through a deep convolutional neural network encoder based on visual perception;

[0057] S102: Acquire joint state information of the left arm and the right arm of the humanoid robot through sensors arranged on the arms of the humanoid robot; convert the joint state information into a second branch feature vector corresponding to the left arm and a second branch feature vector corresponding to the right arm through a multi-layer perceptron network encoder based on joint state perception;

[0058] S103: Fusing the first branch feature vector and the two second branch feature vectors to obtain the target feature vector.

[0059] Specifically, the time-series images acquired by the visual sensor on the head of the humanoid robot are used as input, and the first branch feature vector is obtained through a deep convolutional neural network encoder (VPCNN-Encoder) based on visual perception; the time-series images acquired by the visual sensor include information such as the position, shape, and color of objects in the environment.

[0060] The sensor measurement information of each joint of the left and right arms of the humanoid robot (including joint angle, angular velocity, torque, etc.) is used as input, and the second branch feature vector is obtained through a multi-layer perceptron network encoder (JSP-MLP-Encoder) based on joint state perception (there is one second branch feature vector for each left and right arm);

[0061] The first branch feature vector is fused with the second branch feature vectors of the left and right arms to obtain the effective information of the environment and the state of both arms, that is, the target feature vector. When fusing the feature vectors, it is necessary to consider how to reasonably integrate the visual information and the state information of both arms to provide comprehensive and accurate input for subsequent operations.

[0062] More specifically, the humanoid robot obtains time-series images of the environment through the head vision sensor, and processes them with the visual perception deep convolutional neural network encoder (VPCNN-Encoder) to obtain the first branch feature vector, which contains information such as the position and shape of objects in the environment.

[0063] At the same time, the sensor measurement information of each joint of the left and right arms (including joint angle, angular velocity, torque, etc.) is obtained respectively, and the second branch feature vectors of the left and right arms are obtained through a multi-layer perceptron network encoder (JSP-MLP-Encoder) based on joint state perception.

[0064] The first branch feature vector is fused with the second branch feature vectors of the left and right arms to obtain effective information about the environment and the state of both arms. The fusion process needs to consider the importance of different information, and weighted summation and other methods can be used to ensure the comprehensiveness and accuracy of the information.

[0065] S110: Construct a neural network model and then construct a motion branch model and a trajectory branch model based on the neural network model.

[0066] Furthermore, the step of constructing a neural network model in S110 may include S111 to S113:

[0067] S111: constructing multiple network layers and multiple neurons of different types; wherein the types of neurons include perception neurons, internal neurons, command neurons, and motor neurons;

[0068] S112: configuring each of the neurons to each of the network layers;

[0069] S113: In any two consecutive network layers, insert synapses into any source neurons, and connect the synapses to target neurons to obtain the neural network model; wherein the source neuron is the neuron at the starting point of information transmission, and the target neuron is the neuron at the end point of information transmission.

[0070] Specifically, this embodiment can construct a neural network model suitable for the collaborative operation of two arms of a humanoid robot, including multiple layers of neurons, wherein perception neurons are used to perceive the environment and dual-arm state information, internal neurons are used for information processing and transmission, command neurons are used to generate control instructions for the collaborative operation of two arms, and motor neurons are used to convert control instructions into specific joint movements; between any two consecutive layers, for any source neuron, synapses are inserted according to certain rules, the number and polarity of synapses satisfy a specific distribution, and the target neuron is randomly selected through the corresponding distribution.

[0071] Furthermore, S113 may include: in any two consecutive network layers, inserting the synapses whose quantity and polarity satisfy the Bernoulli distribution into any source neurons, and randomly connecting the synapses to the target neurons according to the binomial distribution to obtain the neural network model.

[0072] As another implementation, the step of constructing a neural network model in S110 may include S114:

[0073] S114: In any two consecutive network layers, each of the instruction neurons is set to be cyclically connected and a preset number of synapses are inserted to obtain the neural network model.

[0074] It is understandable that the source neuron and the target neuron in this embodiment are both command neurons. A loop connection can be set between the command neurons and a suitable number of synapses can be inserted, and their parameters meet the corresponding rules.

[0075] Furthermore, in S110, a motion branch model and a trajectory branch model are respectively constructed based on the neural network model, including S115-S116:

[0076] S115: determining a motion branch loss function according to the difference between the predicted control distribution and the expert distribution and the feature loss between the supervisory signal and the prediction signal of the predicted control distribution and the expert data at the current moment; constructing the motion branch model according to the neural network model and the motion branch loss function;

[0077] S116: Determine a trajectory branch loss function according to the difference between the predicted trajectory point and the true trajectory point and the feature loss between the predicted trajectory point and the supervisory signal and the predicted signal of the expert data at the current moment; construct the trajectory branch model according to the neural network model and the trajectory branch loss function.

[0078] Specifically, the motion branch loss function and the trajectory branch loss function are defined respectively, and the neural network model of this embodiment is used for modeling, wherein the motion branch loss function considers the difference between the predicted control distribution and the expert distribution as well as the feature loss between the supervisory signal and the predicted signal of the expert data at the current moment, and the trajectory branch loss function considers the difference between the predicted trajectory point and the true trajectory point as well as the feature loss between the supervisory signal and the predicted signal of the expert data at the current moment.

[0079] More specifically, the motion branch loss function considers the difference between the predicted control distribution and the expert distribution (which can be obtained from the human dual-arm collaborative operation data) (such as KL divergence) and the feature loss between the supervisory signal and the predicted signal at the current moment of the expert data (which can be measured by calculating the L2 measure between the feature vectors). The trajectory branch loss function considers the difference between the predicted trajectory point (the predicted trajectory point of the dual-arm end effector or joint) and the true trajectory point (such as the spatial position difference, etc.) and the feature loss between the supervisory signal and the predicted signal at the current moment of the expert data (which can be measured by calculating the L2 measure between the feature vectors).

[0080] S120: Input the target feature vector into the motion branch model and the trajectory branch model respectively, and obtain the control instructions of the arms of the humanoid robot output by the motion branch model and the predicted trajectory points of the arms of the humanoid robot output by the trajectory branch model.

[0081] Specifically, the effective information of the fusion environment and the status of the two arms is simultaneously input into the motion branch model and the trajectory branch model; the motion branch model outputs the control instructions of the two arms of the humanoid robot, including the motion angle, speed, torque and other instructions of each joint of the left and right arms; the trajectory branch model outputs the predicted trajectory points of the two arms of the humanoid robot at different times in the future, such as the predicted trajectory points of the end effector of the two arms at the next 1 second, 2 seconds, 3 seconds, etc., to provide a reference for the motion planning of the two arms.

[0082] S130: Generate a control strategy according to the control instruction and the predicted trajectory point.

[0083] Specifically, based on the control instructions of the humanoid robot's arms and the predicted trajectory points at different moments in the future, a fusion mechanism is used to obtain the final control strategy of the humanoid robot's arms to achieve coordinated and accurate operation of the arms.

[0084] Further, S130 may include S131:

[0085] S131: Performing weighted summation on the control instruction and the predicted trajectory point to generate the control strategy.

[0086] For example, different weights can be assigned according to the different motion requirements and importance of the two arms, and then the control instructions and predicted trajectory points can be fused by weighted summation to ensure the coordinated and accurate operation of the two arms, so that the robot can better complete various tasks, such as grasping and assembly.

[0087] S140: Controlling the arms of the humanoid robot to move according to the control strategy.

[0088] It can be understood that in this embodiment, the arms of the humanoid robot can be controlled to move according to the control strategy by controlling the humanoid robot.

[0089] Next, the solution of the embodiment of the present application will be described in detail and explained in conjunction with specific application examples:

[0090] Specifically, refer to Figure 2 , Figure 3 and Figure 4 , this embodiment may include the following solutions:

[0091] (i) Information extraction

[0092] 1.1 Visual information processing.

[0093] The visual sensor on the head of the humanoid robot continuously acquires time-series images of the environment, which contain information such as the position and shape of objects in the environment. These images are input into the deep convolutional neural network encoder (VPCNN-Encoder) based on visual perception, and the first branch feature vector is obtained after encoding processing.

[0094] 1.2 Obtaining dual-arm status information.

[0095] The sensors at each joint of the left and right arms of the humanoid robot continuously measure the angle, angular velocity, torque and other information of the joints. The sensor measurement information of the left and right arms is respectively input into the multi-layer perceptron network encoder (JSP-MLP-Encoder) based on joint state perception, and the second branch feature vectors of the left and right arms are obtained after encoding processing.

[0096] 1.3 Information fusion.

[0097] The first branch feature vector is fused with the second branch feature vector of the left and right arms. In the fusion process, it is necessary to consider how to reasonably integrate the visual information and the state information of the two arms. For example, different weights can be assigned according to the importance of different information, and then fused by weighted summation to obtain effective information about the environment and the state of the two arms. Figure 5 The following is an example flowchart of information fusion.

[0098] (ii) Construction of neural network model.

[0099] 2.1 Neuron definition and connection.

[0100] The constructed neural network model includes sensory neurons, internal neurons, command neurons and motor neurons. The sensory neurons are used to perceive the environment and effective information of the two-arm state, the internal neurons are used for information processing and transmission, the command neurons are used to generate control instructions for the coordinated operation of the two arms, and the motor neurons are used to convert the control instructions into specific executable joint movements. For example, Figure 6 This is an example diagram of the connections between the neurons in the neural network model provided in this embodiment.

[0101] Next, each neuron is described in detail.

[0102] Sensory neurons:

[0103] 1) Environmental information perception:

[0104] Receives the environment-related part of the effective information of the environment and dual-arm status from the fusion unit. For visual information, it parses the feature vector processed by the deep convolutional neural network encoder (VPCNN-Encoder) based on visual perception, and extracts information about the position, shape, color, texture, etc. of objects in the environment, as well as the spatial relationship information between objects. For example, it can identify the position and shape of obstacles in the environment and determine whether they are within the working area of ​​the robot's dual-arm operation.

[0105] The perception of the state information of the two arms is mainly auxiliary. It obtains the encoded features of the approximate position and posture information of the two arms. This information, together with the environmental information, constitutes the initial perception of the overall scene.

[0106] 2) Perception of dual-arm status information:

[0107] From the state information of the arms obtained by the fusion unit, the perception neurons can identify the initial position and general characteristics of the arms. For example, they can determine whether the arms are extended or bent, as well as the approximate angle range, etc., providing a basis for subsequent precise processing.

[0108] Internal Neuron:

[0109] 1) Information integration and preprocessing:

[0110] Receives environmental and arm status information from sensory neurons. For environmental information, it integrates the feature information of different objects, such as combining the position and shape information of multiple objects to build a more complete environmental scene model. At the same time, it further refines the arm status information and analyzes whether the relative position and posture of the arms in the current environment are reasonable in combination with environmental information.

[0111] Internal neurons also pre-process the information transmitted by sensory neurons, such as normalizing data and removing noise, to improve the quality and accuracy of information and provide better input for the processing of command neurons.

[0112] 2) Feature extraction and abstraction:

[0113] Extract higher-level features from the integrated information. For environmental information, it may extract some representative environmental pattern features, such as spatial layout patterns, object combination patterns, etc. For dual-arm state information, it can extract the trend characteristics of dual-arm movement, such as whether the arms tend to move closer or farther away, and the speed trend of movement, etc. These features will help the command neurons generate more reasonable control instructions.

[0114] Instruction neuron:

[0115] 1) Control instruction generation basis:

[0116] Receives information processed by internal neurons. It first determines the goals and requirements of the current task based on the environment information and the state of the two arms. For example, if the task is to grasp an object, it needs to determine the location, shape, size of the object, and the distance and angle relationship between the two arms and the object.

[0117] At the same time, the command neuron takes into account the current state of the arms, such as joint angle, angular velocity, torque, etc., as well as the movement capabilities and limitations of the arms to ensure that the generated control commands are feasible.

[0118] 2) Generation of dual-arm collaborative control instructions:

[0119] Based on the above information, the command neuron generates control instructions for the coordinated operation of both arms. For the left arm, it determines the parameters such as the angle, speed, and torque that need to be adjusted for each joint to achieve the correct movement of the left arm. For the right arm, corresponding control instructions are also generated. These control instructions take into account the collaborative relationship between the two arms, such as the movement sequence and synchronization requirements of the two arms. For example, if you want to grab a larger object, you may need to move the left arm to a specific position first, and then the right arm will cooperate. The command neuron will accurately generate such a control instruction sequence.

[0120] Motor Neuron:

[0121] Converting commands to joint movements: Receive control commands from command neurons. For the motor neurons of the left arm, it converts the parameters such as the angle, speed and torque of each joint of the left arm into specific electrical signals or mechanical movement commands, directly drives the actuators of the left arm joints, and makes the left arm move according to the command requirements.

[0122] Similarly, for the motor neuron of the right arm, it converts the control command of the right arm into actual joint movement, ensuring that the right arm can accurately execute the command and achieve coordinated operation with the left arm. For example, if the command requires the elbow joint of the right arm to bend a certain angle, the motor neuron will convert this angle command into a signal that can be recognized by the elbow joint actuator, so that the elbow joint can be accurately bent to the specified angle.

[0123] Between any two consecutive layers, for any source neuron, synapses are inserted according to certain rules. For example, the number and polarity of synapses satisfy the Bernoulli distribution, and the target neuron is randomly selected through the binomial distribution. A recurrent connection can be set between the instruction neurons to insert an appropriate number of synapses.

[0124] Next, the source neuron is described:

[0125] Source neuron is a concept involved in describing the connection relationship between any two consecutive layers in a neural network. It refers to the neuron that serves as the starting point for information transmission in a certain layer.

[0126] The difference between source neurons and other neurons:

[0127] 1) With sensory neurons:

[0128] The main function of sensory neurons is to perceive and preliminarily process the environment and the status of both arms. They are the first processing link for information entering the neural network. The source neuron is a more general concept used to describe the sending end of information when connecting between layers. It can appear between each consecutive layer of the neural network, not just limited to the layer where the sensory neurons are located. For example, when transmitting information from the sensory neuron layer to the internal neuron layer, the sensory neuron can serve as the source neuron, but when the internal neuron layer transmits information to other layers, the internal neuron can also become the source neuron.

[0129] 2) With internal neurons:

[0130] Internal neurons focus on information integration, preprocessing, feature extraction, and abstraction. Source neurons emphasize the starting point of information transmission. When an internal neuron sends information to other layers as the starting point of information transmission, it becomes a source neuron. This is defined from the perspective of information flow, which is different from the functional processing focus of the internal neuron itself.

[0131] 3) With the instruction neuron:

[0132] The command neuron is mainly responsible for generating control instructions for the coordinated operation of the two arms. The source neuron is a concept used to describe the connection structure of the neural network. When the command neuron transmits information to the motor neuron or other related layers, the command neuron can act as a source neuron. This is a different dimension from the control instruction generation function of the command neuron itself. One is about functional processing and the other is about structural connection.

[0133] 4) With motor neurons:

[0134] The role of motor neurons is to convert control instructions into specific executable joint movements. Source neurons are used to illustrate the starting situation of information transmission in neural networks. Motor neurons can act as source neurons in some cases, such as when the output of motor neurons is passed as feedback information to other layers for further processing, but this is different from its main function, which is to drive joint movement.

[0135] Between any two consecutive layers, for any source neuron, synapses are inserted according to certain rules. For example, the number and polarity of synapses satisfy the Bernoulli distribution, and the target neuron is randomly selected through the binomial distribution. A loop connection can be set between the command neurons, and a suitable number of specific synaptic parameters (such as the number meets certain conditions and the polarity satisfies the Bernoulli distribution) can be inserted to achieve information transmission and interaction between the command neurons.

[0136] Next, the effect of inserting a synapse will be described.

[0137] 1) Information transmission:

[0138] Synapses are key structures for information transmission between neurons. In neural networks, the insertion of synapses enables source neurons to transmit the electrical or chemical signals they generate (electrical pulses and neurotransmitters in biological neural networks, analogous to processed information here) to target neurons. For example, sensory neurons transmit the perceived environment and arm status information to internal neurons through synapses for further processing and integration.

[0139] 2) Construct the connection structure of the neural network:

[0140] The way synapses are inserted determines the connection topology of the neural network. Different neurons are connected by inserting synapses according to specific rules, thus forming a complex network structure. This structure has a crucial impact on the flow and processing of information in the network. For example, reasonable synaptic connections can enable the neural network to effectively learn and recognize different input patterns, just like in the collaborative operation of the humanoid robot's two arms, it can accurately generate appropriate control instructions and predict trajectory points based on the environment and the state information of the two arms.

[0141] Next, the role of the number and polarity of synapses satisfying the Bernoulli distribution is explained.

[0142] 1) Control the complexity and diversity of information transmission:

[0143] Bernoulli distribution is a discrete probability distribution. The number and polarity of synapses satisfying Bernoulli distribution means that the number and polarity of synapses are randomly determined, but follow certain probability rules. This randomness increases the complexity and diversity of neural network connections. Different combinations of synapse numbers and polarities lead to different information transmission paths and strengths, enabling neural networks to learn and process a wider range of input patterns. For example, in the collaborative operation of humanoid robots with two arms, facing different environments and task scenarios, this complex and diverse connection method can enable neural networks to better adapt to and handle various situations, rather than being limited to a fixed connection mode.

[0144] 2) Simulate the characteristics of biological neural networks:

[0145] In biological neural networks, the connections between neurons are also complex and have a certain degree of randomness. The number and polarity of synapses satisfying the Bernoulli distribution can simulate this characteristic of biological neural networks to a certain extent, thereby possibly giving artificial neural networks some advantages similar to biological neural networks, such as better adaptability and learning ability. This helps the neural network of the humanoid robot's dual-arm collaborative operation to better learn the patterns and skills of human dual-arm collaborative operation, and improve its accuracy and flexibility in operation.

[0146] Next, the meaning of the target neuron is explained.

[0147] Definition: Target neuron is the concept corresponding to source neuron in neural network. When source neuron transmits information through synapse, the receiving neuron to which the information is directed is the target neuron.

[0148] 1) Position in the neural network structure:

[0149] The target neurons are located in the layer below the source neurons (in the context of transmitting information from one layer to the next layer). For example, when transmitting information from the sensory neuron layer to the internal neuron layer, if the sensory neurons are the source neurons, then some neurons in the internal neuron layer will become target neurons and receive information from the sensory neurons.

[0150] 2) The role of randomly selecting target neurons through binomial distribution:

[0151] Increase the randomness and diversity of network connections: Binomial distribution is a discrete probability distribution. Randomly selecting target neurons through binomial distribution makes the connection between source neurons and target neurons in the neural network random. This randomness increases the diversity of network connections and avoids fixed, single connection patterns. Different combinations of random connections enable the neural network to explore more different information transmission paths and processing methods during the learning process. For example, in the neural network of the humanoid robot's dual arms, this randomness can enable the network to better adapt to various complex environments and task situations and improve its generalization ability.

[0152] 3) Optimize information processing and learning capabilities:

[0153] Randomly selecting target neurons can prompt the neural network to discover more effective ways of processing information during the learning process. Because the connections are random, the network will constantly adjust its weights and connections during training to adapt to different input and output requirements. This helps the network better learn the relationship between input information (such as environment and arm state information) and output information (such as control instructions and predicted trajectory points), thereby improving the performance and accuracy of the network. For example, when learning how to accurately control the arms of a humanoid robot to perform grasping tasks, random target neuron connections can enable the network to find the optimal control strategy more quickly.

[0154] 2.2 Define loss function.

[0155] Motion branch loss function: Consider the difference between the predicted control distribution and the expert distribution (e.g., the expert distribution obtained from human dual-arm collaborative operation data), which can be measured by KL divergence; and also include the feature loss between the supervision signal and the prediction signal at the current moment of the expert data, which can be measured by calculating the L2 measure between the feature vectors. For example, Figure 7 Flowchart for motion branch loss function calculation.

[0156] Trajectory branch loss function: considers the difference between the predicted trajectory points (the predicted trajectory points of the dual-arm end effectors or joints) and the actual trajectory points, such as the spatial position difference, etc.; it also includes the feature loss between the supervision signal and the predicted signal at the current moment of the expert data, which can be measured by calculating the L2 measure between the feature vectors. For example, Figure 8 Flowchart for trajectory branch loss function calculation.

[0157] (3) Information input and output.

[0158] 3.1 Input:

[0159] The fused effective information of the environment and dual-arm states is simultaneously input into the motion branch model and the trajectory branch model.

[0160] 3.2 Output:

[0161] The motion branch model outputs control instructions for the humanoid robot's arms, including the motion angle, speed, torque, and other instructions for each joint of the left and right arms. For example, for the shoulder joint of the left arm, an angle value and an angular velocity value may be output; for the elbow joint of the right arm, a different angle value and angular velocity value may be output.

[0162] The trajectory branching model outputs the predicted trajectory points of the humanoid robot's arms at different moments in the future, such as the predicted trajectory points of the end effectors of the arms at 1 second, 2 seconds, 3 seconds in the future, etc. These predicted trajectory points can provide a reference for the motion planning of the arms, helping the robot to better plan the motion paths of the arms.

[0163] (IV) Generation of final control strategy.

[0164] Based on the control instructions of the humanoid robot's arms and the predicted trajectory points at different moments in the future, a fusion mechanism is used to generate the final control strategy. For example, a weighted average method can be used to assign different weights according to the different movement requirements and importance of the arms, and then the control instructions and the predicted trajectory points can be fused by weighted summation to obtain the final control strategy. This fusion mechanism can ensure the coordinated and accurate operation of the arms, enabling the robot to better complete various tasks, such as grasping, assembly, and other tasks. For example, Fig. 9 Example flow chart for collaborative operation of two arms of a humanoid robot.

[0165] Reference Fig.10 The embodiment of the present application also provides a humanoid robot dual-arm coordination device, which can implement the above-mentioned humanoid robot dual-arm coordination method, and the device includes:

[0166] A feature vector acquisition unit, used to acquire a target feature vector; wherein the target feature vector includes a feature vector corresponding to the surrounding environment information of the humanoid robot and a feature vector corresponding to the joint state information of both arms of the humanoid robot;

[0167] A model building unit, used to build a neural network model and then build a motion branch model and a trajectory branch model based on the neural network model;

[0168] an output prediction unit, used to input the target feature vector into the motion branch model and the trajectory branch model respectively, to obtain control instructions of the two arms of the humanoid robot output by the motion branch model and predicted trajectory points of the two arms of the humanoid robot output by the trajectory branch model;

[0169] A control strategy generating unit, used for generating a control strategy according to the control instruction and the predicted trajectory point;

[0170] The dual-arm driving unit is used to control the dual arms of the humanoid robot to move according to the control strategy.

[0171] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0172] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned knowledge extraction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0173] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0174] See also Fig.11 , Fig.11 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0175] The processor 1101 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0176] The memory 1102 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 may store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1102, and the processor 1101 calls and executes the knowledge extraction method of the embodiments of this application;

[0177] Input / output interface 1103, used to implement information input and output;

[0178] The communication interface 1104 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0179] A bus 1105 that transmits information between various components of the device (e.g., the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104);

[0180] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via the bus 1105 .

[0181] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned knowledge extraction method is implemented.

[0182] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0183] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0184] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0185] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0186] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0188] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0189] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0190] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0191] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0193] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0194] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A humanoid robot dual-arm collaboration method, characterized in that: The method comprises the following steps: Acquire a target feature vector; wherein the target feature vector includes a feature vector corresponding to the surrounding environment information of the humanoid robot and a feature vector corresponding to the joint state information of both arms of the humanoid robot; Constructing a neural network model and then constructing a motion branch model and a trajectory branch model based on the neural network model; Inputting the target feature vector into the motion branch model and the trajectory branch model respectively, obtaining control instructions of the two arms of the humanoid robot output by the motion branch model and predicted trajectory points of the two arms of the humanoid robot output by the trajectory branch model; Generate a control strategy according to the control instruction and the predicted trajectory point; The arms of the humanoid robot are controlled to move according to the control strategy.

2. The humanoid robot dual-arm collaboration method according to claim 1, characterized in that: The step of obtaining the target feature vector comprises the following steps: Acquire a time-series image of the environment information around the humanoid robot through a sensor arranged on the head of the humanoid robot; convert the time-series image into a first branch feature vector through a deep convolutional neural network encoder based on visual perception; Acquire joint state information of the left arm and the right arm of the humanoid robot through sensors arranged on the arms of the humanoid robot; convert the joint state information into a second branch feature vector corresponding to the left arm and a second branch feature vector corresponding to the right arm through a multi-layer perceptron network encoder based on joint state perception; The first branch feature vector and the two second branch feature vectors are merged to obtain the target feature vector.

3. The humanoid robot dual-arm collaboration method according to claim 1, characterized in that: The construction of the neural network model comprises the following steps: Constructing multiple network layers and multiple different types of neurons; wherein the types of neurons include perception neurons, internal neurons, command neurons and motor neurons; Allocating each of the neurons to each of the network layers; In any two consecutive network layers, synapses are inserted into any source neurons, and the synapses are connected to target neurons to obtain the neural network model; wherein the source neuron is the neuron at the starting point of information transmission, and the target neuron is the neuron at the end point of information transmission.

4. The humanoid robot dual-arm coordination method according to claim 3, characterized in that: The method of inserting synapses into any source neurons in any two consecutive network layers and connecting the synapses to target neurons to obtain the neural network model comprises the following steps: In any two consecutive network layers, the synapses whose quantity and polarity satisfy the Bernoulli distribution are inserted into any source neurons, and the synapses are randomly connected to the target neurons according to the binomial distribution to obtain the neural network model.

5. The humanoid robot dual-arm coordination method according to claim 3, characterized in that: The method of inserting synapses into any source neurons in any two consecutive network layers and connecting the synapses to target neurons to obtain the neural network model comprises the following steps: In any two consecutive network layers, each of the instruction neurons is set to be cyclically connected and a preset number of synapses are inserted to obtain the neural network model.

6. The humanoid robot dual-arm coordination method according to claim 1, characterized in that: The step of constructing a motion branch model and a trajectory branch model based on the neural network model comprises the following steps: Determine the motion branch loss function according to the difference between the predicted control distribution and the expert distribution and the feature loss between the supervisory signal and the prediction signal of the predicted control distribution and the expert data at the current moment; construct the motion branch model according to the neural network model and the motion branch loss function; The trajectory branch loss function is determined according to the difference between the predicted trajectory point and the true trajectory point and the feature loss between the predicted trajectory point and the supervisory signal and the predicted signal of the expert data at the current moment; and the trajectory branch model is constructed according to the neural network model and the trajectory branch loss function.

7. The humanoid robot dual-arm collaboration method according to any one of claims 1 to 6, characterized in that: The generating of the control strategy according to the control instruction and the predicted trajectory point comprises the following steps: The control command and the predicted trajectory point are weightedly summed to generate the control strategy.

8. A humanoid robot dual-arm collaborative device, characterized in that: The device comprises: A feature vector acquisition unit, used to acquire a target feature vector; wherein the target feature vector includes a feature vector corresponding to the surrounding environment information of the humanoid robot and a feature vector corresponding to the joint state information of both arms of the humanoid robot; A model building unit, used to build a neural network model and then build a motion branch model and a trajectory branch model based on the neural network model; an output prediction unit, used to input the target feature vector into the motion branch model and the trajectory branch model respectively, to obtain control instructions of the two arms of the humanoid robot output by the motion branch model and predicted trajectory points of the two arms of the humanoid robot output by the trajectory branch model; A control strategy generating unit, used for generating a control strategy according to the control instruction and the predicted trajectory point; The dual-arm driving unit is used to control the dual arms of the humanoid robot to move according to the control strategy.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images