Robot arm loading method and device

Abstract

The present invention provides a manipulator loading method and device. The method includes: obtaining monitoring data when the manipulator grabs an object with the current load, where the monitoring data includes feedback data of the contact part between the manipulator and the object and feedback data of the driving device for changing the load; using a neural network model to extract a global feature vector from the feedback data, and determining upper load limit information and lower load limit information according to the global feature vector; judging whether the current load will damage the object according to the upper load limit information; if it is determined that the current load will not damage the object, then judging whether the current load is sufficient to pick up the object according to the lower load limit information; and if it is determined that the current load is not sufficient to pick up the object, increasing the load.

Description

Technical Field

[0001] The present invention relates to the field of robotic arm control, and particularly to a method and device for loading a robotic arm. Background Art

[0002] Robotic arms can be applied to various industries. Currently, automated grasping strategies for robotic arms are usually designed for some fixed application scenarios so that they can grasp specific objects. However, in some application scenarios, the grasped objects may have strong diversity. If the adaptability of the grasping strategy is insufficient, it may result in the inability to grasp the target object or damage to the target object during grasping.

[0003] A typical adaptive grasping strategy is to grasp an object with a set smaller load (usually referred to as the loading stage), and then dynamically adjust the load according to the grasping situation. The above-mentioned prior art may not be able to grasp the object at all due to the initially set load being too small, or the load during loading may directly damage the object. Thus, it can be seen that the existing adaptive grasping strategies have insufficient adaptability. Summary of the Invention

[0004] In view of this, the present application provides a method for loading a robotic arm, including:

[0005] Obtaining monitoring data when the robotic arm loads and grasps an object with the current load, where the monitoring data includes feedback data of the contact part between the robotic arm and the object and feedback data of the driving device for changing the load;

[0006] Using a neural network model to extract a global feature vector from the feedback data, and determining load upper limit information and load lower limit information according to the global feature vector;

[0007] Judging whether the current load will damage the object according to the load upper limit information;

[0008] If it is determined that the current load will not damage the object, then judging whether the current load is sufficient to grasp the object according to the load lower limit information;

[0009] If it is determined that the current load is not sufficient to grasp the object, then increase the load.

[0010] Optionally, the feedback data of the contact part between the robotic arm and the object includes pressure data collected by an electronic skin arranged at the contact part between the robotic arm and the object.

[0011] Optionally, the pressure data is array data collected by electronic skins respectively arranged on the palm and multiple fingers of the robotic arm.

[0012] Optionally, the current load includes a first characteristic contact force value and a second characteristic contact force value, where the first characteristic contact force value is the maximum value in all the array data, and the second characteristic contact force value is the relatively smaller one between the maximum value in the array data collected by the e-skin of the thumb and the maximum value in the array data collected by the e-skins of other fingers;

[0013] The first characteristic contact force value is used in the step of judging whether the current load will damage the grasped object according to the load upper limit information, and the second characteristic contact force value is used in the step of judging whether the current load is sufficient to pick up the grasped object according to the load lower limit information.

[0014] Optionally, the feedback data of the contact part between the robotic hand and the grasped object includes the temperature data collected by the e-skin arranged at the contact part between the robotic hand and the grasped object.

[0015] Optionally, the temperature data is the array data collected by the e-skins respectively arranged on the palm and multiple fingers of the robotic hand.

[0016] Optionally, the feedback data of the driving device for changing the load includes the displacement data and force data of each driving device of the robotic hand.

[0017] Optionally, extracting the global feature vector from the feedback data by using a neural network model includes:

[0018] Extracting the e-skin force distribution feature from the pressure data collected by each e-skin respectively, extracting the e-skin temperature distribution feature from the temperature data collected by each e-skin respectively, and fusing the e-skin force distribution feature and the e-skin temperature distribution feature to obtain the e-skin feature vector;

[0019] Extracting the driving device feature vector from the data connected based on the displacement data and force data of the driving device;

[0020] Connecting the e-skin feature vector and the driving device feature vector to obtain the global feature vector.

[0021] Optionally, fusing the e-skin force distribution feature and the e-skin temperature distribution feature to obtain the e-skin feature vector includes:

[0022] Performing pooling processing on the one with higher pixel density among the e-skin force distribution feature and the e-skin temperature distribution feature, so as to adjust the number of pixels to be the same;

[0023] Fusing the adjusted e-skin force distribution feature and the e-skin temperature distribution feature;

[0024] Taking the average of the fused result in the array dimension to obtain the e-skin feature vector.

[0025] Optionally, a drive device feature vector is obtained by extracting data coupled based on the displacement data and force data of the drive device, including:

[0026] Taking the difference between the displacement data at the current moment and the displacement data at the previous moment;

[0027] Coupling the difference result with the force data, and then extracting the drive device feature vector.

[0028] Optionally, determining the load upper limit information and the load lower limit information according to the global feature vector, including:

[0029] Classifying according to the global feature vector to obtain a classification vector regarding the category of the grasped object;

[0030] Determining the probability values of the grasped object belonging to various categories according to the classification vector;

[0031] Determining the load upper limit information and the load lower limit information according to the probability values of the grasped object belonging to various categories and the load upper limit and load lower limit corresponding to various categories.

[0032] Optionally, determining the probability values of the grasped object belonging to various categories according to the classification vector, including:

[0033] Obtaining multi-step joint data according to the classification vector at the current moment and at least one previous moment;

[0034] Obtaining the probability values of the grasped object belonging to various categories according to the multi-step joint data.

[0035] Optionally, obtaining multi-step joint data according to the classification result at the current moment and at least one previous moment, including:

[0036] Multiplying the classification vector at each moment by a preset coefficient respectively, and then summing the results of multiplication at each moment;

[0037] Combining the summation result with a preset background value to obtain the multi-step joint data.

[0038] Optionally, determining the load upper limit information and the load lower limit information according to the probability values of the grasped object belonging to various categories and the load upper limit and load lower limit corresponding to various categories, including:

[0039] Determining the maximum value among the probability values, and then determining the category to which the grasped object belongs;

[0040] Obtaining the load upper limit and load lower limit corresponding to the category to which the grasped object belongs.

[0041] Optionally, judging whether the current load will damage the grasped object according to the load upper limit information, including:

[0042] Determine whether the current load is lower than the load upper limit;

[0043] If the current load is lower than the load upper limit, determine that the current load will not damage the grasped object;

[0044] According to the load lower limit information, determine whether the current load is sufficient to grasp the grasped object, including:

[0045] Determine whether the current load is lower than the load lower limit;

[0046] If the current load is lower than the load lower limit, determine that the current load is not sufficient to grasp the grasped object.

[0047] Optionally, the load upper limit information includes the load upper limits corresponding to various categories and the probabilities that the grasped object belongs to various categories; the load lower limit information includes the load lower limits corresponding to various categories and the probabilities that the grasped object belongs to various categories.

[0048] Optionally, according to the load upper limit information, determine whether the current load will damage the grasped object, including:

[0049] Compare the current load with the load upper limits of each category respectively, and add up the probabilities of the categories exceeding the load upper limit to obtain a first cumulative probability;

[0050] Determine whether the first cumulative probability is lower than a first probability threshold;

[0051] When the first cumulative probability is lower than the first probability threshold, determine that the current load will not damage the grasped object;

[0052] According to the load lower limit information, determine whether the current load is sufficient to grasp the grasped object, including:

[0053] Compare the current load with the load lower limits of each category respectively, and add up the probabilities of the categories exceeding the load lower limit to obtain a second cumulative probability;

[0054] Determine whether the second cumulative probability is lower than a second probability threshold;

[0055] When the second cumulative probability is lower than the second probability threshold, determine that the current load is not sufficient to grasp the grasped object.

[0056] Correspondingly, the present application provides a manipulator loading device, which is characterized by including: a processor and a memory connected to the processor; wherein, the memory stores instructions executable by the processor, and the instructions are executed by the processor to enable the processor to execute the above-mentioned manipulator loading method.

[0057] According to the robotic arm loading method and device provided by the present application, corresponding monitoring data is obtained as the grasping load changes, and then the load upper limit information and load lower limit information corresponding to the current monitoring data are determined. The load is adjusted according to the relationship between the current load and the load upper limit information and load lower limit information, realizing an adaptive grasping and loading process of loading, identifying, and making decisions simultaneously, grasping an object with an appropriate load, which can effectively avoid slipping or damage when picking up the object subsequently, and has strong adaptability. Brief Description of the Drawings

[0058] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0059] Figure 1 It is a flowchart of the robotic arm loading method in the embodiment of the present invention;

[0060] Figure 2 It is a schematic diagram of the neural network model architecture in the embodiment of the present invention;

[0061] Figure 3 It is a schematic diagram of multi-step joint data processing in the embodiment of the present invention;

[0062] Figure 4 It is a load upper limit and probability distribution diagram in the embodiment of the present invention;

[0063] Figure 5 It is a load lower limit and probability distribution diagram in the embodiment of the present invention. Detailed Embodiments

[0064] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the drawings. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0065] In addition, the technical features involved in different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0066] The process of a robotic arm grasping an object can be decomposed into four stages, namely the positioning stage, the loading stage, the grasping stage, and the placing stage. The positioning stage refers to the robotic arm determining the position of the object to be grasped, moving, approaching, and finally contacting the object. The loading stage refers to, after the robotic arm contacts the object, adjusting the load of the robotic arm to determine the load value that can lift the object without damaging it. The grasping stage refers to grasping the object and moving it (moving upward and laterally above the target position). The placing stage refers to placing the object at the target position.

[0067] For the loading stage, this embodiment provides a robotic arm loading method, which can be executed by electronic devices such as a computer or a server, as Figure 1 shown, including the following operations:

[0068] S21, obtain the monitoring data when the robotic arm loads the object with the current load. The monitoring data includes the feedback data of the contact part between the robotic arm and the object and the feedback data of the driving device for changing the load.

[0069] The loading in this method refers to the process of the robotic arm grasping and holding a certain object on a supporting surface (such as the ground, a tabletop, or other platforms), excluding the action of moving the object upward or in other directions after grasping. This method can be executed synchronously with the change process of the robotic arm load. The initial load is denoted as F0. The robotic arm can be made to contact the object with the load F0, and then the load is gradually increased. As the load changes, the corresponding monitoring data also changes.

[0070] The contact part between the robotic arm and the object and the driving device for changing the load depend on the structure of the robotic arm. Taking a humanoid robotic arm as an example, the contact part is the surface of 5 fingers and 1 palm, and the driving device for changing the load can be a device for driving the finger joints to move. Specifically, each movable joint can correspond to an independent driving device.

[0071] The feedback data of the contact part between the robotic arm and the object can specifically include force, which, for a humanoid robotic arm, refers to the force (pressure or pressure) exerted by the object on the surfaces of the fingers and the palm. In an optional embodiment, it can also include temperature.

[0072] The feedback data of the driving device for changing the load can specifically include force and / or displacement, which, for a humanoid robotic arm, refers to the force borne by the driving device and / or the displacement provided to generate the current load.

[0073] As an example, for the load at any moment (time step), the obtained monitoring data can specifically include 、 、 and where The force of the i-th contact part The temperature of the i-th contact part The displacement of the i-th driving device The force of the i-th driving device

[0074] The load can be understood as an eigenvalue related to force. The eigenvalue can be one or more. Specifically, it can be the force data in the monitoring data or the value calculated according to the force data. As an example, the current load can be the value of a certain acting force among the acting forces on the finger and palm surfaces, or the sum of these acting forces, or the force of a certain driving device or the sum of the forces of the driving devices, etc.

[0075] S22, using a neural network model to extract the global feature vector from the feedback data, and determining the load upper limit information and the load lower limit information according to the global feature vector. The feedback data is at least two data from two different parts or driving devices in the robot hand. According to the above example, the feedback data may be dozens. In order to extract the feature vector, it is necessary to fuse these feedback data. The neural network model can adopt various network structures composed of units such as CNN (Convolutional Neural Network) and MLP (Multi-Layer Perceptron) to realize the extraction and fusion of features, and there are various specific implementation manners. For example, it can include the ResNet (Residual Network) structure.

[0076] The load upper limit means that the grasped object is likely to be damaged above this load (which can be called the damage load); the load lower limit means that it is worth trying to grasp the grasped object above this load, or it can be interpreted that the load lower limit is considered to be the lowest load that can pick up the grasped object.

[0077] It should be noted that in this application, the so-called load upper limit and load lower limit are the numerical values of force, while the load upper limit information and the load lower limit information can be the numerical values of force or the information calculated based on the numerical values of force, such as statistical information, etc.

[0078] The load upper limit information and the load lower limit information can be the output results of a neural network model, or the mapping information of the output results of the neural network model. For example, the neural network model can be configured to directly output the value of the load upper limit and the value of the load lower limit, and use a large number of sample data to train it before use so that it can output two values according to the above feedback data input; the neural network model can be configured to output the information of the gripper, such as the category of the gripper, etc., and a corresponding relationship library between this information and the load upper limit and the load lower limit is established in advance. When the neural network model outputs the information of the gripper during use, the corresponding load upper limit and load lower limit can be queried in the corresponding relationship library.

[0079] S23. Judge whether the current load will damage the gripper according to the load upper limit information. If it is determined that the current load will not damage the gripper, execute step S24; otherwise, as an optional solution, step S26 can be executed, or stop executing this method.

[0080] An optional way is to use a hard classification method in step S22 to determine that the gripper belongs to a certain category, and then obtain the corresponding load upper limit, or directly output the load upper limit by the neural network in step S22. In step S23, judge whether the current load is lower than the load upper limit. If the current load is lower than the load upper limit, it is determined that the current load will not damage the gripper.

[0081] S24. Judge whether the current load is sufficient to pick up the gripper according to the load lower limit information. If it is determined that the current load is not sufficient to pick up the gripper, execute step S25; otherwise, attempt to pick up the object with the current load, that is, enter the grasping stage.

[0082] An optional way is to use a hard classification method in step S22 to determine that the gripper belongs to a certain category, and then obtain the corresponding load lower limit, or directly output the load lower limit by the neural network in step S22. In step S24, judge whether the current load is lower than the load lower limit. If the current load is lower than the load upper limit, it is determined that the current load is not sufficient to pick up the gripper.

[0083] S25. Increase the load. Then return to step S21 and execute this method again until the current load meets the conditions of step S24. The increased amount of the load can be fixed, or the amount calculated according to the gap between the current load and the load lower limit information. For example, if the gap between the current load and the load lower limit is large, the increased amount is large, and so on.

[0084] S26. Set the load to the initial value, then return to step S21 and execute this method again. Further, before executing this method again, the posture of the robotic arm can be fine-tuned so that it can re-attempt the loading process from different angles or positions. The fine-tuning of the posture of the robotic arm can be to only fine-tune the angles of the mechanical joints, or to re-select the contact position.

[0085] According to the robotic arm loading method provided by the embodiments of the present invention, corresponding monitoring data is obtained along with the change of the grasping load, and then the load upper limit information and the load lower limit information corresponding to the current monitoring data are determined. The load is adjusted according to the relationship between the current load and the load upper limit information and the load lower limit information, so as to realize an adaptive grasping and loading process of loading, identifying and making decisions simultaneously, and grasping an object with an appropriate load, which can effectively avoid slipping or damage when grasping the object subsequently, and has strong adaptability.

[0086] Regarding the monitoring data in the above steps, in one embodiment, the feedback data of the contact part between the robotic arm and the grasped object includes the pressure data collected by the electronic skin arranged at the contact part between the robotic arm and the grasped object. The pressure data is the array data collected by the electronic skin respectively arranged on the palm and multiple fingers of the robotic arm.

[0087] For example, the th row and the th column of the th piece of electronic skin are recorded as .

[0088] In one embodiment, the feedback data of the contact part between the robotic arm and the grasped object includes the temperature data collected by the electronic skin arranged at the contact part between the robotic arm and the grasped object. The temperature data is the array data collected by the electronic skin respectively arranged on the palm and multiple fingers of the robotic arm.

[0089] For example, the th row and the th column of the th piece of electronic skin are recorded as .

[0090] According to the above example, the sizes of the electronic skins at different positions on the robotic arm or the number of measurement points therein may be different, so the number of rows and columns of the array is different. In one embodiment, the feedback data of the driving device for changing the load includes the displacement data and the force data of each driving device of the robotic arm. For example, the displacement data and the force data of the th driving device are respectively recorded as .

[0091] The above-mentioned various feedback data can accurately reflect the grasping state. By extracting features from these feedback data, the accuracy of the load upper limit information and the load lower limit information can be improved.

[0092] There are various specific structures of the neural network model in the above step S22. This embodiment provides a specific implementation manner. As shown in Figure 2 , step S22 includes the following operations:

[0093] S221. Extract the electronic skin force distribution features from the pressure data collected by each electronic skin respectively, extract the electronic skin temperature distribution features from the temperature data collected by each electronic skin respectively, and fuse the electronic skin force distribution features and the electronic skin temperature distribution features to obtain the electronic skin feature vector.

[0094] Specifically, input the array data of the force monitoring values of each piece of electronic skin into the CNN layer respectively, and then pass through several residual modules respectively. Take the average of the results obtained after processing to obtain the electronic skin force distribution features; input the array data of the temperature monitoring values of each piece of electronic skin into the CNN layer respectively, and then pass through several residual modules respectively. Take the average of the results obtained after processing to obtain the electronic skin temperature distribution features.

[0095] In some embodiments, the scales of the electronic skin temperature distribution features and the electronic skin force distribution features may be the same, so the electronic skin feature vector can be directly fused. If the scales of the two distribution features are different, adjustment is required:

[0096] Perform pooling on the one with higher pixel density among the electronic skin force distribution features and the electronic skin temperature distribution features, so as to adjust the number of pixels to be consistent (connect after unifying the number of pixels); fuse the adjusted electronic skin force distribution features and the electronic skin temperature distribution features (fuse the arrays); take the average of the fused results in the array dimension (array average) to obtain the electronic skin feature vector.

[0097] The above layers process each piece of electronic skin separately, and the parameters of the layers are shared.

[0098] S222. Extract the drive device feature vector from the data obtained by connecting the displacement data and the force data of the drive device. Further, first take the difference between the displacement data of the drive device at the current moment and the displacement data of the drive device at the previous moment ; connect the difference result with the force data of the drive device and then extract the drive device feature vector. In this embodiment, the MLP module is specifically used to obtain the robot drive device feature vector.

[0099] S223. Connect the electronic skin feature vector and the drive device feature vector to obtain the global feature vector.

[0100] It should be noted that the above module architecture is just one of many optional ways and not the only feasible solution. For example, normalization, dropout, etc. can be set at appropriate positions in the above process, and other modules can be used to replace the above MLP module and CNN module, etc., to form a deformed architecture. The difference in the displacement and force of the driving device reflects the relationship between the force and displacement in the clamping direction, the force distribution of the electronic skin reflects the force distribution characteristics along the contact tangential direction, and the temperature reflects information such as heat capacity and heat transfer. The above embodiments fuse these rich information, and the result can comprehensively reflect the material characteristics of the grasped object, thereby making the classification result more accurate.

[0101] In one embodiment, step S22 includes the following operations:

[0102] S224. Classify according to the global feature vector to obtain a classification vector regarding the category of the grasped object. Specifically, the global feature vector can be obtained in the manner of the above steps S221 - S223, or other network structures can be used, such as network structures like LSTM, etc.

[0103] S225. Determine the probability values of the grasped object belonging to various categories according to the classification vector. An optional way is to take the softmax of the component vectors output by the neural network model to obtain the probabilities regarding each category.

[0104] S226. Determine the load upper limit information and load lower limit information according to the probability values of the grasped object belonging to various categories and the load upper limit and load lower limit corresponding to various categories.

[0105] In a preferred embodiment, the probabilities regarding each category are obtained by a multi - step joint method. Combining Figure 3 The specific operations of step S225 shown are as follows:

[0106] Obtain multi - step joint data according to the classification vector at the current moment and at least one previous moment. There is more than one specific calculation method. In this embodiment, the classification vector at each moment is multiplied by a preset coefficient respectively. The preset coefficient in this embodiment is , where M is the number of time steps (the total number of the current moment and at least one previous moment), K and A are preset constants, and then the results of multiplication at each moment are summed. The sum result is combined with a preset background value B to obtain multi - step joint data, which is specifically a vector with a length of the number of time steps + 1. Obtain the probability values of the grasped object belonging to various categories according to the multi - step joint data. In this embodiment, the softmax is taken for the above - processed result. The values corresponding to each item except the constant are the probability values of the corresponding classifications. In addition, in this embodiment, the classification vector at moment is used to obtain multi - step joint data. can be interpreted as the current moment. is interpreted as the initial moment or any moment before Any previous moment. The component vectors obtained through steps S221 - S224 can be regarded as the feature information obtained from the feedback data for only one moment. Based on this, the category of the grasped object at this moment can be obtained. In practical applications, since the feedback data changes continuously over time, different categories of grasped objects are obtained at different moments. The above - mentioned embodiment adopts a multi - part joint method to comprehensively obtain the classification result by integrating the feature information of the current moment and multiple previous moments, which can make the classification result more stable and accurate.

[0107] Suppose there are four possible categories A, B, C, and D in total. The probability values that the grasped object belongs to the four categories obtained through the above - mentioned processing are denoted as PA, PB, PC, and PD. An optional implementation method is that in step S226, determine the maximum value among the probability values, and then determine the category to which the grasped object belongs; obtain the upper load limit and the lower load limit corresponding to the category to which the grasped object belongs.

[0108] As an example, assume that the maximum value among PA, PB, PC, and PD is PA, then it is determined that the grasped object belongs to category A, and obtain the upper load limit F1A and the lower load limit F2A associated with category A in the pre - stored relationship library. In the following steps, it is possible to directly judge the relationship between the current load and F1A and F2A to decide to increase the load, reset, or enter the grasping stage (see steps S23 - S26 above, which will not be elaborated here). This method has a small amount of calculation and a fast response speed, and can meet the needs of general application scenarios.

[0109] In a preferred embodiment, the load - upper - limit information determined in step S226 includes the upper load limits corresponding to various categories and the probabilities that the grasped object belongs to various categories, and the load - lower - limit information includes the lower load limits corresponding to various categories and the probabilities that the grasped object belongs to various categories. As an example, the upper load limit F1A and the lower load limit F2A of category A, the upper load limit F1B and the lower load limit F2B of category B, the upper load limit F1C and the lower load limit F2C of category C, the upper load limit F1D and the lower load limit F2D of category D, and the probability values that the grasped object belongs to the four categories obtained through the above - mentioned processing are PA, PB, PC, and PD.

[0110] Further, step S23 specifically includes: respectively comparing the current load with the upper load limits of each category, adding the probabilities of the categories that exceed the upper load limit to obtain the first cumulative probability; judging whether the first cumulative probability is lower than the first probability threshold; when the first cumulative probability is lower than the first probability threshold, it is determined that the current load will not damage the grasped object.

[0111] Through Figure 4Express the relationship among the load upper limit, probability, and the current load. The abscissa represents the load, and the ordinate represents the probability. Since the current load is greater than the load upper limit F1A of category A and greater than the load upper limit F1B of category B, the probabilities PA and PB are added to obtain the first cumulative probability. If PA + PB is greater than the first probability threshold, it is determined that the current load will damage the gripper; otherwise, it is determined that the current load will not damage the gripper.

[0112] Step S24 specifically includes: comparing the current load with the load lower limits of each category respectively, and adding the probabilities of the categories exceeding the load lower limit to obtain the second cumulative probability; determining whether the second cumulative probability is lower than the second probability threshold; when the second cumulative probability is lower than the second probability threshold, it is determined that the current load is not sufficient to pick up the gripper.

[0113] By Figure 5 Express the relationship among the load lower limit, probability, and the current load. The abscissa represents the load, and the ordinate represents the probability. Since the current load is greater than the load lower limit F2A of category A and greater than the load lower limit F2B of category B, the probabilities PA and PB are added to obtain the second cumulative probability. If PA + PB is greater than the second probability threshold, it is determined that the current load can pick up the gripper; otherwise, it is determined that the current load is not sufficient to pick up the gripper.

[0114] Using the above method of calculating the cumulative probability and comparing it with the probability threshold to judge the current load can improve the accuracy of the judgment result. Especially when the probabilities of the obtained grippers belonging to various categories are relatively close or do not conform to the actual situation, the above preferred solution can avoid mistakenly entering the grasping stage.

[0115] In one embodiment, the current load includes two different values for applying to the above steps S23 and S24 respectively. Specifically, in this embodiment, the forces of the electronic skins of each piece of the robotic hand are array data , represents the number of electronic skins, and j and k represent the rows and columns of the array. For the convenience of explanation, assume that each of the 5 fingers is provided with 1 electronic skin and the palm is provided with 1 electronic skin. Then the first characteristic contact force value is the maximum value in all the array data, that is, max( ); the second characteristic contact force value is the relatively smaller value between the maximum value in the array data collected by the electronic skin of the thumb and the maximum value in the array data collected by the electronic skins of the other fingers. The maximum value in the array data of the thumb is denoted as max( ), and the maximum value in the array data of the other four fingers is denoted as max( , , , ), then the second characteristic contact force value is min(max( ), max( , , , ))

[0116] In step S23, the current load can adopt max( ), and in step S24 above, the current load can adopt min(max( ), max( , , , )), and in step S26, the initial value of the load can be set with any one of the above two characteristic values as the target.

[0117] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 block or multiple blocks.

[0119] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 block or multiple blocks.

[0120] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are executed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions for implementing the steps specified in one process or a plurality of processes and / or blocks Figure 1 one process or a plurality of processes and / or blocks Figure 1 steps of the functions specified in one block or a plurality of blocks.

[0121] Obviously, the above-described embodiments are merely examples for clear illustration and are not intended to limit the implementation manners. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation manners here. And the obvious changes or modifications derived therefrom are still within the protection scope of the present invention.

Claims

1. A robot arm loading method, characterized in that, Including: S21. Obtain the monitoring data when the robotic arm grabs an object with the current load. The monitoring data includes the feedback data of the contact part between the robotic arm and the object to be grabbed and the feedback data of the driving device for changing the load; S22. Use a neural network model to extract global feature vectors from the feedback data, and determine the upper load limit information and the lower load limit information according to the global feature vectors; S23. Judge whether the current load will damage the object to be grabbed according to the upper load limit information. If it is determined that the current load will not damage the object to be grabbed, execute step S24; otherwise, execute step S26; S24. Judge whether the current load is sufficient to grab the object to be grabbed according to the lower load limit information. If it is determined that the current load is not sufficient to grab the object to be grabbed, execute step S25; otherwise, attempt to grab the object with the current load and enter the grasping stage; S25. Increase the load and return to step S21; S26. Set the load to the initial value and return to step S21.

2. The method according to claim 1, wherein The feedback data of the contact part between the robotic arm and the object to be grabbed includes the pressure data collected by the electronic skin arranged at the contact part between the robotic arm and the object to be grabbed.

3. The method according to claim 2, wherein The pressure data is the array data collected by the electronic skins respectively arranged on the palm and multiple fingers of the robotic arm.

4. The method according to claim 3, characterized in that The current load includes a first characteristic contact force value and a second characteristic contact force value. The first characteristic contact force value is the maximum value in all the array data, and the second characteristic contact force value is the relatively smaller one between the maximum value in the array data collected by the electronic skin of the thumb and the maximum value in the array data collected by the electronic skins of other fingers; In the step of judging whether the current load will damage the object to be grabbed according to the upper load limit information, the first characteristic contact force value is adopted; in the step of judging whether the current load is sufficient to grab the object to be grabbed according to the lower load limit information, the second characteristic contact force value is adopted.

5. The method according to claim 2, wherein The feedback data of the contact part between the robotic arm and the object to be grabbed includes the temperature data collected by the electronic skin arranged at the contact part between the robotic arm and the object to be grabbed.

6. The method according to claim 5, characterized in that, The temperature data is the array data collected by the electronic skins respectively arranged on the palm and multiple fingers of the robotic arm.

7. The method according to claim 5, characterized in that, The feedback data of the driving device for changing the load includes the displacement data and force data of each driving device of the robotic arm.

8. The method according to claim 7, wherein Using a neural network model to extract global feature vectors from the feedback data includes: Respectively extract the electronic skin force distribution features from the pressure data collected by each electronic skin, and respectively extract the electronic skin temperature distribution features from the temperature data collected by each electronic skin, and fuse the electronic skin force distribution features and the electronic skin temperature distribution features to obtain an electronic skin feature vector; Extract the driving device feature vector from the data connected based on the displacement data and force data of the driving device; Connect the electronic skin feature vector and the driving device feature vector to obtain a global feature vector.

9. The method according to claim 8, characterized in that, Fusing the electronic skin force distribution features and the electronic skin temperature distribution features to obtain an electronic skin feature vector includes: Perform pooling processing on the one with higher pixel density among the electronic skin force distribution features and the electronic skin temperature distribution features, so as to adjust the number of pixels to be the same; Fuse the adjusted force distribution characteristics and temperature distribution characteristics of the electronic skin; Take the average of the fused results in the array dimension to obtain the electronic skin feature vector.

10. The method according to claim 8, characterized in that, Extract the drive device feature vector from the data joined by the displacement data and force data of the drive device, including: Take the difference between the displacement data at the current moment and the displacement data at the previous moment; Join the difference result with the force data, and then extract the drive device feature vector.

11. The method according to claim 1, characterized in that, Determine the load upper limit information and load lower limit information according to the global feature vector, including: Classify according to the global feature vector to obtain a classification vector about the category of the grasped object; Determine the probability values of the grasped object belonging to various categories according to the classification vector; Determine the load upper limit information and load lower limit information according to the probability values of the grasped object belonging to various categories and the load upper limit and load lower limit corresponding to various categories.

12. The method according to claim 11, wherein, Determine the probability values of the grasped object belonging to various categories according to the classification vector, including: Obtain multi-step joint data according to the classification vector at the current moment and at least one previous moment; Obtain the probability values of the grasped object belonging to various categories according to the multi-step joint data.

13. The method according to claim 12, characterized in that, Obtain multi-step joint data according to the classification vector at the current moment and at least one previous moment, including: Multiply the classification vector at each moment by a preset coefficient respectively, and then sum the results of multiplication at each moment; Combine the sum result with a preset background value to obtain the multi-step joint data.

14. The method according to claim 11, wherein Determine the load upper limit information and load lower limit information according to the probability values of the grasped object belonging to various categories and the load upper limit and load lower limit corresponding to various categories, including: Determine the maximum value among the probability values, and then determine the category to which the grasped object belongs; Obtain the load upper limit and load lower limit corresponding to the category to which the grasped object belongs.

15. The method according to claim 14, wherein Judge whether the current load will damage the grasped object according to the load upper limit information, including: Judge whether the current load is lower than the load upper limit; If the current load is lower than the load upper limit, determine that the current load will not damage the grasped object; Judge whether the current load is sufficient to pick up the grasped object according to the load lower limit information, including: Judge whether the current load is lower than the load lower limit; If the current load is lower than the load lower limit, determine that the current load is not sufficient to pick up the grasped object.

16. The method according to claim 11, characterized in that The load upper limit information includes the load upper limit corresponding to various categories and the probability of the grasped object belonging to various categories; the load lower limit information includes the load lower limit corresponding to various categories and the probability of the grasped object belonging to various categories.

17. The method according to claim 16, wherein, Judge whether the current load will damage the grasped object according to the load upper limit information, including: Compare the current load with the load upper limit of each category respectively, and add the probabilities of the categories exceeding the load upper limit to obtain a first cumulative probability; Judge whether the first cumulative probability is lower than a first probability threshold; When the first cumulative probability is lower than the first probability threshold, determine that the current load will not damage the grasped object; Judge whether the current load is sufficient to pick up the grasped object according to the load lower limit information, including: Compare the current load with the lower limit of the load for each category respectively, and add the probabilities of the categories exceeding the lower limit of the load to obtain a second cumulative probability; Determine whether the second cumulative probability is lower than the second probability threshold; When the second cumulative probability is lower than the second probability threshold, determine that the current load is not sufficient to pick up the object to be grasped.

18. A robotic arm loading device, characterized in that, Comprising: A processor and a memory connected to the processor; wherein, the memory stores instructions executable by the processor, and the instructions are executed by the processor to enable the processor to execute the robotic arm loading method according to any one of claims 1-17.

Images