A behavior test method and related device for a multi-modal perception humanoid robot

Abstract

The application discloses a behavior test method and related equipment of a multi-modal perception humanoid robot, and the method comprises the following steps: acquiring multi-modal perception data and current joint state data, splicing the multi-modal data and the current joint state data to determine a current state vector; sampling actions according to the current state vector and a trained processing model to obtain a plurality of sampling actions; calculating according to the plurality of sampling actions, the current state vector and a preset running model to determine a plurality of candidate action states from the sampling actions; performing discriminant calculation on the candidate action states according to a preset neural network model to obtain a total discriminant score, and determining a target action state according to the total discriminant score; simulating and evaluating the humanoid robot according to a preset scene test case and the target action state to generate a corresponding test report. The embodiment of the application can improve the action adaptability and scene adaptability. The application can be widely applied in the technical field of humanoid robots.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of humanoid robots, and particularly relates to a behavior testing method of a multi-modal perception humanoid robot and related equipment. BACKGROUND

[0002] With the rapid development of robot technology, humanoid robots are increasingly applied in process scenes and home scenes to perform corresponding tasks in complex environments, such as industrial assembly, house cleaning, and the like. Existing technologies are preprogrammed according to tasks, and humanoid robots follow preset programming procedures along fixed trajectories, which is poor in dynamic adaptability and scene adaptability. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a behavior testing method of a multi-modal perception humanoid robot and related equipment, which can improve testing efficiency and improve the safety of simulated behaviors of the humanoid robot.

[0004] To achieve the above purpose, one aspect of the embodiments of the present application provides a behavior testing method of a multi-modal perception humanoid robot, which comprises:

[0005] Obtaining multi-modal perception data and current joint state data, splicing the multi-modal perception data and the joint state data to determine a current state vector;

[0006] Sampling actions according to the current state vector and a trained processing model, determining a plurality of sampling actions, and calculating according to the plurality of sampling actions, the current state vector and a preset motion model to determine a plurality of candidate action states;

[0007] Discriminating and calculating according to a preset neural network model and the plurality of candidate action states to determine a plurality of total discrimination scores, and analyzing according to the plurality of total discrimination scores and the plurality of candidate action states to determine a target behavior state;

[0008] Testing and evaluating according to the target behavior state and a preset scene test case to determine a test report.

[0009] In some embodiments, the current state vector is subjected to feature extraction according to the trained processing model to determine multi-modal features, and the multi-modal features are subjected to feature fusion to determine fusion feature data;

[0010] Performing full connection processing according to a preset robot dual-arm shared weight and the fusion feature data to determine weight feature data;

[0011] According to the weight feature data and a preset activation function, an action space is determined, and according to the action space and a preset degree of freedom parameter, a plurality of sampling actions are sampled to determine the plurality of sampling actions.

[0012] In some embodiments, the plurality of sampling actions are analyzed respectively to determine a plurality of action motor parameters; wherein the action motor parameters include motor currents and reduction ratios corresponding to the sampling actions;

[0013] According to the plurality of action motor parameters and a first preset formula, a plurality of first torque vectors are determined, and according to a preset torque threshold, the plurality of first torque vectors are adjusted respectively to determine a plurality of joint torque vectors;

[0014] The current state vector is analyzed to determine a current joint angle vector and a current joint angular velocity vector, and according to the current joint angle vector and the current joint angular velocity vector, the plurality of joint torque vectors and the preset motion model, a plurality of next time joint angle vectors are determined;

[0015] According to the plurality of next time joint angle vectors and a preset joint limit threshold, a plurality of candidate action states are determined.

[0016] In some embodiments, the plurality of candidate action states are analyzed to determine a plurality of torque data, a plurality of visual data and a plurality of tactile data;

[0017] According to the plurality of torque data and the plurality of visual data, full connection processing is performed to determine first output data, and the first output data is activated to determine a safety sub-score;

[0018] According to the plurality of visual data and the plurality of tactile data, an adaptive sub-score is determined;

[0019] According to the safety sub-score, the adaptive sub-score and a preset weight, the total discrimination score is determined.

[0020] In some embodiments, the plurality of total discrimination scores are compared with a preset scene threshold respectively;

[0021] If the total discrimination score is greater than or equal to the preset scene threshold, the candidate action state corresponding to the total discrimination score is taken as the target behavior state;

[0022] If the total discrimination score is less than the preset scene threshold, the process of sampling actions according to the current state vector and the trained processing model is returned to execute until the total discrimination score is greater than or equal to the preset scene threshold.

[0023] In some embodiments, behavior simulation is performed according to the preset scene test case and the target behavior state, and a simulation result is determined;

[0024] According to the simulation result and a preset evaluation index, an action output quality is calculated, and the test report is determined according to the action output quality.

[0025] To achieve the above-mentioned purposes, another aspect of the embodiments of the present application proposes a behavior test system of a multi-modal perception humanoid robot, which comprises:

[0026] A data acquisition module is configured to acquire multi-modal perception data and current joint state data, splice the multi-modal perception data and the joint state data, and determine a current state vector;

[0027] An action sampling module is configured to sample actions according to the current state vector and a trained processing model, determine a plurality of sampling actions, and calculate a plurality of candidate action states according to the plurality of sampling actions, the current state vector, and a preset motion model;

[0028] A discriminant calculation module is configured to calculate discriminants according to the plurality of candidate action states and a preset neural network model, determine a plurality of total discriminants, and analyze the plurality of total discriminants and the plurality of candidate action states to determine a target behavior state;

[0029] A test evaluation module is configured to evaluate tests according to the target behavior state and a preset scene test case, and determine a test report.

[0030] To achieve the above-mentioned purposes, another aspect of the embodiments of the present application proposes an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned method when executing the computer program.

[0031] To achieve the above-mentioned purposes, another aspect of the embodiments of the present application proposes a computer readable storage medium, which stores a computer program, and the computer program implements the above-mentioned method when executed by a processor.

[0032] To achieve the above-mentioned purposes, another aspect of the embodiments of the present application proposes a computer program product, which comprises a computer program, and the computer program implements the above-mentioned method when executed by a processor.

[0033] The embodiments of the present application at least have the following beneficial effects: The present application provides a behavior test method, system, electronic device, storage medium and program product of a multi-modal perception humanoid robot, which acquires multi-modal perception data and current joint state data of the humanoid robot, splices the multi-modal data and the current joint state data to determine a current state vector, samples actions according to the current state vector and a trained processing model to obtain a plurality of sample actions, calculates the plurality of sample actions, the current state vector obtained by splicing and a preset running model to select a plurality of candidate action states from the obtained sample actions, performs discriminant calculation on the plurality of candidate action states according to a preset neural network model to obtain a total discriminant score, and determines a target action state from the candidate action states according to the total discriminant score, simulates the humanoid robot according to a preset scene test case and the determined target action state, and performs evaluation to generate a corresponding test report, improves the dynamic adaptability of the humanoid robot action by collecting multi-modal perception data and combining the current joint state data of the humanoid robot for action simulation, and provides scene adaptability according to multi-modal perception data feedback real-time environment, multi-modal perception data and current joint state data for action sampling to determine a target behavior state for simulation. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a flowchart of a behavior test method of a multi-modal perception humanoid robot provided by the embodiments of the present application;

[0035] Figure 2 is a flowchart of step S102 in Figure 1

[0036] Figure 3 is another flowchart of step S102 in Figure 1

[0037] Figure 4 is a flowchart of step S103 in Figure 1

[0038] Figure 5 is another flowchart of step S103 in Figure 1

[0039] Figure 6 is a flowchart of step S104 in Figure 1

[0040] Figure 7 is a flowchart of a test process in a specific embodiment provided by the embodiments of the present application;

[0041] Figure 8 ​​​​​is a working process schematic diagram of a multi-modal perception module in an embodiment provided by the present application;

[0042] Figure 9 is a working process schematic diagram of a strategy neural network in an embodiment provided by the present application;

[0043] Figure 10 is a flowchart of calculating a total discrimination score by a discrimination module in an embodiment provided by the present application;

[0044] Figure 11 is a flowchart of performing simulation test in an embodiment provided by the present application;

[0045] Figure 12 is a structural schematic diagram of a behavior test system of a multi-modal perception humanoid robot provided by an embodiment of the present application;

[0046] Figure 13 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not used to limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0048] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0049] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0051] The embodiment of the application provides a behavior test method of a multi-modal perception humanoid robot, and relates to the field of information technology. The behavior test method of the multi-modal perception humanoid robot can be applied to a terminal, can be applied to a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto; the server end can be configured as an independent physical server, can be configured as a server cluster or a distributed system formed by multiple physical servers, can be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and can also be a node server in a blockchain network; and the software can be an application for realizing the behavior test method of the multi-modal perception humanoid robot, and the like, but is not limited to the above forms.

[0052] The application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions executed by a computer, such as a program module. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0053] It should be noted that in various specific embodiments of the present application, when relevant processing needs to be performed on data related to the identity or characteristics of the user, such as user information, user behavior data, user history data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the user's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for the normal operation of the embodiments of the present application will be obtained.

[0054] Figure 1 is an optional flowchart of a behavior test method of a multi-modal perception humanoid robot provided by the embodiments of the present application, Figure 1 The method in can include but is not limited to steps S101-S104.

[0055] Step S101, obtain multi-modal perception data and current joint state data, splice the multi-modal perception data and the joint state data to determine a current state vector;

[0056] Step S102, sample actions according to the current state vector and a trained processing model, determine a plurality of sample actions, and calculate according to the plurality of sample actions, the current state vector, and a preset motion model to determine a plurality of candidate action states;

[0057] Step S103, perform discriminant calculation according to the plurality of candidate action states and a preset neural network model to determine a plurality of total discriminant scores, and analyze the plurality of total discriminant scores and the plurality of candidate action states to determine a target behavior state;

[0058] Step S104, perform test evaluation according to the target behavior state and a preset scene test case to determine a test report.

[0059] The steps S101 to S104 shown in the embodiments of the present application are real-time data collection through the data collection device arranged on the biped humanoid robot, such as real-time collection of scene images through a binocular camera, real-time collection of real-time torque data of each joint of the humanoid robot through a joint torque sensor, real-time collection of pressure distribution through a tactile sensor, acquisition of joint state data of each joint of the humanoid robot through a sensor, including the current activity angle and angular velocity of each joint, and the like; the collected real-time data are respectively preprocessed to obtain multi-modal perception data and current joint data of the robot; in the present embodiment, the collected visual data are subjected to noise reduction processing, and feature processing is performed through a target monitoring model to extract scene feature data; the collected joint torque data are subjected to low-pass filtering to reduce noise interference, and are subjected to normalization processing to obtain multi-dimensional joint torque data, the dimension of the joint torque data being adaptively adjusted according to the joint degrees of freedom of the humanoid robot; the collected joint state data are subjected to normalization processing to obtain multi-dimensional joint state data, the dimension of which is adaptively adjusted according to the joint degrees of freedom; the tactile data are subjected to pressure center calculation and normalization processing; the preprocessed data are subjected to data splicing to obtain a current state vector representing the current state of the biped humanoid robot, the dimension of which is equal to the sum of the dimensions of the collected data; then, the current state vector is subjected to action sampling using the processed model constructed and trained, and is processed in combination with the pre-set dual-arm cooperation mechanism to obtain an executable sampling action of the humanoid robot; then, the high-degree-of-freedom motion model is analyzed based on the current state vector of the robot, the sampling action input, and the established high-degree-of-freedom motion model to determine a sampling action in the sampling action that meets the requirements of the robot for action execution, and the action state at the next moment is calculated according to the sampling action as a candidate action state, so as to calculate a discrimination score of the screened action, judge whether the safety and scene adaptability of the action meet the corresponding requirements according to the calculated total discrimination score, and thus determine whether to simulate test the humanoid robot according to the screened action and generate a corresponding test report.

[0060] Please refer to Figure 2 In some embodiments, step S102 can include but is not limited to steps S201 to S203:

[0061] Step S201, according to the trained processing model, the current state vector is subjected to feature extraction, multi-modal features are determined, and the multi-modal features are subjected to feature fusion to determine fusion feature data;

[0062] Step S202, according to the pre-set robot dual-arm shared weight and the fusion feature data, full connection processing is performed to determine weight feature data;

[0063] In step S203, the weight feature data and the preset activation function are calculated to determine the action space, and the action space and the preset degree of freedom parameter are used for action sampling to determine a plurality of sampling actions.

[0064] In step S201 of some embodiments, the current state vector obtained by splicing the data is input into the trained processing model to extract features, obtain multi-modal feature data including visual features, torque features, and tactile features, and perform feature fusion on the feature data of different modalities to enhance the feature representation and obtain fused feature data. In this embodiment, a strategy neural network is constructed to extract and fuse features of the current state vector. The strategy neural network is configured with three convolutional layers to extract features of the input current state vector, and the extracted feature data is input into two fully connected layers to perform feature fusion and obtain feature fusion data.

[0065] In step S202 of some embodiments, a dual-arm weight sharing mechanism is introduced to improve the coordination of the dual arms of the humanoid robot when performing feature fusion on the extracted multi-modal feature data, and the dual arms of the humanoid robot are shared by weight. Then, the fused feature data is fully connected to obtain weight feature data. In this embodiment, the sharing range of the dual-arm weight in the dual-arm sharing mechanism of the humanoid robot is adaptively adjusted according to the dual-arm degree of freedom.

[0066] In step S203 of some embodiments, the processing model outputs the weight feature data output by the fully connected layer to the output layer for processing, calculates the weight feature data through an activation function, and outputs the probability distribution result of the action space. Then, the probability distribution result of the action space and the preset degree of freedom parameter are used for action sampling to obtain sampling actions. In this embodiment, each degree of freedom of the humanoid robot is provided with three sampling actions, i.e., three actions are sampled for each degree of freedom in the action space.

[0067] Please refer to Figure 3 In some embodiments, step S102 can further include but is not limited to steps S301 to S304:

[0068] Step S301: The plurality of sampling actions are analyzed respectively to determine a plurality of action motor parameters; wherein the action motor parameters include motor currents and reduction ratios corresponding to the sampling actions.

[0069] Step S302: A plurality of first torque vectors are determined according to the plurality of action motor parameters and a first preset formula, and a plurality of joint torque vectors are determined by adjusting the plurality of first torque vectors according to a preset torque threshold.

[0070] In step S303, the current state vector is parsed to determine the current joint angle vector and the current joint angular velocity vector, and the current joint angle vector and the current joint angular velocity vector, a plurality of joint torque vectors, and a preset motion model are calculated to determine a plurality of next time joint angle vectors;

[0071] In step S304, the plurality of next time joint angle vectors are constrained according to a preset joint limit threshold to determine a plurality of candidate action states.

[0072] In step S301 of some embodiments, after the action space is sampled to obtain the sampled action of each degree of freedom of the humanoid robot, the sampled action is parsed to determine the motor current and the reduction ratio corresponding to the sampled action, so that whether the sampled action meets the humanoid robot to execute the action is analyzed subsequently, and the next time action state is calculated according to the sampled action as a candidate action state for subsequent simulation test of the humanoid robot.

[0073] In step S302 of some embodiments, the motor parameters corresponding to the sampled action of each degree of freedom are calculated according to a preset formula to determine the torque vector of the degree of freedom of the humanoid robot at the current time, and whether the torque vector meets the torque threshold set by the corresponding application scenario is determined, and the calculated torque vector is adjusted according to the set torque threshold for subsequent calculation of the joint state data of each degree of freedom at the next time; in this embodiment, the torque vector is calculated by the following formula:

[0074] ,

[0075] wherein, is the sampled action corresponding joint torque vector, is the torque constant of the joint motor, is the sampled action corresponding motor current, is the reduction ratio of the motor.

[0076] In step S303 of some embodiments, the current state vector obtained by data splicing is parsed to determine the current joint angular velocity vector and the joint velocity vector of each degree of freedom of the humanoid robot; then, the current joint angular velocity vector and the joint velocity vector of each degree of freedom, and the torque vector calculated in the foregoing are input into a high degree of freedom motion model for calculation to calculate the joint angle vector of each degree of freedom at the next time; in this embodiment, the high degree of freedom motion model is:

[0077] ,

[0078] wherein, a joint angle vector of a next time instant, a joint angle vector of a current time instant, a time step, a joint angular velocity vector of a current time instant, a diagonal moment coefficient matrix of , which is set according to a joint load difference, a total number of degrees of freedom of the humanoid robot, a joint torque vector corresponding to a current sampling action, a diagonal damping coefficient matrix of a static friction coefficient vector of dimension, a sign function of angular velocity.

[0079] In step S304 of some embodiments, according to the calculated joint angle vector of the next time instant and the preset joint angle threshold, it is determined whether the calculated joint angle vector of the next time instant exceeds the angle limit of the current degree of freedom, and the joint angle vector exceeding the angle limit is truncated to the limit value of the angle limit as the candidate action state of the next time instant.

[0080] Referring to Figure 4 In some embodiments, step S103 can include but is not limited to steps S401 to S404:

[0081] Step S401, a plurality of candidate action states are analyzed to determine a plurality of torque data, a plurality of visual data and a plurality of tactile data;

[0082] Step S402, according to the plurality of torque data and the plurality of visual data, full connection processing is performed to determine first output data, and the first output data is activated to determine a safety sub-score;

[0083] Step S403, according to the plurality of visual data and the plurality of tactile data, calculation is performed to determine an adaptation sub-score;

[0084] Step S404, according to the safety sub-score, the adaptation sub-score and a preset weight, calculation is performed to determine a total discrimination score.

[0085] ​​In step S401 of some embodiments, the total discrimination score is calculated according to the calculated candidate action state, whether the candidate action state meets the safety in the corresponding scene is judged by the total discrimination score, and whether the candidate action adapts to the corresponding scene is judged, so as to select the optimal action from the candidate actions calculated from each degree of freedom according to the total discrimination score for simulation test; in the embodiment, the corresponding torque data, visual data and tactile data are determined by analyzing the candidate actions of each degree of freedom, so as to calculate the total discrimination score according to the torque data, visual data and tactile data.

[0086] In step S402 of some embodiments, the total discrimination score of each candidate action is calculated by the constructed multi-modal discriminator neural network, which is provided with two branches to calculate the total discrimination score according to the input multi-modal data, including calculating the safety sub-score and the adaptation sub-score; the multi-modal discriminator neural network extracts the torque vector from the input torque data, extracts the obstacle distance data from the visual data, processes the extracted torque vector and obstacle distance data through the full connection layer, and calculates through the activation function, and outputs the corresponding safety sub-score; at the same time, according to the torque vector and the set scene threshold, and according to the obstacle distance data and the set distance threshold, the corresponding safety sub-score range is determined by comprehensive judgment; in the embodiment, when the torque vector is less than or equal to the set scene threshold, such as 30N·m for industrial scene and 20N·m for household scene, and the obstacle distance is greater than or equal to 5cm, the safety sub-score is greater than or equal to 0.8, otherwise it is less than or equal to 0.3.

[0087] In step S403 of some embodiments, another branch in the multi-modal discriminator neural network processes the visual data and tactile data to determine the assembly precision in the industrial scene or the action flexibility in the household scene; the corresponding precision threshold is determined according to the degree of freedom of the humanoid robot double arms to ensure that it matches the degree of freedom ability; the corresponding adaptation sub-score is determined according to the assembly precision or the action flexibility.

[0088] In step S404 of some embodiments, the safety sub-score and the adaptation sub-score are calculated according to the calculated safety sub-score and the adaptation sub-score, the weights are dynamically adjusted according to the scene risk level, the safety sub-score and the adaptation sub-score are weighted and summed to obtain the total discrimination score:

[0089] ,

[0090] wherein, the total discrimination score, the weight, the safety sub-score, Adaptation score; in this embodiment, in the industrial scene, the weight value range is 0.7-0.8, to preferentially guarantee the safety in the high-risk scene, and in the home scene, the weight value range is 0.5-0.6, to balance the safety and flexibility in the medium-risk scene.

[0091] Please refer to Figure 5 In some embodiments, step S103 can further include but is not limited to steps S501-S503:

[0092] Step S501, respectively comparing the total discrimination scores with the preset scene threshold values;

[0093] Step S502, if the total discrimination score is greater than or equal to the preset scene threshold value, taking the candidate action state corresponding to the total discrimination score as the target behavior state;

[0094] Step S503, if the total discrimination score is less than the preset scene threshold value, returning to perform action sampling according to the current state vector and the trained processing model to determine a plurality of sampling actions, until the total discrimination score is greater than or equal to the preset scene threshold value.

[0095] In step S501 of some embodiments, the total discrimination scores corresponding to each candidate action are determined through the foregoing calculation process, and the next time action state is selected by comparing the total discrimination scores corresponding to each candidate action with the corresponding scene threshold values.

[0096] In step S502 of some embodiments, if the total discrimination score corresponding to the candidate action is greater than or equal to the corresponding scene threshold value, the candidate action is taken as the target behavior state for subsequent simulation test of the humanoid robot; in this embodiment, the candidate actions corresponding to each degree of freedom can also be sorted according to the total discrimination scores, and the candidate action with the highest total discrimination score is selected as the target action state.

[0097] In step S503 of some embodiments, if the total discrimination score corresponding to the candidate action is less than the corresponding scene threshold value, the action space is returned to perform action sampling, the candidate actions corresponding to the degree of freedom are determined again, and the corresponding total discrimination scores are calculated again to determine the target action state.

[0098] Please refer to Figure 6 In some embodiments, step S104 can further include but is not limited to steps S601-S602:

[0099] Step S601, performing behavior simulation according to the preset scene test case and the target behavior state to determine a simulation result;

[0100] In step S602, the action output quality is determined according to the calculation result and the preset evaluation index, and the test report is determined according to the action output quality.

[0101] In step S601 of some embodiments, after obtaining the target action state corresponding to each degree of freedom, the target action state of each degree of freedom of the humanoid robot is tested according to the preset scene test case, and the corresponding test result is determined, including action execution quality, action safety, etc.; in this embodiment, the scene test case includes industrial assembly part offset, household obstacle shielding, etc., and the action output quality of the humanoid robot is evaluated through multi-time step behavior simulation.

[0102] In step S602 of some embodiments, the target action state of each degree of freedom of the humanoid robot is tested according to the scene test case, the number of actions meeting the scene requirements or test requirements under the test scene is counted, the corresponding evaluation index such as success execution rate and safety violation rate is calculated according to the counted number and the total number of test simulation, and the action output quality is determined, and the corresponding test report is generated.

[0103] Next, the scheme of the embodiments of the present application will be described and explained in detail in combination with specific application examples:

[0104] Please refer to Figure 7 , Figure 7 is a system flowchart of a behavior test method of a multi-modal perception humanoid robot provided by the embodiments of the present application in specific embodiments, a multi-modal perception module is set to collect data, including collecting visual data through a binocular camera, collecting corresponding torque vectors through a corresponding number of joint torque sensors according to the degrees of freedom of the humanoid robot, for example, in an industrial scene of bearing assembly, there are 28 degrees of freedom, the degrees of freedom of the arms are 14, the assembly accuracy is less than or equal to 0.1mm, and the joint torque threshold is 30N·m; 28-dimensional torque vector data are collected through the joint torque sensors; the tactile data of the humanoid robot are collected through the tactile sensors, so as to calculate the pressure barycentric offset data of the humanoid robot; the multi-modal perception module processes the collected multi-modal perception data through a data preprocessing unit, wherein the working process of the multi-modal perception module is as shown in Figure 8 , the torque data or joint data of the corresponding degree of freedom dimension are adapted, and then the output data of the data preprocessing unit are data fused through a data fusion unit to output a 2136-dimensional state vector; the behavior generation module processes the 2136-dimensional state vector through a strategy neural network to generate a corresponding action space; wherein the working process of the strategy neural network is as shown in Figure 9As shown, convolution processing is performed through three convolutional layers, employing a dual-arm degree-of-freedom weight sharing mechanism and fully connected layers. The motion space is output through a softmax output layer, and motion sampling is performed within this space. The adopted motions of the humanoid robot are: left arm shoulder joint -2°, right arm shoulder joint +1°, torso pitch ±10°, lateral tilt ±5°, and lower limb hip joint angle error less than or equal to 2°. The next motion state is calculated using a high-degree-of-freedom motion model, yielding 28.1° (left arm) and 31.2° (right arm) as candidate states. The discrimination module uses... Figure 10 The process shown calculates the total discrimination score. The multimodal discrimination neural network uses a branch network to evaluate candidate states, obtaining a safety sub-score of 0.92 and an adaptation sub-score of 0.88. Based on the weight of 0.75 configured in the score combination unit, the total discrimination score for each candidate state is calculated to be 0.91. The multimodal discrimination neural network is trained using real robot safety behavior data from different scenarios, and cross-entropy loss is used as the training loss function. The cross-entropy loss is as follows:

[0105] ,

[0106] in, For cross-entropy loss, The model outputs a total discrimination score, where y=1 corresponds to truly safe data and y=0 corresponds to unsafe data. Based on preset test case input units and degree-of-freedom adaptation rules, the humanoid robot is input into the control test module to conduct simulated tests. Figure 11 As shown, the assembly of a 50mm diameter bearing and a 50.05mm inner diameter bushing was performed, including gripping, alignment, and pressing. The test results were evaluated by the performance evaluation unit. The force feedback was determined to be 28-dimensional data, the torque vector was 25 N·m, which is less than the scene threshold, the assembly accuracy was 0.1mm, the success rate of the humanoid robot was determined to be 92%, and the safety violation rate was 0.8%, which met the test requirements.

[0107] The embodiments of the present application at least have the following beneficial effects: the present application provides a behavior test method, system, electronic device, storage medium and program product of a multi-modal perception humanoid robot, which acquires multi-modal perception data and current joint state data of the humanoid robot, splices the multi-modal data and the current joint state data to determine a current state vector, samples actions according to the current state vector and a trained processing model to obtain a plurality of sample actions, calculates the plurality of sample actions, the spliced current state vector and a preset running model to select a plurality of candidate action states from the sample actions, performs discriminant calculation on the plurality of candidate action states according to a preset neural network model to obtain a total discriminant score, determines a target action state from the candidate action states according to the total discriminant score, simulates the humanoid robot according to a preset scene test case and the determined target action state, performs evaluation, and generates a corresponding test report; the multi-modal perception data is collected, and the action simulation is performed in combination with the current joint state data of the humanoid robot to improve the dynamic adaptability of the humanoid robot action; the real-time environment is fed back according to the multi-modal perception data, the action sampling is performed based on the multi-modal perception data and the current joint state data to determine the target behavior state for simulation, and the scene adaptability is provided.

[0108] Please refer to Figure 12 The embodiments of the present application also provide a behavior test system of a multi-modal perception humanoid robot, which can implement the above method, and the system comprises:

[0109] A data acquisition module is configured to acquire multi-modal perception data and current joint state data, splice the multi-modal perception data and the joint state data to determine a current state vector;

[0110] An action sampling module is configured to sample actions according to the current state vector and a trained processing model to determine a plurality of sample actions, and calculate the plurality of sample actions, the current state vector and a preset motion model to determine a plurality of candidate action states;

[0111] A discriminant calculation module is configured to perform discriminant calculation on the plurality of candidate action states and a preset neural network model to determine a plurality of total discriminant scores, and analyze the plurality of total discriminant scores and the plurality of candidate action states to determine a target behavior state;

[0112] A test evaluation module is configured to perform test evaluation according to the target behavior state and a preset scene test case to determine a test report.

[0113] It can be understood that the contents in the above method embodiments are all applicable to the device embodiments, the device embodiments specifically implement the functions same as those of the above method embodiments, and achieve the same beneficial effects as those of the above method embodiments.

[0114] The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0115] It can be understood that the contents in the above method embodiments are all applicable to the device embodiments, the device embodiments specifically implement the functions same as those of the above method embodiments, and achieve the same beneficial effects as those of the above method embodiments.

[0116] Please refer to Figure 13 , Figure 13 The electronic device of another embodiment is illustrated, which includes:

[0117] The processor 1301 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0118] The memory 1302 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 1302 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are saved in the memory 1302 and executed by the processor 1301 to implement the above method of the embodiments of the present application.

[0119] The input / output interface 1303 is used to realize information input and output.

[0120] The communication interface 1304 is used to realize the communication interaction between the device and other devices, which can be realized by a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.).

[0121] a bus 1305 for communicating information between the various components (e.g., the processor 1301, the memory 1302, the input / output interface 1303, and the communication interface 1304) of the device;

[0122] The processor 1301, the memory 1302, the input / output interface 1303, and the communication interface 1304 are communicatively connected to each other within the device through the bus 1305.

[0123] The computer program is stored in the computer readable storage medium, and the computer program is executed by the processor to implement the method described above.

[0124] It can be understood that the contents in the above method embodiments are all applicable to the present storage medium embodiment, the present storage medium embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0125] The computer program product includes a computer program, and the computer program is executed by the processor to implement the method described above.

[0126] It can be understood that the contents in the above method embodiments are all applicable to the present program product embodiment, the present program product embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0127] The memory is a non-transitory computer readable storage medium, and can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0128] The embodiments described in the present application embodiments are for more clearly illustrating the technical solutions of the present application embodiments, and do not constitute a limitation on the technical solutions provided by the present application embodiments. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the present application embodiments are also applicable to similar technical problems.

[0129] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation to the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0130] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, that is, can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0131] Those skilled in the art can understand that all or some steps in the above disclosed method, functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0132] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those clearly listed steps or units, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0133] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0134] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0135] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0136] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0137] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0138] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A behavior test method of a multi-modal perception humanoid robot, characterized by, The method comprises: acquiring multi-modal perception data and current joint state data, splicing according to the multi-modal perception data and the joint state data, and determining a current state vector; action sampling is performed according to the current state vector and a trained processing model, a plurality of sampling actions are determined, and a plurality of candidate action states are determined by calculating according to the plurality of sampling actions, the current state vector and a preset motion model; determining a plurality of total discrimination scores by discriminant calculation according to a plurality of candidate action states and a preset neural network model, and determining a target behavior state by analyzing a plurality of total discrimination scores and a plurality of candidate action states; test evaluation is performed according to the target behavior state and a preset scene test case, and a test report is determined.

2. The method of claim 1, wherein, The action sampling according to the current state vector and the trained processing model to determine a plurality of sampling actions specifically comprises: feature extraction is performed on the current state vector according to the trained processing model to determine multi-modal features, and feature fusion is performed on the multi-modal features to determine fusion feature data; full connection processing is performed on the fusion feature data according to a preset robot dual-arm shared weight to determine weight feature data; action space is determined by calculating according to the weight feature data and a preset activation function, and a plurality of sampling actions are determined by action sampling according to the action space and a preset degree of freedom parameter.

3. The method of claim 1, wherein, The calculation according to the plurality of sampling actions, the current state vector and the preset motion model to determine a plurality of candidate action states specifically comprises: a plurality of action motor parameters are determined by analyzing a plurality of sampling actions respectively; wherein the action motor parameters include motor current and reduction ratio corresponding to the sampling actions; a plurality of first torque vectors are determined by calculating according to a plurality of action motor parameters and a first preset formula, and a plurality of joint torque vectors are determined by adjusting a plurality of first torque vectors respectively according to a preset torque threshold value; a current joint angle vector and a current joint angular velocity vector are determined by analyzing the current state vector, and a plurality of next-time joint angle vectors are determined by calculating according to the current joint angle vector and the current joint angular velocity vector, a plurality of joint torque vectors and the preset motion model; a plurality of candidate action states are determined by constraint processing according to a plurality of next-time joint angle vectors and a preset joint limit threshold value.

4. The method of claim 1, wherein, The discriminant calculation according to a plurality of candidate action states and a preset neural network model to determine a plurality of total discrimination scores specifically comprises: a plurality of torque data, a plurality of visual data and a plurality of tactile data are determined by analyzing a plurality of candidate action states; first output data are determined by full connection processing according to a plurality of torque data and a plurality of visual data, and a safety sub-score is determined by activation calculation on the first output data; an adaptation sub-score is determined by calculating according to a plurality of visual data and a plurality of tactile data; The total discrimination score is determined according to the security sub-score, the adaptation sub-score, and a preset weight.

5. The method of claim 1, wherein, The target behavior state is determined according to the total discrimination score and the candidate action state. The total discrimination score is compared with a preset scene threshold. If the total discrimination score is greater than or equal to the preset scene threshold, the candidate action state corresponding to the total discrimination score is taken as the target behavior state. If the total discrimination score is less than the preset scene threshold, the action sampling is performed according to the current state vector and the trained processing model to determine a plurality of sampling actions until the total discrimination score is greater than or equal to the preset scene threshold.

6. The method of claim 1, wherein, The test report is determined according to the target behavior state and a preset scene test case. The simulation result is determined according to the preset scene test case and the target behavior state. The action output quality is determined according to the simulation result and a preset evaluation index, and the test report is determined according to the action output quality.

7. A multi-modal perceptual humanoid robot behavior testing system, characterized by, The system comprises: The data acquisition module is configured to acquire multi-modal perception data and current joint state data, splice the multi-modal perception data and the joint state data to determine a current state vector. The action sampling module is configured to sample actions according to the current state vector and a trained processing model to determine a plurality of sampling actions, and calculate a plurality of candidate action states according to the plurality of sampling actions, the current state vector, and a preset motion model. The discrimination calculation module is configured to perform discrimination calculation according to the plurality of candidate action states and a preset neural network model to determine a plurality of total discrimination scores, and analyze the plurality of total discrimination scores and the plurality of candidate action states to determine a target behavior state. The test evaluation module is configured to perform test evaluation according to the target behavior state and a preset scene test case to determine a test report.

8. An electronic device, comprising: The system comprises: At least one processor; At least one memory configured to store at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 6.

Images