A control method, system, device and medium of a humanoid robot

Abstract

The application discloses a kind of humanoid robot control method, system, equipment and medium, wherein the method obtains the conversation information and working environment information of current round of humanoid robot, and several first structured information and first summary information;The conversation information and the working environment information are input into pre-training language model and are structured extraction, obtain the second structured information output by the pre-training language model, and the second structured information is the structured information of current round;According to the working environment information, all the first structured information and the first summary information, instruction generation is carried out to the second structured information, and action planning instruction is obtained;According to the action planning instruction, the action control of humanoid robot is carried out.The method can improve the task efficiency of humanoid robot in human-machine collaborative control process.The application relates to the technical field of intelligent robot.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent robots, and in particular to a control method, system, device and medium for a humanoid robot. BACKGROUND

[0002] In recent years, the development of deep learning technology has promoted the progress of humanoid robot technology, especially in the human-robot collaborative control of humanoid robots, which has shown significant potential.

[0003] Currently, the related technology usually generates action planning and executes by means of the inference ability of a large model after receiving the instructions (such as voice instructions, text instructions, gesture instructions, etc.) issued by the user. This kind of mode is insufficient in capturing and responding to user intent, and the task efficiency of human-robot collaborative control is low.

[0004] Therefore, the problems of the related technology still need to be solved and optimized. SUMMARY

[0005] The purpose of the present application is to at least partially solve one of the technical problems in the related art.

[0006] To this end, one purpose of the embodiments of the present application is to provide a control method, system, device and medium for a humanoid robot, wherein the method can improve the task efficiency of the humanoid robot in the human-robot collaborative control process.

[0007] In order to achieve the above technical purpose, the technical solutions adopted by the embodiments of the present application include:

[0008] In a first aspect, the embodiments of the present application provide a control method for a humanoid robot, comprising:

[0009] obtaining conversation information and working environment information of a current round of the humanoid robot, as well as a plurality of first structured information and first summary information, the first structured information being structured information of a previous round, and the first summary information being unstructured summary information corresponding to the first structured information;

[0010] inputting the conversation information and the working environment information into a pre-trained language model for structured extraction, to obtain second structured information output by the pre-trained language model, the second structured information being structured information of the current round;

[0011] generating an instruction based on the working environment information, all the first structured information and the first summary information, to obtain an action planning instruction;

[0012] controlling the humanoid robot according to the action planning instruction.

[0013] In addition, the method according to the above-mentioned embodiments of the present application can further have the following additional technical features:

[0014] Further, in an embodiment of the present application, the inputting the conversation information and the working environment information into the pre-trained language model for structured extraction to obtain second structured information output by the pre-trained language model comprises:

[0015] spatiotemporal alignment is performed on the conversation information and the working environment information to obtain alignment information, the alignment information comprising spatiotemporally aligned conversation information and working environment information;

[0016] the alignment information is input into the pre-trained language model for structured extraction to obtain the second structured information.

[0017] Further, in an embodiment of the present application, the instruction generation on the second structured information according to the working environment information, all the first structured information and the first summary information to obtain an action planning instruction comprises:

[0018] obtaining key sensor data of the humanoid robot;

[0019] performing information screening on all the first structured information according to the working environment information to obtain a plurality of third structured information corresponding to the working environment information;

[0020] performing information screening on all the first summary information according to the working environment information to obtain a plurality of second summary information corresponding to the working environment information;

[0021] performing relevance analysis and instruction construction on the second structured information according to the key sensor data, all the third structured information and the second summary information to obtain the action planning instruction.

[0022] Further, in an embodiment of the present application, the relevance analysis and instruction construction on the second structured information according to the key sensor data, all the third structured information and the second summary information to obtain the action planning instruction comprises:

[0023] performing structured feature extraction on the second structured information and all the third structured information to obtain a plurality of structured feature vectors, each of the structured feature vectors comprising a feature vector of the second structured information and a feature vector of one of the third structured information;

[0024] performing unstructured feature extraction on all the second summary information to obtain a summary feature vector, each of the summary feature vectors corresponding to one of the structured feature vectors;

[0025] performing feature fusion on all the structured feature vectors and the summary feature vectors according to the key sensor data to obtain a target fusion feature;

[0026] inputting the target fusion feature into a semantic action mapping model for mapping generation to obtain an action planning instruction output by the semantic action mapping model.

[0027] Further, in an embodiment of the present application, the performing feature fusion on all the structured feature vectors and the summary feature vectors according to the key sensor data to obtain a target fusion feature comprises:

[0028] performing similarity analysis on all the structured feature vectors according to the key sensor data to obtain a plurality of first similarities;

[0029] performing similarity analysis on all the summary feature vectors according to the key sensor data to obtain a plurality of second similarities;

[0030] performing screening fusion on all the structured feature vectors and the summary feature vectors according to all the first similarities and the second similarities to obtain the target fusion feature.

[0031] Further, in an embodiment of the present application, the performing screening fusion on all the structured feature vectors and the summary feature vectors according to all the first similarities and the second similarities to obtain the target fusion feature comprises:

[0032] performing weighted fusion on the corresponding second similarity according to each of the first similarities to obtain a plurality of weighted fusion values;

[0033] comparing all the weighted fusion values to obtain a target fusion value;

[0034] performing fusion splicing on the corresponding structured feature vector and the summary feature vector according to the target fusion value to obtain the target fusion feature.

[0035] Further, in an embodiment of the present application, the method further comprises:

[0036] obtaining a summary prompt word template and action interaction data, the action interaction data comprising the second structured information, the action planning instruction and action execution information of the action planning instruction;

[0037] According to the summary prompt word template, the action interaction data is prompted to obtain an action interaction prompt;

[0038] The action interaction prompt is input into the pre-trained language model for unstructured reasoning to obtain third summary information, the third summary information being unstructured summary information corresponding to the second structured information.

[0039] In a second aspect, an embodiment of the present application provides a control system of a humanoid robot, comprising:

[0040] A first processing unit is configured to acquire session information and working environment information of a current round of the humanoid robot, and a plurality of first structured information and first summary information, the first structured information being structured information of a previous round, and the first summary information being unstructured summary information corresponding to the first structured information;

[0041] A second processing unit is configured to input the session information and the working environment information into a pre-trained language model for structured extraction to obtain second structured information output by the pre-trained language model, the second structured information being structured information of the current round;

[0042] A third processing unit is configured to generate an action planning instruction according to the working environment information, all the first structured information, and the first summary information, and according to the second structured information.

[0043] A fourth processing unit is configured to control the humanoid robot according to the action planning instruction.

[0044] In a third aspect, an embodiment of the present application further provides an electronic device, comprising:

[0045] At least one processor;

[0046] At least one memory configured to store at least one program;

[0047] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0048] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, wherein a processor executable program is stored, and the processor executable program is used to implement the method described above when executed by the processor.

[0049] The advantages and beneficial effects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood by the practice of the present application:

[0050] A control method, system, device and medium of a humanoid robot are disclosed, wherein the method obtains conversation information and working environment information of a current round of the humanoid robot, and a plurality of first structured information and first summary information, the first structured information being structured information of a previous round, and the first summary information being unstructured summary information corresponding to the first structured information; the conversation information and the working environment information are input into a pre-trained language model for structured extraction to obtain second structured information output by the pre-trained language model, the second structured information being structured information of the current round; the second structured information is subjected to instruction generation according to the working environment information, all the first structured information and the first summary information to obtain action planning instructions; and the humanoid robot is subjected to action control according to the action planning instructions. The method generates action planning instructions based on structured information of a previous round, unstructured summary information and structured information of a current round, which can extract working task preferences and scene modes of a user in a human-machine collaborative control process, so that the generated action planning instructions realize personalized collaboration of the humanoid robot and the user, and help to improve the task efficiency of human-machine collaborative control. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of expressing part of the embodiments of the technical solutions of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0052] Figure 1 A flowchart of a control method of a humanoid robot is provided for the embodiments of the present application.

[0053] Figure 2 A framework diagram of a control system of a humanoid robot is provided for the embodiments of the present application.

[0054] Figure 3 A structural diagram of an electronic device is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0055] Embodiments of the present application are described below in detail with reference to the accompanying drawings, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of explanation and description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0057] At present, the related technology is usually to receive the instruction (such as voice instruction, text instruction, gesture instruction, etc.) issued by the user, generate action planning and execute based on the preset action library and the inference ability of the large model. This kind of way is insufficient for capturing and responding to user's intention, and cannot extract and understand the user's working habit preference in the scene mode, and is easy to appear the situation that the action cannot be adjusted in time according to the user's gesture or voice instruction in time, there is a collision risk between the humanoid robot and the user, and the safety and task efficiency of human-machine collaborative control are low.

[0058] In addition, the action planning generated by this kind of way is relatively fixed, and it is difficult to cope with the parameter changes in the work task, and the flexibility of the action planning is not high. In the process of human-machine collaborative control, the user often needs to cooperate with the action of the humanoid robot. The user experience is not good. For example, in the elbow joint assembly work task of the humanoid robot on the industrial production line, this kind of way is difficult to cope with the parameter changes in the joint assembly, such as the slight change of the joint part size caused by temperature change, the action planning executed by the humanoid robot is easy to appear the situation of execution interruption or failure, at this time, the user needs to cooperate with the execution action of the humanoid robot, and the user experience is not good.

[0059] It should be noted that the above-mentioned related technology is only used to assist in understanding the technical solutions of the present application, and does not mean that it belongs to the prior art disclosed.

[0060] Therefore, the embodiments of the present application provide a control method, system, device and medium for a humanoid robot, wherein the method generates an action planning instruction based on structured information and unstructured summary information of previous rounds and structured information of the current round, which can extract the work task preference and scene mode rule of the user in the human-robot collaborative control process, so that the generated action planning instruction can better adapt to the work habit preference of the user, reduce the risk of collision between the humanoid robot and the user during action execution, effectively improve the safety of human-robot collaborative control, and realize personalized collaboration between the humanoid robot and the user, which is conducive to improving the task efficiency of human-robot collaborative control.

[0061] In addition, the method infers unstructured summary information of each previous round through action interaction data, and generates an instruction based on the unstructured summary information of the previous round, which can also extract the parameter variation rule of the humanoid robot or the scene variation rule in the scene mode, so that the subsequent generated action planning instruction takes into account the variation rule, and the flexibility of the action planning instruction is higher, thereby being conducive to improving the user experience of the user in the human-robot collaborative control process.

[0062] With reference to Figure 1 In the embodiments of the present application, a control method for a humanoid robot comprises:

[0063] In step 110, the conversation information and the work environment information of the current round of the humanoid robot, and a plurality of first structured information and first summary information are obtained, wherein the first structured information is the structured information of the previous round, and the first summary information is the unstructured summary information corresponding to the first structured information;

[0064] In the embodiments of the present application, the conversation information can be the user voice collected by the humanoid robot through a sound collection device (such as a microphone), or the user voice text obtained through voice-to-text conversion; the work environment information can be environment data related to the work task; and the first structured information can be the structured information of the previous round of the humanoid robot, and the first summary information can be the unstructured summary information of the same round as the first structured information, which contains the scene mode rule and the work task preference. The round can be the round in which the humanoid robot executes the action planning instruction.

[0065] For example, the embodiments of the present application take the assembly of the robot elbow joint of the work task as an example, at this time the user can be the operator of the industrial production line, and the conversation information can be the instruction provided by the operator to "assemble the robot elbow joint and correct the range of motion"; and the work environment information can be three-dimensional visual data, force sensor signal data, etc. of the robot elbow joint.

[0066] In step 120, the conversation information and the working environment information are input into a pre-trained language model for structured extraction to obtain second structured information output by the pre-trained language model, the second structured information being structured information of the current round.

[0067] In the embodiments of the present application, the humanoid robot can input the conversation information and the working environment information into a pre-trained language model, which can be a large language model. The pre-trained language model extracts structured information from the conversation information and the working environment information to obtain structured information of the current round, denoted as second structured information.

[0068] In some embodiments, the inputting of the conversation information and the working environment information into the pre-trained language model for structured extraction to obtain the second structured information output by the pre-trained language model comprises:

[0069] temporally and spatially aligning the conversation information and the working environment information to obtain alignment information, the alignment information comprising temporally and spatially aligned conversation information and working environment information;

[0070] inputting the alignment information into the pre-trained language model for structured extraction to obtain the second structured information.

[0071] In the embodiments of the present application, before the conversation information and the working environment information are input into the pre-trained language model, the conversation information and the working environment information can be temporally and spatially aligned based on a Temporal and Spatial Alignment (TMRoPE) algorithm to fuse semantic features in the conversation information and sensor features in the working environment information, thereby obtaining alignment information. Then, the alignment information is input into the pre-trained language model to obtain second structured information output by the pre-trained language model, which can be {“task type”: “elbow assembly and calibration”, “component model”: “RJ-ELBOW-007”, “installation coordinates”: “(200mm, 150mm, 80mm)”, “torque requirement”: “12±0.5N·m”, “range of motion”: “120°±2°”}.

[0072] It can be understood that, before the conversation information and the working environment information are temporally and spatially aligned, the conversation information and the working environment information can be preprocessed. The preprocessing of the conversation information can include data cleaning, noise filtering, etc., and the preprocessing of the working environment information can include data calibration, outlier rejection, etc.

[0073] Step 130, generating an action planning instruction according to the working environment information, all the first structured information and the first summary information, and obtaining the action planning instruction according to the second structured information;

[0074] In the embodiment of the present application, the action planning instruction corresponding to the second structured information can be generated based on the working environment information, the working task preferences and the scene mode rules recorded in the structured information and the unstructured summary information of the previous round.

[0075] In some embodiments, the generating an action planning instruction according to the working environment information, all the first structured information and the first summary information, and obtaining the action planning instruction according to the second structured information, comprises:

[0076] Obtaining the key sensor data of the humanoid robot;

[0077] Filtering all the first structured information according to the working environment information to obtain a plurality of third structured information corresponding to the working environment information;

[0078] Filtering all the first summary information according to the working environment information to obtain a plurality of second summary information corresponding to the working environment information;

[0079] In the embodiment of the present application, the key sensor data of the humanoid robot can be the sensor data required by the humanoid robot to execute the action planning instruction, and the key sensor data can be at least one of force sensor data, visual data, distance sensor data, etc.

[0080] It can be understood that for the first structured information, the information filtering can be to filter out the first structured information with similar environment parameters to the working environment information from all the first structured information of the previous round, and the first structured information is recorded as the third structured information, wherein the similar environment parameters in the robot elbow joint assembly application scene can be at least one of the assembly environment temperature, the assembly environment humidity, the three-dimensional visual data of the elbow joint, the force sensor signal data, etc. For the first summary information, the information filtering content is similar to the foregoing first structured information, and the same can be deduced by analogy, which will not be described herein.

[0081] Performing correlation analysis and instruction construction on the second structured information according to the key sensor data, all the third structured information and the second summary information to obtain the action planning instruction.

[0082] Further, the performing correlation analysis and instruction construction on the second structured information according to the key sensor data, all the third structured information and the second summary information to obtain the action planning instruction, comprises:

[0083] performing structured feature extraction on the second structured information and all the third structured information to obtain a plurality of structured feature vectors, each of the structured feature vectors comprising a feature vector of the second structured information and a feature vector of one of the third structured information;

[0084] performing unstructured feature extraction on all the second summary information to obtain summary feature vectors, each of the summary feature vectors corresponding to one of the structured feature vectors;

[0085] In the embodiments of the present application, the structured feature extraction can be to extract the feature vectors of the second structured information and each of the third structured information respectively. For example, for the structured information {task type: elbow joint assembly, torque requirement: 12 ± 0.5 N·m, temperature compensation: 0.3% / ℃}, the feature vector in the form of a multi-dimensional vector can be [task type, torque value, temperature coefficient]. Then, the feature vector of the second structured information is combined with the feature vector of each of the third structured information to obtain a plurality of structured feature vectors. The number of the structured feature vectors is the same as the number of the third structured information.

[0086] It can be understood that, for any one of the second summary information, the unstructured feature extraction can be to extract a text semantic vector of the second summary information by using a Bert model, denoted as a summary feature vector. The text semantic vector can be specifically a feature vector of a work task preference and a feature vector of a scene mode rule in the second summary information. For example, in the application scenario of robot elbow joint assembly, the feature vector of the assembly habit keyword "step-by-step tightening method" in the work task preference can be determined, and the feature vector of the first scene mode rule "temperature ≥ 30℃ needs to be compensated" and the feature vector of the second scene mode rule "the accuracy of the torque wrench decreases by 1% after 50 consecutive assemblies" can be obtained. The remaining second summary information can be similarly deduced.

[0087] performing feature fusion on all the structured feature vectors and the summary feature vectors according to the key sensing data to obtain target fusion features;

[0088] Further, the performing feature fusion on all the structured feature vectors and the summary feature vectors according to the key sensing data to obtain target fusion features comprises:

[0089] performing similarity analysis on all the structured feature vectors according to the key sensing data to obtain a plurality of first similarities;

[0090] performing similarity analysis on all the summary feature vectors according to the key sensing data to obtain a plurality of second similarities;

[0091] In the embodiments of the present application, the similarity analysis can be to calculate the similarity between the feature vector of the key sensor data and each structured feature vector, and there are various specific similarity calculation methods, such as cosine similarity calculation method, distance similarity calculation method, etc., so as to obtain the first similarity between each structured feature vector and the key sensor feature vector; and the content of the second similarity is similar to that of the first similarity, which can be simply analogized, and the present application will not be repeated here.

[0092] According to all the first similarities and the second similarities, all the structured feature vectors and the summary feature vector are screened and fused to obtain the target fusion feature.

[0093] Further, the screening and fusion of all the structured feature vectors and the summary feature vector according to all the first similarities and the second similarities to obtain the target fusion feature comprises:

[0094] According to each first similarity, the corresponding second similarity is weighted and fused to obtain a plurality of weighted fusion values;

[0095] The target fusion value is obtained by comparing all the weighted fusion values;

[0096] According to the target fusion value, the corresponding structured feature vector and the summary feature vector are fused and spliced to obtain the target fusion feature.

[0097] In the embodiments of the present application, the weighted fusion can be to weight and fuse the first similarity and the second similarity under the same round based on the preset weight parameter, so as to obtain the weighted fusion value under each round. Specifically, for a previous round t-1, the second similarity can be determined based on the summary feature vector under the previous round t-1 and the key sensor data feature vector; and the first similarity can be determined based on the feature vector of the key sensor data and the target structured feature vector, which includes the feature vector of the second structured information and the feature vector of the third structured information under the previous round t-1.

[0098] It can be understood that after obtaining the weighted fusion value under each round, the size relationship between the weighted fusion values under all rounds can be compared, and the maximum weighted fusion value among all the weighted fusion values is determined as the target fusion value; then, based on the target fusion value, the target fusion feature of the current round is determined from the corresponding structured feature vector and the summary feature vector.

[0099] The target fusion features are input into the semantic action mapping model for mapping generation, and the action planning instructions output by the semantic action mapping model are obtained.

[0100] In this embodiment, the target fusion features can be input into a semantic action mapping model. This semantic action mapping model records several semantic-action mapping relationships. The semantic action mapping model generates original action planning instructions based on the structured feature vectors in the target fusion features. Specifically, the original action planning instructions are determined by querying the semantic-action mapping relationships through the structured information of the current round and the structured information of the previous round in the structured feature vectors. Then, the original action planning instructions are corrected by summarizing the work task preferences and scene pattern rules contained in the feature vectors, thereby obtaining the final action planning instructions output by the semantic action mapping model.

[0101] For example, in the application scenario of robot elbow joint assembly in industrial production lines, the original motion planning instructions can be modified based on the "step-by-step tightening method" and "operator assembly habits" in the work task preferences summarized in the feature vector, as well as the "torque wrench accuracy decreases by 1% after 50 consecutive assemblies" in the scene pattern rules. Specifically, the original motion avoidance instructions can be readjusted by summarizing the work task preferences in the feature vector so that the humanoid robot's robotic arm path avoids the operator's operating area while using the step-by-step tightening method; and the angle parameters with torque in the adjusted motion planning instructions can be updated by updating the scene pattern rules to obtain the final motion planning instructions.

[0102] Step 140: Perform motion control on the humanoid robot according to the motion planning instructions.

[0103] In this embodiment of the application, the humanoid robot can obtain motion planning instructions and execute corresponding actions. For example, in the application scenario of robot elbow joint assembly in industrial production lines, the robot arm can be driven to assemble elbow joint products on the production line based on motion planning instructions.

[0104] In some embodiments, the method further includes:

[0105] Obtain summary prompt word templates and action interaction data, wherein the action interaction data includes the second structured information, the action planning instruction, and the action execution information of the action planning instruction;

[0106] Based on the summarized prompt word template, prompts are constructed from the action interaction data to obtain action interaction prompts;

[0107] The action interaction prompt is input into the pre-trained language model for unstructured inference, and third summary information is obtained, which is unstructured summary information corresponding to the second structured information.

[0108] In the embodiment of the present application, for the current round, the humanoid robot can generate corresponding action execution information based on the execution state (such as execution failure, execution interruption, execution success, etc.) of the action planning instruction after executing the action planning instruction; then, the action execution information, the second structured information of the current round and the action planning instruction are added to the summary prompt template to construct the generated action interaction prompt. There are many specific construction methods of the summary prompt template, which will not be repeated here.

[0109] It can be understood that the action interaction prompt can be input into the pre-trained language model for unstructured inference, and the unstructured information in the action interaction prompt is extracted through the pre-trained language model, and is recorded as the unstructured summary information of the second structured information in the current round.

[0110] A control system of a humanoid robot according to an embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0111] Referring to Figure 2 , the control system of the humanoid robot according to the embodiment of the present application comprises:

[0112] The first processing unit 101 is configured to obtain the conversation information and the working environment information of the current round of the humanoid robot, and a plurality of first structured information and first summary information, the first structured information being the structured information of the previous round, and the first summary information being the unstructured summary information corresponding to the first structured information;

[0113] The second processing unit 102 is configured to input the conversation information and the working environment information into a pre-trained language model for structured extraction, and obtain the second structured information output by the pre-trained language model, the second structured information being the structured information of the current round;

[0114] The third processing unit 103 is configured to generate an action planning instruction based on the working environment information, all the first structured information and the first summary information, and the second structured information;

[0115] The fourth processing unit 104 is configured to control the humanoid robot according to the action planning instruction.

[0116] It can be understood that the contents in the above method embodiments are applicable to the present system embodiments, the present system embodiments specifically implement the functions same as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0117] With reference to Figure 3 The present application embodiment further provides an electronic device, comprising:

[0118] at least one processor 201;

[0119] at least one memory 202, configured to store at least one program;

[0120] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the above method embodiments.

[0121] Similarly, it can be understood that the contents in the above method embodiments are applicable to the present device embodiments, the present device embodiments specifically implement the functions same as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0122] The present application embodiment further provides a computer readable storage medium, wherein a program executable by the processor 201 is stored, and the program executable by the processor 201 is used to implement the above method embodiments when executed by the processor 201.

[0123] Similarly, the contents in the above method embodiments are applicable to the present computer readable storage medium embodiments, the present computer readable storage medium embodiments specifically implement the functions same as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0124] The present application embodiment further provides a computer program product, comprising a computer program, which is executed by a processor to implement the steps in the above method embodiments.

[0125] Those skilled in the art can understand that all or part of the processes in the above method embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above method embodiments.

[0126] In alternative embodiments, the functions / operations in the flow diagrams can occur in sequences other than those depicted. For example, two operations shown in succession can in fact be executed substantially concurrently or the operations sometimes can be executed in the reverse order depending upon the functionality / operations involved. Also, embodiments presented and described in this application are provided by way of example only. The

[0127] Moreover, while this application has been described in the context of functional modules, it is to be understood that one or more of the functions and / or features can be integrated in a single physical device and / or software module or one or more functions and / or features can be implemented in separate physical devices or software modules. It will also be appreciated that detailed discussion of the actual implementation of each module is not necessary for an understanding of the application. Rather, the actual implementation is a matter of choice apart from the conception and the disclosure provided that the attributes of each module remain intact. Accordingly, the disclosure is to be understood in a manner that

[0128] If the functions are implemented in software, the functions can be stored in or implemented as one or more software modules on a computer-readable medium. The computer-readable medium can include a floppy disk, a CD-ROM, a DVD, a Blu-ray Disc, a hard disk, a magnetic tape, a computer memory, or any other computer-readable medium. The software modules include instructions that, when executed by a computer, cause the computer to perform the specific functions disclosed in this specification. The software modules can include one or more instructions that, when executed by a computer, cause the computer to perform the steps of the methods disclosed in this specification. The software modules can include one or more instructions that, when executed by a computer, cause the computer to perform the steps of the methods disclosed in this specification.

[0129] The logic and / or steps represented in the flow diagrams or otherwise described herein, for example, can be embodied in non-transitory computer-readable media in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or in conjunction with which the instructions can be executed. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical connections), a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and stored in a computer memory.

[0130] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical connections), a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and stored in a computer memory.

[0131] It should be understood that aspects of the present application can be implemented in hardware, software, firmware or combinations thereof. In the above described embodiments, the various steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or combinations thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0132] In the above description of the present specification, the description using the terms "one embodiment / one example", "another embodiment / another example" or "some embodiments / some examples" and the like means that the relevant described specific features, structures, materials or characteristics are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the described specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in an appropriate manner.

[0133] While the embodiments of the application have been shown and described, it is to be understood that the embodiments can be varied, modified, substituted and changed by those skilled in the art without departing from the principles and spirit of the application, the scope of which is defined by the claims and their equivalents.

[0134] The above is a specific description of the preferred embodiments of the application, but the application is not limited to the embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the application, and these equivalent modifications or substitutions are all included in the scope defined by the claims of the application.

Claims

1. A control method for a humanoid robot, characterized in that, include: Acquire the conversation information and working environment information of the humanoid robot in the current round, as well as several first structured information and first summary information. The first structured information is the structured information of the previous round, and the first summary information is the unstructured summary information of the corresponding first structured information. The conversation information and the working environment information are input into a pre-trained language model for structured extraction to obtain the second structured information output by the pre-trained language model. The second structured information is the structured information of the current round. Based on the work environment information, all the first structured information, and the first summary information, instructions are generated from the second structured information to obtain action planning instructions; The humanoid robot is controlled to perform motion according to the motion planning instructions; The step of generating instructions from the second structured information based on the work environment information, all the first structured information, and the first summary information to obtain action planning instructions includes: Acquire key sensor data of the humanoid robot; Based on the work environment information, all the first structured information is filtered to obtain several third structured information corresponding to the work environment information; Based on the work environment information, all the first summary information is filtered to obtain several second summary information corresponding to the work environment information; Based on the key sensor data, all the third structured information and the second summary information, correlation analysis and instruction construction are performed on the second structured information to obtain the action planning instruction; The step of performing correlation analysis and instruction construction on the second structured information based on the key sensor data, all the third structured information, and the second summary information to obtain the action planning instruction includes: Structural feature extraction is performed on the second structured information and all the third structured information to obtain several structured feature vectors. Each structured feature vector includes a feature vector of the second structured information and a feature vector of the third structured information. Unstructured features are extracted from all the second summary information to obtain summary feature vectors, and each summary feature vector corresponds to one structured feature vector; Based on the key sensing data, feature fusion is performed on all the structured feature vectors and the summarized feature vectors to obtain the target fused features; The target fusion features are input into the semantic action mapping model for mapping generation, and the action planning instructions output by the semantic action mapping model are obtained.

2. The method according to claim 1, characterized in that, The step of inputting the conversation information and the working environment information into a pre-trained language model for structured extraction to obtain the second structured information output by the pre-trained language model includes: The session information and the work environment information are spatiotemporally aligned to obtain alignment information, which includes the spatiotemporally aligned session information and work environment information. The alignment information is input into the pre-trained language model for structured extraction to obtain the second structured information.

3. The method according to claim 1, characterized in that, The step of fusing all the structured feature vectors and the summarized feature vectors based on the key sensing data to obtain the target fused features includes: Based on the key sensing data, a similarity analysis is performed on all the structured feature vectors to obtain several first similarities; Based on the key sensing data, a similarity analysis is performed on all the summarized feature vectors to obtain several second similarities. Based on all the first similarity and the second similarity, all the structured feature vectors and the summarized feature vectors are filtered and fused to obtain the target fused feature.

4. The method according to claim 3, characterized in that, The step of filtering and fusing all the structured feature vectors and the summarized feature vectors based on all the first similarity and the second similarity to obtain the target fused feature includes: Based on each first similarity, the corresponding second similarity is weighted and fused to obtain several weighted fusion values; By comparing all the weighted fusion values, the target fusion value is obtained; Based on the target fusion value, the corresponding structured feature vector and the summary feature vector are fused and concatenated to obtain the target fusion feature.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Obtain summary prompt word templates and action interaction data, wherein the action interaction data includes the second structured information, the action planning instruction, and the action execution information of the action planning instruction; Based on the summarized prompt word template, prompts are constructed from the action interaction data to obtain action interaction prompts; The action interaction prompt is input into the pre-trained language model for unstructured reasoning to obtain third summary information, which is the unstructured summary information corresponding to the second structured information.

6. A control system for a humanoid robot, characterized in that, include: The first processing unit is used to acquire the conversation information and working environment information of the humanoid robot in the current round, as well as several first structured information and first summary information. The first structured information is the structured information of the previous round, and the first summary information is the unstructured summary information of the corresponding first structured information. The second processing unit is used to input the conversation information and the working environment information into a pre-trained language model for structured extraction, and obtain the second structured information output by the pre-trained language model, wherein the second structured information is the structured information of the current round; The third processing unit is used to generate instructions from the second structured information based on the work environment information, all the first structured information and the first summary information, to obtain action planning instructions; The fourth processing unit is used to control the motion of the humanoid robot according to the motion planning instructions; The step of generating instructions from the second structured information based on the work environment information, all the first structured information, and the first summary information to obtain action planning instructions includes: Acquire key sensor data of the humanoid robot; Based on the work environment information, all the first structured information is filtered to obtain several third structured information corresponding to the work environment information; Based on the work environment information, all the first summary information is filtered to obtain several second summary information corresponding to the work environment information; Based on the key sensor data, all the third structured information and the second summary information, correlation analysis and instruction construction are performed on the second structured information to obtain the action planning instruction; The step of performing correlation analysis and instruction construction on the second structured information based on the key sensor data, all the third structured information, and the second summary information to obtain the action planning instruction includes: Structural feature extraction is performed on the second structured information and all the third structured information to obtain several structured feature vectors. Each structured feature vector includes a feature vector of the second structured information and a feature vector of the third structured information. Unstructured features are extracted from all the second summary information to obtain summary feature vectors, and each summary feature vector corresponds to one structured feature vector; Based on the key sensing data, feature fusion is performed on all the structured feature vectors and the summarized feature vectors to obtain the target fused features; The target fusion features are input into the semantic action mapping model for mapping generation, and the action planning instructions output by the semantic action mapping model are obtained.

7. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1-5.

8. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to implement the method as described in any one of claims 1-5.

Images