Humanoid robot control method, system, equipment and medium

Abstract

The invention discloses a control method, system and device of a humanoid robot and a medium, and the method comprises the steps: obtaining the session information and working environment information of the current round of the humanoid robot, and a plurality of pieces of first structured information and first summary information; the session information and the working environment information are input into a pre-training language model for structured extraction, second structured information output by the pre-training language model is obtained, and the second structured information is structured information of the current round; according to the working environment information, all the first structured information and the first summary information, performing instruction generation on the second structured information to obtain an action planning instruction; and performing action control on the humanoid robot according to the action planning instruction. According to the method, the task efficiency of the humanoid robot in the man-machine cooperative control process can be improved. The invention relates to the technical field of intelligent robots.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and in particular to a control method, system, equipment and medium for a humanoid robot. Background Art

[0002] In recent years, the development of deep learning technology has promoted the advancement of humanoid robot technology, especially in the human-machine collaborative control of humanoid robots, showing significant potential.

[0003] At present, relevant technologies usually generate action plans and execute them based on the reasoning capabilities of large models after receiving instructions from users (such as voice instructions, text instructions, gesture instructions, etc.). This method is insufficient in capturing and responding to user intentions, and the task efficiency of human-computer collaborative control is low.

[0004] Therefore, the problems existing in related technologies still need to be solved and optimized urgently. Summary of the Invention

[0005] The purpose of the present invention is to solve one of the technical problems existing in the related art to at least a certain extent.

[0006] To this end, an object of an embodiment of the present invention is to provide a control method, system, device and medium for a humanoid robot, wherein the method can improve the task efficiency of the humanoid robot in the process of human-machine collaborative control.

[0007] In order to achieve the above technical objectives, the technical solutions adopted in the embodiments of the present application include:

[0008] In a first aspect, an embodiment of the present application provides a method for controlling a humanoid robot, comprising:

[0009] Acquiring conversation information and work environment information of the humanoid robot in a current round, as well as a plurality of first structured information and first summary information, where the first structured information is structured information from a previous round, and the first summary information is unstructured summary information corresponding to the first structured information;

[0010] Inputting the session information and the work environment information into a pre-trained language model for structured extraction to obtain second structured information output by the pre-trained language model, where the second structured information is structured information for the current round;

[0011] Generating instructions for the second structured information according to the working environment information, all the first structured information, and the first summary information to obtain an action planning instruction;

[0012] The humanoid robot is controlled in motion according to the motion planning instruction.

[0013] In addition, the method according to the above embodiment of the present application may also have the following additional technical features:

[0014] Furthermore, in one embodiment of the present application, inputting the session information and the work environment information into a pre-trained language model for structured extraction to obtain second structured information output by the pre-trained language model includes:

[0015] Performing spatiotemporal alignment on the session information and the work environment information to obtain alignment information, the alignment information including the spatiotemporally aligned session information and work environment information;

[0016] The alignment information is input into the pre-trained language model for structured extraction to obtain the second structured information.

[0017] Furthermore, in one embodiment of the present application, generating an instruction for the second structured information based on the work environment information, all the first structured information, and the first summary information to obtain an action planning instruction includes:

[0018] Acquiring key sensor data of the humanoid robot;

[0019] According to the working environment information, all the first structured information is screened to obtain a plurality of third structured information corresponding to the working environment information;

[0020] Filtering all the first summary information according to the working environment information to obtain a plurality of second summary information corresponding to the working environment information;

[0021] According to the key sensor data, all the third structured information and the second summary information, correlation analysis and instruction construction are performed on the second structured information to obtain the action planning instruction.

[0022] Furthermore, in one embodiment of the present application, performing correlation analysis and instruction construction on the second structured information based on the key sensor data, all the third structured information, and the second summary information to obtain the action planning instruction includes:

[0023] Performing structured feature extraction on the second structured information and all the third structured information to obtain a plurality of structured feature vectors, each of the structured feature vectors including a feature vector of the second structured information and a feature vector of the third structured information;

[0024] Performing unstructured feature extraction on all the second summary information to obtain summary feature vectors, each of the summary feature vectors corresponding to one of the structured feature vectors;

[0025] performing feature fusion on all the structured feature vectors and the summary feature vector according to the key sensor data to obtain a target fusion feature;

[0026] The target fusion feature is input into a semantic action mapping model for mapping generation, and an action planning instruction output by the semantic action mapping model is obtained.

[0027] Furthermore, in one embodiment of the present application, performing feature fusion on all the structured feature vectors and the summary feature vectors based on the key sensor data to obtain target fusion features includes:

[0028] performing similarity analysis on all the structured feature vectors according to the key sensor data to obtain a plurality of first similarities;

[0029] performing similarity analysis on all the summarized feature vectors according to the key sensor data to obtain a plurality of second similarities;

[0030] According to all the first similarities and the second similarities, all the structured feature vectors and the summary feature vectors are screened and fused to obtain the target fused feature.

[0031] Furthermore, in one embodiment of the present application, the step of screening and fusing all the structured feature vectors and the summary feature vectors according to all the first similarities and the second similarities to obtain the target fusion feature includes:

[0032] According to each first similarity, performing weighted fusion on the corresponding second similarity to obtain a plurality of weighted fusion values;

[0033] Comparing all the weighted fusion values ​​to obtain a target fusion value;

[0034] According to the target fusion value, the corresponding structured feature vector and the summary feature vector are fused and spliced ​​to obtain the target fusion feature.

[0035] Furthermore, in one embodiment of the present application, the method further includes:

[0036] Acquire a summary prompt word template and action interaction data, wherein the action interaction data includes the second structured information, the action planning instruction, and action execution information of the action planning instruction;

[0037] Constructing prompts for the action interaction data according to the summary prompt word template to obtain action interaction prompts;

[0038] The action interaction prompt is input into the pre-trained language model for unstructured reasoning to obtain third summary information, where the third summary information is unstructured summary information corresponding to the second structured information.

[0039] In a second aspect, an embodiment of the present application provides a control system for a humanoid robot, comprising:

[0040] a first processing unit configured to obtain conversation information and work environment information of the humanoid robot in a current round, as well as a plurality of first structured information and first summary information, wherein the first structured information is structured information of a previous round, and the first summary information is unstructured summary information of the corresponding first structured information;

[0041] a second processing unit, configured to input the session information and the work environment information into a pre-trained language model for structured extraction, and obtain second structured information output by the pre-trained language model, where the second structured information is structured information of a current round;

[0042] a third processing unit, configured to generate an instruction for the second structured information based on the work environment information, all the first structured information, and the first summary information, to obtain an action planning instruction;

[0043] The fourth processing unit is used to control the motion of the humanoid robot according to the motion planning instruction.

[0044] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0045] at least one processor;

[0046] at least one memory for storing at least one program;

[0047] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0048] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program executable by a processor, and the program executable by the processor is used to implement the above method when executed by the processor.

[0049] The advantages and benefits of this application will be partially given in the following description, and partially become apparent from the following description, or learned through practice of this application:

[0050] Embodiments of the present application disclose a control method, system, device, and medium for a humanoid robot. The method obtains conversation information and work environment information for the current round of the humanoid robot, as well as a plurality of first structured information and first summary information, wherein the first structured information is structured information from a previous round, and the first summary information is unstructured summary information of the corresponding first structured information. The conversation information and work environment information are input into a pre-trained language model for structured extraction, thereby obtaining second structured information output by the pre-trained language model, wherein the second structured information is structured information for the current round. Based on the work environment information, all of the first structured information, and the first summary information, instructions are generated for the second structured information to obtain action planning instructions. The humanoid robot is then motion-controlled based on the action planning instructions. The method generates action planning instructions based on the structured information and unstructured summary information from the previous round, as well as the structured information from the current round. This method can refine the user's work task preferences and scenario patterns during human-robot collaborative control, so that the generated action planning instructions achieve personalized collaboration between the humanoid robot and the user, thereby improving the task efficiency of human-robot collaborative control. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present application or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly expressing some embodiments of the technical solutions of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0052] Figure 1 A schematic flow chart of a control method for a humanoid robot provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of a framework of a control system of a humanoid robot provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application. For the step numbers in the following embodiments, they are provided only for the convenience of explanation and are not intended to limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0057] At present, relevant technologies usually generate and execute action plans based on a preset action library and the reasoning ability of a large model after receiving instructions from the user (such as voice instructions, text instructions, gesture instructions, etc.). This method is insufficient in capturing and responding to user intentions, and is unable to refine and understand the user's work habits and preferences in scene modes. It is easy to fail to adjust the action in time according to the user's gestures or voice instructions. There is a risk of collision between the humanoid robot and the user, and the safety and task efficiency of human-machine collaborative control are low.

[0058] Furthermore, the motion plans generated by this method are relatively fixed and difficult to cope with parameter changes in the work tasks. The motion planning is not very flexible, and in the process of human-machine collaborative control, the user is often required to cooperate with the humanoid robot's movements. This results in a poor user experience. For example, in the task of assembling the elbow joint of a humanoid robot on an industrial production line, this method is difficult to cope with parameter changes in the joint assembly. For example, when the size of the joint components changes slightly due to temperature changes, the motion planning executed by the humanoid robot is prone to execution interruption or failure. At this time, the user is required to cooperate with the execution of the humanoid robot's movements, which results in a poor user experience.

[0059] It should be noted that the above-mentioned related technologies are only used to assist in understanding the technical solutions of this application and do not mean that they belong to the disclosed prior art.

[0060] In view of this, an embodiment of the present application provides a control method, system, device and medium for a humanoid robot, wherein the method generates action planning instructions based on structured information and unstructured summary information of previous rounds, as well as structured information of the current round, which can refine the user's work task preferences and scene mode rules in the process of human-machine collaborative control, so that the generated action planning instructions can better adapt to the user's work habits and preferences, reduce the risk of the humanoid robot colliding with the user when performing actions, effectively improve the safety of human-machine collaborative control, and realize personalized collaboration between the humanoid robot and the user, which is conducive to improving the task efficiency of human-machine collaborative control.

[0061] In addition, this method infers the unstructured summary information of each previous round through action interaction data, and generates instructions based on the unstructured summary information of the previous round. It can also extract the parameter change rules of the humanoid robot or the scene change rules under the scene mode, so that the subsequently generated action planning instructions can take into account the change rules, and the action planning instructions are more flexible, which is conducive to improving the user experience in the process of human-machine collaborative control.

[0062] Reference Figure 1 In an embodiment of the present application, a method for controlling a humanoid robot includes:

[0063] Step 110: Acquire conversation information and work environment information of the humanoid robot for the current round, as well as a plurality of first structured information and first summary information, where the first structured information is structured information from a previous round, and the first summary information is unstructured summary information corresponding to the first structured information;

[0064] In an embodiment of the present application, the conversation information may be user voice collected by the humanoid robot through a sound collection device (such as a microphone), or it may be user voice text obtained by voice-to-text conversion; the work environment information may be environmental data related to the work task; and the first structured information may be the structured information of the humanoid robot in the previous round, and the first summary information may be unstructured summary information of the same round as the first structured information, and the first summary information includes scene pattern rules and work task preferences. The round may specifically be the round in which the humanoid robot executes action planning instructions.

[0065] For example, the embodiment of the present application takes the assembly of robot elbow joints for an industrial production line as an example. In this case, the user may be an operator of the industrial production line, and the conversation information may be the instruction provided by the operator to "assemble the robot elbow joint and correct the range of motion"; and the working environment information may be the three-dimensional visual data of the robot elbow joint, force sensor signal data, etc.

[0066] Step 120: Input the session information and the work environment information into a pre-trained language model for structured extraction to obtain second structured information output by the pre-trained language model, where the second structured information is structured information for the current round.

[0067] In an embodiment of the present application, the humanoid robot can input conversation information and work environment information into a pre-trained language model, which can specifically be a large language model. The structured information in the conversation information and work environment information is extracted through the pre-trained language model to obtain the structured information of the current round, which is recorded as the second structured information.

[0068] In some embodiments, inputting the session information and the work environment information into a pre-trained language model for structured extraction to obtain second structured information output by the pre-trained language model includes:

[0069] Performing spatiotemporal alignment on the session information and the work environment information to obtain alignment information, the alignment information including the spatiotemporally aligned session information and work environment information;

[0070] The alignment information is input into the pre-trained language model for structured extraction to obtain the second structured information.

[0071] In an embodiment of the present application, before inputting the session information and the work environment information into the pre-trained language model, the session information and the work environment information can be spatiotemporally aligned based on a temporal and spatial alignment (TMRoPE) algorithm to fuse the semantic features in the session information with the sensor features in the work environment information, thereby obtaining alignment information; then, the alignment information is input into the pre-trained language model to obtain second structured information output by the pre-trained language model. For example, the second structured information can be {“task type”: “elbow joint assembly and calibration”, “component model”: “RJ-ELBOW-007”, “installation coordinates”: “(200mm, 150mm, 80mm)”, “torque requirement”: “12±0.5N·m”, “motion range”: “120°±2°”}.

[0072] It is understandable that before the session information and the work environment information are aligned in time and space, the session information and the work environment information can be preprocessed. The preprocessing operation of the session information can specifically include data cleaning, noise filtering and other operations, while the preprocessing operation of the work environment information can include data calibration, outlier removal and the like.

[0073] Step 130: Generate an instruction for the second structured information based on the work environment information, all the first structured information, and the first summary information to obtain an action planning instruction;

[0074] In an embodiment of the present application, action planning instructions corresponding to the second structured information can be generated based on the work environment information, the work task preferences and scene mode rules recorded in the previous round of structured information and unstructured summary information.

[0075] In some embodiments, generating an instruction for the second structured information based on the work environment information, all the first structured information, and the first summary information to obtain an action planning instruction includes:

[0076] Acquiring key sensor data of the humanoid robot;

[0077] According to the working environment information, all the first structured information is screened to obtain a plurality of third structured information corresponding to the working environment information;

[0078] Filtering all the first summary information according to the working environment information to obtain a plurality of second summary information corresponding to the working environment information;

[0079] In an embodiment of the present application, the key sensor data of the humanoid robot may be the sensor data required for the humanoid robot to execute motion planning instructions. The key sensor data may specifically be at least one of force sensor data, visual data, distance sensor data, etc.

[0080] It is understood that for the first structured information, information screening can be performed by filtering out first structured information with similar environmental parameters to the working environment information from all previous rounds of first structured information, and recording this as third structured information. In the robot elbow joint assembly application scenario, this similar environmental parameter can be at least one of the elbow joint assembly environment temperature, assembly environment humidity, elbow joint 3D visual data, force sensor signal data, etc. The first summary information is similar to the information screening content of the aforementioned first structured information and can be simply inferred. This application will not elaborate on this.

[0081] According to the key sensor data, all the third structured information and the second summary information, correlation analysis and instruction construction are performed on the second structured information to obtain the action planning instruction.

[0082] Furthermore, performing correlation analysis and instruction construction on the second structured information based on the key sensor data, all the third structured information and the second summary information to obtain the action planning instruction includes:

[0083] Performing structured feature extraction on the second structured information and all the third structured information to obtain a plurality of structured feature vectors, each of the structured feature vectors including a feature vector of the second structured information and a feature vector of the third structured information;

[0084] Performing unstructured feature extraction on all the second summary information to obtain summary feature vectors, each of the summary feature vectors corresponding to one of the structured feature vectors;

[0085] In an embodiment of the present application, structured feature extraction can be to extract the feature vectors of the second structured information and each third structured information respectively. For example, for the structured information {task type: elbow joint assembly, torque requirement: 12±0.5N·m, temperature compensation: 0.3% / ℃}, its feature vector in the form of a multidimensional vector can be [task type, torque value, temperature coefficient]; then, the feature vector of the second structured information is combined with the feature vector of each third structured information respectively to obtain a number of structured feature vectors, and the number of the structured feature vectors is the same as the number of the third structured information.

[0086] It is understandable that for any second summary information, unstructured feature extraction can be to extract the text semantic vector of the second summary information through the Bert model, recorded as the summary feature vector. The text semantic vector can specifically be the feature vector of the work task preference and the feature vector of the scene pattern rule in the second summary information. For example, in the robot elbow joint assembly application scenario, its feature vector can be determined from the assembly habit keyword "step-by-step tightening method" in the work task preference, as well as the feature vector of the first scene pattern rule "compensation is required when the temperature is ≥30°C" and the feature vector of the second scene pattern rule "the torque wrench accuracy decreases by 1% after 50 consecutive assemblies". The same is true for the remaining second summary information, which can be simply deduced by analogy.

[0087] performing feature fusion on all the structured feature vectors and the summary feature vector according to the key sensor data to obtain a target fusion feature;

[0088] Furthermore, the step of performing feature fusion on all the structured feature vectors and the summary feature vectors based on the key sensor data to obtain target fusion features includes:

[0089] performing similarity analysis on all the structured feature vectors according to the key sensor data to obtain a plurality of first similarities;

[0090] performing similarity analysis on all the summarized feature vectors according to the key sensor data to obtain a plurality of second similarities;

[0091] In an embodiment of the present application, similarity analysis can be to calculate the similarity between the feature vector of the key sensor data and each structured feature vector. There are many specific similarity calculation methods, for example, it can be a cosine similarity calculation method, a distance similarity calculation method, etc., so as to obtain the first similarity between each structured feature vector and the key sensor feature vector; and the content of the second similarity is similar to the content of the aforementioned first similarity, which can be simply inferred and will not be repeated in this application.

[0092] According to all the first similarities and the second similarities, all the structured feature vectors and the summary feature vectors are screened and fused to obtain the target fused feature.

[0093] Furthermore, the filtering and fusing of all the structured feature vectors and the summary feature vectors according to all the first similarities and the second similarities to obtain the target fusion feature includes:

[0094] According to each first similarity, performing weighted fusion on the corresponding second similarity to obtain a plurality of weighted fusion values;

[0095] Comparing all the weighted fusion values ​​to obtain a target fusion value;

[0096] According to the target fusion value, the corresponding structured feature vector and the summary feature vector are fused and spliced ​​to obtain the target fusion feature.

[0097] In an embodiment of the present application, weighted fusion may be performed by weighted fusion of the first and second similarities in the same round based on a preset weight parameter, thereby obtaining a weighted fusion value in each round. Specifically, for a previous round t-1, its second similarity may be determined based on the summary feature vector and the key sensor data feature vector in the previous round t-1; and its first similarity may be determined based on the feature vector of the key sensor data and the target structured feature vector, where the target structured feature vector includes the feature vector of the second structured information and the feature vector of the third structured information in the previous round t-1.

[0098] It can be understood that after obtaining the weighted fusion value in each round, the size relationship between the weighted fusion values ​​in all rounds can be compared, and the largest weighted fusion value among all weighted fusion values ​​can be determined as the target fusion value; then, based on the target fusion value, the target fusion feature of the current round is determined from the corresponding structured feature vector and summary feature vector.

[0099] The target fusion feature is input into a semantic action mapping model for mapping generation, and an action planning instruction output by the semantic action mapping model is obtained.

[0100] In an embodiment of the present application, the target fusion feature can be input into a semantic-action mapping model, which records a number of mapping relationships between semantics and actions. The semantic-action mapping model generates original action planning instructions based on the structured feature vector in the target fusion feature. Specifically, the mapping relationship between semantics and actions is queried through the structured information of the current round and the structured information of the previous round in the structured feature vector to determine the original action planning instructions; then, the original action planning instructions are corrected by summarizing the work task preferences and scene pattern rules contained in the feature vector, so as to obtain the final action planning instructions output by the semantic-action mapping model.

[0101] For example, in the application scenario of robot elbow joint assembly in an industrial production line, the original motion planning instructions can be corrected based on the "step-by-step tightening method" and "assembly habits of the operator" in the work task preferences in the summarized eigenvectors, and the "torque wrench accuracy drops by 1% after 50 consecutive assemblies" in the scene pattern rules. Specifically, the original motion avoidance instructions can be readjusted by summarizing the work task preferences of the eigenvectors so that the adjusted motion avoidance instructions make the humanoid robot's robotic arm path avoid the operator's operating area while adopting the step-by-step tightening method; and the angle parameters with torque in the adjusted motion planning instructions can be updated through the scene pattern rules to obtain the final motion planning instructions.

[0102] Step 140: Control the motion of the humanoid robot according to the motion planning instruction.

[0103] In an embodiment of the present application, a humanoid robot can obtain motion planning instructions and perform corresponding actions. For example, in the application scenario of robot elbow joint assembly on an industrial production line, the robot arm can be driven to assemble elbow joint products on the production line based on the motion planning instructions.

[0104] In some embodiments, the method further comprises:

[0105] Acquire a summary prompt word template and action interaction data, wherein the action interaction data includes the second structured information, the action planning instruction, and action execution information of the action planning instruction;

[0106] Constructing prompts for the action interaction data according to the summary prompt word template to obtain action interaction prompts;

[0107] The action interaction prompt is input into the pre-trained language model for unstructured reasoning to obtain third summary information, where the third summary information is unstructured summary information corresponding to the second structured information.

[0108] In an embodiment of the present application, for the current round, after executing the action planning instruction, the humanoid robot can generate corresponding action execution information based on the execution status of the action planning instruction (such as execution failure, execution interruption, execution success, etc.); then, the action execution information is added to the summary prompt word template together with the second structured information and action planning instruction of the current round to construct the generated action interaction prompt. There are many specific ways to construct the summary prompt word template, which will not be repeated here in this application.

[0109] It is understandable that the action interaction prompt input value can be unstructured inferred in the pre-trained language model, and the unstructured information in the action interaction prompt can be extracted through the pre-trained language model and recorded as the unstructured summary information of the second structured information in the current round.

[0110] A control system of a humanoid robot proposed according to an embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0111] Reference Figure 2 , a control system of a humanoid robot proposed in an embodiment of the present application includes:

[0112] A first processing unit 101 is configured to obtain conversation information and work environment information of a current round of the humanoid robot, as well as a plurality of first structured information and first summary information, wherein the first structured information is structured information of a previous round, and the first summary information is unstructured summary information of the corresponding first structured information;

[0113] A second processing unit 102 is configured to input the session information and the work environment information into a pre-trained language model for structured extraction, thereby obtaining second structured information output by the pre-trained language model, where the second structured information is structured information for the current round;

[0114] a third processing unit 103, configured to generate an instruction for the second structured information based on the work environment information, all the first structured information, and the first summary information, to obtain an action planning instruction;

[0115] The fourth processing unit 104 is configured to control the motion of the humanoid robot according to the motion planning instruction.

[0116] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0117] Reference Figure 3 , an embodiment of the present application further provides an electronic device, including:

[0118] at least one processor 201;

[0119] At least one memory 202, configured to store at least one program;

[0120] When the at least one program is executed by the at least one processor 201 , the at least one processor 201 implements the above method embodiment.

[0121] Similarly, it can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0122] An embodiment of the present application further provides a computer-readable storage medium, in which a program executable by the processor 201 is stored. The program executable by the processor 201 is used to implement the above-mentioned method embodiment when executed by the processor 201.

[0123] Similarly, the contents of the above method embodiments are applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0124] An embodiment of the present application further provides a computer program product, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0125] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0126] In some optional embodiments, the functions / operations mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, the two boxes shown in succession may actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flow chart of the present application are provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logic flows presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0127] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present application. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the routine skills of an engineer. Therefore, a person skilled in the art can implement the present application as set forth in the claims using ordinary techniques without undue experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the appended claims and their equivalents.

[0128] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the embodiment method of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0129] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0130] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0131] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0132] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples.

[0133] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and intent of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0134] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present application, and these equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for controlling a humanoid robot, characterized in that: include: Acquiring conversation information and work environment information of the humanoid robot in a current round, as well as a plurality of first structured information and first summary information, where the first structured information is structured information from a previous round, and the first summary information is unstructured summary information corresponding to the first structured information; Inputting the session information and the work environment information into a pre-trained language model for structured extraction to obtain second structured information output by the pre-trained language model, where the second structured information is structured information for the current round; Generating instructions for the second structured information according to the working environment information, all the first structured information, and the first summary information to obtain an action planning instruction; The humanoid robot is controlled in motion according to the motion planning instruction.

2. The method according to claim 1, characterized in that The step of inputting the session information and the work environment information into a pre-trained language model for structured extraction to obtain second structured information output by the pre-trained language model includes: Performing spatiotemporal alignment on the session information and the work environment information to obtain alignment information, the alignment information including the spatiotemporally aligned session information and work environment information; The alignment information is input into the pre-trained language model for structured extraction to obtain the second structured information.

3. The method according to claim 1, characterized in that Generating an instruction for the second structured information according to the working environment information, all the first structured information, and the first summary information to obtain an action planning instruction includes: Acquiring key sensor data of the humanoid robot; According to the working environment information, all the first structured information is screened to obtain a plurality of third structured information corresponding to the working environment information; Filtering all the first summary information according to the working environment information to obtain a plurality of second summary information corresponding to the working environment information; According to the key sensor data, all the third structured information and the second summary information, correlation analysis and instruction construction are performed on the second structured information to obtain the action planning instruction.

4. The method according to claim 3, characterized in that The step of performing correlation analysis and instruction construction on the second structured information based on the key sensor data, all the third structured information, and the second summary information to obtain the action planning instruction includes: Performing structured feature extraction on the second structured information and all the third structured information to obtain a plurality of structured feature vectors, each of the structured feature vectors including a feature vector of the second structured information and a feature vector of the third structured information; Performing unstructured feature extraction on all the second summary information to obtain summary feature vectors, each of the summary feature vectors corresponding to one of the structured feature vectors; performing feature fusion on all the structured feature vectors and the summary feature vector according to the key sensor data to obtain a target fusion feature; The target fusion feature is input into a semantic action mapping model for mapping generation, and an action planning instruction output by the semantic action mapping model is obtained.

5. The method according to claim 4, characterized in that The step of performing feature fusion on all the structured feature vectors and the summary feature vectors according to the key sensor data to obtain target fusion features includes: performing similarity analysis on all the structured feature vectors according to the key sensor data to obtain a plurality of first similarities; performing similarity analysis on all the summarized feature vectors according to the key sensor data to obtain a plurality of second similarities; According to all the first similarities and the second similarities, all the structured feature vectors and the summary feature vectors are screened and fused to obtain the target fused feature.

6. The method according to claim 5, characterized in that The step of screening and fusing all the structured feature vectors and the summary feature vectors according to all the first similarities and the second similarities to obtain the target fusion feature includes: According to each first similarity, performing weighted fusion on the corresponding second similarity to obtain a plurality of weighted fusion values; Comparing all the weighted fusion values ​​to obtain a target fusion value; According to the target fusion value, the corresponding structured feature vector and the summary feature vector are fused and spliced ​​to obtain the target fusion feature.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Acquire a summary prompt word template and action interaction data, wherein the action interaction data includes the second structured information, the action planning instruction, and action execution information of the action planning instruction; Constructing prompts for the action interaction data according to the summary prompt word template to obtain action interaction prompts; The action interaction prompt is input into the pre-trained language model for unstructured reasoning to obtain third summary information, where the third summary information is unstructured summary information corresponding to the second structured information.

8. A control system for a humanoid robot, characterized in that: include: a first processing unit configured to obtain conversation information and work environment information of the humanoid robot in a current round, as well as a plurality of first structured information and first summary information, wherein the first structured information is structured information of a previous round, and the first summary information is unstructured summary information of the corresponding first structured information; a second processing unit, configured to input the session information and the work environment information into a pre-trained language model for structured extraction, and obtain second structured information output by the pre-trained language model, where the second structured information is structured information of a current round; a third processing unit, configured to generate an instruction for the second structured information based on the work environment information, all the first structured information, and the first summary information, to obtain an action planning instruction; The fourth processing unit is used to control the motion of the humanoid robot according to the motion planning instruction.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images