A humanoid robot multi-modal interaction and dynamic task execution method and humanoid robot

Abstract

The application relates to the technical field of humanoid robots, in particular to a humanoid robot multi-modal interaction and dynamic task execution method and a humanoid robot. The method receives a multi-modal interaction request, obtains environmental characteristic information in combination with a world model, performs joint coding by using a multi-modal embedding model, and retrieves target information in a scenario-based knowledge database; further, the request is analyzed by combining a multi-modal large model and a VLA / VLM end-to-end model, action execution instructions are generated, and the robot is driven to execute; meanwhile, fine motion control is realized based on joint degree of freedom execution coefficients, and the instructions are dynamically updated when the environment changes, so that adaptive adjustment and stable execution of the task are realized. The application significantly improves the multi-modal understanding accuracy, task decision-making ability and adaptive execution ability of the humanoid robot in a complex dynamic environment, thereby improving the accuracy and flexibility of task execution.

Description

Technical Field

[0001] This application relates to the field of humanoid robot technology, and in particular to a method for multimodal interaction and dynamic task execution of a humanoid robot, as well as a humanoid robot. Background Technology

[0002] In related technologies, with the rapid development of artificial intelligence and robotics, humanoid robots are increasingly being applied in industrial manufacturing and home services, becoming an important vehicle for achieving intelligent operations and human-machine collaboration. These humanoid robots typically integrate multimodal interaction capabilities such as speech recognition, visual perception, and tactile feedback. By parsing multimodal interaction requests input by users and combining environmental perception information, they perform task planning and action execution, thereby completing various tasks such as parts assembly, item retrieval, and daily living assistance. Simultaneously, to improve task execution efficiency, some systems introduce scenario knowledge bases and learning models to support operational procedures and parameters in different scenarios, enabling humanoid robots to make autonomous decisions and execute actions in complex environments.

[0003] However, existing methods for multimodal interaction and task execution in humanoid robots still have significant shortcomings: First, multimodal information is typically processed separately by modules such as speech and vision and then fused in the backend, lacking an end-to-end integrated understanding mechanism, which can easily lead to information fusion bias and thus affect the accuracy of task parsing; Second, existing systems mostly rely on static knowledge bases or offline data, lacking a scenario-based knowledge management mechanism that supports dynamic updates, making it difficult to adapt to the ever-changing task requirements and environmental information in industrial and home scenarios; In addition, traditional robots mostly execute actions through overall degree of freedom control or fixed control strategies, unable to perform fine-tuning of individual joints, making it difficult to achieve an effective balance between high-precision industrial operation and high-flexibility home operation; At the same time, there is a lack of an effective closed-loop linkage mechanism between environmental perception and task execution, making it difficult for robots to adjust their action strategies in a timely manner when the environment changes, which can easily lead to task failure or collision risks; Furthermore, existing models lack the injection of domain knowledge specific to the application scenario during training, resulting in insufficient model generalization ability and difficulty in adapting to the actual application needs of multiple scenarios and multiple tasks.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The main objective of this application is to propose a method for multimodal interaction and dynamic task execution of a humanoid robot, as well as a humanoid robot, which can significantly improve the accuracy of multimodal understanding, task decision-making ability, and adaptive execution ability of the humanoid robot in complex dynamic environments, thereby improving the accuracy and flexibility of task execution.

[0006] To achieve the above objectives, one aspect of this application proposes a method for multimodal interaction and dynamic task execution of a humanoid robot, the method comprising the following steps: The system receives multimodal interaction requests and obtains corresponding environmental feature information based on a world model. The multimodal interaction requests include one or more combinations of voice requests, visual requests, and tactile requests. The environmental feature information includes one or more of the following: three-dimensional spatial information of the environment, obstacle distribution information, task object feature information, and scene lighting information. The multimodal interaction request and the environmental feature information are jointly encoded to obtain a joint encoding result; the joint encoding is implemented based on a pre-trained multimodal embedding model. Based on the joint encoding results, a search is performed in the contextualized knowledge database to obtain the target search results. The contextualized knowledge database supports dynamic updates and includes an industrial scenario sub-database and a household scenario sub-database. Based on a multimodal large model combined with an end-to-end VLA / VLM model, the multimodal interaction request is parsed according to the target retrieval results and the environmental feature information to generate action execution instructions with joint degrees of freedom execution coefficients; The task action is executed according to the action execution command, and the motion state of each joint is adjusted based on the joint degree of freedom execution coefficient. The joint degree of freedom execution coefficient can be adapted to humanoid robot models with different basic degrees of freedom. When the environmental feature information changes according to a preset time, the action execution instruction is regenerated and executed based on the updated environmental feature information, and the task execution result is output.

[0007] In some embodiments, the joint encoding of the multimodal interaction request and the environmental feature information to obtain a joint encoding result includes: The multimodal interaction request is preprocessed to obtain standardized multimodal data; The environmental feature information is processed in a structured manner to obtain structured environmental data; Based on the pre-trained multimodal embedding model, feature mapping is performed on the standardized multimodal data and the structured environment data respectively to obtain the corresponding multimodal request vector and environment feature vector; The multimodal request vector and the environmental feature vector are fused and mapped to a unified semantic vector space to generate a joint encoding vector, which serves as the joint encoding result.

[0008] In some embodiments, the step of retrieving the target retrieval result from the contextualized knowledge database based on the joint encoding result includes: Based on the joint encoding result, similarity matching calculation is performed with each index in the scenario-based knowledge database to obtain the corresponding similarity result; the industrial scenario sub-library and the household scenario sub-library in the scenario-based knowledge database each contain multiple indexes and data blocks corresponding to each index, and the indexes are obtained by embedding the data blocks; Based on the similarity results, the indexes are sorted, and a preset number of indexes with the highest similarity are selected as target indexes. Based on the target index, locate the corresponding data block in the contextualized knowledge database; The data blocks are fused to obtain the target retrieval results.

[0009] In some embodiments, the dynamic updating of the contextualized knowledge database includes: Receive newly added industrial scenario knowledge data or home scenario knowledge data; The newly added knowledge data is split into corresponding data blocks according to the rules of a single process or a single life service scenario; The data block is subjected to embedding encoding to generate a corresponding index; The index and the data block are written into the corresponding scenario sub-library to complete the dynamic update of the scenario-based knowledge database.

[0010] In some embodiments, the step of parsing the multimodal interaction request based on the multimodal large model combined with the VLA / VLM end-to-end model, according to the target retrieval results and the environmental feature information, and generating action execution instructions with joint degrees of freedom execution coefficients, includes: Based on the preset prompt information template, the multimodal interaction request, the target retrieval result, and the environmental feature information are input into the multimodal large model and the VLA / VLM end-to-end model to generate model input; Based on the model input, slot identification is performed on the multimodal interaction request to obtain task semantic information; Based on the task semantic information, the target retrieval results, and the environmental feature information, task action prediction is performed to obtain an action sequence; Configure the execution coefficients of each joint's degrees of freedom based on the aforementioned action sequence; Based on the action sequence and the degree of freedom execution coefficients, action execution instructions are generated.

[0011] In some embodiments, the construction process of the multimodal large model and the VLA / VLM end-to-end model includes: Obtain multimodal interaction request samples, environmental feature samples, and corresponding task action labels and joint degree of freedom execution coefficient labels; Based on the multimodal interaction request samples and the environmental feature samples, the contextualized knowledge database is searched to obtain the corresponding sample search results; Based on the multimodal interaction request samples, the environmental feature samples, the sample retrieval results, and the preset prompt information template, the model training input is constructed; Based on the model training input, a multi-task joint training method is used to train the multimodal large model and the VLA / VLM end-to-end model. Specifically, slot recognition and task action prediction training are performed based on the cross-entropy loss function, and joint degree of freedom execution coefficient prediction training is performed based on the mean square error loss function. The losses are then weighted and fused to obtain the trained multimodal large model and VLA / VLM end-to-end model.

[0012] In some embodiments, the construction process of the multimodal large model and the VLA / VLM end-to-end model further includes: Pre-configured domain knowledge of humanoid robots, process knowledge of industrial scenarios, and life knowledge of household scenarios are injected into the basic multimodal model and the basic VLA / VLM end-to-end model. The knowledge injection methods include constructing domain knowledge into a knowledge graph and fusing it into the weights of the basic model through the embedding layer, or injecting domain knowledge into the pre-training process of the model through scenario-based prompt word templates. The injected base model is pre-trained to obtain a pre-trained multimodal model and a VLA / VLM end-to-end model.

[0013] In some embodiments, the process of adjusting the joint degree-of-freedom execution coefficients includes: Based on the task type and scenario type, each joint of the robot is configured with an independent degree of freedom execution coefficient k. i ,0 <k i ≤1, where i is the joint number; The motion accuracy P of the core operating joint i =k i ×P0, the mobility of the joint F i =k i ×F0; where P0 is the basic motion accuracy of the joint, and F0 is the basic motion flexibility of the joint; The degree of freedom execution coefficient k i It is decoupled from the number of basic degrees of freedom of the robot, and can be adapted to humanoid robot models with any basic degrees of freedom.

[0014] In some embodiments, when the environmental feature information undergoes a preset change, regenerating and executing the action execution instruction based on the updated environmental feature information, and outputting the task execution result, includes: During the task execution, the environmental feature information is updated in real time based on the world model. When the detected position deviation of the work object, change of obstacles, or abnormal operation of the equipment reaches a preset threshold, the current task action is paused. Based on the updated environmental feature information, joint encoding and contextualized knowledge database retrieval are performed again to obtain updated target retrieval results; Based on the updated target retrieval results and environmental feature information, the action execution instructions are regenerated and executed, and the task execution results are output.

[0015] To achieve the above objectives, another aspect of this application provides a humanoid robot, which includes: The multimodal perception module is used to collect external multimodal interaction requests; The world model perception module is used to perceive the physical environment around the robot and generate and update environmental feature information in real time. The joint encoding module is used to jointly encode the multimodal interaction request and the environmental feature information to obtain the joint encoding result; The scenario knowledge retrieval module is used to retrieve the target retrieval result from the scenario-based knowledge database based on the joint encoding result; the scenario-based knowledge database supports dynamic updates and includes an industrial scenario sub-database and a home scenario sub-database. The instruction generation module is used to parse the multimodal interaction request based on the multimodal large model combined with the VLA / VLM end-to-end model, according to the target retrieval results and the environmental feature information, and generate action execution instructions with joint degrees of freedom execution coefficients; and when the environmental feature information undergoes a preset change, it regenerates the action execution instructions based on the updated environmental feature information. The joint drive module is used to drive each joint of the robot to perform task actions according to the action execution command and the degree of freedom execution coefficient. The joint degree of freedom execution coefficient can be adapted to humanoid robot models with different basic degrees of freedom. The output module is used to regenerate and execute action execution instructions based on the updated environmental feature information when the environmental feature information undergoes a preset change, and output the task execution result. The communication module is used to enable the robot to interact with external systems and provide feedback on execution results.

[0016] To achieve the above objectives, another aspect of this application provides a humanoid robot, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0017] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0018] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0019] The embodiments of this application include at least the following beneficial effects: This application provides a method for multimodal interaction and dynamic task execution of a humanoid robot, as well as a humanoid robot. This solution, by receiving multimodal interaction requests and combining them with a world model to obtain environmental feature information, can uniformly model speech, text, images, and scene states, thereby improving the robot's comprehensive understanding of complex task semantics and environmental context. Furthermore, by using a pre-trained multimodal embedding model to jointly encode multimodal interaction requests and environmental feature information, and combining this with a scenario-based knowledge database for retrieval, the robot's accuracy in recognizing task objectives, operational objects, and execution constraints can be enhanced, improving the pertinence and accuracy of task decision-making. Simultaneously, based on multimodal... The large model combines VLA / VLM end-to-end models to generate action execution instructions and regulates the motion state of each joint through joint degree-of-freedom execution coefficients. This facilitates efficient connection from semantic understanding to action control, improving the coordination, stability, and precision of action execution. Furthermore, when environmental feature information changes according to preset parameters, action execution instructions can be regenerated and executed based on the updated environmental feature information. This enables the humanoid robot to have stronger environmental adaptability and dynamic task adjustment capabilities, solving the problems of insufficient interactive understanding, rigid task execution, and untimely response to environmental changes in existing robots in complex open scenarios. As a result, the success rate, flexibility, and intelligence level of task execution are improved. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a method for multimodal interaction and dynamic task execution of a humanoid robot provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the scenario-based knowledge database provided in the embodiments of this application; Figure 3 This is a schematic diagram illustrating the association between the configuration of degree-of-freedom coefficients and the execution of actions provided in an embodiment of this application; Figure 4 This is a flowchart illustrating an industrial scenario (component assembly) provided in an embodiment of this application; Figure 5 This is a flowchart illustrating a home scenario (home item retrieval) provided in an embodiment of this application; Figure 6This is a schematic diagram of a humanoid robot provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0022] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0023] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0025] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application shall be interpreted as follows; A multimodal interaction request (MRP) is a task request message sent by a user to a humanoid robot through one or more sensory means such as voice, vision, and touch. It is used to describe the task objective, the object to be manipulated, and the interaction intent. Environmental feature information refers to structured data obtained after the robot perceives its environment through sensors. This includes three-dimensional spatial information of the environment, obstacle distribution information, task object feature information, and scene state information, which are used to characterize the current task execution environment. A world model is a cognitive model used to model and dynamically update the physical environment in which a robot is located. It constructs the spatial structure, object relationships, and state changes of the environment by integrating multi-source sensor data to support task decision-making and action planning. A multimodal embedding model is a model used to map different modal data (such as speech, image, tactile and environmental data) to a unified semantic vector space, and achieves a unified expression of cross-modal information through feature extraction and fusion. A scenario-based knowledge base is a structured collection of knowledge organized and stored according to different application scenarios (such as industrial scenarios and home scenarios). It includes indexes and corresponding data blocks to support task understanding and decision retrieval. An index (Index Vector) is a vector representation formed by embedding and encoding data blocks in a knowledge database, used for similarity matching calculations with the joint encoding results; Similarity matching refers to the process of determining the knowledge data most relevant to the current task by calculating the semantic distance (such as cosine similarity) between the joint encoding vector and the index. A multimodal large model refers to a large-scale deep learning model that can simultaneously process multimodal information such as text, speech, and images, and perform semantic understanding and task reasoning. The VLA / VLM end-to-end model (Vision-Language-Action / Vision-Language End-to-End Model) refers to a deep learning model that unifies the modeling of visual, linguistic, and motion control information. It achieves a direct mapping from multimodal inputs to action execution commands through a single model framework. The VLM model (Vision-Language Model) is used for visual and linguistic semantic alignment and understanding, while the VLA model (Vision-Language-Action Model) further enables action sequence generation and joint control parameter configuration. The two models work collaboratively: VLM handles the semantic understanding of multimodal requests, while VLA maps the understanding results into executable robot action logic. The joint degree-of-freedom execution coefficient is a parameter used to regulate the amplitude, speed, and direction of motion of each joint of a robot, in order to achieve fine motion control.

[0026] This application provides a method for multimodal interaction and dynamic task execution of a humanoid robot, as well as the humanoid robot itself. This method receives multimodal interaction requests and combines them with a world model to obtain environmental feature information. It can uniformly model speech, text, images, and scene states, improving the robot's comprehensive understanding of complex task semantics and environmental context. Furthermore, by using a pre-trained multimodal embedding model to jointly encode multimodal interaction requests and environmental feature information, and combining this with a contextualized knowledge database for retrieval, the robot's accuracy in recognizing task objectives, operational objects, and execution constraints can be enhanced, improving the relevance and accuracy of task decisions. Simultaneously, based on a large multimodal model combined with VL (Virtual World Model), the method further enhances the robot's ability to perform multimodal interaction and dynamic task execution. The A / VLM end-to-end model generates action execution instructions and regulates the motion state of each joint through joint degree-of-freedom execution coefficients. This facilitates efficient connection from semantic understanding to action control, improving the coordination, stability, and precision of action execution. Furthermore, when environmental feature information changes according to preset parameters, it can regenerate and execute action execution instructions based on the updated environmental feature information. This enables the humanoid robot to have stronger environmental adaptability and dynamic task adjustment capabilities, solving the problems of insufficient interactive understanding, rigid task execution, and untimely response to environmental changes in existing robots in complex open scenarios. As a result, it improves the success rate, flexibility, and intelligence level of task execution.

[0027] This application provides a method for multimodal interaction and dynamic task execution of a humanoid robot, relating to the field of humanoid robot technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited thereto. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing a method for multimodal interaction and dynamic task execution of a humanoid robot, but is not limited to the above forms.

[0028] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0029] Figure 1 This is an optional flowchart of a humanoid robot multimodal interaction and dynamic task execution method provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S1 to S6: S1: Receive multimodal interaction requests and obtain corresponding environmental feature information based on the world model; the multimodal interaction request includes one or more combinations of voice requests, visual requests, and tactile requests; the environmental feature information includes one or more of environmental three-dimensional spatial information, obstacle distribution information, work object feature information, and scene lighting information.

[0030] In this embodiment, the humanoid robot is pre-configured with a multimodal perception module and a world model perception module.

[0031] The humanoid robot collects external multimodal interaction requests through a multimodal perception module (including a microphone array, a high-definition camera, a tactile sensor, and a gesture recognition module), and performs preprocessing such as noise reduction and normalization on the raw request data to obtain standardized multimodal interaction requests. Humanoid robots use a world model perception module (including lidar, depth camera, and IMU) to perceive and model the surrounding physical environment in real time, extracting environmental feature information, including: 3D spatial information of the environment, obstacle distribution information, position / size / material characteristics of the work object, and scene lighting / temperature and humidity information; in industrial scenarios, it also includes the operating status information of production line equipment, and in home scenarios, it also includes the layout / model information of home appliances.

[0032] S2: Jointly encode the multimodal interaction requests and environmental feature information to obtain the joint encoding result; this joint encoding is based on a pre-trained multimodal embedding model. The process involves jointly encoding multimodal interaction requests and environmental feature information to obtain the joint encoding results, including: The multimodal interaction requests are preprocessed to obtain standardized multimodal data; The environmental feature information is processed in a structured manner to obtain structured environmental data; Based on the pre-trained multimodal embedding model, feature mapping is performed on standardized multimodal data and structured environmental data respectively to obtain the corresponding multimodal request vector and environmental feature vector; The multimodal request vector and the environmental feature vector are fused and mapped to a unified semantic vector space to generate a joint encoding vector, which serves as the joint encoding result.

[0033] In this embodiment, after acquiring a multimodal interaction request, the humanoid robot first preprocesses the request to obtain standardized multimodal data. Specifically, for voice requests, speech denoising, endpoint detection, and speech recognition are used to convert the speech signal into a text sequence, followed by keyword extraction and semantic normalization. For visual requests, visual semantic information such as the target object's position and pointing relationship is extracted through target detection and gesture recognition. For tactile requests, contact position, contact intensity, and contact type are extracted through tactile sensors and mapped to confirmation or control semantic tags. Furthermore, time alignment and semantic fusion are performed on the different modal data to form standardized multimodal data containing task intent, target object, and spatial pointing information. Simultaneously, environmental feature information is structured, converting the three-dimensional spatial information of the environment, obstacle distribution information, task object feature information, and scene lighting information into structured environmental data. This structured environmental data includes fields such as object position coordinates, size parameters, material properties, obstacle relationships, and environmental constraint parameters.

[0034] Building upon this foundation, a pre-trained multimodal embedding model is used to perform feature mapping processing on standardized multimodal data and structured environmental data, respectively, to obtain corresponding multimodal request vectors and environmental feature vectors. Specifically, the multimodal embedding model extracts task semantic features, object semantic features, and action relationship features from the standardized multimodal data, and integrates speech, visual, and tactile information to generate a unified multimodal request vector. Simultaneously, it embeds and encodes the structured environmental data, extracting environmental spatial topological features, obstacle distribution features, object attribute features, and scene constraint features to generate an environmental feature vector. Preferably, the multimodal embedding model has integrated humanoid robot domain knowledge and industrial and household scenario knowledge during pre-training, thereby enhancing its ability to express task semantics and environmental constraints.

[0035] Subsequently, the multimodal request vector and the environmental feature vector are fused and mapped to a unified semantic vector space to generate a joint encoding vector, which serves as the joint encoding result. Specifically, the task intent information and environmental constraint information can be semantically aligned through vector concatenation, weighted fusion, or cross-modal alignment based on an attention mechanism. Thus, the joint encoding result can comprehensively reflect the matching relationship between the current task request and the actual environment, providing a unified and highly expressive input representation for accurate retrieval of subsequent contextualized knowledge databases and generation of action execution instructions.

[0036] S3: Based on the joint encoding results, perform a search in the contextualized knowledge database to obtain the target search results; Specifically, based on the joint coding results, a search is performed in the contextualized knowledge database to obtain the target search results, including: Based on the joint encoding results, similarity matching calculations are performed with each index in the contextual knowledge database to obtain the corresponding similarity results. The contextual knowledge database includes an industrial scenario sub-database and a household scenario sub-database. Each sub-database contains multiple indexes and data blocks corresponding to each index. The indexes are obtained by embedding the data blocks. Based on the similarity results, the indexes are sorted, and a preset number of indexes with the highest similarity are selected as target indexes. Based on the target index, locate the corresponding data block in the contextualized knowledge database; The data blocks are fused to obtain the target retrieval results.

[0037] Specifically, the dynamic updating of the contextualized knowledge database includes: Receive newly added industrial scenario knowledge data or home scenario knowledge data; The newly added knowledge data is split into corresponding data blocks according to the rules of a single process or a single life service scenario; The data blocks are embedded and encoded to generate corresponding indexes; The index and data blocks are written to the corresponding scenario sub-database to complete the dynamic update of the scenario-based knowledge database.

[0038] In this embodiment, after obtaining the joint encoding result, the humanoid robot performs retrieval processing in the contextualized knowledge database based on the joint encoding result. Specifically, the contextualized knowledge database includes an industrial scenario sub-database and a home scenario sub-database, each sub-database consisting of multiple indexes and corresponding data blocks.

[0039] Specifically, refer to Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of the scenario-based knowledge database in this embodiment. The scenario-based knowledge database adopts a hierarchical structure design. The top layer is a unified scenario-based knowledge database, which is used to centrally store various scenario knowledge related to the humanoid robot's task execution. The knowledge content is expanded and maintained in real time through a dynamic update interface, thereby ensuring the timeliness and applicability of the knowledge data.

[0040] Specifically, the scenario-based knowledge database is logically divided into two main sub-databases: an industrial scenario sub-database and a household scenario sub-database. The industrial scenario sub-database stores knowledge data related to industrial production, including but not limited to component assembly processes, equipment operating procedures, process parameters, and safety rules. The household scenario sub-database stores knowledge data related to home services, including home layout information, daily service processes, equipment usage rules, and safety constraints. This sub-database division enables the categorized management and rapid retrieval of knowledge across different application scenarios. Furthermore, by incorporating actual environmental characteristics, the search scope can be limited, improving search efficiency and accuracy.

[0041] Within each sub-database, an index and data block organization method are employed. The index is generated from the corresponding data block through embedded encoding, representing the semantic features of the data block and supporting similarity matching calculations with the joint encoding results. The data block serves as the actual storage unit for knowledge, further subdivided into semantic units and scenario parameter units. The semantic unit describes semantic information such as task objects, process names, equipment types, or service categories, while the scenario parameter unit describes specific parameters and constraints related to task execution, such as assembly torque, gripping force, operating steps, path restrictions, and safety rules. This structural design enables structured expression and efficient retrieval of knowledge.

[0042] Furthermore, the dynamic update interface receives new knowledge data during system operation, splits and embeds this data according to preset rules, generates corresponding indexes and data blocks, and then writes them into the corresponding scene sub-library, enabling continuous updating and expansion of the knowledge database. Based on this structure, after the robot obtains the joint encoding results, it can perform similarity matching in the corresponding sub-library using the index, quickly locate relevant data blocks, and perform fusion processing to obtain target retrieval results that highly match the current task, providing accurate knowledge support for the generation of subsequent action execution instructions.

[0043] The humanoid robot uses the joint encoding result as a query vector and performs similarity matching calculation with each index in the contextualized knowledge database to obtain the similarity result corresponding to each index. Preferably, the similarity matching calculation adopts the cosine similarity calculation method to characterize the semantic closeness between the current task request and each contextual knowledge item.

[0044] After obtaining the similarity results, the robot sorts the indices according to their similarity values ​​and selects a preset number of indices with the highest similarity as target indices. The preset number is preferably the first m, where m is a positive integer and 1≤m≤5. In this embodiment, m=3 can be used. Subsequently, the robot locates the corresponding data blocks in the contextualized knowledge database based on the target indices and retrieves the located data blocks to form a candidate knowledge set directly related to the current task.

[0045] After determining the target index, the corresponding data block is located in the contextualized knowledge database based on the target index, and the data block is fused to obtain the target retrieval result. Specifically, the semantic units and scene parameter units in each data block are summarized, deduplicated, and correlated to integrate the scattered knowledge information into a unified task knowledge representation.

[0046] S4: Based on a multimodal large model combined with an end-to-end VLA / VLM model, the multimodal interaction request is parsed according to the target retrieval results and environmental feature information to generate action execution instructions with joint degree of freedom execution coefficients; the VLM is responsible for multimodal semantic understanding, and the VLA is responsible for action sequence planning and joint degree of freedom execution coefficient configuration. Specifically, based on a multimodal large model combined with an end-to-end VLA / VLM model, the multimodal interaction request is parsed according to the target retrieval results and environmental feature information to generate action execution instructions with joint degrees of freedom execution coefficients, including: Based on the preset prompt information template, the multimodal interaction request, target retrieval results and environmental feature information are input into the multimodal large model and the VLA / VLM end-to-end model to generate model input; Based on the model input, slot identification is performed on multimodal interaction requests to obtain task semantic information; Task action prediction is performed based on task semantic information, target retrieval results, and environmental feature information to obtain action sequences; The execution coefficients of each joint's degrees of freedom are configured based on the motion sequence; Based on the action sequence and the execution coefficients of the degrees of freedom, action execution instructions are generated.

[0047] In this embodiment, after obtaining the target retrieval result, the humanoid robot, based on a multimodal large model combined with an end-to-end VLA / VLM model, parses the multimodal interaction request and generates action execution instructions. Specifically, firstly, based on a preset prompt information template, the multimodal interaction request, target retrieval result, and environmental feature information are uniformly encapsulated to form the model input. The prompt information template is pre-configured with task parsing rules, action planning specifications, freedom degree execution coefficient configuration rules, and instruction output format requirements, thereby guiding the model to generate standardized results according to the humanoid robot's execution needs. For example, in an industrial scenario, this includes assembly process parameters and precision requirements, while in a home scenario, it includes path constraints and grasping force limitations. Subsequently, based on the model input, the model performs slot identification on the multimodal interaction request, extracting key information such as the work object, task type, execution position, target position, and precision or force requirements to obtain task semantic information.

[0048] After obtaining the task semantic information, task action prediction is performed by combining the target retrieval results and environmental feature information to generate an action sequence. Specifically, the multimodal large model is responsible for task semantic understanding and completion, the VLM is used for multimodal information parsing and scene knowledge association, and the VLA is used to map the parsing results into specific action planning logic. Through the collaborative processing of these three components, user requests, scene knowledge, and environmental constraints are fused and reasoned to obtain an action sequence that conforms to actual execution conditions.

[0049] After the motion sequence is determined, the execution coefficients of each joint's degrees of freedom are configured based on the motion sequence, and motion execution instructions are generated. Specifically, according to the task's requirements for precision or flexibility, corresponding execution coefficients k are set for each joint of the arms, legs, torso, and head. i This is to achieve differentiated control. For example, in high-precision operations such as parts assembly and precision testing in industrial settings, the k-axis of the core operating joints... i The value ranges from 0.9 to 1.0, and k is the value of the moving joint. i The value ranges from 0.7 to 0.9; in flexible operation tasks such as furniture retrieval and spatial movement in home scenarios, k is the core operation joint. i The value of k for the moving joint is between 0.7 and 0.9. i The value is set to 0.8~1.0 to improve movement flexibility. Finally, the motion sequence is combined with the execution coefficients of each joint degree of freedom to generate standardized motion execution instructions. These instructions include the task type, motion steps, and k values ​​for each joint. i Information such as execution parameters can be directly used to drive the robot to complete task actions.

[0050] Specifically, the construction process of multimodal large models and VLA / VLM end-to-end models includes: Obtain multimodal interaction request samples, environmental feature samples, and corresponding task action labels and joint degree of freedom execution coefficient labels; Based on multimodal interaction request samples and environmental feature samples, the scenario-based knowledge database is searched to obtain the corresponding sample retrieval results; The model training input is constructed based on multimodal interaction request samples, environmental feature samples, sample retrieval results, and preset prompt information templates; Based on the model training input, a multi-task joint training method is used to train the multimodal large model and the VLA / VLM end-to-end model. Specifically, slot recognition and task action prediction training are performed based on the cross-entropy loss function, and joint degree of freedom execution coefficient prediction training is performed based on the mean square error loss function. The losses are then weighted and fused to obtain the trained multimodal large model and VLA / VLM end-to-end model.

[0051] The construction process of multimodal large models and VLA / VLM end-to-end models also includes: Pre-configured domain knowledge of humanoid robots, process knowledge of industrial scenarios, and life knowledge of household scenarios are injected into the basic multimodal model and the basic VLA / VLM end-to-end model. The knowledge injection methods include constructing domain knowledge into a knowledge graph and fusing it into the weights of the basic model through the embedding layer, or injecting domain knowledge into the pre-training process of the model through scenario-based prompt word templates. The injected base model is pre-trained to obtain a pre-trained multimodal model and a VLA / VLM end-to-end model.

[0052] Specifically, the process of adjusting the joint degree-of-freedom execution coefficients includes: Based on the task type and scenario type, each joint of the robot is configured with an independent degree of freedom execution coefficient k. i ,0 <k i ≤1, where i is the joint number; The motion accuracy P of the core operating joint i =k i ×P0, the mobility of the joint F i =k i ×F0; where P0 is the basic motion accuracy of the joint, and F0 is the basic motion flexibility of the joint; Degrees of freedom execution coefficient k i It is decoupled from the number of basic degrees of freedom of the robot, and can be adapted to humanoid robot models with any basic degrees of freedom.

[0053] In this embodiment, the construction process of the multimodal large model and the VLA / VLM end-to-end model includes a knowledge injection pre-training stage and a supervised joint training stage. First, in the knowledge injection pre-training stage, knowledge from the humanoid robot domain, industrial scenario processes, and household scenario life knowledge are injected into the basic multimodal model and the basic VLA / VLM end-to-end model. Specifically, the humanoid robot domain knowledge includes joint kinematics, dynamic constraints, and degree-of-freedom control rules; the industrial scenario knowledge includes assembly processes, process parameters, and safety regulations; and the household scenario knowledge includes home layout, service processes, and operational constraints. This can be achieved by constructing a knowledge graph and using an embedding layer to integrate knowledge information into the model weights, or by injecting knowledge into the model pre-training process in the form of rule constraints using scenario-based prompt word templates. This allows the model to possess a basic understanding of task semantics, scenario rules, and action control relationships during the pre-training stage. After knowledge injection, the basic model is pre-trained to obtain a multimodal model and a VLA / VLM end-to-end model with domain adaptability.

[0054] After pre-training, the supervised joint training phase begins. First, multimodal interaction request samples, environmental feature samples, and corresponding task action labels and joint degree-of-freedom execution coefficient labels are acquired. The multimodal interaction request samples include combinations of voice, visual, and tactile requests; the environmental feature samples include environmental spatial information, obstacle distribution information, task object features, and scene state information; task action labels represent task types and action sequences; and joint degree-of-freedom execution coefficient labels represent the target control parameters of each joint in different tasks. Subsequently, based on the multimodal interaction request samples and environmental feature samples, a scenario-based knowledge database is searched to obtain corresponding sample retrieval results. These results are then combined with a pre-set prompt information template to uniformly encapsulate the multimodal interaction request samples, environmental feature samples, sample retrieval results, and template constraint information, constructing the model training input. This enables the model to learn the mapping relationship between task semantics, environmental constraints, and scene knowledge.

[0055] Based on this, a multi-task joint training approach is adopted to train the multimodal large model and the VLA / VLM end-to-end model. Specifically, for slot recognition and task action prediction tasks, the cross-entropy loss function is used for optimization to improve the accuracy of the model in task semantic parsing and action sequence generation; for the joint degree of freedom execution coefficient prediction task, the mean squared error loss function is used for optimization to reduce the deviation between the predicted coefficients and the label values.

[0056] Furthermore, the aforementioned multi-task losses are weighted and fused to construct a unified total loss function. The model parameters are then iteratively updated based on this total loss function until the model converges, resulting in a trained multimodal large model and a VLA / VLM end-to-end model. Through this construction process, the model can output action sequences and joint degree-of-freedom execution coefficients that meet the robot's execution requirements, based on the fusion of multimodal information, scene knowledge, and environmental constraints. This improves the accuracy and adaptability of task execution in different scenarios.

[0057] Specifically, the AdamW optimizer is used for gradient descent optimization. The initial learning rate is set to 3e-5, and the learning rate decays by 10% every 10 rounds. The total number of training rounds is 50, the batch size is 32, and the weight decay coefficient is 0.01. The loss function includes the cross-entropy loss function for slot identification and task action prediction, and the mean squared error loss function for configuring the degree of freedom execution coefficients. The total loss value is the weighted sum of the two types of loss values. The model parameters are updated in reverse by calculating the loss value between the model output and the sample label until the total loss value converges to below 0.01.

[0058] refer to Figure 3 As shown, Figure 3This is a schematic diagram of the joint degree-of-freedom execution coefficient configuration and control mechanism of the humanoid robot in this invention. The humanoid robot has 29 basic degrees of freedom (excluding dexterous hands). By classifying each joint and configuring differentiated coefficients, coordinated control of motion accuracy and motion flexibility can be achieved during task execution.

[0059] Specifically, based on 29 fundamental degrees of freedom, the robot joints are first functionally classified into two main categories: core manipulator joints and locating joints. Core manipulator joints primarily include the shoulder, elbow, and wrist joints of both arms, used for performing delicate tasks such as grasping, assembly, and manipulation. Locating joints primarily include the hip, knee, and ankle joints of both legs, used for achieving overall robot positional movement and posture adjustment. This joint classification allows different task requirements to be mapped to different joint types, thus providing a foundation for subsequent differentiated control.

[0060] Based on the joint classification, the execution coefficients for the degrees of freedom of core manipulation joints and translation joints are configured separately. For core manipulation joints, the corresponding core joint coefficient k is configured. i This is done to adjust the motion precision of the joints, enabling them to have higher positioning accuracy and trajectory stability when performing delicate operations; for movable joints, corresponding movable joint coefficients k are configured. i This allows for the regulation of joint mobility, enabling the robot to have higher dynamic response capabilities during path movement, obstacle avoidance, and posture adjustment. By configuring execution coefficients for different joint types, a hierarchical control strategy prioritizing accuracy and flexibility is achieved.

[0061] Furthermore, based on the execution coefficients of the degrees of freedom, the motion precision of the core operating joints is adjusted, and the motion flexibility of the movable joints is adjusted. The motion precision of the core operating joints is positively correlated with the execution coefficient; the higher the execution coefficient, the more stable the joint movement and the smaller the error. Similarly, the motion flexibility of the movable joints is also positively correlated with the execution coefficient; the higher the execution coefficient, the more sensitive the joint response and the stronger the range of motion and adaptability. The results of these two types of adjustments are ultimately converged in the joint drive module. The joint drive module, based on the action execution command and the corresponding execution coefficient, performs coordinated drive control on each joint, thereby achieving a dynamic balance between task execution precision and flexibility in different scenarios.

[0062] Specifically, in this embodiment, based on the scenario type and task requirements, independent k-joints are configured for each of the robot's 29 basic degrees of freedom. i The configuration rules are as follows: Industrial application: Core operating joints of both arms (shoulder, elbow, and wrist joints) i =0.95~1.0, corresponding to an improvement in motion accuracy to 95%~100% of the basic accuracy, trunk joint k i =0.9, head joint ki =0.8, k of the two leg joints i =0.75~0.8, corresponding to 75%~80% of the basic flexibility of motion; the total execution accuracy meets the requirements of industrial precision assembly, while also taking into account the needs of small movements; Home use scenario: Dual-arm core operation joint k i =0.8~0.9, corresponding to 80%~90% of the basic accuracy of motion, trunk joint k i =0.9, head joint k i =0.95, k of the two leg movement joints i =0.9~1.0, corresponding to an improvement in motion flexibility to 90%~100% of the basic flexibility; the total execution flexibility meets the needs of home use for picking up and putting down items, while also taking into account operational precision.

[0063] S5: Executes task actions according to action execution instructions, and adjusts the motion state of each joint based on the joint degree of freedom execution coefficient. This joint degree of freedom execution coefficient can be adapted to humanoid robot models with different basic degrees of freedom. In this embodiment, after receiving the action execution command generated in step S4, the humanoid robot sequentially drives the joints of its head, arms, torso, and legs to execute the target action according to the action sequence. During execution, the joints do not operate with uniform control parameters, but rather according to their corresponding degree of freedom execution coefficients k. i Adjust motion accuracy, speed response, trajectory smoothness, and posture flexibility separately. In other words, when a humanoid robot performs the same type of task, different joints can bear different levels of accuracy or flexibility requirements according to their respective roles, thereby achieving finer-grained motion control. The essence of step S5 is to refine the motion execution instructions generated at the task layer to the joint layer, and use the joint degree-of-freedom execution coefficients to achieve differentiated execution control according to the needs of the scenario.

[0064] S6: When the environmental feature information changes according to a preset time, the action execution instruction is regenerated and executed based on the updated environmental feature information, and the task execution result is output.

[0065] When the environmental feature information changes according to a preset timeframe, the action execution command is regenerated and executed based on the updated environmental feature information, and the task execution result is output, including: During the mission execution, environmental feature information is updated in real time based on the world model; When the detected position deviation of the work object, change of obstacles, or abnormal operation of the equipment reaches a preset threshold, the current task action is paused. Based on the updated environmental feature information, joint encoding and contextualized knowledge database retrieval are performed again to obtain updated target retrieval results; Based on the updated target retrieval results and environmental feature information, the action execution instructions are regenerated and executed, and the task execution results are output.

[0066] In this embodiment, step S6 is used to achieve dynamic closed-loop control of environmental perception and action execution during task execution. Specifically, during task execution, the robot updates environmental feature information in real time through a world model. The world model, based on sensors such as LiDAR, depth cameras, and IMUs, continuously perceives three-dimensional spatial information of the environment, obstacle distribution information, the location of the work object, and the operating status of equipment, and refreshes at a preset frequency, thereby forming continuously changing environmental feature data. In this way, the robot can obtain the current environmental state in real time, providing a basis for subsequent dynamic adjustments.

[0067] Based on real-time updates of environmental characteristic information, key environmental changes are detected and judged. When a preset threshold is reached, such as a shift in the position of the work object, the addition or movement of obstacles, or abnormal equipment operation, a valid environmental change is determined, and the current task is immediately paused. The preset threshold can be set according to different application scenarios, such as a shift in the position of the work object exceeding a preset distance range, changes in the size or position of obstacles exceeding preset conditions, or equipment switching from a normal state to an abnormal state. Through this mechanism, timely responses to environmental changes can be made during task execution, avoiding deviations in action execution or potential collision risks, and improving the safety and stability of task execution.

[0068] After pausing the current task action, the robot re-encodes and retrieves the contextualized knowledge database based on the updated environmental feature information to obtain updated target retrieval results. Specifically, the updated environmental information is re-fused with the original multimodal interaction request to generate new joint encoding results. Similarity matching and index filtering are then performed in the contextualized knowledge database to obtain knowledge data matching the current environmental state. Based on this, the robot regenerates action execution instructions using a multimodal large model and a VLA / VLM end-to-end model, combining the updated target retrieval results and environmental feature information, and continues task execution. Finally, the robot completes the task and outputs the task execution results, including the task completion status and related execution information. Through this process, adaptive task adjustment driven by environmental changes is achieved, significantly improving the robot's execution reliability and flexibility in dynamic environments.

[0069] Specifically, this embodiment is described based on industrial and home scenarios.

[0070] 1. Industrial scenario: Assembly of parts on a production line; In this embodiment, the humanoid robot is applied to bearing assembly tasks on an automotive parts production line, such as... Figure 4As shown, the robot's action sequence for completing bearing assembly in a production line scenario is "move to the workbench → left arm picks up the bearing → bearing positioning → wrist joint rotation assembly → torque detection". The specific execution process is as follows: The user inputs a multimodal interaction request to the robot: a voice request to "assemble the left wheel hub bearing" + a visual request (pointing to the wheel hub workpiece on the production line). The world model perception unit perceives the production line environment and obtains environmental feature information: the wheel hub workpiece is located on the left side of the production line workbench, the workbench is free of obstacles, the production line equipment is in standby mode, and the bearing assembly torque requirement is 50 N·m. After joint encoding, the industrial scenario sub-database was searched to obtain the target search results: assembly steps of the left wheel hub bearing, assembly torque parameter of 50 N·m, and safety rules for production line assembly. The multimodal large model and the VLA / VLM end-to-end model are analyzed: VLM is responsible for slot identification, and the analysis yields the work object = left wheel hub bearing, task type = assembly, execution position = production line workbench, and accuracy requirement = torque 50 N·m; VLA is responsible for task motion prediction and degree-of-freedom execution coefficient configuration, predicting the above assembly motion sequence and configuring the dual-arm core joint k. i =0.98 (corresponding to 98% of the basic accuracy), k for both leg movement joints i =0.75 (corresponding to 75% of basic mobility); The robot performs assembly actions according to the action execution instructions. The world model perceives the production line environment in real time at a frequency of 10Hz. There are no environmental changes during the assembly process. Finally, the bearing assembly is completed, the torque detection meets the standard, and the robot feeds back "assembly completed, accuracy qualified" to the production line control system.

[0071] 2. Example of a home scenario: taking and putting away household items; In this embodiment, the humanoid robot is used for tasks such as retrieving and placing water cups in a household setting. Figure 5 As shown, the robot's sequence of actions for retrieving and placing a water cup in a home setting is "moving from the charging station to the coffee table → gently grasping the water cup with its left arm → standing up and moving to the dining table → placing the water cup down → returning to the charging station." The specific execution process is as follows: The user inputs a multimodal interaction request to the robot: a voice request "Take the ceramic water cup on the coffee table in the living room and put it on the dining table" + a visual request (pointing to the water cup on the coffee table). The world model perception unit perceives the home environment and obtains environmental feature information: the coffee table is located in the center of the living room, the water cup is to the right of the coffee table, there are no obstacles between the coffee table and the dining table, the ceramic water cup is made of fragile material, and the gripping force requirement is ≤5N. After joint encoding, the household scene sub-database was searched to obtain the target search results: the gripping force parameters of ceramic water cups, the home layout of the living room and dining room, and the action specifications for picking up and placing fragile items. The multimodal large model and the VLA / VLM end-to-end model are analyzed: VLM is responsible for slot identification, and the analysis yields "Work object = ceramic water cup, task type = pick and place, starting position = living room coffee table, target position = dining table, accuracy requirement = gripping force ≤ 5N"; VLA is responsible for task motion prediction and degree-of-freedom execution coefficient configuration, predicting the above pick and place motion sequence and configuring the core joints k of both arms. i =0.85 (corresponding to motion accuracy of 85% of basic accuracy), k for both leg movement joints i =0.95 (corresponding to 95% of basic mobility), head joint k i =0.95; The robot performs the picking and placing actions according to the action execution instructions. During the movement, the world model detects that a child briefly passes between the coffee table and the dining table. The robot immediately stops moving and emits a soft prompt. The world model continuously detects the environment at a frequency of 10Hz. When it cannot detect the outline of the child for 3 consecutive seconds and there are no new obstacles in the path, it replans the shortest movement path from the current position to the dining table and continues to perform the action. Finally, it successfully places the water cup on the dining table and gives the user voice feedback that "the water cup has been picked up and placed on the dining table".

[0072] Please see Figure 6 This application also provides a humanoid robot, which includes: The multimodal perception module is used to collect external multimodal interaction requests; The world model perception module is used to perceive the physical environment around the robot and generate and update environmental feature information in real time. The joint encoding module is used to jointly encode multimodal interaction requests and environmental feature information to obtain the joint encoding result; The scenario knowledge retrieval module is used to retrieve target search results from the scenario-based knowledge database based on the joint encoding results. The scenario-based knowledge database supports dynamic updates and includes industrial scenario sub-databases and household scenario sub-databases. The instruction generation module is used to parse multimodal interaction requests based on a multimodal large model combined with a VLA / VLM end-to-end model, according to the target retrieval results and environmental feature information, and generate action execution instructions with joint degrees of freedom execution coefficients; and when the environmental feature information changes by a preset, it regenerates the action execution instructions based on the updated environmental feature information. The joint drive module is used to drive each joint of the robot to perform task actions according to the action execution command and the degree of freedom execution coefficient. The joint degree of freedom execution coefficient can be adapted to humanoid robot models with different basic degrees of freedom. The output module is used to regenerate and execute action execution instructions based on the updated environmental feature information when the environmental feature information changes by a preset time, and output the task execution result. The communication module is used to enable the robot to interact with external systems and provide feedback on execution results.

[0073] This application also provides a humanoid robot, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0074] It is understood that the content of the above method embodiments is applicable to the humanoid robot embodiments. The specific functions implemented by the humanoid robot embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0075] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0076] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0077] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0078] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0079] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0080] This application provides a method for multimodal interaction and dynamic task execution of a humanoid robot, as well as the humanoid robot itself. This method receives multimodal interaction requests and combines them with a world model to obtain environmental feature information. It can uniformly model speech, text, images, and scene states, improving the robot's comprehensive understanding of complex task semantics and environmental context. Furthermore, by using a pre-trained multimodal embedding model to jointly encode multimodal interaction requests and environmental feature information, and combining this with a contextualized knowledge database for retrieval, the robot's accuracy in recognizing task objectives, operational objects, and execution constraints can be enhanced, improving the relevance and accuracy of task decisions. Simultaneously, based on a large multimodal model combined with VL (Virtual World Model), the method further enhances the robot's ability to perform multimodal interaction and dynamic task execution. The A / VLM end-to-end model generates action execution instructions and regulates the motion state of each joint through joint degree-of-freedom execution coefficients. This facilitates efficient connection from semantic understanding to action control, improving the coordination, stability, and precision of action execution. Furthermore, when environmental feature information changes according to preset parameters, it can regenerate and execute action execution instructions based on the updated environmental feature information. This enables the humanoid robot to have stronger environmental adaptability and dynamic task adjustment capabilities, solving the problems of insufficient interactive understanding, rigid task execution, and untimely response to environmental changes in existing robots in complex open scenarios. As a result, it improves the success rate, flexibility, and intelligence level of task execution.

[0081] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0082] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0083] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0084] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0085] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for multimodal interaction and dynamic task execution of a humanoid robot, characterized in that, The method includes the following steps: The system receives multimodal interaction requests and obtains corresponding environmental feature information based on a world model. The multimodal interaction requests include one or more combinations of voice requests, visual requests, and tactile requests. The environmental feature information includes one or more of the following: three-dimensional spatial information of the environment, obstacle distribution information, task object feature information, and scene lighting information. The multimodal interaction request and the environmental feature information are jointly encoded to obtain a joint encoding result; the joint encoding is implemented based on a pre-trained multimodal embedding model. Based on the joint encoding results, a search is performed in the contextualized knowledge database to obtain the target search results. The contextualized knowledge database supports dynamic updates and includes an industrial scenario sub-database and a household scenario sub-database. Based on a multimodal large model combined with an end-to-end VLA / VLM model, the multimodal interaction request is parsed according to the target retrieval results and the environmental feature information to generate action execution instructions with joint degrees of freedom execution coefficients; The task action is executed according to the action execution command, and the motion state of each joint is adjusted based on the joint degree of freedom execution coefficient. The joint degree of freedom execution coefficient can be adapted to humanoid robot models with different basic degrees of freedom. When the environmental feature information changes according to a preset time, the action execution instruction is regenerated and executed based on the updated environmental feature information, and the task execution result is output.

2. The method according to claim 1, characterized in that, The step of jointly encoding the multimodal interaction request and the environmental feature information to obtain the joint encoding result includes: The multimodal interaction request is preprocessed to obtain standardized multimodal data; The environmental feature information is processed in a structured manner to obtain structured environmental data; Based on the pre-trained multimodal embedding model, feature mapping is performed on the standardized multimodal data and the structured environment data respectively to obtain the corresponding multimodal request vector and environment feature vector; The multimodal request vector and the environmental feature vector are fused and mapped to a unified semantic vector space to generate a joint encoding vector, which serves as the joint encoding result.

3. The method according to claim 1, characterized in that, The step of retrieving the target search results from the contextualized knowledge database based on the joint encoding results includes: Based on the joint encoding result, similarity matching calculation is performed with each index in the scenario-based knowledge database to obtain the corresponding similarity result; the industrial scenario sub-library and the household scenario sub-library in the scenario-based knowledge database each contain multiple indexes and data blocks corresponding to each index, and the indexes are obtained by embedding the data blocks; Based on the similarity results, the indexes are sorted, and a preset number of indexes with the highest similarity are selected as target indexes. Based on the target index, locate the corresponding data block in the contextualized knowledge database; The data blocks are fused to obtain the target retrieval results.

4. The method according to claim 3, characterized in that, The dynamic updating of the contextualized knowledge database includes: Receive newly added industrial scenario knowledge data or home scenario knowledge data; The newly added knowledge data is split into corresponding data blocks according to the rules of a single process or a single life service scenario; The data block is subjected to embedding encoding to generate a corresponding index; The index and the data block are written into the corresponding scenario sub-library to complete the dynamic update of the scenario-based knowledge database.

5. The method according to claim 1, characterized in that, The method, based on a multimodal large model combined with an end-to-end VLA / VLM model, parses the multimodal interaction request according to the target retrieval results and the environmental feature information, generating action execution instructions with joint degrees of freedom execution coefficients, including: Based on the preset prompt information template, the multimodal interaction request, the target retrieval result, and the environmental feature information are input into the multimodal large model and the VLA / VLM end-to-end model to generate model input; Based on the model input, slot identification is performed on the multimodal interaction request to obtain task semantic information; Based on the task semantic information, the target retrieval results, and the environmental feature information, task action prediction is performed to obtain an action sequence; Configure the execution coefficients of each joint's degrees of freedom based on the aforementioned action sequence; Based on the action sequence and the degree of freedom execution coefficients, action execution instructions are generated.

6. The method according to claim 5, characterized in that, The construction process of the multimodal large model and the VLA / VLM end-to-end model includes: Obtain multimodal interaction request samples, environmental feature samples, and corresponding task action labels and joint degree of freedom execution coefficient labels; Based on the multimodal interaction request samples and the environmental feature samples, the contextualized knowledge database is searched to obtain the corresponding sample search results; Based on the multimodal interaction request samples, the environmental feature samples, the sample retrieval results, and the preset prompt information template, the model training input is constructed; Based on the model training input, a multi-task joint training method is used to train the multimodal large model and the VLA / VLM end-to-end model. Specifically, slot recognition and task action prediction training are performed based on the cross-entropy loss function, and joint degree of freedom execution coefficient prediction training is performed based on the mean square error loss function. The losses are then weighted and fused to obtain the trained multimodal large model and VLA / VLM end-to-end model.

7. The method according to claim 6, characterized in that, The construction process of the multimodal large model and the VLA / VLM end-to-end model also includes: Pre-configured domain knowledge of humanoid robots, process knowledge of industrial scenarios, and life knowledge of household scenarios are injected into the basic multimodal model and the basic VLA / VLM end-to-end model. The knowledge injection methods include constructing domain knowledge into a knowledge graph and fusing it into the weights of the basic model through the embedding layer, or injecting domain knowledge into the pre-training process of the model through scenario-based prompt word templates. The injected base model is pre-trained to obtain a pre-trained multimodal model and a VLA / VLM end-to-end model.

8. The method according to claim 7, characterized in that, The process of adjusting the joint degree of freedom execution coefficient includes: Based on the task type and scenario type, each joint of the robot is configured with an independent degree of freedom execution coefficient k. i ,0 <k i ≤1, where i is the joint number; The motion accuracy P of the core operating joint i =k i ×P0, the mobility of the joint F i =k i ×F0; where P0 is the basic motion accuracy of the joint, and F0 is the basic motion flexibility of the joint; The degree of freedom execution coefficient k i It is decoupled from the number of basic degrees of freedom of the robot, and can be adapted to humanoid robot models with any basic degrees of freedom.

9. The method according to claim 1, characterized in that, When the environmental feature information undergoes a preset change, the action execution instruction is regenerated and executed based on the updated environmental feature information, and the task execution result is output, including: During the task execution, the environmental feature information is updated in real time based on the world model. When the detected position deviation of the work object, change of obstacles, or abnormal operation of the equipment reaches a preset threshold, the current task action is paused. Based on the updated environmental feature information, joint encoding and contextualized knowledge database retrieval are performed again to obtain updated target retrieval results; Based on the updated target retrieval results and environmental feature information, the action execution instructions are regenerated and executed, and the task execution results are output.

10. A humanoid robot, characterized in that, The humanoid robot includes: The multimodal perception module is used to collect external multimodal interaction requests; The world model perception module is used to perceive the physical environment around the robot and generate and update environmental feature information in real time. The joint encoding module is used to jointly encode the multimodal interaction request and the environmental feature information to obtain the joint encoding result; The scenario knowledge retrieval module is used to retrieve the target retrieval result from the scenario-based knowledge database based on the joint encoding result; the scenario-based knowledge database supports dynamic updates and includes an industrial scenario sub-database and a home scenario sub-database. The instruction generation module is used to parse the multimodal interaction request based on the multimodal large model combined with the VLA / VLM end-to-end model, according to the target retrieval results and the environmental feature information, and generate action execution instructions with joint degrees of freedom execution coefficients; and when the environmental feature information undergoes a preset change, it regenerates the action execution instructions based on the updated environmental feature information. The joint drive module is used to drive each joint of the robot to perform task actions according to the action execution command and the degree of freedom execution coefficient. The joint degree of freedom execution coefficient can be adapted to humanoid robot models with different basic degrees of freedom. The output module is used to regenerate and execute action execution instructions based on the updated environmental feature information when the environmental feature information undergoes a preset change, and output the task execution result. The communication module is used to enable the robot to interact with external systems and provide feedback on execution results.

Images