Robot impromptu performance method and system based on large model and knowledge base

Abstract

The invention discloses a robot impromptu performance method and system based on a large model and a knowledge base. The method comprises the steps that an action library containing basic actions and control instructions of the basic actions is constructed, and multidimensional semantic information is labeled through an RAG editor to form an action RAG knowledge base; acquiring environment information through a sensor, and analyzing and generating an environment understanding text by using a VLM large model; a query vector is generated based on the text, and a candidate action set with semantic matching is retrieved in an action RAG knowledge base; fusing the environment text and the candidate action information into a structured cue word, and inputting the structured cue word into an LLM large model to generate a structured action sequence; and analyzing the sequence and calling a bottom-layer control instruction to drive the robot to execute. The system correspondingly comprises an action library module, an environment perception module, a candidate action retrieval module, an impromptu performance dialogue module, a large model interaction module, an action execution engine module and the like. The dynamic and creative performance of the robot according to the real-time environment is realized, and the context fitting degree and the interaction naturalness of the action are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot technology, and more specifically to a method and system for robot improvisation based on a large model and knowledge base. Background Technology

[0002] With the maturation of artificial intelligence and robotics technologies, robots are moving from factories and warehouses to stages, exhibition halls, shopping malls, and even homes. Their role is expanding from simple production tools to companions and performers with emotional interaction and entertainment functions. "Robot performances," as one of the most publicly perceptible application scenarios, not only vividly showcase the charm of technology but also create unique art forms and interactive experiences. It is an important frontier for promoting the popularization of robotics technology and building a human-machine harmonious society.

[0003] However, the current methods of implementing robot performances limit their expressiveness, interactivity, and accessibility. Existing mainstream technologies primarily rely on the following two models: 1. Real-time control based on remote control or teach pendant: The operator controls every movement of the robot in real time, like manipulating a puppet. This method relies heavily on the operator's skill and on-the-spot reaction, and cannot reflect the robot's intelligence. It is essentially a labor-intensive method.

[0004] 2. Based on pre-programmed fixed program playback: Technicians pre-write detailed sequences of actions, and the robot strictly follows the script during the performance. While this method is stable and reliable, the content is completely fixed and lacks flexibility. The robot cannot make any adaptive adjustments to the on-site environment, audience reactions, or impromptu dialogues, resulting in monotonous performances with almost zero interactivity, which also weakens its appeal as an intelligent agent.

[0005] Both of these models expose a fundamental flaw in existing technology: robot performances lack true "intelligence" and "improvisation." They cannot understand the context and emotions of a performance, nor can they dynamically generate appropriate and novel actions based on real-time input (such as audience language, music, and the atmosphere of the event). This makes current robot performances more like a precise but cold mechanical display than a soulful, interactive artistic creation or entertainment experience. To improve interactivity, some research has attempted to pre-set simple action libraries for robots and trigger fixed actions based on keywords. However, this method has a limited action library, stiff combinations, and falls far short of achieving a level of "natural fluency and creative improvisation," and it cannot understand complex semantics. Some cutting-edge work explores the use of generative models to create animations or music, but these results mostly remain at the level of digital virtual characters and have not yet effectively solved the severe challenges of ontological constraints, dynamic safety, and real-time computation in safely and feasiblely mapping generated actions onto physical robot entities.

[0006] Specifically, the existing technology mainly has the following problems: 1. Rigid behavior and lack of environmental awareness and adaptability: The performance actions of existing robots are mostly pre-programmed fixed sequences, which cannot be dynamically adjusted and adapted to the real-time changing surrounding environment (such as different characters, scenes and atmospheres), resulting in stiff and monotonous performances and dull interactive experience.

[0007] 2. Disconnect between actions and semantics, lack of contextual understanding: Traditional robot action libraries are merely collections of low-level motion instructions, lacking high-level, rich semantic information descriptions. The system cannot understand "when, where, to whom, and why" to use a certain action, resulting in a serious disconnect between action selection and the current context.

[0008] 3. The generation process is blind and lacks controllability and feasibility assurance: If the action instructions are generated directly using the large language model of LLM, it is very easy to produce "illusion" actions that exceed the physical capabilities of the robot or are inconsistent with the logic of the current scene. The generation results are uncontrollable and unreliable, and it is difficult to use them directly for actual robot driving.

[0009] Therefore, in the process of robots moving towards a broader consumer market, existing technologies cannot achieve real-time creative generation and safe execution of actions based on natural language understanding and context awareness. This results in robot performances remaining at the stage of automation rather than intelligence, hindering the full realization of their artistic value and commercial potential. How to endow them with the advanced intelligent interactive capability of improvisation has become the key to breaking through the current application ceiling, and it is also a problem that urgently needs to be solved by those in the field. Summary of the Invention

[0010] In view of the above problems, the present invention provides a robot improvisation method and system based on a large model and knowledge base to overcome or at least partially solve the above problems.

[0011] To achieve the above objectives, this invention discloses a robot improvisation method based on a large model and knowledge base, comprising the following steps: S1: Establish an action library containing basic actions that the robot can execute and their corresponding underlying control instructions; label each basic action with multi-dimensional semantic information, and convert the multi-dimensional semantic information into a semantic vector, and store it to form an action knowledge base; S2: The robot acquires environmental information through its onboard sensors and uses a first artificial intelligence model to analyze the environmental information and generate environmental understanding text. S3: Generate a query vector based on the environmental understanding text, perform similarity retrieval in the action knowledge base, and obtain a set of candidate actions that semantically match the current environmental information. The set of candidate actions includes one or more basic actions and their corresponding multidimensional semantic information. S4: The information of the environmental understanding text and the candidate action set are fused to obtain structured prompt words; the structured prompt words are input into the second artificial intelligence model for reasoning to generate a structured action sequence composed of the basic actions; S5: Parse the structured action sequence, and call the corresponding underlying control instructions from the action library based on the basic actions in it, so as to drive the robot body to execute the structured action sequence.

[0012] Preferably, the multidimensional semantic information includes at least one of the following: application scenario, suitable object, suitable atmosphere, venue requirements, and performance duration.

[0013] Preferably, in step S2, the sensor is a visual sensor, the first artificial intelligence model is a VLM large model, and the environmental understanding text is natural language text that describes at least one of the scene, object, and atmosphere in the environmental information.

[0014] Preferably, in step S2, the acquisition of environmental information is triggered at a preset period or in response to an interactive event.

[0015] Preferably, step S3 specifically includes: S31: Encode the environment understanding text into the query vector using the same embedding model as when constructing the action knowledge base; S32: Calculate the similarity between the query vector and each semantic vector in the action knowledge base; S33: Sort by cosine similarity, return the top K basic actions with the highest similarity and their multidimensional semantic information to form the candidate action set.

[0016] Preferably, after step S5 is completed, after a preset time interval or in response to a new external interaction event, the process automatically returns to step S2 and starts a new round of processing including steps S2 to S5.

[0017] On the other hand, based on the same inventive concept, a robot improvisation system based on a large model and knowledge base is also disclosed, which is used to implement any of the above methods, the system comprising: The motion library and motion knowledge base are used to store the basic motions that the robot can perform and their corresponding low-level control instructions, multi-dimensional semantic information and corresponding semantic vectors. An environmental perception and understanding module, deployed on the robot body, is used to acquire environmental information through sensors; and to call a first artificial intelligence model to parse the environmental information and generate environmental understanding text. The candidate action retrieval module is used to generate a query vector based on the environment understanding text, perform similarity retrieval in the action knowledge base, and obtain a set of candidate actions that semantically match the current environment information. The large model interaction module is used to receive the environmental understanding text and the candidate action set, construct structured prompt words and call the second artificial intelligence model to generate a structured action sequence; A robot motion execution engine, deployed on the robot body or near-end control computer, is used to parse and execute the structured motion sequence, and to call the corresponding low-level control instructions from the motion library based on the basic motions therein, driving the robot body to execute the structured motion sequence.

[0018] Preferably, the action knowledge base is an action RAG knowledge base, which includes an action RAG editor for annotating the multidimensional semantic information of the basic action and managing the generation and updating of the semantic vector.

[0019] Preferably, the candidate action retrieval module includes a query vectorization unit and a vector similarity retrieval engine.

[0020] Preferably, the system further includes an improvisational dialogue module for receiving natural language instructions from the user and providing the natural language instructions to the large model interaction module to participate in the construction of the structured prompt words.

[0021] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following: 1. Achieved highly contextualized adaptive performance: The robot can dynamically generate performance content that fits the current scene, object and atmosphere, just like an actor, based on real-time "observation" (visual perception) and "understanding" (large model analysis) of the environment.

[0022] 2. It endows the robot with "impromptu creativity" capabilities: the system does not simply play preset clips, but rather creates new and unique action sequences in real time based on the unique context and a large model. This makes each interaction unique and surprising, greatly enhancing the freshness and appeal of the user experience.

[0023] 3. Achieved semantic-level human-computer interaction understanding: Based on the action knowledge base, the robot not only "can perform" actions, but also "understands" the social meaning and applicable context of each action (such as "waving" representing "greeting" or "saying goodbye"). This elevates the robot's behavior from "mechanical movement" to "meaningful social signals," achieving a deeper level of intent alignment and emotional resonance.

[0024] 4. It has opened up a new paradigm for robot applications: enabling robots to move beyond being merely tools for completing functional tasks, and instead become social entities that provide emotional value, entertainment, and creative content. This opens up highly imaginative high-end application scenarios for service robots, educational robots, entertainment robots, and metaverse digital humans, such as intelligent companionship, interactive navigation, personalized theaters, and dynamic content generation.

[0025] 5. Lowering the threshold for designing complex robot behaviors: This invention transforms professional script writing and animation production into semantic annotation of actions and environmental descriptions in natural language, reducing the professional difficulty of behavior design and allowing non-domain experts (such as directors and teachers) to indirectly "program" robot behaviors by describing scenarios and intentions. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0027] Figure 1 This is an overall flowchart of a robot improvisation method based on a large model and knowledge base provided in an embodiment of the present invention; Figure 2 This is a framework diagram of a robot improvisation system based on a large model and knowledge base provided in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Example 1 like Figure 1 As shown, this invention discloses a method for robot improvisation based on a large model and knowledge base, including the following steps: Step S1: Build an action library / action knowledge base S1.1 Action Primitive Encapsulation: Decompose and encapsulate all possible physical actions (including dance, martial arts, simple movement actions, etc.) exposed by the robot based on the SDK to form a series of standard basic actions, assign a basic action identifier to each basic action, and store its control instruction set in the action library.

[0030] It should be noted that the basic action identifier includes, but is not limited to, using a unique ID corresponding to the basic action.

[0031] S1.2 Action Semantic Annotation: Using the action RAG editor, create a multi-dimensional semantic description (application scenario, object, atmosphere, etc.) for each basic action ID to form structured text.

[0032] S1.3 Vectorized storage: Use a text embedding model (such as Sentence-BERT) to convert the semantic description text of each basic action into a high-dimensional semantic vector to build an action RAG knowledge base (vector database).

[0033] Step S2: Real-time environmental perception and understanding S2.1 Image Acquisition: The robot acquires environmental images through its onboard vision sensors at a preset period (e.g., once per second) or when triggered by interactive events.

[0034] S2.2 Visual Semantic Analysis: Input the image into a large VLM model with visual capabilities, and generate a natural language description of the environment. For example: "In the image, there is a child of about 5 years old sitting on the carpet playing with building blocks. The child's expression is focused. The room is brightly lit and the atmosphere is quiet." Step S3: Context-related action retrieval S3.1 Query Vector Generation: The contextual understanding text obtained in S2.2 is converted into query vectors using the same text embedding model as in S1.3.

[0035] S3.2 Similarity retrieval: In the action RAG knowledge base, calculate the cosine similarity between the query vector and all action semantic vectors in the base.

[0036] S3.3 Search Results Return: Return the Top-K (e.g., 3-5) most relevant basic action IDs and their complete semantic meta-information text, sorted by similarity from high to low.

[0037] Step S4: Action Sequence Generation S4.1 Cue Word Template Population: Uses a predefined structured cue word template. Contextual understanding text and a list of retrieved action metadata are used as variables to populate the specified locations in the template.

[0038] S4.2 Large Model Inference: Send the completed prompts to the LLM large language model, requiring it to create a coherent, reasonable, and somewhat creative sequence of actions based on the provided context. The instructions explicitly require that the output must be in a structured list format, and that list items must use action IDs existing in the action library or natural language descriptions that can be mapped to action IDs.

[0039] S4.3 Sequence Parsing and Standardization: Receives the output of the large model and parses out the structured action sequence. If the output is natural language, it is mapped back to standard action IDs and time interval instructions through keyword matching or a small classification model.

[0040] Step S5: Robot action execution S5.1 Instruction Serialization: The robot motion execution engine parses the structured motion sequence obtained in S4 into specific motion IDs in sequence.

[0041] S5.2 Low-level instruction call: Based on each action ID, query and load the corresponding low-level robot control instruction set (such as action messages in ROS) from the action library.

[0042] S5.3 Drive Execution and Synchronization: The engine sends the instruction set sequentially to the actuators of each joint of the robot according to the time rhythm specified in the sequence, controlling the robot to complete the entire performance sequence accurately and coherently. After execution is completed, the process can return to step S2 to start the next perception-performance cycle.

[0043] Example 2 like Figure 2 As shown, this invention also discloses a robot improvisation system based on a large model and knowledge base. The system mainly adopts the following modular design: 1. Motion library and motion RAG knowledge base: This module contains two closely related databases and a set of editing and management tools.

[0044] Motion Library: A database storing all the basic motions that the robot can perform. Each entry contains a unique motion ID, the part that performs the motion (left arm, right arm, whole body, etc.), and a motion description.

[0045] Action RAG Knowledge Base: This is a vector database. Each entry corresponds one-to-one with an action ID in the action library, but stores a multi-dimensional semantic vector of the action and its structured metadata. The semantic vector is generated by encoding pre-annotated text descriptions (e.g., "Action: Waving. Applicable scenarios: Welcoming guests, saying goodbye. Suitable for: Everyone. Atmosphere: Friendly, cheerful. Duration: Short (2-4 seconds)...") through an embedding model (e.g., text-embedding-ada-002).

[0046] 2. Environmental Perception and Understanding Module: Composed of visual sensors (such as RGB-D cameras) deployed on the robot body, this module periodically (e.g., at a frequency of 1Hz) acquires environmental image data. Furthermore, the environment understanding unit communicates with the environment perception module. The environment understanding unit calls upon a large model (VLM) with visual understanding capabilities to generate structured environmental description text (e.g., "A child smiles at the cake at a birthday party; the atmosphere is cheerful"). This text is simultaneously sent to the cue word construction unit and the candidate action retrieval module. Thus, the system obtains descriptive text about the environment from the large model, providing raw environmental perception signals and serving as the starting point for triggering the improvisation loop.

[0047] 3. Candidate Action Retrieval Module: This is a standalone retrieval service module that integrates a query vectorization unit and a vector similarity retrieval engine. It receives environment description text from the periodic environment awareness module. The query vectorization unit uses the same text embedding model as when building the action RAG knowledge base to convert this environment description text into a query vector V_q. Within the action RAG knowledge base, it calculates the cosine similarity between V_q and the semantic vectors of all actions in the base, performs a k-nearest neighbor (k-NN) search, and finds the top K most similar (e.g., Top-5) action entries. Finally, it outputs a list of IDs for these K actions and their complete semantic metadata, as a candidate action set, to the prompt word construction unit of the large model interaction module.

[0048] 4. Impromptu Dialogue Module: This is a preferred implementation, aiming to provide a human-computer dialogue interface that, based on user input, such as "Please perform a martial arts routine to entertain us," integrates environmental description text from the environmental understanding unit and a candidate action list (containing action IDs and their semantic metadata) from the action RAG retrieval module. These three pieces of information are then fused together to generate context-rich, structured prompts.

[0049] 5. Large Model Interaction Module: Receive the output from the improvisational dialogue module, call the plain text large model LLM, generate a structured action performance sequence consisting of basic action IDs and time parameters (such as JSON format: [{"action_id": "clap", "duration": 2.0}, {"action_id": "dance_simple", "duration": 5.0}]), and output it to the robot action execution engine.

[0050] 6. Robot motion execution engine: Real-time control software deployed on the robot body or near-end control computer. It receives structured motion performance sequences from the large model interaction module, parses the action_id in the sequence one by one, queries the corresponding low-level control instructions from the motion library, and sends them to the robot body's driver according to the timing and parameter requirements to complete the physical execution of the motion.

[0051] Overview of Working Principle: After system startup, periodic environmental perception triggers a loop. After the large model understands the environmental image, it generates environmental description text and drives candidate action retrieval. The retrieved list of candidate actions that best matches the semantics of the current scene is woven into the prompts given to the large model along with the environmental description. It's worth noting that this can also be combined with the human-computer interaction function of the improvisation dialogue module. Real-time natural language commands input by the user (such as "Please dance a lively dance" or "Imitate animal movements") can be integrated with the environmental description text and the candidate action list to construct richer and more directional structured prompts. This guides the large model to generate a smooth sequence of actions that fits the scene, responds to user intent, and is creative. Finally, the robot executes this sequence, completing an improvisation performance.

[0052] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0053] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for robot improvisation based on a large model and knowledge base, characterized in that, Includes the following steps: S1: Establish an action library containing basic actions that the robot can execute and their corresponding underlying control instructions; label each basic action with multi-dimensional semantic information, and convert the multi-dimensional semantic information into a semantic vector, and store it to form an action knowledge base; S2: The robot acquires environmental information through its onboard sensors and uses a first artificial intelligence model to analyze the environmental information and generate environmental understanding text. S3: Generate a query vector based on the environmental understanding text, perform similarity retrieval in the action knowledge base, and obtain a set of candidate actions that semantically match the current environmental information. The set of candidate actions includes one or more basic actions and their corresponding multidimensional semantic information. S4: The information from the environmental understanding text and the candidate action set is fused to obtain structured prompt words; The structured prompts are input into a second artificial intelligence model for reasoning to generate a structured action sequence composed of the basic actions; S5: Parse the structured action sequence, and call the corresponding underlying control instructions from the action library based on the basic actions in it, so as to drive the robot body to execute the structured action sequence.

2. The robot improvisation method based on a large model and knowledge base according to claim 1, characterized in that, The multidimensional semantic information includes at least one of the following: application scenario, suitable object, suitable atmosphere, venue requirements, and performance duration.

3. The robot improvisation method based on a large model and knowledge base according to claim 1, characterized in that, In step S2, the sensor is a visual sensor, the first artificial intelligence model is a VLM large model, and the environmental understanding text is natural language text that describes at least one of the scene, object, and atmosphere in the environmental information.

4. The robot improvisation method based on a large model and knowledge base according to claim 1, characterized in that, In step S2, the acquisition of environmental information is triggered at a preset period or in response to an interactive event.

5. The robot improvisation method based on a large model and knowledge base according to claim 1, characterized in that, Step S3 specifically includes: S31: Encode the environment understanding text into the query vector using the same embedding model as when constructing the action knowledge base; S32: Calculate the similarity between the query vector and each semantic vector in the action knowledge base; S33: Sort by cosine similarity, return the top K basic actions with the highest similarity and their multidimensional semantic information to form the candidate action set.

6. The robot improvisation method based on a large model and knowledge base according to claim 1, characterized in that, After step S5 is completed, after a preset time interval or in response to a new external interaction event, the system automatically returns to step S2 and starts a new round of processing including steps S2 to S5.

7. A robot improvisation system based on a large model and knowledge base, characterized in that, The system for implementing the method as described in any one of claims 1 to 6 comprises: The motion library and motion knowledge base are used to store the basic motions that the robot can perform and their corresponding low-level control instructions, multi-dimensional semantic information and corresponding semantic vectors. An environmental perception and understanding module, deployed on the robot body, is used to acquire environmental information through sensors; and to call a first artificial intelligence model to parse the environmental information and generate environmental understanding text. The candidate action retrieval module is used to generate a query vector based on the environment understanding text, perform similarity retrieval in the action knowledge base, and obtain a set of candidate actions that semantically match the current environment information. The large model interaction module is used to receive the environmental understanding text and the candidate action set, construct structured prompt words and call the second artificial intelligence model to generate a structured action sequence; A robot motion execution engine, deployed on the robot body or near-end control computer, is used to parse and execute the structured motion sequence, and to call the corresponding low-level control instructions from the motion library based on the basic motions therein, driving the robot body to execute the structured motion sequence.

8. A robot improvisation system based on a large model and knowledge base according to claim 7, characterized in that, The action knowledge base is an action RAG knowledge base, which includes an action RAG editor for annotating the multidimensional semantic information of the basic action and managing the generation and updating of the semantic vector.

9. A robot improvisation system based on a large model and knowledge base according to claim 7, characterized in that, The candidate action retrieval module includes a query vectorization unit and a vector similarity retrieval engine.

10. A robot improvisation system based on a large model and knowledge base according to claim 7, characterized in that, The system also includes an improvisational dialogue module, which receives natural language commands from users and provides the natural language commands to the large model interaction module to participate in the construction of the structured prompt words.

Images