Model training method, robot control method, and electronic device

Abstract

This disclosure relates to a model training method, a robot control method, and an electronic device. The model training method includes: acquiring a first model; wherein the first model is trained based on supervised data from multiple tasks of different types; performing reinforcement learning training on the first model using training data from the first task among the multiple tasks to obtain a second model corresponding to the first task; controlling the robot to execute interactive tasks using the first model, and training the first model using the execution results of the interactive tasks to obtain a third model; fusing multiple second models corresponding to multiple tasks with the third model to obtain a target model; wherein the target model is deployed in the robot. This disclosure can improve the comprehensive decision-making ability of the target model in robot interactive tasks.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model training method, a robot control method, and an electronic device. Background Technology

[0002] With the development of artificial intelligence technology, robot intelligence is gradually evolving from traditional rule-based control methods to intelligent decision-making methods based on large models. However, most of the large models currently used for robots are based on general language tasks and lack comprehensive interactive cognitive capabilities, which restricts the robot's comprehensive decision-making ability in interactive tasks. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides a model training method, a robot control method, and an electronic device.

[0004] According to one aspect of this disclosure, a model training method is provided, the method comprising: Obtain a first model; wherein the first model is trained based on supervised data from multiple tasks, and the multiple tasks belong to different types of tasks; Using the training data of the first task among the multiple tasks, the first model is trained by reinforcement learning to obtain the second model corresponding to the first task. The robot is controlled to perform interactive tasks by the first model, and the first model is trained using the execution results of the interactive tasks to obtain the third model; The multiple second models corresponding to the multiple tasks are fused with the third model to obtain the target model; wherein the target model is used to deploy in the robot.

[0005] According to another aspect of this disclosure, a robot control method is also provided, the method comprising: Obtain the target task for the robot; The robot is controlled to perform the target task by a target model; wherein the target model is trained according to the model training method described above.

[0006] According to one aspect of this disclosure, a model training apparatus is provided, the apparatus comprising: The model acquisition module is used to acquire a first model; wherein the first model is trained based on supervised data from multiple tasks, and the multiple tasks belong to different types of tasks; The first training module is used to perform reinforcement learning training on the first model using the training data of the first task among the plurality of tasks, so as to obtain the second model corresponding to the first task. The second training module is used to control the robot to perform interactive tasks through the first model, and to train the first model using the execution results of the interactive tasks to obtain the third model; The fusion module is used to fuse multiple second models corresponding to the multiple tasks with the third model to obtain a target model; wherein the target model is used to deploy in the robot.

[0007] According to another aspect of this disclosure, a robot control device is also provided, the device comprising: The task acquisition module is used to acquire the target task for the robot; The task execution module is used to control the robot to perform the target task through the target model; wherein the target model is trained according to the model training method described above.

[0008] According to another aspect of this disclosure, an electronic device is also provided, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the above-described model training method or robot control method.

[0009] According to another aspect of this disclosure, a computer-readable storage medium is also provided, the storage medium storing a computer program for executing the above-described model training method or robot control method.

[0010] The technical solution provided in this disclosure has the following advantages compared with the prior art: The model training method provided in this disclosure first obtains a first model, which is trained based on supervised data from multiple tasks. This first model possesses basic capabilities for various general tasks, providing a unified starting point for subsequent training of capabilities in different directions. Based on this, on one hand, reinforcement learning training is performed on the first model using training data from the first task among multiple tasks to obtain a second model corresponding to the first task. This allows the second model to achieve targeted improvements in its processing capabilities for the corresponding first task. On the other hand, the first model controls the robot to execute interactive tasks, and the execution results of the interactive tasks are used to train the first model, resulting in a third model specifically designed for robot interaction scenarios. This allows the third model to possess specialized capabilities adaptable to dynamic interactive environments. Then, the multiple second models and the third model corresponding to multiple tasks are fused to obtain a target model. This target model is deployed in the robot. By integrating specialized reasoning capabilities from multiple directions with robot interaction capabilities into a single target model, the comprehensive decision-making capability of the target model in robot interaction tasks is improved, and the computational overhead and response latency caused by the separate deployment and invocation of different task processing capabilities are reduced.

[0011] The robot control method provided in this disclosure includes: acquiring a target task for the robot, and controlling the robot to execute the target task through a target model; wherein, after acquiring basic capabilities through multi-task supervised training, the target model acquires specialized reasoning capabilities and interactive cognitive capabilities through multi-directional reinforcement learning training and robot interaction training, and integrates multiple capabilities into the same model. Therefore, when controlling the robot to execute the target task, the target model can simultaneously mobilize general reasoning capabilities and robot interactive cognitive capabilities to make comprehensive decisions, thereby improving the success rate and efficiency of the robot in executing the target task. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0013] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of the model training method described in the embodiments of this disclosure; Figure 2 This is a schematic diagram of the model training process described in the embodiments of this disclosure; Figure 3This is a schematic diagram of the model fusion method described in the embodiments of this disclosure; Figure 4 This is a schematic diagram of the structure of the model training device described in the embodiments of this disclosure; Figure 5 This is a schematic diagram of the structure of the electronic device described in an embodiment of this disclosure. Detailed Implementation

[0015] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0016] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0017] In related technologies, the intelligent control methods for robots mainly fall into the following categories.

[0018] The first category is end-to-end learning methods based on Vision-Language-Action (VLA) models. These methods train the model using a large amount of robot operation data, directly learning the mapping relationship from visual input to robot action output, making them suitable for executing specific tasks. However, these methods typically rely on large-scale robot teaching data, and when the task scenario changes, the model's generalization ability and cross-scenario adaptability are somewhat limited.

[0019] The second category is agent-based methods based on world models. These methods improve a robot's adaptability to complex environments by having the model learn dynamic environmental models for prediction, planning, and decision-making. However, these methods require a complex environmental modeling process, have high training costs, and the quality of decisions is highly dependent on the accuracy of the world model's predictions. When there are discrepancies between the dynamic environmental model and the actual environment, the decision-making performance deteriorates significantly.

[0020] The third category comprises robot systems based on Large Language Models (LLMs) or Vision Language Models (VLMs). This type of approach typically uses the large model as the robot's high-level cognitive center, responsible for task understanding of user commands, planning and reasoning, tool invocation, and multi-module collaboration between the perception and motion control modules. It then integrates with the underlying perception and motion control modules to complete the robot's tasks. Due to the strong knowledge reasoning capabilities and good generalization ability of large language models and vision language models, this type of approach has gradually become an important development direction in the field of embodied intelligence in robotics.

[0021] However, the aforementioned large-model-based robotics technologies still have many shortcomings.

[0022] First, current models are primarily trained on general tasks such as natural language understanding, question answering, mathematical reasoning, and code generation, with optimization geared towards general language intelligence. In contrast, embodied intelligence in robots requires models to continuously interact with the physical environment, performing cognitive processes such as state maintenance, long-term memory, spatial cognition, proactive information acquisition, and long-term task planning under partially observable conditions. Existing post-training processes lack specific optimization mechanisms for these robotic cognitive abilities, resulting in models that, while possessing strong general reasoning capabilities, struggle to directly meet the specific needs of high-level cognitive decision-making in robots.

[0023] Secondly, current inference models generally employ reinforcement learning training methods primarily aimed at improving task accuracy. Under this training approach, models often achieve higher accuracy by generating longer inference processes. However, this method of generating longer inference processes simultaneously introduces problems such as increased inference latency, increased computational resource consumption, and slower response speed. These issues are clearly contradictory to scenarios involving real-time robot interaction and edge deployment, significantly limiting the usability of the models.

[0024] Furthermore, large robot models typically train different capabilities such as code generation, tool invocation, proxy tasks, and robot interaction separately during the training process. The lack of a unified training framework among these capabilities can easily lead to mutual interference between them, resulting in high model maintenance costs. It also makes it difficult to form a unified cognitive model that combines general intelligence and robot interactive intelligence.

[0025] In summary, how to construct a model training method for embodied intelligence in robots, so that the model can maintain strong general intelligence while possessing robot interactive cognitive abilities, and taking into account both reasoning efficiency and deployment efficiency, has become a technical problem that urgently needs to be solved in the field of large robot models.

[0026] To improve at least one of the above problems, this disclosure provides a model training method, a robot control method, and an electronic device; wherein the model training method takes a first model with multi-task processing capability as a unified basis, performs reinforcement learning training and training using the execution results of interactive tasks, and then merges the trained multiple second models and a third model, thereby enabling the integration of specialized reasoning capabilities and robot interaction capabilities in multiple directions in a single target model, thereby improving the comprehensive decision-making capability of the target model in robot interaction tasks.

[0027] For ease of understanding, the solution provided in this disclosure will be described in detail below with reference to the accompanying drawings.

[0028] Figure 1 This flowchart illustrates a model training method provided in an embodiment of the present disclosure. This method is applicable to model training processes in robot-controlled scenarios; the robot is generally a full-body, multi-degree-of-freedom robot with a physical body. The model training method can be executed by a model training device, which can be implemented using software and / or hardware, specifically, for example, an electronic device or a server.

[0029] like Figure 1 As shown, in this embodiment, the model training method may include the following steps S102 to S108.

[0030] S102, Obtain the first model; wherein, the first model is trained based on supervised data from multiple tasks, and the multiple tasks belong to different types of tasks.

[0031] This embodiment may include: obtaining a large language model. Specifically, a basic large language model can be used as the initial model to be trained; the large language model mentioned above is, for example, the Qwen series model, the Llama series model, the Gemma series model, or other pre-trained language models with general language understanding capabilities. It is not limited here, but only its general language understanding and generation capabilities are used as the initialization parameters for subsequent training.

[0032] Based on supervised data from multiple tasks, the large language model is fine-tuned under supervision to obtain the first model.

[0033] For example, supervised data for multiple different types of tasks may include supervised data for mathematical reasoning tasks, supervised data for code generation tasks, supervised data for instruction following tasks, supervised data for tool invocation tasks, supervised data for knowledge question answering tasks, and supervised data for multilingual tasks, etc.

[0034] The system covers multiple tasks, including language understanding, logical reasoning, structured output, external tool invocation, knowledge retrieval, and cross-language processing. The first model, trained using supervised data from multiple tasks, can establish a stable and transferable foundation for general capabilities.

[0035] The process of supervised fine-tuning of a large language model based on supervised data from multiple tasks can include: mixing the supervised data from each task into a fine-tuning dataset according to a preset data ratio, and using the fine-tuning dataset to perform supervised instruction fine-tuning of the large language model. During training, the large language model learns the mapping relationship from input to standard output, and updates the model parameters by minimizing the loss function between the predicted output and the standard output.

[0036] After the above-mentioned supervised fine-tuning training, the first model is able to perform a variety of basic tasks, such as language understanding, logical reasoning, structured output, external tool invocation, knowledge retrieval and cross-language processing, and establish basic capabilities for multiple tasks. This provides a unified capability foundation and a unified parameter initialization anchor point for subsequent expert capability training in different directions.

[0037] S104. Using the training data of the first task among multiple tasks, the first model is trained by reinforcement learning to obtain the second model corresponding to the first task.

[0038] Specifically, for each of the various task types, such as language understanding, logical reasoning, and external tool invocation, the first model is trained using the training data corresponding to that task, resulting in a second model for that task. Thus, an independent second model is obtained for each task.

[0039] Taking the first task as an example, the first task can be any of the tasks mentioned above. Using the training data from the first task, the first model is trained through reinforcement learning to obtain the second model corresponding to the first task, which may include: The training data for the first task is obtained. This training data includes multiple training samples, each containing the task input and a corresponding reference answer or evaluation criterion. The form of the reference answer or evaluation criterion varies depending on the type of task. For example, for a mathematical reasoning task, the training data includes a mathematical problem and its standard answer; for a code generation task, the training data includes a description of the programming problem and its corresponding set of test cases; for an instruction following task, the training data includes the instruction text and rules for judging whether the answer meets the instruction requirements.

[0040] Then, the first model is trained using reinforcement learning on the training data to obtain the second model corresponding to the first task.

[0041] Specifically, during reinforcement learning training, the first model autonomously generates answers based on the task information of the first task. The reinforcement learning algorithm then generates a reward signal based on the answers generated by the first model and optimizes the model parameters with the goal of maximizing the cumulative reward. The reward signal is used to evaluate the quality of the answers generated by the first model for the current input. Through this method, the first model continuously optimizes its strategy through ongoing interaction with the training data of the first task, gradually improving its performance in processing the first task.

[0042] After completing the reinforcement learning training for each of the multiple tasks, a second model corresponding to each task is obtained. For example, for the language understanding task, there is a corresponding second model, which serves as a language understanding expert model; for the logical reasoning task, there is a corresponding second model, which serves as a logical reasoning expert model; and for the code generation task, there is a corresponding second model, which serves as a code generation expert model.

[0043] The multiple second models corresponding to the aforementioned tasks all start from the same first model, but each has been specifically enhanced in different task directions, thus possessing strong specific performance in different tasks. At the same time, they share consistent basic language expression capabilities and parameter space structure, laying a unified foundation for subsequent multi-model fusion.

[0044] S106, the robot is controlled to perform interactive tasks through the first model, and the first model is trained using the results of the interactive tasks to obtain the third model.

[0045] In this embodiment, a human-computer interaction environment can be constructed first, and in the human-computer interaction environment, the robot can be controlled to perform interactive tasks through the first model, and the execution results of the interactive tasks can be used to train the first model to obtain the third model.

[0046] The human-computer interaction environment is used to simulate the application scenarios in which the robot performs tasks, providing the first model with an interactive experience similar to the real world. The human-computer interaction environment can be any combination of one or more of the following: a text-based interaction environment, a simulation environment, a digital twin environment, or a real robot environment. These human-computer interaction environments simulate museum scenes, supermarket scenes, home scenes, or other robot application scenarios. In these scenarios, the robot continuously interacts with the environment to complete specified interactive tasks.

[0047] During training, a closed-loop interaction is formed between the first model and the interactive environment. Specifically, in each round of interaction, the first model acquires the state information of the environment, such as scene description, object location, task instructions, and historical interaction records; the first model makes decisions based on the current state information and outputs the next action instruction; after the robot executes the action instruction, the state information of the environment changes, and the new state information is fed back to the first model; the first model continues to make decisions based on the new state information, and this cycle repeats until the interaction task is completed or the termination condition is met.

[0048] In the aforementioned closed-loop interaction process, the first model needs to solve the task under partially observable conditions. Partially observable conditions mean that the first model cannot obtain all the state information of the environment and can only make decisions based on currently observable local information and historical interaction records. This setting is to simulate the actual situation of incomplete information in real-world robot application scenarios.

[0049] During the interaction, the first model gradually learns the interactive cognitive abilities required to perform the interactive task. These interactive cognitive abilities include: environmental state tracking ability, i.e., continuously tracking the state of objects and scene changes in the environment; long-term memory maintenance ability, i.e., maintaining the memory of key information throughout the task execution process; spatial cognitive ability, i.e. understanding the spatial layout of the environment and the spatial relationships between objects; proactive exploration ability, i.e. taking proactive actions to obtain more environmental information when information is insufficient; multi-step task planning ability, i.e. planning a multi-step action sequence based on the current state and the target state; and dynamic decision-making ability, i.e. adjusting decision-making strategies in real time based on environmental feedback.

[0050] During training, the results of the interactive tasks are used as training signals to update the model parameters of the first model.

[0051] After completing the above training, a third model is obtained. This third model has specialized cognitive capabilities for robot interaction scenarios and can serve as a robot interaction expert model for subsequent model fusion processes, providing the target model with cognitive decision-making capabilities in dynamic interaction scenarios.

[0052] S108, multiple second models and third models corresponding to multiple tasks are fused to obtain the target model; the target model is used to deploy in the robot.

[0053] In this embodiment, multiple second and third models are trained using the same first model as their initialization starting point, thus sharing consistent basic language expression capabilities and parameter space structures. This unified initialization source ensures that the parameter updates of each model are within the same coordinate system, laying the foundation for subsequent fusion operations.

[0054] In some implementations, model parameter fusion, task vector fusion, LoRA (Low-Rank Adaptation) weight fusion, or other model fusion methods can be used to fuse multiple second models and third models corresponding to multiple tasks to obtain the target model.

[0055] Taking model parameter fusion as an example, this embodiment may include: weighting and combining the model parameters of multiple second models with the model parameters of a third model according to preset weight coefficients to obtain the fused target model. Since multiple second and third models share the same initialization point and their model parameters are in the same parameter space, mathematical operations can be performed directly using the above-mentioned model parameter fusion method.

[0056] In this embodiment, the model parameters of the multiple second models and the third model mentioned above are fused in the parameter space to obtain the target model. Through fusion, the different task processing capabilities learned by the multiple second models and the interactive cognitive capabilities learned by the third model are integrated into a single model, so that the final target model can simultaneously possess multi-domain specialized reasoning capabilities and robot interactive cognitive capabilities.

[0057] The target model is deployed in robots as a high-level cognitive decision-making module within the robot system. This target model possesses general language understanding and reasoning capabilities, multi-domain specialized reasoning capabilities (such as mathematical reasoning, code generation, scientific reasoning, and instruction following), and robot interactive cognitive capabilities (such as environmental state tracking, long-term memory maintenance, spatial cognition, active exploration, multi-step task planning, and dynamic decision-making), enabling robots to perform complex tasks with unified intelligent decision-making capabilities.

[0058] The model training method provided in this embodiment first obtains a first model, which is trained based on supervised data from multiple tasks. This first model possesses basic capabilities for various general tasks, providing a unified starting point for subsequent training of capabilities in different directions. Based on this, on one hand, reinforcement learning is performed on the first model using training data from the first task among multiple tasks to obtain a second model corresponding to the first task. This allows the second model to achieve targeted improvements in its processing capabilities for the corresponding first task. On the other hand, the first model controls the robot to execute interactive tasks, and the execution results of the interactive tasks are used to train the first model, resulting in a third model specifically designed for robot interaction scenarios. This allows the third model to possess specialized capabilities adaptable to dynamic interactive environments. Then, the multiple second models and the third model corresponding to multiple tasks are fused to obtain a target model. This target model is deployed in the robot. By integrating specialized reasoning capabilities from multiple directions with robot interaction capabilities into a single target model, the comprehensive decision-making capability of the target model in robot interaction tasks is improved, and the computational overhead and response latency caused by the separate deployment and invocation of different task processing capabilities are reduced.

[0059] Combination Figure 2 The following provides a more detailed description of the model training method provided in this disclosure.

[0060] like Figure 2 The model initialization section shown uses a basic large language model as the initial model to be trained, such as the Qwen series, Llama series, and Gemma series models. During the general ability training phase, the large language model is fine-tuned under supervised conditions based on supervised data from multiple tasks, resulting in the first model.

[0061] In the expert capability enhancement training section, this embodiment uses the training data of the first task from multiple tasks to perform reinforcement learning training on the first model to obtain the second model corresponding to the first task, which may include: The first model outputs a reward signal based on the training data of the first task, under the constraint of the number of lexical units. The first task can be any one of multiple tasks. The reward signal represents the success rate of the first model in outputting correct answers within the given number of lexical units. The first model is then optimized using the reward signal to obtain the second model corresponding to the first task.

[0062] The first task can be any of the following: mathematical reasoning, code generation, scientific reasoning, or instruction following. Training data for the first task typically includes the task input and the corresponding reference answer or test conditions.

[0063] For example, for mathematical reasoning tasks, training data includes mathematical problems and their corresponding standard answers; for code generation tasks, training data includes code writing requirements and their corresponding test case sets; for instruction following tasks, training data includes instruction descriptions and their corresponding format or content constraints.

[0064] During the reinforcement learning training of the first model using the training data from the first task, a budget is set for the number of tokens for the first model. The token budget represents the maximum number of tokens that the first model is allowed to consume when generating a response for a training sample, for example, N.

[0065] For each training sample in the training data for the first task, the task input is provided to the first model, which then generates the corresponding response data. During the generation process, if the first model fails to output response data within the preset word count constraint, the reward signal for that sample is directly set to zero or a negative value, regardless of whether the response data generated by the first model is correct.

[0066] If the first model outputs response data under the constraint of a preset number of tokens, the success level of the response data is determined based on the standard answer, and then this success level is converted into a positive reward signal. The method for determining the success level of the response data based on the standard answer differs for different types of tasks.

[0067] For example, for a mathematical reasoning task, the final answer output by the first model is compared with the standard answer in the training data. If they match, the success rate is determined to be 100%; if they do not match, the success rate is determined to be 0%.

[0068] For the code generation task, the code generated by the first model is run on the set of test cases provided by the training data, and the proportion of test cases that pass is used as the correctness score, which is used to represent the correctness of the code generated by the first model.

[0069] For tasks that follow instructions, the system automatically determines whether the first model's response data meets the instruction requirements in terms of format, structure, or content constraints through code rules. If it does, the success rate is determined to be 100%; otherwise, the success rate is determined to be 0%.

[0070] Next, the model parameters of the first model are updated and optimized using a reinforcement learning algorithm, leveraging the reward signal. In a specific example, Proximal Policy Optimization (PPO) or other suitable reinforcement learning algorithms can be employed to iteratively update the model parameters of the first model with the goal of maximizing the reward signal.

[0071] During training, high-reward generation strategies are reinforced, meaning the first model tends to use reasoning methods that correctly complete the task within the constraint of the number of lexical units; low-reward generation strategies are suppressed, meaning the first model gradually reduces reasoning methods that exceed the constraint of the number of lexical units or output errors. After multiple rounds of iterative training, the first model maintains task accuracy while gradually reducing the number of lexical units required, thus reducing inference latency and computational overhead.

[0072] The average reward curve is continuously monitored during training. When the reward curve stabilizes and no longer rises significantly, the first model is considered to have converged, training is stopped, and the second model corresponding to the first task is obtained. This second model possesses both specialized reasoning capabilities for the first task and efficient reasoning capabilities suitable for real-time robot deployment.

[0073] Using the same method, we obtained the second model for each task.

[0074] The above embodiments output a reward signal under the constraint of the number of lexical units, and optimize the first model based on the reward signal. This ensures that during reinforcement learning training, the first model not only needs to pursue the correctness of the output result, but also needs to complete the reasoning and output the correct result within the limited number of lexical units. Therefore, the second model corresponding to the first task trained can effectively reduce the overhead of the number of lexical units while maintaining the task accuracy, thereby reducing model inference latency and computational resource consumption, and improving the model's response speed and deployment efficiency in robot real-time decision-making scenarios.

[0075] In this embodiment, the robot is controlled to perform interactive tasks by the first model, and the first model is trained using the execution results of the interactive tasks to obtain the third model. The following steps (1)-(3) can be referred to.

[0076] (1) Obtain the task description text; wherein, the task description text is used to describe the information required to perform the interactive task. The task description text includes at least: the detection conditions for the successful execution of the interactive task. The detection conditions are used to determine whether the first model has successfully completed the interactive task during subsequent training, so that the first model can clearly understand the standard for the successful execution of the interactive task before executing the interactive task, and provide clear and explicit optimization guidance for subsequent training.

[0077] The task description text may also include one or more of the following information: information about the robot's current environment, the robot's pose information, and a description of the interactive task. For example, environmental scene information may be information obtained by capturing images of the surrounding environment and converting them into text format; pose information may include the robot's position in the human-computer interaction environment and the posture of key robot parts (such as the arm); and the description of the interactive task may be an instruction to the robot to retrieve a loaf of bread from the shelf and pay at the checkout counter.

[0078] (2) In the human-computer interaction environment, the robot is controlled to perform interactive tasks according to the task description text through the first model, and the task execution results are obtained.

[0079] Specifically, in the human-computer interaction environment, the first model outputs decision actions based on the task description text, controlling the robot to execute the interactive task according to the decision actions. After the robot completes the interactive task, the human-computer interaction environment returns the task execution result to the first model. Since the first model cannot obtain all the information in the environment, such as the position of occluded objects or the state of areas not in the field of vision, it needs to make decisions under partially observable conditions during the interaction process. This requires the first model to have the ability to actively explore and complete information.

[0080] (3) The first model is trained based on the task execution results and the detection conditions for successful execution of interactive tasks, and the third model is obtained after training.

[0081] Specifically, after obtaining the task execution result, the system determines whether the interactive task was successfully completed based on the result and the detection conditions for successful interactive task execution provided in the task description text. If the task execution result meets the detection conditions, a positive reward signal is assigned to the interactive process; if the result does not meet the detection conditions, a negative reward signal or a zero reward signal is assigned to the interactive process.

[0082] Based on the aforementioned reward signals, the parameters of the first model are updated using a reinforcement learning algorithm, making the first model more inclined to adopt decision-making strategies that lead to success in subsequent interactions.

[0083] During the training process, the first model continuously interacts with the human-computer interaction environment in a closed loop, gradually learning and mastering various cognitive abilities required for the robot to perform interactive tasks, such as environmental state tracking, long-term memory maintenance, spatial cognition, active exploration, multi-step task planning, and dynamic decision-making.

[0084] Repeat steps (1)-(3) above until the performance of the first model on the interaction task reaches the preset convergence condition or meets the preset training round requirements. The first model after training is used as the third model, which is the robot interaction expert model.

[0085] In the above embodiments, the robot is controlled by the first model to perform interactive tasks and obtain execution results in the human-computer interaction environment, so that the first model can accumulate execution experience in dynamic and partially observable real interactive scenarios; then, the first model is trained according to the execution results and detection conditions, so that the trained third model can have multiple interactive cognitive abilities, thereby improving the adaptability and success rate of the third model in performing complex long-range tasks in robot interaction scenarios.

[0086] Reference Figure 3 In one embodiment, multiple second models and a third model corresponding to multiple tasks are fused to obtain a target model, including: A preset fusion algorithm is used to fuse multiple second and third models corresponding to multiple tasks to obtain the target model; the fusion algorithm includes parameter pruning, symbol election and parameter merging.

[0087] In this embodiment, multiple second and third models all originate from a unified first model, thus sharing consistent basic language capabilities. Furthermore, the parameter changes generated by each expert model during training all reside within the same parameter space. Within this same parameter space, for any given parameter position, the parameter changes of different expert models at that position can be compared and measured using numerical signs and magnitudes.

[0088] Therefore, this embodiment employs a fusion algorithm to combine multiple second and third models corresponding to multiple tasks to obtain the target model. This fusion algorithm alleviates parameter conflicts between different expert models by identifying parameter updates with consistent directions in different expert models and suppressing conflicting parameter updates, thereby integrating the knowledge of multiple expert models while minimizing mutual interference. This fusion algorithm retains the advantages gained by each expert model at different training stages without requiring additional multi-task joint optimization.

[0089] In practical implementation, parameter pruning, symbol election, and parameter merging are performed on the model parameters of multiple second models and the model parameters of the third model to obtain the target model. Parameter pruning means that for each expert model, parameter updates with parameter changes greater than a threshold are retained, while parameter updates with parameter changes not greater than the threshold are discarded. Each expert model refers to the model parameters of multiple second models and each model in the third model.

[0090] Symbol election means that at each parameter position, the sign direction of parameter updates from all expert models at that position is counted (whether positive or negative), and a dominant sign direction is elected from among these sign directions. Specifically, for each parameter position, the dominant sign direction at that position is determined by majority vote, that is, the sign direction that appears most frequently at that parameter position is selected as the dominant sign direction for that position.

[0091] Parameter merging means that during the parameter merging stage, for each parameter position, only the model parameter updates whose sign direction is consistent with the dominant sign direction are retained, and the retained model parameter updates are merged to obtain the fused parameter update at that parameter position.

[0092] After performing the above operations on all parameter positions, a complete fusion parameter update is obtained. This fusion parameter update is then superimposed onto the model parameters of the first model to obtain the target model.

[0093] Through the above-mentioned parameter pruning, symbol election and parameter merging processes, the conflicting parameter updates among multiple second and third models and all expert models are effectively suppressed, and the complementary capabilities of each expert model are integrated into a single model, thereby obtaining a target model that simultaneously possesses multi-domain specialized reasoning capabilities and robot interactive cognitive capabilities.

[0094] To better understand, the fusion algorithm, which includes parameter pruning, symbol election, and parameter merging, will be described in more detail below, with reference to the following examples.

[0095] The steps of parameter pruning are as follows: Take each expert model in the multiple second models and third models as the current model, and prune the parameters of the current model according to the magnitude of parameter changes before and after the update, so as to obtain multiple fourth models corresponding to multiple second models and a fifth model corresponding to the third model.

[0096] Specifically, for each current model, the magnitude of the parameter changes before and after training is obtained. The magnitude of the parameter changes is the difference between the parameter values ​​of the current model and the parameter values ​​of the first model, which reflects the amount of parameter changes that occur in the current model during reinforcement learning training or robot interaction training.

[0097] For each parameter position in the current model, the magnitude of parameter change at that position is obtained and compared with a preset change threshold. In the current model, model parameter updates with a parameter change magnitude greater than the change threshold are retained, while model parameter updates with a parameter change magnitude not greater than the change threshold are pruned, specifically by setting the parameter change at that position to zero.

[0098] By using the above method, each parameter position in the current model is processed one by one, completing the parameter pruning of the current model. After parameter pruning, only parameter updates with large parameter change amplitudes are retained in the current model, while parameter updates with small parameter change amplitudes are removed.

[0099] The above parameter pruning process is performed on each of the multiple second models and the third model to obtain multiple fourth models corresponding to the multiple second models after parameter pruning, and a fifth model corresponding to the third model after parameter pruning. The parameter updates in the fourth and fifth models are all selected parameter changes that contribute significantly to the model's capabilities.

[0100] Through the above pruning operation, parameter updates with small amplitudes and weak contributions in each expert model are pruned away, and only parameter updates with large amplitudes and significant contributions are retained to participate in the subsequent fusion process, thereby reducing the interference of noise parameters on the fusion results.

[0101] The steps of symbol election are as follows: In multiple fourth and fifth models, the dominant symbol direction is elected based on the symbol direction of the model parameter update at each parameter position.

[0102] Specifically, after obtaining multiple fourth and fifth models, the sign direction of the parameter update at each parameter position in each fourth and fifth model is obtained. The sign direction of the parameter update indicates whether the parameter change at that position is positive or negative; a positive value indicates that the model parameter has been strengthened, and a negative value indicates that the model parameter has been weakened.

[0103] For each parameter position, the sign direction of parameter updates for all fourth and fifth models at that position is statistically analyzed. The sign direction that occurs most frequently is determined as the dominant sign direction at that parameter position. For example, if the sign direction of most expert models is positive at a certain parameter position, then the dominant sign direction for that parameter position is elected as positive; conversely, if the sign direction of most expert models is negative, then it is elected as negative.

[0104] Perform the above processing on each parameter position to obtain the dominant symbol direction corresponding to each parameter position.

[0105] The parameter merging steps are as follows: At each parameter position corresponding to multiple fourth and fifth models, the model parameters that are consistent with the direction of the dominant symbol are merged to obtain the target model.

[0106] Specifically, the sign direction of the parameter updates for each of the fourth and fifth models at that parameter position is compared with the dominant sign direction. If the sign direction of a certain expert model is consistent with the dominant sign direction (i.e., both are positive or both are negative), the updated value of the model parameter at that parameter position is retained. If the sign direction of a certain expert model is inconsistent with the dominant sign direction (i.e., opposite to the dominant sign direction), the updated value of the model parameter at that parameter position is discarded, meaning that the model parameter does not participate in the subsequent merging calculation.

[0107] For each parameter location, the update values ​​of all model parameters whose sign direction aligns with the dominant sign direction are merged to obtain the fused parameter update for that parameter location. This merging can be achieved using methods such as averaging or weighted averaging.

[0108] Perform the above processing on each parameter position one by one to obtain the fusion parameter update at each parameter position.

[0109] After merging all parameter positions, the fused parameters at each position are updated and superimposed onto the corresponding model parameters of the first model to obtain the target model. Based on this, the target model can integrate the complementary capabilities of multiple second and third models corresponding to multiple tasks, and possesses multi-domain specialized reasoning capabilities and robot interaction and cognition capabilities.

[0110] In the above embodiments, firstly, parameter pruning is used to remove parameter updates with smaller amplitudes from each expert model, thereby reducing the interference of noisy parameters on the fusion result. Then, symbol election is used to determine the dominant symbol direction at each parameter position, ensuring that the fusion direction determination incorporates the common choices of multiple expert models. Finally, parameter merging is used to retain only model parameter updates consistent with the dominant symbol direction for fusion, thereby reducing the mutual cancellation of capabilities between different expert models due to conflicting update directions. Through this method, the specialized capabilities learned by multiple second and third models can be integrated into a single model with lower interference, improving the overall performance of the fused target model in multi-task scenarios.

[0111] Understandably, although multiple second and third models originate from the same first model, differences in training data and training objectives may lead to varying degrees of displacement of different expert models relative to the first model in the parameter space. This difference in distance between different expert models in the parameter space has a significant impact on the choice of fusion method.

[0112] Based on this, combined Figure 3 In another embodiment, multiple second models corresponding to multiple tasks are fused with a third model to obtain a target model, including: For each expert model in the multiple second and third models corresponding to multiple tasks, determine the model distance with the first model in the parameter space.

[0113] A preset fusion algorithm is used to fuse expert models whose model distance is less than or equal to a distance threshold to obtain a fused model; A linear interpolation algorithm is used to fuse the fusion model with expert models whose model distance is greater than a distance threshold to obtain the target model. In the linear interpolation algorithm, the weights of expert models whose model distance is greater than the distance threshold are less than the weights of the fusion model.

[0114] In a specific embodiment, for each expert model among the multiple second and third models, the model distance between the expert model and the first model in the parameter space is determined. The model distance is used to represent the degree of difference between the parameter distribution of the expert model and the parameter distribution of the first model.

[0115] The calculated distances between each model are compared with a preset distance threshold. For expert models whose distances are less than or equal to the distance threshold, since these expert models are close to the first model and fall within the local parameter neighborhood of the first model, their parameter updates have good comparability and consistency. Therefore, a fusion algorithm can be used to fuse them, thereby eliminating parameter conflicts between the expert models and achieving effective capability superposition. The specific implementation method is the same as described in the aforementioned embodiment and will not be repeated here.

[0116] For expert models whose model distance is greater than a distance threshold (referred to as long-distance models), their parameter updates include significant parameter shifts. These shifts relative to the first model encompass both training lineage differences and task-specific differences. If these models are directly incorporated into a pre-defined fusion algorithm for symbol election and parameter merging with other expert models, their large parameter updates may dominate the election results, drowning out the contributions of other closer expert models and leading to a decline in the performance of the fused model.

[0117] Therefore, for such long-range models, this embodiment can use a linear interpolation algorithm to fuse them with the fusion model to obtain the target model. Specifically, the model parameters of the long-range model and the model parameters of the fusion model are linearly combined according to preset weights, wherein the weight of the long-range model is less than the weight of the fusion model. By introducing this low-weight approach, the existing parameter structure and performance of the fusion model can be preserved, while complementary capabilities can be transferred from the long-range model.

[0118] The target model obtained through the above-mentioned phased fusion is then integrated with expert models of different parameter distances using appropriate fusion methods. This ensures both the fine fusion of capabilities between near-distance expert models and the safe introduction of complementary capabilities from far-distance expert models. As a result, while making full use of the complementary capabilities of each expert model, the stability of the fusion process and the overall decision-making performance of the fused target model in robot deployment scenarios are improved.

[0119] In this embodiment, the model training method may further include: Obtain the sixth model for processing the third task; use a fusion algorithm or a linear interpolation algorithm to fuse the sixth model with the target model to obtain a new target model.

[0120] In real-world applications, in addition to the aforementioned task processing capabilities such as mathematical reasoning, code generation, scientific reasoning, instruction following, and robot interaction, other specialized capabilities may be required. The specific capabilities a robot needs may differ depending on the application scenario.

[0121] Based on the above requirements, this embodiment can further extend the model fusion. Specifically, a sixth model is obtained for processing the third task. The sixth model is an expert model trained for the third task, which is a task different from the aforementioned tasks.

[0122] The model distance between the sixth model and the first model in the parameter space is determined. If the model distance between the sixth model and the first model is less than or equal to the distance threshold, it indicates that the sixth model is within the local parameter neighborhood of the first model, and its parameter updates are comparable to those of the existing expert models. Therefore, the aforementioned fusion algorithm is used to fuse the sixth model with the target model.

[0123] If the model distance between the sixth model and the first model is greater than a distance threshold, it indicates that the parameter update magnitude of the sixth model is large and there is a certain difference between its parameter structure and that of the target model. Therefore, a linear interpolation algorithm is used to fuse the sixth model and the target model. In this linear interpolation algorithm, the interpolation weight of the sixth model is smaller than that of the target model. That is, the capabilities of the sixth model are introduced into the target model with a lower weight, avoiding the impact of large parameter updates of distant models on the existing capability structure of the target model.

[0124] After the above fusion process, a new target model is obtained. This new target model, based on the multi-domain specialized reasoning ability and robot interaction cognitive ability of the original target model, further incorporates specialized processing capabilities for the third task.

[0125] This embodiment continuously introduces new expert models based on actual needs, expanding the functionality of the target model to adapt to the differentiated requirements of different application scenarios. Simultaneously, since the newly added sixth model is trained independently and can be directly fused with the target model, there is no need to retrain all models, reducing training costs.

[0126] According to the above embodiments, since the consensus-reached parts of the parameter update directions of each expert model are preserved during the fusion process, and conflicting updates are effectively suppressed, the fused target model can simultaneously integrate specialized capabilities from multiple directions. Specifically, the target model possesses the following capabilities.

[0127] First, general language understanding and reasoning ability. This ability stems from the general language foundation established during the basic large language model and supervised fine-tuning phase, enabling the model to understand natural language instructions, perform logical reasoning, and process various types of language input.

[0128] Second, multi-domain specialized reasoning capabilities. This capability stems from the fusion of secondary models corresponding to multiple tasks, including mathematical reasoning, code generation, scientific reasoning, and instruction following. These specialized capabilities enable the model to achieve high accuracy and reliability when handling tasks involving mathematical calculations, code writing, scientific problem solving, and complex instruction execution.

[0129] Third, efficient reasoning capability. Because the second model incorporates a constraint on the number of lexical units as an optimization objective for reasoning efficiency during reinforcement learning training, it maintains task accuracy while reducing the number of lexical units generated during the reasoning process. This lowers reasoning latency and computational resource consumption, enabling the fused target model to better adapt to the requirements of response speed and computational resources in actual robot deployment scenarios.

[0130] Fourth, the robot's interactive cognitive capabilities. This capability stems from the fusion of the third model, including environmental state tracking, long-term memory maintenance, spatial cognition, proactive exploration, and long-term task planning. These capabilities enable the target model to continuously track scene changes in dynamic interactive environments, maintain key information during task execution, understand spatial layout and object positional relationships, proactively acquire environmental information when information is insufficient, and plan multi-step action sequences based on the current and target states.

[0131] The target model obtained after the above fusion process can serve as the robot's high-level cognitive decision-making module. When the robot performs interactive tasks, the target model makes comprehensive decisions based on its internal general reasoning ability and the robot's interactive cognitive ability, and outputs action commands to control the robot to perform the interactive tasks. This target model is deployed in the robot as a single model, which can reduce the system complexity and resource overhead caused by maintaining and calling multiple independent models in the robot at the same time, and provide a unified intelligent decision-making capability for the robot to perform complex tasks.

[0132] In some embodiments of this disclosure, a robot control method may also be provided, which can be executed by a robot control device that can be applied to an end-side device of a robot or an edge computing device that is communicatively connected to the robot.

[0133] In this embodiment, the robot control method may include: Obtain the target task for the robot; control the robot to execute the target task through the target model; wherein the target model is trained according to the model training method provided in the above embodiments.

[0134] In one example, the target task is a natural language command given by the user to the robot, such as: taking a beverage out of the refrigerator for the user in a home environment, leading visitors to a specific exhibit in a museum setting, or purchasing a certain type of goods in a supermarket setting.

[0135] Then, the robot is controlled to perform the target task through the target model. The target model is trained according to the aforementioned model training method implementation example and can be pre-deployed on the robot's end or edge.

[0136] When robots face complex, long-term target tasks that require dynamic strategy adjustments, the target model can simultaneously possess general language understanding and reasoning capabilities, multi-domain specialized reasoning capabilities, efficient reasoning capabilities, and robot interactive cognitive capabilities. Therefore, a unified intelligent decision can be made through a single target model to control the robot to execute the target task. This can reduce the reasoning delay caused by calling multiple independent models for decision-making and improve the robot's response speed and task execution success rate in real-world application scenarios.

[0137] The robot control method provided in this disclosure includes: acquiring a target task for the robot, and controlling the robot to execute the target task through a target model; wherein, after acquiring basic capabilities through multi-task supervised training, the target model acquires specialized reasoning capabilities and interactive cognitive capabilities through multi-directional reinforcement learning training and robot interaction training, and integrates multiple capabilities into a single model. Therefore, when controlling the robot to execute the target task, the target model can simultaneously mobilize general reasoning capabilities and robot interactive cognitive capabilities for comprehensive decision-making, thereby improving the robot's success rate and efficiency in executing the target task. Furthermore, since multiple task processing capabilities are integrated into a single target model, the reasoning latency and computational resource consumption caused by calling multiple independent models during actual deployment can be reduced, enabling the robot to respond more efficiently to dynamic changes in real-time interactive scenarios.

[0138] Figure 4 This is a schematic diagram of a model training apparatus provided in an embodiment of the present disclosure. The model training apparatus is used to implement the model training method in the above embodiments. The model training apparatus may include the following modules: The model acquisition module 402 is used to acquire a first model; wherein the first model is trained based on supervised data from multiple tasks, and the multiple tasks belong to different types of tasks; The first training module 404 is used to perform reinforcement learning training on the first model using the training data of the first task among the plurality of tasks, so as to obtain the second model corresponding to the first task. The second training module 406 is used to control the robot to perform interactive tasks through the first model, and to train the first model using the execution results of the interactive tasks to obtain the third model. The fusion module 408 is used to fuse multiple second models corresponding to the multiple tasks with the third model to obtain a target model; wherein the target model is used to deploy in the robot.

[0139] In one embodiment, the fusion module 408 is further configured to: A preset fusion algorithm is used to fuse multiple second models corresponding to the multiple tasks with the third model to obtain a target model; wherein, the fusion algorithm includes: parameter pruning, symbol election and parameter merging.

[0140] In one embodiment, the fusion module 408 is further configured to: Each of the multiple second models and the expert model in the third model is taken as the current model. Based on the magnitude of the parameter change of the model parameters in the current model before and after the update, the parameters of the current model are pruned to obtain multiple fourth models corresponding to the multiple second models and a fifth model corresponding to the third model. In the plurality of fourth models and the fifth model, a dominant sign direction is elected based on the sign direction of the model parameter update at each parameter position; At each parameter position corresponding to the plurality of fourth models and the fifth model, the model parameters that are consistent with the direction of the dominant symbol are fused to obtain the target model.

[0141] In one embodiment, the fusion module 408 is further configured to: In the current model, model parameter updates with parameter change magnitudes greater than the change threshold are retained, while model parameter updates with parameter change magnitudes not greater than the change threshold are pruned.

[0142] In one embodiment, the fusion module 408 is further configured to: For each expert model in the multiple second models and the third models corresponding to the multiple tasks, the model distance with the first model in the parameter space is determined respectively; A preset fusion algorithm is used to fuse expert models whose model distance is less than or equal to a distance threshold to obtain a fused model; A linear interpolation algorithm is used to fuse the fusion model with the expert models whose model distance is greater than a distance threshold to obtain the target model; wherein, in the linear interpolation algorithm, the weights of the expert models whose model distance is greater than the distance threshold are less than the weights of the fusion model.

[0143] In one embodiment, the fusion module 408 is further configured to: Obtain the sixth model for processing the third task; The sixth model is fused with the target model using the fusion algorithm or the linear interpolation algorithm to obtain a new target model.

[0144] In one embodiment, the second training module 406 is further configured to: Obtain task description text; wherein, the task description text is used to describe the information required to execute the interactive task, and the task description text includes at least: the detection conditions for the successful execution of the interactive task; In a human-computer interaction environment, the first model controls the robot to perform the interactive task according to the task description text, and the task execution result is obtained. The first model is trained based on the task execution result and the detection conditions for successful execution of the interactive task, and a third model is obtained after training.

[0145] In one embodiment, the first training module 404 is further configured to: Using the first model, under the constraint of the number of lexical units, a reward signal is output based on the training data of the first task; wherein, the first task is any one of the plurality of tasks; The first model is optimized using the reward signal to obtain the second model corresponding to the first task.

[0146] The model training device provided in this embodiment has the same implementation principle and technical effect as the aforementioned model training method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0147] In one embodiment, a robot control device is also provided, the robot control device comprising: The task acquisition module is used to acquire the target task for the robot; The task execution module is used to control the robot to perform the target task through the target model; wherein the target model is trained according to the model training method described above.

[0148] The robot control device provided in this embodiment has the same implementation principle and technical effect as the robot control method embodiment described above. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0149] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Figure 5As shown, the electronic device 500 includes one or more processors 501 and memory 502.

[0150] The processor 501 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 500 to perform desired functions.

[0151] The memory 502 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 501 may execute the program instructions to implement the model training method or robot control method of the embodiments of this disclosure described above, and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0152] In one example, the electronic device 500 may also include an input device 503 and an output device 504, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0153] In addition, the input device 503 may also include, for example, a keyboard, a mouse, etc.

[0154] The output device 504 can output various information to the outside, including determined distance information, direction information, etc. The output device 504 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0155] Of course, for the sake of simplicity, Figure 5 Only some of the components of the electronic device 500 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 500 may include any other suitable components depending on the specific application.

[0156] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program for executing the above-described model training method or robot control method.

[0157] The computer program product of a model training method or robot control method, device, electronic device and medium provided in this disclosure includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0158] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0159] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model training method, characterized in that, The method includes: Obtain a first model; wherein the first model is trained based on supervised data from multiple tasks, and the multiple tasks belong to different types of tasks; Using the training data of the first task among the multiple tasks, the first model is trained by reinforcement learning to obtain the second model corresponding to the first task. The robot is controlled to perform interactive tasks by the first model, and the first model is trained using the execution results of the interactive tasks to obtain the third model; The multiple second models corresponding to the multiple tasks are fused with the third model to obtain the target model; wherein the target model is used to deploy in the robot.

2. The method according to claim 1, characterized in that, The step of fusing the multiple second models corresponding to the multiple tasks with the third model to obtain the target model includes: A preset fusion algorithm is used to fuse multiple second models corresponding to the multiple tasks with the third model to obtain a target model; wherein, the fusion algorithm includes: parameter pruning, symbol election and parameter merging.

3. The method according to claim 2, characterized in that, The step involves employing a preset fusion algorithm to fuse multiple second models corresponding to the multiple tasks with the third model to obtain a target model, including: Each of the multiple second models and the expert model in the third model is taken as the current model. Based on the magnitude of the parameter change of the model parameters in the current model before and after the update, the parameters of the current model are pruned to obtain multiple fourth models corresponding to the multiple second models and a fifth model corresponding to the third model. In the plurality of fourth models and the fifth model, a dominant sign direction is elected based on the sign direction of the model parameter update at each parameter position; At each parameter position corresponding to the plurality of fourth models and the fifth model, the model parameters that are consistent with the direction of the dominant symbol are fused to obtain the target model.

4. The method according to claim 3, characterized in that, The step of pruning the parameters of the current model based on the magnitude of parameter changes before and after the update includes: In the current model, model parameter updates with parameter change magnitudes greater than the change threshold are retained, while model parameter updates with parameter change magnitudes not greater than the change threshold are pruned.

5. The method according to claim 1, characterized in that, The step of fusing the multiple second models corresponding to the multiple tasks with the third model to obtain the target model includes: For each expert model in the multiple second models and the third models corresponding to the multiple tasks, the model distance in the parameter space with the first model is determined respectively; A preset fusion algorithm is used to fuse expert models whose model distance is less than or equal to a distance threshold to obtain a fused model; A linear interpolation algorithm is used to fuse the fusion model with the expert models whose model distance is greater than a distance threshold to obtain the target model; wherein, in the linear interpolation algorithm, the weights of the expert models whose model distance is greater than the distance threshold are less than the weights of the fusion model.

6. The method according to claim 5, characterized in that, The method further includes: Obtain the sixth model for processing the third task; The sixth model is fused with the target model using the fusion algorithm or the linear interpolation algorithm to obtain a new target model.

7. The method according to claim 1, characterized in that, The process of controlling the robot to perform interactive tasks through the first model and training the first model using the execution results of the interactive tasks to obtain the third model includes: Obtain task description text; wherein, the task description text is used to describe the information required to execute the interactive task, and the task description text includes at least: the detection conditions for the successful execution of the interactive task; In a human-computer interaction environment, the first model controls the robot to perform the interactive task according to the task description text, and the task execution result is obtained. The first model is trained based on the task execution result and the detection conditions for successful execution of the interactive task, and a third model is obtained after training.

8. The method according to claim 1, characterized in that, The step of training the first model using training data from the first task among the plurality of tasks to obtain a second model corresponding to the first task includes: Using the first model, under the constraint of the number of lexical units, a reward signal is output based on the training data of the first task; wherein, the first task is any one of the plurality of tasks; The first model is optimized using the reward signal to obtain the second model corresponding to the first task.

9. A robot control method, characterized in that, The method includes: Obtain the target task for the robot; The robot is controlled to perform the target task by a target model; wherein the target model is trained by the model training method according to any one of claims 1 to 8.

10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the above-described model training method or robot control method.

Images