A robot model training method, apparatus, equipment and medium

Abstract

This application provides a robot model training method, apparatus, device, and medium, relating to the field of robot model training technology, and can solve the problems of poor generalization and low accuracy in the training process of existing robot models. The training method includes: determining multiple target values ​​and a sequence of models to be trained based on task matching values ​​and their corresponding model efficiency values; generating target simulation data and calculating similarity based on the required actions and objects to obtain target historical data; and sequentially training and testing the models to be trained in the sequence based on the target simulation data, target historical data, and real-time requirement data until a training test value greater than a preset threshold is obtained, at which point the training test is complete. Based on the above method, the generalization and accuracy of robot training are improved.

Description

Technical Field

[0001] This application relates to the field of robot model training technology, and in particular to a robot model training method, apparatus, device and medium. Background Technology

[0002] Intelligent robot models are artificial intelligence models that integrate multimodal perception, decision-making, and control capabilities, enabling robots to understand the physical world, respond to task requirements, and autonomously execute actions. This allows them to replace or assist humans in completing complex, repetitive, dangerous, or high-precision physical world tasks. Application scenarios can cover multiple fields, including industrial production scenarios such as automated assembly, parts grasping and handling, and precision machining quality inspection; service robot scenarios such as home cleaning and food preparation; and special environmental operations such as detection and operation in high-temperature, high-pressure, and radiation environments.

[0003] Existing robot model training methods generally rely on a large amount of real-world data, which is cumbersome and costly to collect. Although simulation platforms are used to generate training data, experts usually need to manually configure the environment parameters, task scripts and physics engines of the simulation platform, and it can only be used for training specific models or fixed architectures.

[0004] Therefore, existing robot model training methods suffer from low generalization ability, low training efficiency, and low training accuracy. Summary of the Invention

[0005] This application provides a robot model training method, apparatus, device, and medium to solve the problems of low generalization ability, low training efficiency, and low training accuracy in existing robot model training methods.

[0006] Firstly, this application provides a robot model training method, the method comprising:

[0007] Based on the task matching values ​​of each model task vector set and the demand vector, and their respective model efficiency values, multiple target values ​​are determined, and based on each target value, the sequence of models to be trained is determined.

[0008] Based on the required actions and the required objects, target simulation data is generated in the simulation environment, and the similarity between the target fused feature vector and the historical data vector is calculated to obtain the target historical data.

[0009] Based on the target simulation data, target historical data, and real-time demand data, the models to be trained in the training model sequence are trained and tested sequentially until a training test value greater than the preset threshold is obtained, at which point the training and testing are completed.

[0010] In some embodiments of this application, multiple target values ​​are determined based on the task matching values ​​of each model task vector set and the demand vector, and their respective corresponding model efficiency values, including:

[0011] The user requirement text is mapped to obtain a requirement vector, and the similarity value between the requirement vector and each model task vector is calculated; each model task vector set includes at least one model task vector.

[0012] Based on at least one corresponding similarity value, the largest similarity value is determined as the task matching value of the model task vector set, and based on the initial model corresponding to each model task vector set, the number of parameters and the memory value of the initial model are determined.

[0013] The parameter values ​​and memory values ​​are normalized separately to obtain the target parameter values ​​and target memory values ​​corresponding to each initial model. The target parameter values ​​and target memory values ​​are then weighted and summed to obtain the model efficiency value.

[0014] The target value for each initial model is obtained by weighted summation of the task matching value and the model efficiency value.

[0015] In some embodiments of this application, determining the sequence of models to be trained based on various target values ​​includes:

[0016] Sort multiple target values ​​from largest to smallest, and then determine the sorted target values ​​in sequence;

[0017] Based on the initial model corresponding to each sorting target value, determine the sequence of models to be trained.

[0018] In some embodiments of this application, target simulation data is generated in a simulation environment based on the required action and the required object, including:

[0019] Based on the visual language model, semantic analysis is performed on user requirement text and user requirement image to obtain requirement actions and requirement objects. Then, based on the simulation environment, a simulation scenario corresponding to the requirement actions and requirement objects is established.

[0020] Import the user requirements file into the simulation scene, and add a physics engine and scene noise to obtain the target simulation environment;

[0021] Based on the target simulation environment, generate target simulation data.

[0022] In some embodiments of this application, the similarity between the target fused feature vector and the historical data vector is calculated to obtain the target historical data, including:

[0023] Based on the dual-tower Transformer architecture, user requirement text, user requirement image, and user requirement file are encoded separately to obtain corresponding requirement feature vectors and model feature vectors. Then, based on the cross-attention mechanism, the requirement feature vectors and model feature vectors are fused to obtain the target fused feature vector.

[0024] Based on multiple historical data, the corresponding historical data vectors are determined, and the similarity between the target fusion feature vector and the historical data vector is calculated to obtain the corresponding similarity value.

[0025] The similarity values ​​that are greater than a preset similarity threshold are determined as target similarity values, and the historical data corresponding to the target similarity values ​​are determined as target historical data.

[0026] In some embodiments of this application, based on target simulation data, target historical data, and real-time demand data, the models to be trained in the training model sequence are sequentially trained and tested until a training test value greater than a preset threshold is obtained, at which point the training test is completed, including:

[0027] Acquire real-time demand data, and determine the merged data of target simulation data and target historical data as the first dataset, and the real-time demand data as the second dataset;

[0028] Based on the first dataset, the first model to be trained in the training model sequence is subjected to multiple rounds of training and testing to obtain the validation value after each round of training. Based on the validation value with the largest value and its corresponding model parameters, the parameters of the model to be trained are adjusted.

[0029] Based on the second dataset, the parameters of the model to be trained are adjusted to obtain the target model to be trained, and the target model to be trained is tested to obtain the corresponding training test values.

[0030] Compare the training test value with the preset threshold, and obtain a training test value that is greater than the preset threshold. The training test is then complete.

[0031] In some embodiments of this application, comparing the training test value with a preset threshold to obtain a training test value greater than the preset threshold completes the training test, including:

[0032] Compare the training test values ​​with the preset threshold to obtain the corresponding comparison results;

[0033] If the comparison result shows that the training test value is greater than the preset threshold, then the training test is completed, and the first model to be trained is determined as the target requirement model.

[0034] If the comparison result shows that the training test value is not greater than the preset threshold, then based on the sequence of models to be trained, the next model to be trained and its corresponding training test value are determined sequentially. When all training test values ​​are not greater than the preset threshold, a prompt message is output. Based on the prompt message, augmented training data is obtained, and based on the augmented training data, the models to be trained are trained and tested sequentially until a training test value greater than the preset threshold and its corresponding target requirement model are obtained.

[0035] Secondly, this application provides a robot model training device, the device comprising:

[0036] The sequence determination module is used to determine multiple target values ​​based on the task matching values ​​of each model task vector set and the demand vector, as well as their respective model efficiency values, and to determine the sequence of models to be trained based on each target value.

[0037] The calculation module is used to generate target simulation data in the simulation environment based on the required actions and the required objects, and to calculate the similarity between the target fused feature vector and the historical data vector to obtain the target historical data.

[0038] The training and testing module is used to sequentially train and test the models to be trained in the training model sequence based on the target simulation data, target historical data, and real-time data required, until a training and testing value greater than a preset threshold is obtained, at which point the training and testing is completed.

[0039] Thirdly, this application provides a computer device, including: a processor, and a memory communicatively connected to the processor;

[0040] The memory stores the instructions that the computer executes;

[0041] The processor executes computer execution instructions stored in memory to implement the method of this application.

[0042] Fourthly, this application provides a computer-readable storage medium storing program code, which, when executed by a processor, is used to implement the method of this application.

[0043] Compared with existing technologies, the method in this application determines the task matching value and corresponding model efficiency value of each model task vector set and demand vector, calculates the target value, and thus determines the corresponding training model sequence. This effectively solves the problems of traditional solutions lacking an adaptive model selection mechanism and difficulty in matching and adapting models. Furthermore, based on the demand actions and demand objects, it automatically generates noisy target simulation data in the simulation environment and generates target fusion feature vectors through a dual-tower Transformer architecture. It also retrieves target historical data by combining similarity with historical data vectors, avoiding the cumbersome process of manual configuration for simulation data generation, lowering the technical threshold, improving the richness and realism of training data, and enhancing the utilization rate of historical data and model generalization ability. Based on the target simulation data… Based on historical data and real-time demand data, the first model to be trained is trained through a two-stage training process. The training test value is compared with the preset threshold, achieving accurate model optimization based on a small amount of real data, which significantly reduces data collection costs. If the training test value is greater than the preset threshold, the target demand model is directly determined. If it is not greater, the model to be trained is tried in sequence. Finally, the target demand model is determined by combining all training test values, preset thresholds, and accurate data supplementary prompts. This solves the technical problems of high data cost, high technical threshold, poor model adaptability, weak generalization ability, and insufficient reuse of historical data in traditional robot model training. It improves the generalization of robot model training, expands the actual application scenarios of the model, and improves training efficiency and accuracy. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0045] Figure 1 A flowchart illustrating a robot model training method provided in an embodiment of this application;

[0046] Figure 2 A schematic diagram illustrating the framework of a robot model training method provided in an embodiment of this application;

[0047] Figure 3 This is a schematic diagram of the structure of a robot model training device provided in an embodiment of this application;

[0048] Figure 4 This is a structural block diagram of an apparatus for performing a robot model training method according to an embodiment of this application. Detailed Implementation

[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0050] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0051] Figure 1 This is a flowchart illustrating a robot model training method provided in an embodiment of this application. Figure 1 As shown, this robot model training method may include the following steps:

[0052] S110. Based on the task matching values ​​of each model task vector set and the demand vector, and their respective model efficiency values, determine multiple target values, and determine the sequence of models to be trained based on each target value.

[0053] The model task vector set refers to the set of high-dimensional feature vectors generated by encoding all task texts officially claimed to be supported by each candidate model. For example, in practical applications, the official task set of a candidate model can be extracted first, such as {"making sandwiches", "wiping tables"}, and then each task text can be converted into a high-dimensional feature vector through a sentence embedding model, such as Sentence-BERT. Based on all these vectors, the task vector set of the candidate model can be formed together.

[0054] A demand vector is a single high-dimensional feature vector obtained by transforming the user's input task description through the same sentence embedding model used to generate the task vector set of the model. In other words, the user's task description, such as "take the lobster out of the bucket and wash it", is transformed into a high-dimensional feature vector after being transformed by the same sentence embedding model. This transforms the user's vague and unstructured natural language instructions into a precise and structured numerical representation.

[0055] Task matching value is a quantitative indicator that measures the degree of fit between user needs and a candidate model. It quantifies the semantic similarity between a candidate model and a user's task and can intuitively reflect the potential ability of a candidate model to complete the user's current task. The higher the specific value of the task matching value, the more the candidate model's existing capabilities match the user's needs and the more accurately it can complete the user's current task requirements.

[0056] Model efficiency is a quantitative indicator that evaluates the resource consumption and deployment adaptability of a candidate model. It quantifies the degree of computing resource consumption of a candidate model during deployment and operation. The higher the specific value of the model efficiency value, the lighter and more efficient the candidate model is. In practical applications, selecting a candidate model with a high efficiency value can reduce deployment costs, reduce energy consumption, and improve response speed.

[0057] The target value refers to the comprehensive score corresponding to each candidate model. It can comprehensively quantify the degree of matching between each candidate model and the current task requirements, as well as the execution efficiency of the candidate model itself, etc. Thus, through a specific numerical value, it represents the comprehensive execution capability of the candidate model for the task. The higher the actual value of the target value, the better the performance of the candidate model in meeting the current actual task requirements.

[0058] The training model sequence refers to a list of candidate models sorted from high to low according to the target value of the candidate models. This is to determine the priority order of model training, so that the candidate model with the highest overall adaptability is trained first. If its performance does not meet the requirements after training, the next candidate model in the sequence is tried in turn. This not only ensures training efficiency, but also improves the probability of achieving the task target through multiple model alternatives.

[0059] Based on this, by determining the model task vector set corresponding to each candidate model, the matching value between the candidate model and the current actual task to be executed can be quantitatively determined through vector similarity calculation. Based on the execution efficiency value of the candidate model itself, the target value corresponding to each candidate model is calculated comprehensively. The models are then sorted according to the magnitude of the target value to obtain the sequence of models to be trained.

[0060] S120. Based on the required actions and the required objects, generate target simulation data in the simulation environment, and calculate the similarity between the target fusion feature vector and the historical data vector to obtain the target historical data.

[0061] Among them, the required actions refer to the core operational behaviors that the robot needs to perform, which are parsed from the user task description. They are the key execution links of the user task. For example, in the user task "take the lobster out of the bucket and wash it", the parsed "take out" and "wash" are the required actions corresponding to this task.

[0062] The requirement object refers to the core object directly related to the required action, which is parsed from the user task description and input image, including the operation object, tool, scene carrier, etc.; for example, in "scooping out the lobster from the bucket and washing it", the parsed "lobster", "bucket", "brush" and "operating table" are all requirement objects.

[0063] A simulation environment refers to a virtual environment automatically built by a server that simulates a real task scenario. It includes a 3D model of the required object, thereby replacing the real physical environment to complete the training data collection. There is no need to manually build a real scene, which reduces the cost of data collection. For example, it can be the Isaac Sim simulation environment.

[0064] Target simulation data refers to multimodal training data generated when a robot performs required actions in an automatically constructed simulation environment. Input data can include three-view images of the front, left-wrist, and right-wrist, as well as the current joint angle state. Output data is the change in joint angle at the next moment or additionally includes the platform pose, thus serving as one of the data sources for model training and solving the problems of cumbersome and costly real data collection.

[0065] The target fusion feature vector refers to a high-dimensional vector generated by fusing task features and robot ontology features. It comprehensively expresses the representation of different dimensions of information corresponding to the user task and the current robot ontology, realizes the joint quantitative representation of task requirements and robot hardware characteristics, solves the matching inaccuracy problem caused by single feature retrieval, and thus ensures that historical data that can be matched with both the current task and the robot's own features are found.

[0066] Historical data vectors refer to the feature vectors corresponding to each historical data point in a historical dataset. They serve as a quantitative representation of historical data, providing a comparable standard for accurate retrieval of historical data and enabling efficient matching of historical data with current needs.

[0067] Target historical data refers to historical data retrieved from historical datasets that have a high cosine similarity to the target fused feature vector. In other words, it is the historical data with the highest similarity selected by calculating the cosine similarity between the target fused feature vector and all historical data vectors in the database.

[0068] Based on this, by determining the required actions and objects corresponding to the current required task, a corresponding simulation scenario is built in the simulation environment to generate target simulation data. The task features and robot body features are then fused together to generate a high-dimensional vector, which is the target fused feature vector. Based on the similarity between the target fused feature vector and the historical data vector, the target historical data is obtained.

[0069] S130. Based on the target simulation data, target historical data, and demand real-time data, train and test the models to be trained in the training model sequence in sequence until a training test value greater than the preset threshold is obtained, and the training test is completed.

[0070] Among them, real-time data refers to high-quality real data uploaded by users that is directly related to the current task. It is the core real-world sample for model training. The data type can be multimodal, including front, left-wrist, and right-wrist three-view images, robot joint angle data, task names, etc. It makes up for the difference between the simulation environment and the real scene, ensuring that the model can accurately respond to task requirements in actual applications, while significantly reducing the amount of real data that users need to collect and reducing data collection costs.

[0071] Training and testing refer to the performance evaluation of a model trained on a training dataset and then performed on a test dataset. The quantitative metric obtained from the performance evaluation is the training-test value, which is the success rate of the trained model in completing the target task on the test set. The higher the training-test value, the better the trained model performs on the current task.

[0072] The preset threshold is a pre-determined critical value used to judge whether the training test value of the trained model meets the actual requirements. If the training test value of the model is greater than the preset threshold, it means that the trained model can accurately complete the task and the training effect is very good. If the training test value of the model is not greater than the preset threshold, it means that the trained model still cannot complete the task and cannot meet the actual needs, and further adjustments are needed.

[0073] Based on this, by acquiring multi-dimensional target simulation data, target historical data, and real-time demand data, the first model to be trained in the sequence is trained, and the training test value of the model to be trained on the test set is determined. The training test value is then compared with a preset threshold, so as to determine whether the trained model can complete the task requirements based on the comparison results.

[0074] Based on the feasible implementation of S110 described above, this application further provides a method for determining multiple target values ​​based on the task matching values ​​of each model task vector set and the demand vector, and their respective corresponding model efficiency values, including:

[0075] The user requirement text is mapped to obtain a requirement vector, and the similarity value between the requirement vector and each model task vector is calculated; each model task vector set includes at least one model task vector.

[0076] Based on at least one corresponding similarity value, the largest similarity value is determined as the task matching value of the model task vector set, and based on the initial model corresponding to each model task vector set, the number of parameters and the memory value of the initial model are determined.

[0077] The parameter values ​​and memory values ​​are normalized separately to obtain the target parameter values ​​and target memory values ​​corresponding to each initial model. The target parameter values ​​and target memory values ​​are then weighted and summed to obtain the model efficiency value.

[0078] The target value for each initial model is obtained by weighted summation of the task matching value and the model efficiency value.

[0079] User requirement text refers to the natural language expression entered by the user to describe the task objective that needs to be performed, such as "take the lobster out of the bucket and wash it" or "make a hamburger".

[0080] The initial model refers to a mature embodied intelligent model with specific task execution capabilities stored in a dynamically updated model library on the server. Each model has a clearly defined set of officially supported tasks, a known number of parameters, and a minimum amount of GPU memory required for deployment. By calculating the adaptability of the initial model to user needs and its own efficiency, multiple models with the best overall performance are selected to form a sequence of models to be trained.

[0081] Normalization refers to the process of using linear normalization to convert the parameters and minimum memory required for deployment of different initial models, which are two dimensions that differ significantly, into standardized values ​​with a uniform range.

[0082] Based on this, by acquiring user input information and identifying the user requirement text within it, a mapping is performed to obtain a requirement vector. Further, the model task vector set corresponding to each initial model is determined. This model task vector set typically includes multiple model task vectors. By calculating the similarity value between the requirement vector and each model task vector in the same model task vector set, multiple similarity values ​​are obtained. The largest similarity value is determined as the task matching value for that model task vector set, i.e., the task matching value between the initial model and the task requirement. Based on the parameter count and memory value of each initial model, the normalized target parameter value and target memory value are weighted and summed to obtain the model efficiency value. This weighted sum of the task matching value and the model efficiency value yields the target value corresponding to each initial model.

[0083] Based on the feasible implementation of S110 described above, this application further provides a method for determining the sequence of models to be trained according to each target value, including:

[0084] Sort multiple target values ​​from largest to smallest, and then determine the sorted target values ​​in sequence;

[0085] Based on the initial model corresponding to each sorting target value, determine the sequence of models to be trained.

[0086] The sorting target value refers to the target value determined sequentially after arranging all the target values ​​corresponding to the initial models in descending order of numerical value. This is so that the trainable models that need to be trained can be determined based on the initial models corresponding to each sorted target value. For example, if the sorted target values ​​are 27, 25, 22, 17, 11, then the first three target values, namely 27, 25, and 22, can be selected as the three sorted target values. In practical applications, the number of sorted target values ​​that need to be determined can be determined according to the user's needs.

[0087] Based on this, the target values ​​corresponding to each initial model are determined and sorted. Then, according to the user's actual needs, multiple sorted target values ​​are determined from the sorted target values ​​so as to further determine the sequence of models to be trained based on the initial models corresponding to each sorted target value.

[0088] Based on the feasible implementation of S120 described above, this application further provides a method for generating target simulation data in a simulation environment according to the required action and the required object, including:

[0089] Based on the visual language model, semantic analysis is performed on user requirement text and user requirement image to obtain requirement actions and requirement objects. Then, based on the simulation environment, a simulation scenario corresponding to the requirement actions and requirement objects is established.

[0090] Import the user requirements file into the simulation scene, and add a physics engine and scene noise to obtain the target simulation environment;

[0091] Based on the target simulation environment, generate target simulation data.

[0092] Among them, visual language models are artificial intelligence models that can simultaneously understand visual information in images and linguistic information in text, and achieve cross-modal semantic association. These include CLIP, ViT-BERT, etc., thereby breaking down the information barriers between text and images. They can not only parse the semantic logic in the user's task description, but also identify the visual elements in the requirement image, and thus accurately extract the required actions and the required objects.

[0093] User-defined images are real-world images provided by the user that are relevant to the current task. These images can be viewed from multiple perspectives, such as front, left-wrist, and right-wrist, and are a visual representation of the user's task scenario.

[0094] Semantic parsing refers to the process of performing cross-modal information fusion analysis on user-input requirement text and requirement images based on visual language models, in order to extract requirement actions and requirement objects from text semantic logic and image visual features.

[0095] User requirement files refer to key configuration files provided by the user that are related to the current task and the robot. These include the robot's URDF file, which contains hardware information such as the robot's joint structure, range of motion, and physical properties. It is the core basis for building robot models in simulation scenarios.

[0096] A physics engine is a computational module embedded in a simulation environment to simulate the physical laws of the real world. Scene noise refers to random interference factors automatically introduced into the simulation scene to enhance the diversity and robustness of simulation data. These include visual noise (such as changes in light intensity, image blurring, and shadow shift), physical noise (such as slight fluctuations in friction, gravity deviation, and changes in joint damping), and task noise (such as slight shifts in object position and fluctuations in the timing of action execution).

[0097] Based on this, semantic parsing of user requirement text and user requirement images is performed to obtain requirement actions and requirement objects. Then, according to the simulation environment, a simulation scene corresponding to the requirement actions and requirement objects is established. Furthermore, the user requirement file is imported into the simulation scene, and a physics engine and scene noise are added to obtain the target simulation environment, so as to generate target simulation data based on the target simulation environment.

[0098] Based on the feasible implementation of S120 described above, this application further provides methods for calculating the similarity between the target fusion feature vector and the historical data vector to obtain target historical data, including:

[0099] Based on the dual-tower Transformer architecture, user requirement text, user requirement image, and user requirement file are encoded separately to obtain corresponding requirement feature vectors and model feature vectors. Then, based on the cross-attention mechanism, the requirement feature vectors and model feature vectors are fused to obtain the target fused feature vector.

[0100] Based on multiple historical data, the corresponding historical data vectors are determined, and the similarity between the target fusion feature vector and the historical data vector is calculated to obtain the corresponding similarity value.

[0101] The similarity values ​​that are greater than a preset similarity threshold are determined as target similarity values, and the historical data corresponding to the target similarity values ​​are determined as target historical data.

[0102] Among them, the dual-tower Transformer architecture is a Transformer network structure that includes two independent feature extraction towers. The two towers are responsible for processing different types of input data respectively, and then the outputs are integrated through a specific fusion mechanism. This allows for the collaborative processing of user task-related information and robot ontology information, solving the problem that single feature extraction cannot take into account both task requirements and robot hardware characteristics. Through parallel encoding of the two towers, task semantic features and robot structural features are captured separately, and then fused to generate a joint vector that simultaneously associates the two, ensuring that historical data retrieval matches both the task type and the robot hardware.

[0103] The requirement feature vector refers to a high-dimensional feature vector generated by the task feature extraction tower in the dual-tower Transformer architecture. The input data are user requirement text and user requirement image.

[0104] The model feature vector refers to the high-dimensional feature vector generated by the robot feature extraction tower in the dual-tower Transformer architecture. The input data is the user requirement file, which is obtained after being parsed by the graph neural network (GNN) encoder and can carry the robot's ontological features.

[0105] Cross-attention is an attention mechanism used to fuse feature vectors from two different sources. By calculating the attention weights between the two vectors, the features of each vector can pay attention to the key information related to it in the other vector, thereby generating a joint vector that fuses the core features of both vectors.

[0106] A preset similarity threshold is a pre-defined similarity criterion used to filter valid historical data. For example, it can be 0.7, so that only high-quality, highly relevant historical data with similarity values ​​exceeding the preset similarity threshold are allowed to enter the training set.

[0107] Based on this, using a dual-tower Transformer architecture, user requirement text and user requirement images are encoded to obtain corresponding requirement feature vectors, and user requirement files are encoded to obtain model feature vectors. Then, based on a cross-attention mechanism, the requirement feature vectors and model feature vectors are fused to obtain a target fused feature vector. This allows for further similarity calculation between the target fused feature vector and historical data vectors to obtain the similarity values ​​corresponding to each historical data point. Historical data with similarity values ​​greater than a preset similarity threshold are then identified as target historical data.

[0108] Based on the feasible implementation of S130 described above, this application further provides a method for sequentially training and testing the models to be trained in the training model sequence according to the target simulation data, target historical data, and real-time demand data until a training test value greater than a preset threshold is obtained, thus completing the training test, including:

[0109] Acquire real-time demand data, and determine the merged data of target simulation data and target historical data as the first dataset, and the real-time demand data as the second dataset;

[0110] Based on the first dataset, the first model to be trained in the training model sequence is subjected to multiple rounds of training and testing to obtain the validation value after each round of training. Based on the validation value with the largest value and its corresponding model parameters, the parameters of the model to be trained are adjusted.

[0111] Based on the second dataset, the parameters of the model to be trained are adjusted to obtain the target model to be trained, and the target model to be trained is tested to obtain the corresponding training test values.

[0112] Compare the training test value with the preset threshold, and obtain a training test value that is greater than the preset threshold. The training test is then complete.

[0113] The validation value refers to the quantitative metric obtained by evaluating the model's performance on the first validation dataset after each round of training. For example, it can be the validation set loss value, such as classification loss or regression loss. The value of the validation value can characterize the degree of fit of the model on the validation data. The higher the validation value, the better the model fits the validation data and the stronger its generalization ability.

[0114] Model parameters refer to the core learnable parameters in the model to be trained, including the weights and biases of the neural network. Their values ​​can be continuously updated and optimized by the backpropagation algorithm during the training process. By tracking the changes in the validation values ​​of multiple training rounds, the optimal iteration round of model training can be further determined, that is, the actual model parameters corresponding to the maximum validation value. The larger the validation value of this round, the better the model's fitting effect, and the corresponding model parameters are the optimal model parameters.

[0115] Parameter tuning refers to the process of updating model parameters by optimizing algorithms, such as gradient descent, based on the model's performance during training.

[0116] Based on this, by merging the target simulation data and the target historical data (i.e., data fusion), a first dataset is obtained. In practical applications, the first dataset can be divided into a first training dataset and a first validation dataset. The first training dataset is used for model learning, and the first validation dataset is used to determine the optimal parameters of the model. Thus, the first round of model training is completed based on the target simulation data and the target historical data. The required real-time data is determined as the second dataset, and the model is trained in the second round in the same way, so as to integrate multi-dimensional data and achieve accurate model training.

[0117] Based on the feasible implementation of S130 described above, this application further provides a method for comparing training test values ​​with a preset threshold to obtain training test values ​​greater than the preset threshold, thereby completing the training test, including:

[0118] Compare the training test values ​​with the preset threshold to obtain the corresponding comparison results;

[0119] If the comparison result shows that the training test value is greater than the preset threshold, then the training test is completed, and the first model to be trained is determined as the target requirement model.

[0120] If the comparison result shows that the training test value is not greater than the preset threshold, then based on the sequence of models to be trained, the next model to be trained and its corresponding training test value are determined sequentially. When all training test values ​​are not greater than the preset threshold, a prompt message is output. Based on the prompt message, augmented training data is obtained, and based on the augmented training data, the models to be trained are trained and tested sequentially until a training test value greater than the preset threshold and its corresponding target requirement model are obtained.

[0121] Among them, the target requirement model refers to the intelligent robot model that is generated after training based on multi-dimensional training data and can accurately complete the current user's task requirements.

[0122] The prompt message refers to the targeted prompt content generated when the training and test values ​​corresponding to all models in the training model sequence do not exceed the preset threshold, which is used to clearly inform the user that specific types of data need to be supplemented. For example, when the model has a high failure rate in the "bucket edge grabbing" scenario, the prompt message is "Please provide more grabbing data of lobsters located at the bucket edge", emphasizing that it is to supplement the data of lobsters at the bucket edge, rather than to supplement general data, such as the conventional and fixed requirement of "please supplement data".

[0123] Enhanced training data refers to the process where, after two stages of training, all candidate models in the training model sequence have not exceeded a preset threshold, users supplement their training data with high-quality real data to strengthen the model's weak points, based on targeted prompts from the system. This data typically focuses on specific scenarios, actions, or data types where the model fails to meet the requirements. For example, if the model has a high failure rate in the "grabbing at the edge of the bucket" scenario, the user can supplement the data with "grabbing data of lobsters at the edge of the bucket," which has the same data type as the initially uploaded real-time data and precisely matches the model's performance gap.

[0124] Based on this, if the training and test values ​​of all models in the training model sequence are not greater than the preset threshold, the corresponding prompt information can be output to clearly inform the user that specific types of data need to be supplemented. The user-supplemented data can then be added to the existing data. Based on the supplemented and enhanced data, the training and testing of the models in the training model sequence can be repeated in the same way as before, until a model with a corresponding training and test value greater than the preset threshold appears, and this model is determined to be the target model.

[0125] Please refer to Figure 2 , Figure 2 A schematic diagram of the framework of a robot model training method provided in this application embodiment; as shown Figure 2 As shown, the user uploads the robot configuration file, task description, a small amount of real data, and accuracy requirements. The system first adaptively matches and selects the top-3 models with the highest fit, prioritizing the model with the highest fit. Then, it automatically builds a simulation environment to generate data and retrieves and fuses historical data. After training the model through a two-stage process, the accuracy is evaluated. If the accuracy meets the requirements, the final model is output; otherwise, the next suitable model in the sequence is tried. If there are no candidate models, the user is asked to supplement data. The process is repeated after the user submits new data. In this way, the system can generate robot models that meet user needs with low threshold and low cost through adaptive model selection, automatic data generation and reuse, and iterative training and verification. At the same time, the data complementation mechanism ensures that the final performance of the model meets the standards.

[0126] Based on the above steps, it can be seen that this application determines the corresponding training model sequence by determining the task matching value and corresponding model efficiency value of each model task vector set and demand vector, and calculating the target value. This effectively solves the problem of traditional solutions lacking an adaptive model selection mechanism and having difficulty matching and adapting models. Furthermore, based on the demand actions and demand objects, it automatically generates noisy target simulation data in the simulation environment, and generates target fusion feature vectors through a dual-tower Transformer architecture. It also retrieves target historical data by combining similarity with historical data vectors, avoiding the cumbersome process of manual configuration for simulation data generation, lowering the technical threshold, improving the richness and realism of training data, and enhancing the utilization rate of historical data and model generalization ability. Based on target simulation... The system trains the first model to be trained through a two-stage training process, combining historical data, target data, and real-time demand data. It compares the training test values ​​with preset thresholds, achieving precise model optimization based on a small amount of real data and significantly reducing data collection costs. If the training test value exceeds the preset threshold, the target demand model is directly determined; otherwise, the system tries different models sequentially. Finally, by combining all training test values, preset thresholds, and precise data supplementary information, the target demand model is determined. This solves the technical problems of high data costs, high technical barriers, poor model adaptability, weak generalization ability, and insufficient reuse of historical data in traditional robot model training. It improves the generalization of robot model training, expands the practical application scenarios of the model, and enhances training efficiency and accuracy.

[0127] Figure 3 This is a schematic diagram of the structure of a robot model training device provided in an embodiment of this application. Figure 3 As shown, the robot model training device includes: a sequence determination module, a calculation module, a comparison module, a model determination module, and a training test value determination module; wherein:

[0128] The sequence determination module is used to determine multiple target values ​​based on the task matching values ​​of each model task vector set and the demand vector, as well as their respective model efficiency values, and to determine the sequence of models to be trained based on each target value.

[0129] The calculation module is used to generate target simulation data in the simulation environment based on the required actions and the required objects, and to calculate the similarity between the target fused feature vector and the historical data vector to obtain the target historical data.

[0130] The training and testing module is used to sequentially train and test the models to be trained in the training model sequence based on the target simulation data, target historical data, and real-time data required, until a training and testing value greater than a preset threshold is obtained, at which point the training and testing is completed.

[0131] In this embodiment of the application, the sequence determination module can also be specifically used for:

[0132] The user requirement text is mapped to obtain a requirement vector, and the similarity value between the requirement vector and each model task vector is calculated; each model task vector set includes at least one model task vector.

[0133] Based on at least one corresponding similarity value, the largest similarity value is determined as the task matching value of the model task vector set, and based on the initial model corresponding to each model task vector set, the number of parameters and the memory value of the initial model are determined.

[0134] The parameter values ​​and memory values ​​are normalized separately to obtain the target parameter values ​​and target memory values ​​corresponding to each initial model. The target parameter values ​​and target memory values ​​are then weighted and summed to obtain the model efficiency value.

[0135] The target value for each initial model is obtained by weighted summation of the task matching value and the model efficiency value.

[0136] In this embodiment of the application, the calculation module can also be specifically used for:

[0137] Sort multiple target values ​​from largest to smallest, and then determine the sorted target values ​​in sequence;

[0138] Based on the initial model corresponding to each sorting target value, determine the sequence of models to be trained.

[0139] In this embodiment of the application, the sequence determination module can also be specifically used for:

[0140] Based on the visual language model, semantic analysis is performed on user requirement text and user requirement image to obtain requirement actions and requirement objects. Then, based on the simulation environment, a simulation scenario corresponding to the requirement actions and requirement objects is established.

[0141] Import the user requirements file into the simulation scene, and add a physics engine and scene noise to obtain the target simulation environment;

[0142] Based on the target simulation environment, generate target simulation data.

[0143] In this embodiment of the application, the calculation module can also be specifically used for:

[0144] Based on the dual-tower Transformer architecture, user requirement text, user requirement image, and user requirement file are encoded separately to obtain corresponding requirement feature vectors and model feature vectors. Then, based on the cross-attention mechanism, the requirement feature vectors and model feature vectors are fused to obtain the target fused feature vector.

[0145] Based on multiple historical data, the corresponding historical data vectors are determined, and the similarity between the target fusion feature vector and the historical data vector is calculated to obtain the corresponding similarity value.

[0146] The similarity values ​​that are greater than a preset similarity threshold are determined as target similarity values, and the historical data corresponding to the target similarity values ​​are determined as target historical data.

[0147] In this embodiment of the application, the training and testing module can also be specifically used for:

[0148] Acquire real-time demand data, and determine the merged data of target simulation data and target historical data as the first dataset, and the real-time demand data as the second dataset;

[0149] Based on the first dataset, the first model to be trained in the training model sequence is subjected to multiple rounds of training and testing to obtain the validation value after each round of training. Based on the validation value with the largest value and its corresponding model parameters, the parameters of the model to be trained are adjusted.

[0150] Based on the second dataset, the parameters of the model to be trained are adjusted to obtain the target model to be trained, and the target model to be trained is tested to obtain the corresponding training test values.

[0151] Compare the training test value with the preset threshold, and obtain a training test value that is greater than the preset threshold. The training test is then complete.

[0152] In this embodiment of the application, the training and testing module can also be specifically used for:

[0153] Compare the training test values ​​with the preset threshold to obtain the corresponding comparison results;

[0154] If the comparison result shows that the training test value is greater than the preset threshold, then the training test is completed, and the first model to be trained is determined as the target requirement model.

[0155] If the comparison result shows that the training test value is not greater than the preset threshold, then based on the sequence of models to be trained, the next model to be trained and its corresponding training test value are determined sequentially. When all training test values ​​are not greater than the preset threshold, a prompt message is output. Based on the prompt message, augmented training data is obtained, and based on the augmented training data, the models to be trained are trained and tested sequentially until a training test value greater than the preset threshold and its corresponding target requirement model are obtained.

[0156] Figure 4 This is a schematic diagram of the structure of an apparatus for performing a robot model training method according to an embodiment of this application. Figure 4 As shown, the device includes:

[0157] The device may include one or more processors with processing cores, one or more computer-readable storage media such as memory, communication components, etc. The processor, memory, and communication components are connected via a bus.

[0158] In the specific implementation process, at least one processor executes computer execution instructions stored in memory, causing at least one processor to execute the robot model training method described above.

[0159] The specific implementation process of the processor can be found in the above method embodiments, and its implementation principle and technical effect are similar, so it will not be repeated here.

[0160] Furthermore, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0161] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0162] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0163] In some embodiments, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps in any of the above-described robot model training methods.

[0164] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0165] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0166] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of program codes, which can be loaded by a processor to execute the steps in any of the robot model training methods provided in embodiments of this application.

[0167] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0168] According to one aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium.

[0169] Since the instructions stored in the storage medium can execute the steps in any of the robot model training methods provided in the embodiments of this application, the beneficial effects that any of the robot model training methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0170] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the appended claims.

[0171] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A robot model training method characterized by comprising: The method comprises: According to the task matching value of each model task vector set and the demand vector, and the respective corresponding model efficiency value, a plurality of target values are determined, and according to each of the target values, a sequence of to-be-trained models is determined; the target value refers to the comprehensive score corresponding to each candidate model; According to the demand action and the demand object, target simulation data is generated in a simulation environment, and the similarity between the target fusion feature vector and the historical data vector is calculated to obtain target historical data; the target fusion feature vector refers to a high-dimensional vector generated by fusing task features and robot ontology features; Wherein, according to the demand action and the demand object, the target simulation data is generated in the simulation environment, comprising: According to the visual language model, the semantic analysis of the user demand text and the user demand image is carried out to obtain the demand action and the demand object, and the simulation scene corresponding to the demand action and the demand object is established according to the simulation environment; The user demand file is imported into the simulation scene, and a physical engine and scene noise are added to obtain a target simulation environment; According to the target simulation environment, the target simulation data is generated; The calculation of the similarity between the target fusion feature vector and the historical data vector to obtain the target historical data comprises: Based on the double-tower Transformer architecture, the user demand text and the user demand image, and the user demand file are encoded respectively to obtain corresponding demand feature vectors and model feature vectors, and based on the cross-attention mechanism, the demand feature vectors and the model feature vectors are fused to obtain the target fusion feature vector; According to a plurality of historical data, the respective corresponding historical data vectors are determined, and the similarity between the target fusion feature vector and the historical data vector is calculated to obtain the corresponding similarity value; The similarity value greater than the preset similarity threshold value is determined as the target similarity value, and the historical data corresponding to the target similarity value is determined as the target historical data; According to the target simulation data, the target historical data and the demand real-time data, the to-be-trained models in the sequence of to-be-trained models are sequentially trained and tested until a training test value greater than a preset threshold value is obtained, and the training and testing are completed.

2. The method of claim 1, wherein, According to the task matching value of each model task vector set and the demand vector, and the respective corresponding model efficiency value, a plurality of target values are determined, comprising: The user demand text is mapped to obtain the demand vector, and the similarity between the demand vector and each model task vector is calculated; each of the model task vector set comprises at least one model task vector; According to the corresponding at least one similarity value, the similarity value with the maximum value is determined as the task matching value of the model task vector set, and the parameter quantity and the video memory value of the initial model corresponding to each of the model task vector set are determined; The parameter quantity and the video memory value are normalized respectively to obtain the target parameter value and the target video memory value corresponding to each of the initial models, and the target parameter value and the target video memory value are weighted and summed to obtain the model efficiency value; The task matching value and the model efficiency value are weighted and summed to obtain the target value corresponding to each initial model.

3. The method of claim 1, wherein, The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined.

4. The method of claim 1, wherein, The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined.

5. The method of claim 4, wherein, The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined.

6. A robot model training apparatus characterized by comprising: The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The target value corresponding to each initial model is determined. The computing module is configured to generate target simulation data in a simulation environment according to a demand action and a demand object, and calculate the similarity between a target fusion feature vector and a historical data vector to obtain target historical data. The target fusion feature vector refers to a fusion task feature and a robot ontology feature, and is a high-dimensional vector. The target simulation data is generated in the simulation environment according to the demand action and the demand object, including: According to a visual language model, the user demand text and the user demand image are semantically analyzed to obtain the demand action and the demand object, and a simulation scene corresponding to the demand action and the demand object is established according to the simulation environment; The user demand file is imported into the simulation scene, and a physical engine and scene noise are added to obtain a target simulation environment; The target simulation data is generated according to the target simulation environment; The similarity between the target fusion feature vector and the historical data vector is calculated to obtain the target historical data, including: Based on a double-tower Transformer architecture, the user demand text and the user demand image, and the user demand file are encoded to obtain corresponding demand feature vectors and model feature vectors, and the demand feature vectors and the model feature vectors are fused based on a cross-attention mechanism to obtain the target fusion feature vector; According to a plurality of historical data, the historical data vectors corresponding to the historical data are determined, and the target fusion feature vector and the historical data vectors are similarity calculated to obtain corresponding similarity values; The similarity values greater than a preset similarity threshold value are determined as target similarity values, and the historical data corresponding to the target similarity values are determined as the target historical data; The training test module is configured to sequentially train and test the to-be-trained models in the to-be-trained model sequence according to the target simulation data, the target historical data and demand real-time data, until a training test value greater than a preset threshold value is obtained, and the training test is completed.

7. A computer device, comprising: It includes: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The program code is stored in the computer readable storage medium and can be called and executed by the processor to execute the method of any one of claims 1 to 5.

Images