Model strengthening fine tuning method, device and equipment and storage medium thereof

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a model enhancement fine tuning method, device and equipment and a storage medium thereof. Inputting the model strategy into the constructed target model, and performing offline initial training on the target model to obtain the target model after model strategy initialization; obtaining task guidance data collected in real time; and inputting into the target model after model strategy initialization, and carrying out on-line intensified training to obtain the target model after model strategy fine adjustment. Firstly, model initial training is carried out in an offline mode to obtain an initialized model, and then online reinforcement learning training is carried out in combination with actually acquired data, so that the finally trained model is more in line with an actual application scene. The method is specifically applied to the field of medical services, such as micro-robot surgery or financial services, for example, a robot carries out physical credit card making, and it is guaranteed that the finally trained model is fully combined with a large amount of task operation real operation data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and is applied to the scenario of model enhancement and fine-tuning. It relates to a model enhancement and fine-tuning method, device, equipment and its storage medium. Background Technology

[0002] Currently, fine-tuning of visual-language-action (VLA) models primarily relies on supervised learning methods, such as behavior cloning, trained using human demonstration data. While this approach is simple and straightforward, its performance is heavily dependent on the quality and quantity of the demonstration data.

[0003] Visual-Language-Action (VLA) models are primarily deployed as operation guidance models in robotic robots to replace manual tasks. With the rapid development of fintech and healthcare businesses, more and more manual operations are being replaced by intelligent robotic robots. Examples include intelligent credit card issuing robots in the financial sector and intelligent surgical robots in the healthcare sector. A key challenge is how to efficiently and effectively train operation guidance models for these robots, moving beyond the current reliance on supervised learning methods. Summary of the Invention

[0004] The purpose of this application is to propose a model enhancement and fine-tuning method, apparatus, device and storage medium, so as to achieve reasonable and efficient training of an operation guidance model for an operating robot in an unsupervised learning mode.

[0005] Firstly, embodiments of this application provide a model enhancement and fine-tuning method, which employs the following technical solution:

[0006] A model enhancement and fine-tuning method includes the following steps:

[0007] Obtain the target number of task demonstration data;

[0008] The task demonstration data is input into the constructed target model, and the target model is initially trained offline to obtain the target model after the model policy is initialized.

[0009] Obtain real-time task guidance data;

[0010] The task guidance data is input into the target model after the model policy is initialized, and the target model is subjected to online reinforcement training to obtain the target model after the model policy is fine-tuned.

[0011] Secondly, this application also provides a model enhancement and fine-tuning device, which adopts the following technical solution:

[0012] A model enhancement and fine-tuning device, comprising:

[0013] The task demonstration data acquisition module is used to acquire a target number of task demonstration data.

[0014] The offline initial training module is used to input the task demonstration data into the constructed target model, perform offline initial training on the target model, and obtain the target model after model policy initialization.

[0015] The online task guidance data acquisition module is used to acquire real-time task guidance data.

[0016] The online reinforcement training module is used to input the task guidance data into the target model after the model policy is initialized, and to perform online reinforcement training on the target model to obtain the target model after the model policy is fine-tuned.

[0017] Thirdly, embodiments of this application also provide a computer device that adopts the technical solution described below:

[0018] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the model enhancement fine-tuning method described above.

[0019] Fourthly, embodiments of this application also provide a computer-readable storage medium, which adopts the technical solutions described below:

[0020] A computer-readable storage medium storing computer-readable instructions that, when executed by a processor, implement the steps of the model enhancement fine-tuning method described above.

[0021] Compared with the prior art, the embodiments of this application have the following main advantages:

[0022] The model enhancement and fine-tuning method described in this application involves acquiring a target amount of task demonstration data; inputting this data into a constructed target model; performing offline initial training on the target model to obtain a target model with initialized model policies; acquiring real-time task guidance data; inputting this data into the target model with initialized model policies; and performing online reinforcement training to obtain a target model with fine-tuned model policies. First, the model is initially trained offline to obtain an initialized model. Then, online reinforcement learning training is performed using actual collected data to optimize the model, making the final trained model more suitable for real-world application scenarios. Applying this model enhancement and fine-tuning method to the medical field, such as micro-robotic surgery, or the financial field, such as robots producing physical credit cards, ensures that the final trained model fully incorporates a large amount of real-world task operation data. Attached Figure Description

[0023] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0025] Figure 2 This is a flowchart of an embodiment of a model enhancement fine-tuning method according to this application;

[0026] Figure 3 This is a flowchart of a specific embodiment of the post-processing of task demonstration data in the model enhancement and fine-tuning method described in this application;

[0027] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 202 shown;

[0028] Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 203 shown;

[0029] Figure 6 This is a flowchart of a specific embodiment of the model enhancement and fine-tuning method described in this application, which involves filtering and statistically analyzing task guidance data;

[0030] Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 204 shown;

[0031] Figure 8 yes Figure 7 A flowchart of a specific embodiment of step 703 shown;

[0032] Figure 9 This is a schematic diagram of one embodiment of a model enhancement and fine-tuning device according to this application;

[0033] Figure 10 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0035] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0036] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0037] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0038] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0039] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptop computer 1011, tablet computer 1012 or mobile phone 1013, terminal device 101 can also be e-book reader, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer and desktop computer, etc.

[0040] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0041] It should be noted that the model enhancement and fine-tuning method provided in this application embodiment is generally executed by a server, and correspondingly, a model enhancement and fine-tuning device is generally set in the server.

[0042] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0043] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of a model enhancement fine-tuning method according to this application. The model enhancement fine-tuning method includes the following steps:

[0044] Step 201: Obtain the target number of task demonstration data.

[0045] Specifically, the task demonstration data includes pre-designed animated task simulation demonstration data, which is different from the actual task operation data collected.

[0046] In this embodiment, the number of task demonstration data to be used is not too large, since the target model will be trained offline using the task demonstration data. In order to ensure the efficiency of offline initialization training, the number of targets should be as small as possible.

[0047] Step 202: Input the task demonstration data into the constructed target model, perform offline initial training on the target model, and obtain the target model after model policy initialization.

[0048] In this embodiment, the target model includes the VLA model, short for Vision-Language-Motion Big Data Model, a multimodal artificial intelligence model that integrates visual perception, language understanding, and action generation capabilities. It can handle images, text instructions, and operational tasks in complex scenes, forming a closed loop from perception to decision-making to execution. For example, the VLA model can understand user-provided natural language instructions (such as "grab the raw materials on the workbench"), analyze visual information (such as the position and features of the raw materials in the scene), and generate corresponding operation plans (such as the gripping path of the robotic arm). This model is widely used in robot control, industrial automation, and other fields, achieving natural interaction and efficient task execution capabilities through a unified multimodal processing framework.

[0049] Specifically, by utilizing the aforementioned limited task demonstration data—that is, a small amount of pre-designed animated task simulation data—the constructed VLA model is subjected to offline initialization training, enabling the initially trained VLA model to possess the corresponding task execution capabilities. The resulting target model after model policy initialization is the VLA model with initialized model parameters.

[0050] Step 203: Obtain real-time task guidance data.

[0051] Specifically, due to the pre-designed animated task simulation data, in complex task processing scenarios, there will inevitably be certain differences compared with the actual manual processing methods, both in terms of execution steps and execution speed. Therefore, the VLA model trained solely through the above offline methods will be too mechanical and have considerable room for optimization.

[0052] In this embodiment, the acquisition of real-time task guidance data can be achieved by video recording or camera tracking to obtain actual operation videos of the operators; the task guidance data is then filtered out based on the actual operation videos.

[0053] Specifically, by acquiring real-time online videos of operators performing their tasks, task guidance data is obtained, which can then be used to optimize and adjust the VLA model that has been trained offline.

[0054] Step 204: Input the task guidance data into the target model after the model strategy is initialized, and perform online reinforcement training on the target model to obtain the target model after the model strategy is fine-tuned.

[0055] Specifically, the actual task operation videos of the operators, obtained through video recording or camera tracking, are input into the VLA model after the model strategy is initialized. By using the actual task operation videos collected one after another, the VLA model can be continuously optimized and fine-tuned, thereby gradually optimizing the model (strategy) parameters and obtaining a gradually improving VLA model.

[0056] In this application, firstly, a small amount of virtual task operation data is used for offline initialization training of the VLA model to obtain a preliminary usable VLA model. Then, an online real-time acquisition method is used to collect real task operation data. Based on this real operation data, the model parameters of the VLA model are fine-tuned and optimized step by step. This avoids overly complex model initialization and fully incorporates real-time task operation data for model fine-tuning and improvement, ensuring that the final fine-tuned VLA model fully incorporates a large amount of real task operation data. In the medical field, the aforementioned model enhancement and fine-tuning method can be applied to the learning process of robots replacing humans in simple surgeries. First, a preliminary operating robot is trained offline using virtual data. Then, based on a large amount of pre-collected actual surgical videos or data, the pre-trained operating robot is optimized and fine-tuned to obtain a more complete operating robot. In the financial technology field, for example, in the actual credit card manufacturing process, the aforementioned model enhancement and fine-tuning method can be applied to the learning process of robots replacing humans in credit card manufacturing. First, a preliminary operating robot is trained offline using virtual data. Then, based on a large amount of pre-collected actual human card manufacturing videos or data, the pre-trained operating robot is optimized and fine-tuned to obtain a more complete credit card manufacturing robot.

[0057] In this embodiment, a target amount of task demonstration data is acquired and input into the constructed target model for offline initial training to obtain a target model with initialized model strategy. Real-time task guidance data is then acquired and input into the target model with initialized model strategy for online reinforcement training to obtain a target model with fine-tuned model strategy. First, the model is initially trained offline to obtain an initialized model. Then, online reinforcement learning training is performed using actual collected data to optimize the model, making the final trained model more suitable for real-world application scenarios. This model reinforcement and fine-tuning method can be applied to the medical field, such as micro-robotic surgery, or the financial field, such as robots producing physical credit cards, ensuring that the final trained model fully incorporates a large amount of real-world task operation data.

[0058] Continue to refer to Figure 3In some optional implementations, after step 201, the method further includes a step of post-processing the task demonstration data. Figure 3 This is a flowchart of a specific embodiment of the model enhancement and fine-tuning method described in this application, which involves post-processing task demonstration data, and includes the following steps:

[0059] Step 301: Configure natural language instructions for the task demonstration data;

[0060] Specifically, based on the operational actions involved in the task demonstration data, corresponding natural language commands are set for each operational action;

[0061] For example, the specific process of credit card production involves a series of actions, such as placing the raw materials on the cutting table, cutting them according to specifications, pressing the card number sequence after cutting, spraying the surface according to different card types, and drying after spraying. Corresponding natural language instructions are set for each of these actions.

[0062] Step 302: Extract visual information from the task demonstration data to extract the visual information corresponding to each of the task demonstration data.

[0063] Specifically, based on the operational actions involved in the task demonstration data, visual information is extracted from the task demonstration data to identify the visual position changes of the target object in the scene caused by the operational actions, as the target object changes from the previous action to the next action.

[0064] For example, in the specific card production process of the credit card mentioned above, between the two actions of cutting according to the specifications and pressing the card number sequence after cutting, there must be a change in the position of the target credit card from the cutting table to the pressing table, which will cause a change in visual position information.

[0065] Step 303: Construct the correspondence between natural language instructions and visual information.

[0066] By constructing a correspondence between natural language instructions and visual information, we can provide more precise operational control basis when performing offline initialization training of the VLA model.

[0067] Continue to refer to Figure 4 , Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 202 shown includes the following steps:

[0068] Step 401: Identify the natural language instructions corresponding to the current task demonstration data;

[0069] Step 402: Based on the correspondence between natural language instructions and visual information, filter out the visual information corresponding to the current task demonstration data;

[0070] Specifically, by identifying the natural language instructions contained in each task demonstration data, the visual information in the task demonstration data corresponding to all natural language instructions can be determined.

[0071] Step 403: Using the natural language understanding component in the completed VLA model, the natural language instructions are understood to obtain the target task intent of the current task demonstration data;

[0072] Specifically, the natural language understanding component in the VLA model is used to understand all natural language instructions and parse out the target task intent contained in each task demonstration data.

[0073] Step 404: Using the visual information analysis component in the completed VLA model, analyze the visual information to determine the task processing scenario of the current task demonstration data and all objects contained in the task processing scenario.

[0074] Specifically, by using the visual information analysis component in the VLA model, the specific task processing scenario and all objects contained in each processing scenario are analyzed.

[0075] Step 405: Based on the target task intent, identify the target task object and the interfering task object from all the objects;

[0076] Specifically, based on the task intent corresponding to each step, the task object (target task object) operated by each step of the task action and the non-operational background object (interference task object) are identified.

[0077] Step 406: Generate an operation plan for the target task object based on the target task intent;

[0078] Specifically, the operation plan refers to the sequential execution of each task action to obtain a complete execution chain.

[0079] Step 407: Establish the association and mapping relationship between task demonstration data and operation plan;

[0080] Specifically, it involves generating object mapping relationships between task demonstration data and execution chains.

[0081] Step 408: Based on the operation plans corresponding to all task demonstration data, perform offline initial training on the VLA model to obtain the VLA model after model policy initialization.

[0082] Specifically, the VLA model is initially trained offline using the complete execution chain corresponding to all task demonstration data to obtain the VLA model after model policy initialization, that is, the VLA model after model parameter initialization.

[0083] Through offline initial training, a VLA model trained with a small amount of virtual data is obtained, reaching a preliminary usable level. This avoids using too much data during initial training, thus improving the efficiency of initial training.

[0084] Continue to refer to Figure 5 , Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 203 shown includes the following steps:

[0085] Step 501: Based on the preset acquisition camera, real-time acquisition video of each step of the operation change of the target operator when performing the target operation, wherein the video of each step of the operation change is used to jointly record the continuous change process of the operated object in the target operation scenario.

[0086] Specifically, in the target operation scenario, a high-magnification camera acquisition (head) component capable of capturing and integrating the operation change process is installed. According to the high-magnification camera acquisition (head) component, video of each step of the operation change of the target operator is captured in real time. The high-magnification camera acquisition (head) component can be a single acquisition camera or an acquisition phase array component composed of multiple acquisition cameras, depending on actual needs.

[0087] Step 502: Summarize the video of each step of operation change according to the acquisition time and operation steps, and mark each summarized video data as task guidance data.

[0088] By collecting real-time video of each step of the target operation by the operator, and summarizing each step of the video according to the collection time and operation steps, the data is compiled and marked as task guidance data. This allows for the use of real operation videos as a basis for model improvement when the VLA model is subsequently improved after offline initial training.

[0089] Continue to refer to Figure 6 In some optional implementations, after step 203, the method further includes a step of filtering and statistically analyzing the task guidance data. Figure 6 This is a flowchart of a specific embodiment of the model enhancement and fine-tuning method described in this application, which involves filtering and statistically analyzing task guidance data, and includes the following steps:

[0090] Step 601: According to the preset annotation strategy, perform usability annotation on the real-time collected task guidance data;

[0091] Specifically, since the task guidance data is real-time video of each step of the target operator's operation, there may be issues with whether the data is standard or usable. For example, if a certain step is not standard, it is necessary to label the real-time task guidance data for usability to ensure that the video data subsequently used and input into the VLA model conforms to the specified standards.

[0092] Step 602: Count the number of items in the real-time collected task guidance data, and use this as the first item count;

[0093] Step 603: Count the number of entries that are labeled as usable task guidance data, and use this count as the second number of entries;

[0094] Specifically, by statistically analyzing the total number of task guidance data entries collected and the number of task guidance data entries whose labels were deemed usable, the amount of data input into the VLA model at one time was ensured to be sufficient.

[0095] Step 604: By comparison, determine whether the number of the first entries exceeds a preset threshold for the number of the first entries;

[0096] Step 605: By comparison, determine whether the number of the second entries exceeds the preset threshold for the number of the second entries;

[0097] Step 606: If the number of the first entries exceeds the first entry number threshold and the number of the second entries exceeds the second entry number threshold, then all task guidance data with usable annotation results are added to the target preset set to generate a task guidance dataset.

[0098] Of course, if the number of the first items does not exceed the first item number threshold, or if the number of the second items does not exceed the second item number threshold, task guidance data will be continuously collected in real time.

[0099] The above is merely a summary and organization of input data for subsequent training of the VLA model initially trained offline. The purpose is to ensure that the task guidance data input at one time is sufficient. Of course, the above organization method can also be omitted. For example, each time a piece of usable task guidance data is obtained, it can be immediately sent to the current VLA model for incremental learning and training, thereby updating the VLA model in more detail each time.

[0100] Continue to refer to Figure 7 , Figure 7 yes Figure 2A flowchart of a specific embodiment of step 204 shown includes the following steps:

[0101] Step 701: Based on the VLA model initialized by the model strategy, identify the actual task operation plan contained in the task guidance data;

[0102] Specifically, since the VLA model initialized by the model strategy has initially acquired the ability to recognize task operation plans, task intentions, and task execution actions, the actual task operation plan contained in the task guidance data can be directly identified using the VLA model initialized by the model strategy.

[0103] Step 702: Based on the actual operation plan of the task, generate model strategy improvement data corresponding to the task guidance data;

[0104] Specifically, the model strategy improvement data is generated by comparing the task guidance data and the task demonstration data in actual operation scenarios and steps through model processing. For example, the operation time interval between the second and third operation actions in the task demonstration data is 5 seconds, while the operation time interval between the second and third operation actions in the task guidance data is 8 seconds. This 8 seconds is used as an improvement point in the model strategy improvement data.

[0105] Step 703: Improve the data using the model strategy and perform model strategy enhancement and improvement on the VLA model after the model strategy initialization using an online reinforcement training method.

[0106] Continue to refer to Figure 8 , Figure 8 yes Figure 7 A flowchart of a specific embodiment of step 703 shown includes the following steps:

[0107] Step 801: According to a preset comparison strategy, continuously compare the changes of the model strategy. The preset comparison strategy includes continuously acquiring the model strategy at the target time point at preset time intervals and comparing the model strategy acquired at the current target time point with the model strategy acquired at the previous target time point.

[0108] Step 802: If the comparison results of the model strategy changes in N consecutive times are all unchanged, then the reinforcement and improvement is completed, where N is a preset positive integer.

[0109] Step 803: Obtain the VLA model that has been enhanced and improved in this step, as the fine-tuned VLA model.

[0110] Specifically, if the model parameters change after N consecutive comparisons of the model strategy (parameters), new task guidance data is collected to perform online reinforcement and improvement training; if all model parameters remain unchanged after N consecutive comparisons of the model strategy (parameters), the VLA model is considered to have been reinforced and improved, and the VLA model corresponding to the current model strategy is obtained as the fine-tuned VLA model.

[0111] In this embodiment, a target amount of task demonstration data is acquired and input into the constructed target model for offline initial training to obtain a target model with initialized model strategy. Real-time task guidance data is then acquired and input into the target model with initialized model strategy for online reinforcement training to obtain a target model with fine-tuned model strategy. First, the model is initially trained offline to obtain an initialized model. Then, online reinforcement learning training is performed using actual collected data to optimize the model, making the final trained model more suitable for real-world application scenarios. This model reinforcement and fine-tuning method can be applied to the medical field, such as micro-robotic surgery, or the financial field, such as robots producing physical credit cards, ensuring that the final trained model fully incorporates a large amount of real-world task operation data.

[0112] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0113] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0114] In this embodiment, a target amount of task demonstration data is acquired and input into the constructed target model for offline initial training to obtain a target model with initialized model strategy. Real-time task guidance data is then acquired and input into the target model with initialized model strategy for online reinforcement training to obtain a target model with fine-tuned model strategy. First, the model is initially trained offline to obtain an initialized model. Then, online reinforcement learning training is performed using actual collected data to optimize the model, making the final trained model more suitable for real-world application scenarios. This model reinforcement and fine-tuning method can be applied to the medical field, such as micro-robotic surgery, or the financial field, such as robots producing physical credit cards, ensuring that the final trained model fully incorporates a large amount of real-world task operation data.

[0115] Further reference Figure 9 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a model enhancement fine-tuning device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0116] like Figure 9 As shown, the model enhancement and fine-tuning device 900 described in this embodiment includes: a task demonstration data acquisition module 901, an offline initial training module 902, a task guidance data online acquisition module 903, and an online enhancement training module 904. Wherein:

[0117] The task demonstration data acquisition module 901 is used to acquire a target number of task demonstration data.

[0118] The offline initial training module 902 is used to input the task demonstration data into the constructed target model, perform offline initial training on the target model, and obtain the target model after model policy initialization.

[0119] The online task guidance data acquisition module 903 is used to acquire real-time task guidance data.

[0120] The online reinforcement training module 904 is used to input the task guidance data into the target model after the model policy is initialized, and to perform online reinforcement training on the target model to obtain the target model after the model policy is fine-tuned.

[0121] This application obtains a target amount of task demonstration data; inputs it into a constructed target model, performs offline initial training on the target model to obtain a target model with initialized model strategy; then obtains real-time collected task guidance data; inputs it into the target model with initialized model strategy for online reinforcement training to obtain a target model with fine-tuned model strategy. First, the model is initially trained offline to obtain an initialized model. Then, online reinforcement learning training is performed using actual collected data to optimize the model, making the final trained model more suitable for real-world application scenarios. This model reinforcement and fine-tuning method can be applied to the medical field, such as micro-robotic surgery, or the financial field, such as robots producing physical credit cards, ensuring that the final trained model fully incorporates a large amount of real-world task operation data.

[0122] In this embodiment, the model enhancement and fine-tuning device 900 further includes a natural language instruction configuration module, a visual information extraction module, and a correspondence construction module. Wherein:

[0123] The natural language instruction configuration module is used to configure natural language instructions for the task demonstration data.

[0124] The visual information extraction module is used to extract visual information from the task demonstration data, and extract the visual information corresponding to each of the task demonstration data.

[0125] The mapping construction module is used to construct the mapping relationship between natural language instructions and visual information.

[0126] In this embodiment, the offline initial training module 902 includes a natural language instruction recognition unit, a visual information filtering unit, a target task intent recognition unit, a visual information analysis unit, a task object recognition unit, an operation plan generation unit, an association mapping unit, and an offline initial training unit.

[0127] in:

[0128] The natural language instruction recognition unit is used to identify the natural language instructions corresponding to the current task demonstration data;

[0129] The visual information filtering unit is used to filter out the visual information corresponding to the current task demonstration data based on the correspondence between natural language instructions and visual information.

[0130] The target task intent recognition unit is used to understand the natural language instructions using the natural language understanding component in the constructed VLA model, and obtain the target task intent of the current task demonstration data.

[0131] The visual information analysis unit is used to analyze the visual information using the visual information analysis component in the completed VLA model, and to analyze the task processing scenario of the current task demonstration data and all objects contained in the task processing scenario.

[0132] The task object identification unit is used to identify the target task object and the interfering task object from all the objects according to the target task intent;

[0133] An operation plan generation unit is used to generate an operation plan for the target task object based on the target task intent.

[0134] The association mapping unit is used to build the association mapping relationship between task demonstration data and operation plans;

[0135] The offline initial training unit is used to perform offline initial training on the VLA model based on the operation plans corresponding to all task demonstration data, so as to obtain the VLA model after model policy initialization.

[0136] In this embodiment, the online task guidance data acquisition module 903 includes a real-time acquisition unit and a summary and marking unit. Wherein:

[0137] The real-time acquisition unit is used to acquire video of every step of the target operator's operation in real time, based on the preset acquisition camera.

[0138] The summary and marking unit is used to summarize each step of the operation change video according to the acquisition time and operation steps, and mark each summarized video data as task guidance data.

[0139] In this embodiment, the model enhancement and fine-tuning device 900 further includes a usability labeling module, a first statistics module, a second statistics module, a first comparison module, a second comparison module, and a usability data processing module. Wherein:

[0140] The availability labeling module is used to label the real-time collected task guidance data with availability according to a preset labeling strategy.

[0141] The first statistics module is used to count the number of items in the real-time collected task guidance data, which is then used as the first item count.

[0142] The second statistics module is used to count the number of entries whose labeled results are usable task guidance data, which is then used as the second entry count.

[0143] The first comparison module is used to determine whether the number of the first entries exceeds a preset threshold number of the first entries by comparison.

[0144] The second comparison module is used to determine whether the number of the second entries exceeds a preset threshold for the number of the second entries by comparison.

[0145] The availability data processing module is used to add all task guidance data marked as available to the target preset set and generate a task guidance dataset if the number of the first entries exceeds the first entry number threshold and the number of the second entries exceeds the second entry number threshold.

[0146] In this embodiment, the online reinforcement training module 904 includes a task actual operation plan identification unit, a model strategy improvement data generation unit, and an online reinforcement training unit. Wherein:

[0147] The task actual operation plan identification unit is used to identify the task actual operation plan contained in the task guidance data based on the VLA model initialized by the model strategy.

[0148] The model strategy improvement data generation unit is used to generate model strategy improvement data corresponding to the task guidance data based on the actual operation plan of the task.

[0149] The online reinforcement training unit is used to improve the data using the model strategy and to perform model strategy reinforcement and improvement on the VLA model after the model strategy is initialized using an online reinforcement training method.

[0150] In this embodiment, the online reinforcement training unit includes a model policy continuous comparison subunit, a reinforcement improvement completion judgment subunit, and a fine-tuned model acquisition subunit. Wherein:

[0151] The model strategy continuous comparison subunit is used to continuously compare the changes of the model strategy according to a preset comparison strategy. The preset comparison strategy includes continuously acquiring the model strategy at the target time point at a preset time interval, and comparing the model strategy acquired at the current target time point with the model strategy acquired at the previous target time point.

[0152] The reinforcement and improvement completion judgment subunit is used to determine if the model strategy change comparison results are unchanged for N consecutive times, and then the reinforcement and improvement is completed. Here, N is a preset positive integer.

[0153] The fine-tuned model obtains sub-units, which are used to obtain the VLA model completed in this enhancement and improvement, as the fine-tuned VLA model.

[0154] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0155] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0156] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 10 , Figure 10 This is a basic structural block diagram of the computer device in this embodiment.

[0157] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c, which are interconnected via a system bus. It should be noted that... Figure 10 Only a computer device 10 with component memory 10a, processor 10b, and network interface 10c is shown. However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0158] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0159] The memory 10a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10a may be an internal storage unit of the computer device 10, such as the hard disk or memory of the computer device 10. In other embodiments, the memory 10a may also be an external storage device of the computer device 10, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory 10a may include both internal storage units and external storage devices of the computer device 10. In this embodiment, the memory 10a is typically used to store the operating system and various application software installed on the computer device 10, such as computer-readable instructions for a model enhancement fine-tuning method. In addition, the memory 10a can also be used to temporarily store various types of data that have been output or will be output.

[0160] In some embodiments, the processor 10b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 10b is typically used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to execute computer-readable instructions stored in the memory 10a or to process data, for example, to execute computer-readable instructions for the model enhancement fine-tuning method described above.

[0161] The network interface 10c may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 10 and other electronic devices.

[0162] The computer device proposed in this embodiment belongs to the field of artificial intelligence technology and is applied in the scenario of model reinforcement and fine-tuning. This application obtains a target amount of task demonstration data; inputs it into a constructed target model, performs offline initial training on the target model to obtain a target model after model policy initialization; obtains real-time collected task guidance data; inputs it into the target model after model policy initialization, performs online reinforcement training, and obtains a target model after model policy fine-tuning. First, the model is initially trained offline to obtain an initialized model. Then, online reinforcement learning training is performed to optimize the model, making the final trained model more consistent with the actual application scenario. Applying this model reinforcement and fine-tuning method to the medical field, such as micro-robot surgery, or the financial field, such as robots producing physical credit cards, ensures that the final trained model fully incorporates a large amount of real-world task operation data.

[0163] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the model enhancement fine-tuning method described above.

[0164] The computer-readable storage medium proposed in this embodiment belongs to the field of artificial intelligence technology and is applied in model reinforcement and fine-tuning scenarios. This application obtains a target amount of task demonstration data; inputs it into a constructed target model, performs offline initial training on the target model to obtain a target model after model policy initialization; obtains real-time collected task guidance data; inputs it into the target model after model policy initialization, performs online reinforcement training, and obtains a target model after model policy fine-tuning. First, the model is initially trained offline to obtain an initialized model. Then, online reinforcement learning training is performed to optimize the model, making the final trained model more consistent with actual application scenarios. This model reinforcement and fine-tuning method can be applied to the medical field, such as micro-robot surgery, or the financial field, such as robots producing physical credit cards, ensuring that the final trained model fully incorporates a large amount of real-world task operation data.

[0165] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0166] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to make the disclosure of this application more thorough and comprehensive. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

Claims

1. A model enhancement and fine-tuning method, characterized in that, Includes the following steps: Obtain the target number of task demonstration data; The task demonstration data is input into the constructed target model, and the target model is subjected to offline initial training to obtain the target model after model policy initialization. Obtain real-time task guidance data; The task guidance data is input into the target model after the model policy is initialized, and the target model is subjected to online reinforcement training to obtain the target model after the model policy is fine-tuned.

2. The model enhancement and fine-tuning method according to claim 1, characterized in that, After performing the step of obtaining the target amount of task demonstration data, the method further includes: Configure natural language instructions for the task demonstration data; Visual information is extracted from the task demonstration data to extract the visual information corresponding to each of the task demonstration data. Construct a correspondence between natural language instructions and visual information.

3. The model enhancement and fine-tuning method according to claim 2, characterized in that, The target model includes a VLA model. The step of inputting the task demonstration data into the constructed target model and performing offline initial training on the target model to obtain the target model after model policy initialization specifically includes: Identify the natural language commands corresponding to the current task demonstration data; Based on the correspondence between natural language instructions and visual information, the visual information corresponding to the current task demonstration data is filtered out. The natural language understanding component in the completed VLA model is used to understand the natural language instructions and obtain the target task intent of the current task demonstration data; The visual information analysis component in the completed VLA model is used to analyze the visual information, thereby analyzing the task processing scenario of the current task demonstration data and all objects contained in the task processing scenario. Based on the stated target task intent, identify the target task object and the interfering task object from all the objects; Generate an operation plan for the target task object based on the target task intent; Establish a mapping relationship between task demonstration data and operation plans; Based on the operation plans corresponding to all task demonstration data, the VLA model is initially trained offline to obtain the VLA model after model policy initialization.

4. The model enhancement and fine-tuning method according to claim 1, characterized in that, The steps for obtaining real-time collected task guidance data specifically include: Based on the preset acquisition camera, real-time video of every step of the target operator's operation is captured. The video of each step of the operation is summarized as a whole according to the acquisition time and operation steps, and each summarized video data is marked as task guidance data.

5. The model enhancement and fine-tuning method according to any one of claims 1 or 4, characterized in that, After performing the step of acquiring real-time task guidance data, the method further includes: According to the preset annotation strategy, the real-time collected task guidance data is labeled with usability. The number of items in the real-time collected task guidance data is used as the first item count; The number of entries labeled as available task guidance data is used as the second number of entries. By comparison, it is determined whether the number of the first entries exceeds a preset threshold for the number of the first entries; By comparison, it is determined whether the number of the second entries exceeds a preset threshold for the number of the second entries; If the number of the first entries exceeds the first entry number threshold, and the number of the second entries exceeds the second entry number threshold, then all task guidance data with usable annotation results are added to the target preset set to generate a task guidance dataset.

6. The model enhancement and fine-tuning method according to claim 3, characterized in that, The step of inputting the task guidance data into the target model after the model policy is initialized, and performing online reinforcement training on the target model to obtain the target model after the model policy is fine-tuned, specifically includes: Based on the VLA model initialized by the model strategy, identify the actual task operation plan contained in the task guidance data; Based on the actual operation plan of the task, model strategy improvement data corresponding to the task guidance data is generated; The model strategy is used to improve the data, and an online reinforcement training method is adopted to enhance and improve the VLA model after the model strategy is initialized.

7. The model enhancement and fine-tuning method according to claim 6, characterized in that, The step of using the model strategy to improve the data and employing online reinforcement training to enhance and improve the target model after the model strategy initialization includes: According to a preset comparison strategy, the model strategy is continuously compared for changes. The preset comparison strategy includes continuously acquiring the model strategy at the target time point at preset time intervals and comparing the model strategy acquired at the current target time point with the model strategy acquired at the previous target time point for consistency. If the comparison results of N consecutive model strategy changes are all unchanged, then the reinforcement and improvement are completed, where N is a preset positive integer; The VLA model obtained after this enhancement and improvement is used as the fine-tuned VLA model.

8. A model enhancement and fine-tuning device, characterized in that, include: The task demonstration data acquisition module is used to acquire a target number of task demonstration data. The offline initial training module is used to input the task demonstration data into the constructed target model, perform offline initial training on the target model, and obtain the target model after model policy initialization. The online task guidance data acquisition module is used to acquire real-time task guidance data. The online reinforcement training module is used to input the task guidance data into the target model after the model policy is initialized, and to perform online reinforcement training on the target model to obtain the target model after the model policy is fine-tuned.

9. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the model enhancement fine-tuning method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the model enhancement fine-tuning method as described in any one of claims 1 to 7.

Images