Fine adjustment method and system for large model of domain modeling

Abstract

The invention provides a field modeling large model fine tuning method and system, and the method comprises the steps: extracting core elements and main tasks in a field business from the demand description of a business field, starting with the elements and the tasks, defining instruction modes of sub-tasks, and quantifying and evaluating the completeness of an automatic modeling result. And the fine-tuned large model is optimized according to the evaluation result, the completeness of the model is effectively improved by utilizing a fine-tuning-quantification-evaluation-optimization iteration process, and the method can be suitable for field scenes with relatively high reliability requirements.

Description

Technical Field

[0001] The present invention belongs to the technical field of domain modeling and large language models in software development, and specifically relates to a domain modeling large model fine-tuning method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the improvement of computing power and the reduction of storage costs, software-defined technology has developed rapidly and been widely used. Systems in various fields have gradually evolved into large-scale interactive software networks, which has greatly improved flexibility and scalability. However, this has also brought new challenges: non-professional developers need to continuously connect with business personnel to meet professional product requirements and program or refactor large amounts of tedious code, which seriously affects the accuracy and speed of system development.

[0004] Therefore, researchers have introduced automatic code construction technology into the system design and development process in various fields. The most commonly used implementation method is based on the model-driven approach. This method clearly expresses business logic by modeling various field elements and modular system design, and ultimately converts pre-defined abstract models into code, providing an important foundation for enterprises to achieve intelligent and agile transformation and upgrading.

[0005] However, as the demand for software or code construction increases, so does the need for model construction, and the model structure becomes increasingly complex. This complex and specialized modeling process often requires sufficient domain knowledge and experience, relying on professionals to complete it manually or semi-manually. Due to its inefficiency and fixed patterns, this approach cannot adapt to the trend of efficient development of complex systems. The development of artificial intelligence (AI) technology has made automated modeling possible. Automatic modeling based on natural language processing is a common and relatively mature technology. This technology primarily relies on methods such as text analysis and relationship recognition to extract key information (such as entities and relationships) from natural language text (typically requirement text). However, domain modeling often relies on a precise understanding of the business context. Due to the lack of domain-specific knowledge, existing NLP methods, such as Parts of Speech (POS) tagging, Ontology, and the Stanford CoreNLP library, cannot fully cover the domain knowledge required to meet business process requirements when extracting elements such as domain entities and relationships.

[0006] In recent years, large language models (LLMs) such as ChatGPT and Deepseek have developed rapidly. LLMs are generative pre-trained language models that possess powerful semantic understanding capabilities thanks to large-scale data and knowledge accumulation. Through targeted fine-tuning, LLMs can quickly adapt to different scenarios, specific tasks, and requirements. Its capabilities in the field of information extraction allow it to be applied to the automatic construction of domain models. Some researchers have used Prompt engineering to guide large models to generate domain models with complex structures. However, without fine-tuning and optimization, the models constructed by this automatic modeling method have problems such as insufficient completeness (the accuracy and integrity of the model structure and model content cannot be guaranteed), and therefore cannot be applied to domain scenarios with high reliability requirements.

[0007] In summary, existing domain modeling methods lack effective and reliable automation means, and extracting key information and elements from complex requirement texts still relies on a lot of manual work; on the one hand, domain models usually contain multi-level elements and complex relationship networks, and existing methods are difficult to automatically construct these complex structures accurately and completely; on the other hand, automatically generated domain models often have problems with completeness and accuracy, and there is a lack of effective mechanisms for quantitative evaluation and continuous optimization of automatically generated domain models, which makes it difficult to ensure model quality and meet the requirements of practical applications. Summary of the Invention

[0008] In order to solve the above problems, the present invention proposes a method and system for fine-tuning a large model for domain modeling. The present invention extracts the core elements and main tasks in the domain business from the demand description of the business domain, and uses these elements and tasks to define the instruction mode of sub-tasks, and quantifies and evaluates the completeness of the automatic modeling results. The fine-tuned large model is optimized according to the evaluation results, and the iterative process of fine-tuning-quantification-evaluation-optimization is used to effectively improve the completeness of the model, which can be applied to domain scenarios with high reliability requirements.

[0009] According to some embodiments, the present invention adopts the following technical solutions:

[0010] A method for fine-tuning a large domain model includes the following steps:

[0011] Based on the elements and tasks of the target domain, a corresponding abstract model structure is constructed, the abstract model structure is divided into multiple subtasks for extracting elements, and corresponding instruction patterns are established;

[0012] Based on the established instruction model, the elements and relationships between the elements in the requirement text covering various scenarios in the target field are annotated to form an instruction dataset;

[0013] Fine-tune the large model based on the instruction dataset to obtain a large model for domain automatic modeling that is suitable for various element extraction tasks;

[0014] Constructing prompt words for the target domain, which include task descriptions, requirement text for the corresponding scenario, and examples. Based on the prompt words, the domain automatic modeling model is used to extract the elements of each subtask and obtain the inference results.

[0015] The completeness and accuracy of the inference results are quantified, and the parameters of the large model for automatic domain modeling are optimized based on the quantified results.

[0016] The optimized big model can generate a domain model with better completeness and accuracy, and the optimized big model / domain model can be used for information query, question and answer, etc.

[0017] As an optional implementation method, the process of constructing a corresponding abstract model structure based on the elements and tasks of the target domain includes: the abstract model structure includes multiple layers of elements, the parent structure of the entity is the module and the scene, the substructure of the entity is the attribute, event and state, the substructure of the state is the action, and the tasks of the elements of each layer are defined as subtasks of the modeling task.

[0018] As an optional implementation, the process of dividing the abstract model structure into a plurality of subtasks for extracting elements and establishing corresponding instruction patterns includes:

[0019] According to the abstract model structure, seven subtasks are divided. Each subtask constructs a corresponding instruction data annotation mode. The description of each subtask and its instruction mode are as follows:

[0020] For a given requirement text T, it is necessary to identify the domain scenario D to which the text belongs. The instruction annotation pattern of D is: D(T):{domain:D}, where domain is used to define the domain scenario;

[0021] According to the different business and functions, the domain scenario D is divided into multiple modules C1, C2, C3, ..., C m , included in the module set C, the instruction pattern of C is: C(D):{context:[C i ],1≤i≤m}, where context is the configuration context module;

[0022] Each module C i Contains at least one entity, according to the identified module C i Continue to identify the included entities E1, E2, E3, ..., E n , these entities are included in the entity set E(C i ), E(C i ) is:

[0023] E(C i ):{context:C i ,entity:[{name:E j ,namespace:DC i}],1≤j≤n}, where namespace is the representation of the domain scenario and module to which an entity belongs, entity is the entity representation, and name is the name;

[0024] Each entity E j Contains multiple attributes A1, A2, A3, ..., A p , contained in the attribute set A(E j ), attribute A(E j ) instruction mode is: A(E j ): {entity:E j ,cattr:[A x ],1≤x≤p}, cattr is used to define attributes;

[0025] Each entity E j Contains a preceding trigger event EV, the instruction mode of EV is: EV(E j ):{entity:E j ,event:EV}, event is used to define events;

[0026] Each entity E j Contains a set of states S1, S2, S3, ... S q , contained in the state set S(E j ), S(E j ) instruction mode is: S(E j ):{entity:E j ,state:[S y ],1≤y≤q}, state is used to define the state;

[0027] Each state S y Triggered by an event EV, a series of actions will be executed after the trigger, state S y Contains an EV and 0 to r actions AC1, AC2, AC3, ...AC r , the above actions are included in the action set AC(S y ), for state S y , the command mode for triggering events and actions is: AC(S y ):{state:S y ,actions:[{action:ACk ,params:P},preEvent:EV,1≤k≤r], where P is the action AC k The parameter set P1, P2, P3, ... P t , action is used to define actions, and params is used to define parameters.

[0028] As an optional implementation, based on the established instruction pattern, each element and the relationship between the elements in the requirement text covering each scenario in the target field are annotated, and the process of forming the instruction data set includes: annotating multiple data as seed data sets based on the instruction pattern, including annotating each element and the relationship between the elements;

[0029] Various sub-instruction data sets are formed according to the division of subtasks. Each data set contains three fields, namely instructions, input and output. The instructions are used to clarify the functions of the large model and the tasks it needs to complete. They are regulated by the instruction pattern and vary according to the requirements of each subtask in the instruction pattern. They also include a structured description of the output format defined by the instruction pattern, which is used to indicate how the large model presents its extracted model elements; the input is the input of the large model, that is, the requirement text of various domain scenarios; the output is the domain elements extracted by the corresponding subtask, and the final domain model is obtained through information mapping and combination.

[0030] As an optional implementation method, the process of fine-tuning the large model based on the instruction data set to obtain a large model for domain automatic modeling that is suitable for various element extraction tasks includes: annotating the elements of the input text based on the instruction annotation mode of the subtask to obtain an instruction data set containing the requirement text and domain model annotation information, using the instruction data set to fine-tune the parameters of the pre-trained model, incrementally performing fine-tuning on the instruction data corresponding to each subtask in sequence, optimizing the parameters by calculating the loss between the generated output and the standard output, and finally obtaining a large model for domain automatic modeling that is suitable for various element extraction tasks of the model.

[0031] As an optional implementation method, based on the prompt words, the domain automatic modeling model is used to extract the elements of each subtask, and the process of obtaining the inference result includes: constructing prompt words based on real-time input text, scheduling the automatic modeling model to extract model elements of each subtask in turn, the inference result contains the corresponding model elements, and the inference result is parsed, mapped and converted.

[0032] As an optional implementation method, the process of quantifying the completeness and accuracy of the inference results includes: quantifying the completeness and accuracy based on four indicators: coverage of element information itself, coverage of information on relationships between elements, element grammatical accuracy, and element accuracy; using a comprehensive evaluation model based on a random forest regression model, randomly dividing the training data set containing the four indicators and randomly selecting feature subsets to construct a decision tree; and obtaining the final prediction result by fusing multiple prediction results of several decision trees.

[0033] A domain modeling large model fine-tuning system, comprising:

[0034] a subtask construction module configured to construct a corresponding abstract model structure based on the elements and tasks of the target domain, divide the abstract model structure into a plurality of subtasks for extracting elements, and establish corresponding instruction patterns;

[0035] The annotation module is configured to annotate the elements and the relationships between the elements in the requirement text covering various scenarios in the target domain according to the established instruction model to form an instruction data set;

[0036] The fine-tuning module is configured to fine-tune the large model based on the instruction dataset to obtain a large model for domain automatic modeling that is suitable for various element extraction tasks;

[0037] The reasoning module is configured to construct prompt words for the target domain, which include task descriptions, requirement text for corresponding scenarios, and examples. Based on the prompt words, the module uses the domain automatic modeling model to extract elements of each subtask and obtain reasoning results.

[0038] The quantification and optimization module is configured to quantify the completeness and accuracy of the inference results, and optimize the parameters of the domain automatic modeling model based on the quantification results.

[0039] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps in the above method are completed.

[0040] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the above method are completed.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention decomposes complex domain modeling tasks into multiple subtasks and constructs a corresponding instruction data annotation model for each subtask, fully ensuring the large model's ability to understand the domain business during the fine-tuning process; then, through the design of fine-tuning sub-pipelines and automatic modeling sub-pipelines, the domain modeling process is automated, improving modeling efficiency and reducing labor costs; the design of an entity-centric abstract model structure including multi-layer elements can universally describe different domain scenarios, enhancing the applicability and scalability of the method.

[0043] The present invention achieves an objective evaluation of the completeness and accuracy of the automatically generated model by designing quantitative indicators and a comprehensive evaluation model based on the random forest regression model.

[0044] This paper adopts a feedback optimization method based on learning, introduces completeness assessment results and expert annotation feedback as reward signals, and realizes continuous self-optimization of large models for automatic modeling; combined with the construction of prompt words, the accuracy of large model reasoning is further improved; and finally realizes a complete automatic modeling process from large model fine-tuning to reasoning to optimization, ensuring the standardization, consistency, completeness and accuracy of the automatic modeling process.

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0047] Figure 1 A flow chart of a domain modeling large model fine-tuning method according to an embodiment;

[0048] Figure 2 A schematic diagram of an abstract model structure that an instruction mode of an embodiment relies on;

[0049] Figure 3 A flowchart of an automatic modeling pipeline according to an embodiment;

[0050] Figure 4 A schematic diagram of the optimization process of fine-tuning a large model according to an embodiment. DETAILED DESCRIPTION

[0051] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0052] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0053] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0054] In the absence of conflict, the embodiments and features in the embodiments of this application can be combined with each other.

[0055] Example 1

[0056] A domain modeling large model fine-tuning method, such as Figure 1 As shown, the following steps are included:

[0057] Step 1: Based on the elements and tasks of the target domain, a corresponding abstract model structure is constructed, the abstract model structure is divided into multiple subtasks for extracting elements, and a corresponding instruction pattern is established.

[0058] In one embodiment, the abstract model structure includes multiple layers of elements. The parent structure of the entity is module and scene, the substructure of the entity is attribute, event and state, the substructure of the state is action, and the tasks of each layer of elements are defined as subtasks of the modeling task.

[0059] Specifically, seven subtasks are divided according to the abstract model structure. Each subtask constructs a corresponding instruction data annotation mode. The description of each subtask and its instruction mode are as follows:

[0060] like Figure 2 Shown, including:

[0061] a) Domain scene identification: For a given requirement text T, we first need to identify the domain scene D to which the text belongs. The instruction annotation pattern of D is: D(T):{domain:D};

[0062] b) Module division: Divide the domain scenario D into multiple modules C1, C2, C3, ..., C according to different businesses and functions. m , included in the module set C, the instruction pattern of C is: C(D):{context:[C i ],1≤i≤m};

[0063] c) Entity recognition: Each module Ci Each contains at least one entity, and continues to identify the entities E1, E2, E3, ..., E according to the identified module Ci. n , these entities are included in the entity set E(C i ), E(C i ) instruction mode is: E(C i ):{context:C i ,entity:[{name:E j ,namespace:DC i}],1≤j≤n}, where namespace is the representation of the domain scenario and module to which an entity belongs;

[0064] d) Attribute extraction: Each E j Contains multiple attributes A1, A2, A3, ..., A p , contained in the attribute set A(E j ), A(E j ) instruction mode is: A(E j ): {entity:E j ,cattr:[A x ],1≤x≤p};

[0065] e) Event extraction: Each E j Contains a preceding trigger event EV, the instruction mode of EV is: EV(E j ):{entity:E j ,event:EV};

[0066] f) State extraction: Each E j Contains a set of states S1, S2, S3, ... S q , contained in the state set S(E j ), S(E j ) instruction mode is: S(E j ):{entity:E j ,state:[S y ],1≤y≤q};

[0067] g) State triggering events and action extraction: Each state S y It is triggered by an event EV, and a series of actions will be performed after the trigger, so S y Contains an EV and 0 to r actions AC1, AC2, AC3, ...AC r , these actions are included in the action set AC(S y ), for state S y, the command mode for triggering events and actions is: AC(S y ):{state:S y ,actions:[{action:AC k ,params:P},preEvent:EV,1≤k≤r], where P is the action AC k The parameter set P1, P2, P3, ... P t .

[0068] Step 2: Based on the established instruction model, mark the elements and the relationships between the elements in the requirement text covering each scenario in the target field to form an instruction dataset.

[0069] In one embodiment, multiple data are labeled as seed data sets according to the instruction pattern, including the labeling of each element and the relationship between the elements; various sub-instruction data sets are formed according to the division of subtasks, and each data set contains three fields, namely instruction (Instruction), input (Input) and output (Output), wherein the instruction is used to clarify the function of the large model and the tasks to be completed, and is regulated by the instruction pattern. It varies according to the requirements of each subtask in the instruction pattern, and also includes a structured description of the output format defined by the instruction pattern, which is used to indicate how the large model presents its extracted model elements; the input is the input of the large model, that is, the requirement text of various domain scenarios; the output is the domain elements extracted by the corresponding subtask, and the final domain model is obtained through information mapping and combination.

[0070] Because manual annotation of requirement text is time-consuming and expensive, self-instruction technology can be used to augment a seed dataset that provides high-quality feature annotation and modeling results. Using the API provided by the open-source big model, the remaining requirement text is iteratively generated into new instruction data using the seed data as a reference. Standardized procedures and expert screening are then used to maintain overall data quality and format consistency, primarily by eliminating low-quality datasets with grammatical errors or missing content.

[0071] Specifically, taking the intelligent manufacturing field as an example, for the domain scene recognition subtask, an example of the instruction dataset is as follows:

[0072] Instructions: You are a domain information extraction model. Given a production plan text for a process scenario, please gradually extract domain knowledge from the given text based on the questions asked by the user. The first step is to identify the domain scenario to which the text belongs. Please output in JSON format, the format is: {domain:domain scenario name}

[0073] Input: The automotive stamping production plan involves multiple steps, from raw material preparation to finished product inspection. Each step requires specific equipment and precise operating procedures to ensure product quality and production efficiency. The following is a detailed description of the production plan: First, during the material preparation phase, the primary equipment used is an automatic loader. Its key attributes include a maximum load capacity of 1000 kg and a loading speed of 1 m / s. Upon receiving the "start loading" signal, the automatic loader switches from idle mode to loading mode, automatically feeding steel sheets into the press line until the preset number of sheets is reached. At this point, the automatic loader enters a wait mode, awaiting the next loading instruction. Next, during the stamping process, we use a high-speed precision press with a punching force of 2000 tons and a punching speed of 2 times per second. After receiving the signal indicating the steel sheet is ready, the press moves from its idle state to the punching position (X=500mm, Y=500mm) and performs the punching operation. Each punching operation lasts approximately 0.5 seconds. After completion, the press automatically adjusts to the next punching coordinate until all predetermined punching points are completed. After punching is complete, the press returns to its idle state. After punching, the steel sheet undergoes deburring, using a deburring machine. Key features of this equipment include a processing speed of 0.5m / s and a deburring width of 100mm. The deburring machine begins operating upon arrival of the stamped part, switching from its idle state to its deburring state, continuously removing burrs from the stamped part's edges. This operation lasts for 30 seconds, adjusted according to the size of the steel sheet. After completion, the machine returns to its idle state. After deburring, the stamped part undergoes quality inspection, which uses a 3D scanner. Its attributes include a scanning accuracy of 0.01mm and a scanning speed of 5 seconds per part. After the stamped part is in place, the 3D scanner switches from idle mode to scanning mode, fully scanning the part to verify that its size and shape meet standards. After the scan is complete, the device returns to idle mode. Finally, after the stamped parts pass quality inspection, an automated transport robot transports qualified parts to the storage area. The robot's attributes include a load capacity of 500kg and a maximum speed of 2m / s. Upon receiving a "transport request" signal, the robot switches from idle mode to run mode, moving to the designated storage location along a predetermined path (X=0->X=1000mm, Y=0->Y=1000mm). This operation takes approximately 30 seconds. After placing the stamped part, the robot returns to its starting position to prepare for the next task and returns to idle mode.Through the precise process flow and equipment operation described above, our automotive stamping production solution can efficiently process everything from raw material processing to finished product warehousing, ensuring product quality while achieving full automation from start to finish, reflecting the modern manufacturing industry's pursuit of high efficiency and quality.

[0074] Output: {domain:"Intelligent Manufacturing-Automobile Stamping"}

[0075] For the module identification subtask, a sample instruction dataset is as follows:

[0076] Instructions: Please analyze the following requirement text for the automotive stamping forming scenario in the intelligent manufacturing industry and identify the functional modules involved. Please output it in JSON format: {context:[module name 1, module name 2,...]}

[0077] Input: Same as above.

[0078] Output: {context:['Material Preparation', 'Stamping', 'Deburring', 'Quality Inspection', 'Warehousing']}

[0079] For the entity recognition subtask, a sample of the instruction dataset is as follows:

[0080] Instructions: Please analyze the following requirements text for the automotive stamping scenario in the intelligent manufacturing industry and identify the entities contained in this module. Please output in JSON format: {context:"module name",entity:[{name:"entity name 1",namespace:"domain.module"},{name:"entity name 2",namespace:"domain.module"},...]}

[0081] Input: Same as above.

[0082] Output: {'entities':[{'name':'Material preparation','entity':[{'name':'Automatic loader','namespace':'Automobile stamping.Material preparation'}]},{'name':'Stamping','entity':[{'name':'High-speed precision stamping machine','namespace':'Automobile stamping.Stamping'}]},{'name':'Deburring','entity':[{'name':'Deburring machine','namespace':'Automobile stamping.Deburring'}]},{'name':'Quality inspection','entity':[{'name':'3D scanner','namespace':'Automobile stamping.Quality inspection'}]},{'name':'Warehousing','entity':[{'name':'Automatic handling robot','namespace':'Automobile stamping.Warehousing'}]}]}

[0083] For the attribute recognition subtask, a sample instruction dataset is as follows:

[0084] Please analyze the following requirements text for the automotive stamping scenario in the intelligent manufacturing industry and extract the unique attributes of each entity. Please output in JSON format: {entity:"entity name",cattr:["attribute 1","attribute 2",...]}

[0085] Input: Same as above.

[0086] Output: {'entity':'High-speed precision stamping machine','cattr':['Punching force','Punching speed']},{'entity':'Deburring machine','cattr':['Processing speed','Deburring width']},{'entity':'3D scanner','cattr':['Scanning accuracy','Scanning speed']},{'entity':'Automatic handling robot','cattr':['Load capacity','Maximum speed']}

[0087] For the event recognition subtask, a sample instruction dataset is as follows:

[0088] Instructions: Please analyze the following requirements text for the automotive stamping forming scenario in the intelligent manufacturing industry and identify the preceding events for each entity. Please output in JSON format: {entity:"entity name", event:"event name"}

[0089] Input: Same as above.

[0090] Output: {'entity':'Automatic loader','event':'Loading started'}, {'entity':'High-speed precision stamping machine','event':'Steel plate ready'}, {'entity':'Deburring machine','event':'Stamping parts arrived'}, {'entity':'3D scanner','event':'Stamping parts in place'}, {'entity':'Automatic handling robot','event':'Transportation request'}

[0091] For the state recognition subtask, a sample instruction dataset is as follows:

[0092] Instructions: Please analyze the following requirements text for the automotive stamping forming scenario in the intelligent manufacturing industry and identify the states of each entity. Please output in JSON format: {entity:"entity name",state:["state 1","state 2",...]}

[0093] Input: Same as above.

[0094] Output: {'entity':'Automatic loader','states':['Standby','Loading']},{'entity':'High-speed precision stamping machine','states':['Standby','Stamping']},{'entity':'Deburring machine','states':['Standby','Deburring']},{'entity':'3D scanner','states':['Standby','Scanning']},{'entity':'Automatic handling robot','states':['Standby','Running']} For the action recognition subtask, an example of the instruction dataset is as follows:

[0095] Instructions: Please analyze the following requirement text for the automotive stamping forming scenario in the intelligent manufacturing industry. Based on the input text and the identified preceding events for each state, as well as each action in each state, identify the parameters for each action. Please output in JSON format: {state:"state name", actions:[{action:"action name", params:["param 1","param 2",...]}, preEvent:"pre-event"]}

[0096] Input: Same as above.

[0097] Output: {'state':'Standby','actions':[],'preEvent':['Reach preset']},{'state':'Loading','actions':[{'action':'Feed in','params':[]}],'preEvent':['Start loading']},{'state':'Standby','actions':[],'preEvent':[]},{'state':'Punching','actions':[{'action':'Punching','params':['Duration']},{'action':'Punching','params':['Duration']}],'preEvent':['Steel plate ready']},{'state':'Standby','actions':[],'preEvent':['Complete']},{'state':'Deburring','actions': [{'action':'Remove','params':['Duration']}],'preEvent':['Stamping arrived']},{'state':'Standby','actions':[],'preEvent':['Scanning completed']},{'state':'Scanning','actions':[{'action':'Scanning','params':['Standard']}],'preEvent':['Stamping in place']},{'state':'Standby','actions':[],'preEvent':['Robot returns to the starting position']},{'state':'Running','actions':[{'action':'Return','params':['Starting position']},{'action':'Return','params':['Starting position']}],'preEvent':['Transport request']}

[0098] Step 3: Fine-tune the large model based on the instruction dataset to obtain a large domain automatic modeling model that is suitable for various element extraction tasks.

[0099] Specifically, in this embodiment, the fine-tuning process includes the following steps:

[0100] In step 3.1, select an open-source pre-trained large model as the base model, such as LLaMA, Qwen, etc.

[0101] Step 3.2: Prepare the fine-tuning dataset and divide the instruction dataset formed in step 2 into training set, validation set, and test set in a ratio of 8:1:1.

[0102] In step 3.3, set the fine-tuning parameters, including learning rate, batch size, number of training rounds, etc. For models of different sizes, the parameter settings are as follows:

[0103] For the 14B parameter scale model, the learning rate is set to 1e-5, the batch size is set to 4, and the number of training rounds is set to 80 rounds;

[0104] In step 3.4, LoRA (Low-Rank Adaptation) technology is used to efficiently fine-tune parameters. The configuration parameters of LoRA are as follows:

[0105] r (rank): set to 8, indicating the rank of the low-rank matrix;

[0106] alpha (scaling factor): set to 16 to control the strength of LoRA adaptation;

[0107] Dropout (dropout rate): set to 0.05 to prevent overfitting;

[0108] target_modules: Set to ["k_proj","gate_proj","v_proj","q_proj","o_proj","down_proj","up_proj"].

[0109] In step 3.5, fine-tune each subtask in order from simple to complex, in the following order: domain scene recognition, module recognition, entity recognition, attribute recognition, event recognition, state recognition, and action recognition.

[0110] In step 3.6, merge the fine-tuned model parameters with the pre-trained model and save them as a large domain automatic modeling model.

[0111] Step 4: Construct prompt words for the target domain. The prompt words include task description, requirement text and examples for the corresponding scenario. Based on the prompt words, the domain automatic modeling model is used to extract the elements of each subtask to obtain the inference results.

[0112] In one embodiment, prompt words are constructed based on real-time input text, and the automatic modeling model is scheduled to extract model elements of each subtask in turn. The inference results contain corresponding model elements, and the inference results are parsed, mapped and converted.

[0113] In other embodiments, steps 3 and 4 can be implemented through an automatic modeling pipeline. In this embodiment, an automatic modeling pipeline from fine-tuning to inference to post-processing can be designed to implement a streamlined fine-tuning and automatic modeling process. To make the process clearer, the pipeline is divided into two sub-pipelines:

[0114] (1) Fine-tuning the sub-pipeline:

[0115] a) Subtask division: The domain task is converted into a domain information extraction task and divided into seven subtasks to ensure the accuracy of information extraction;

[0116] b) Instruction data annotation: Annotate the elements of the input text based on the instruction annotation mode of the subtask, and finally obtain an instruction dataset containing the requirement text and domain model annotation information for fine-tuning;

[0117] c) Fine-tuning: Fine-tune the parameters of the pre-trained model. This process needs to be performed incrementally on the instruction data corresponding to the seven subtasks in sequence, thereby enhancing the pre-trained model's ability to extract information in each specific field. The open source Qwen2.5-14b / Qwen2.5-32b is selected as the pre-training base model and the LoRA method is used for incremental multi-subtask fine-tuning. The parameters are optimized by continuously calculating the loss between the generated output and the standard output, and finally a large domain automatic modeling model that adapts to various element extraction tasks of the model is obtained.

[0118] (2) Automatic modeling sub-pipeline:

[0119] a) Subtask Division: The fine-tuned large model also requires subtasks when performing reasoning to effectively utilize the information extraction patterns learned after fine-tuning the large model, thereby ensuring the accuracy of reasoning;

[0120] b) Prompt Construction: To further improve the inference accuracy of the large model, a few samples are provided for the large model to learn from and Prompt is constructed using a few-shot approach. Prompt consists of three parts: first, task descriptions that vary depending on the subtask, supplemented with more detailed tasks based on the annotation model mentioned above; second, the real-time requirement text of a single scenario is integrated into Prompt as input text; and third, 3-5 examples are provided in Prompt for reference by the large model.

[0121] c) Inference and Post-Processing: After receiving input text, the inference engine constructs Prompt in real time and schedules the large automatic model to sequentially extract model elements across seven subtasks. The inference results contain seven categories of model elements. To integrate the final domain model, the inference results require post-processing such as parsing, mapping, and transformation. Because Prompt integrates extraction patterns and defines standardized inference outputs, automated post-processing can be achieved through the development of fixed rules, which are scripted and integrated into the implementation of the automatic modeling sub-pipeline.

[0122] In one embodiment, the elements of the input text are annotated based on the instruction annotation mode of the subtask to obtain an instruction dataset containing the requirement text and domain model annotation information. The instruction dataset is used to fine-tune the parameters of the pre-trained model, and the fine-tuning is performed incrementally on the instruction data corresponding to each subtask. The parameters are optimized by calculating the loss between the generated output and the standard output, and finally a large domain automatic modeling model that is suitable for various element extraction tasks of the model is obtained.

[0123] Step 5: quantify the completeness and accuracy of the inference results, and optimize the parameters of the domain automatic modeling model based on the quantification results.

[0124] In one embodiment, the completeness and accuracy are quantified based on the four indicators of coverage of the element itself, coverage of the relationship information between elements, element grammatical accuracy and element accuracy. A comprehensive evaluation model based on the random forest regression model is used to randomly divide the training data set containing the four indicators and randomly select feature subsets to construct a decision tree. The final prediction result is obtained by fusing multiple prediction results of several decision trees.

[0125] Specifically, in some embodiments, the quantitative evaluation and optimization process includes the following steps:

[0126] Step 5.1, define four evaluation metrics:

[0127] A. Information coverage (IC): The degree of completeness of each element identified by the model compared to the requirements document. The calculation formula is:

[0128] IC(m)=f(I(m),C(p(m)),C(s(m)))

[0129] Where m includes the domain scenario D, context C, entity E, etc., and I(m) evaluates the completeness of each element individually. The final value is a binary score (0 / 1), calculated based on the existence score. The existence score indicates whether the element in the domain model is accurately defined in the requirements document. We use natural language processing technology to search for specific patterns through text search, context extraction, and definition detection (dependency parsing) to determine whether the extracted context contains the description or definition of a certain element.

[0130] B. Structural compliance (SC): The accuracy of the overall structure of the domain model. The SC score of a constructed domain model is calculated as follows:

[0131]

[0132] where s i Represents the i-th element level (element level represents the level divided according to the hierarchical structure of the domain model, such as the first element level is "domain" and the second element level is "context") i The score is 0 or 1, determined by the structural evaluation criteria of the element level.

[0133] C. Grammatical compliance GC: The degree of compliance with the specifications of various elements defined by the abstract model structure is determined by the constraints defined for each element (including the attributes of elements or attributes, and the restrictions of elements or attributes). Each element EL of the domain model is subject to n constraints. The i-th constraint C of the i-th element ELi is ij The satisfaction of can be represented by an indicator function f j (EL i )express:

[0134]

[0135] D. Information alignment (IA): This is determined by the degree to which the information of each element is aligned with the information in the requirements document. A matching function is defined to indicate whether entities and sub-elements are accurately identified:

[0136]

[0137] In step 5.2, we collect evaluation data, including the model’s inference results and manually annotated standard results.

[0138] In step 5.3, the four index values of each inference result are calculated.

[0139] In step 5.4, a random forest regression model is constructed to comprehensively evaluate the model performance. The configuration parameters of the random forest model are as follows:

[0140] n_estimators (number of decision trees): set to 100, indicating that 100 decision trees will be built;

[0141] max_depth (maximum depth): set to 10 to limit the maximum depth of the decision tree;

[0142] min_samples_split (minimum number of split samples): set to 2, indicating the minimum number of samples required for node splitting;

[0143] min_samples_leaf (minimum number of leaf node samples): set to 1, indicating the minimum number of samples required for a leaf node;

[0144] random_state (random seed): Set to 42 to ensure repeatable results.

[0145] In step 5.5, the four indicators are used as features and the comprehensive reliability ST of the domain model automatically generated by the large model is used as a label to train the random forest regression model. The ST calculation method is as follows:

[0146]

[0147] in represents the prediction function of the i-th decision tree, and ST is the final prediction result. Since the four indicators are independent of each other, there is no need to consider the impact of interactive features on the prediction results.

[0148] In step 5.6, the trained random forest model is used to comprehensively score the new inference results, with a score range of 0-1.

[0149] In step 5.7, based on the two metrics of "accuracy" and "completeness," the evaluation model and humans were asked to rank the outputs of the preference dataset. These two scoring datasets were used to train two reward models: a reward model based on ST scores and a reward model based on human scores.

[0150] In step 5.8, during the RLHF-PPO phase, the reward model scores the new instruction data and uses this score to continuously iteratively optimize the parameters of the large automatic model.

[0151] In step 5.9, the domain automatic modeling model is retrained using the optimized parameters.

[0152] In step 5.10, the optimized model is evaluated to compare the performance changes before and after optimization.

[0153] In step 5.11, if the performance improvement is not significant, repeat steps 5.7 to 5.10 until a satisfactory performance level is achieved.

[0154] In some embodiments, two typical situations that affect the completeness of the domain model are also defined: incompleteness and inaccuracy. Four quantitative indicators are designed based on these two aspects, and the four indicators are finally combined into an evaluation score, such as Figure 3 shown.

[0155] (1) Completeness Quantification: The integrity of the domain model is mainly reflected in the element level of the model. The coverage of the information of each element to the requirements document determines the integrity of the domain model. The coverage is mainly reflected in two aspects: one is the coverage of the information of the element itself, and the other is the coverage of the relationship information between elements. Therefore, according to the division of the model structure, the completeness of each element compared with the requirements document is counted separately, and the element information coverage rate IC is used to recursively represent the integrity of the element structure el in the domain model (the element structure represents the level divided according to the hierarchical structure of the domain model, such as the first element structure is "domain", the second element structure is "module"..., and the seventh element structure is state action and state event).

[0156] (2) Accuracy quantification: The accuracy of the domain model is another key factor in evaluating whether the model is effective. It is not only reflected in the overall structure of the domain model, but also involves the details and functions of each individual element in the model.

[0157] a) At the overall structural level, the accuracy of a domain model primarily refers to the correctness of its structural syntax, which is determined by whether the model adheres to the specifications and framework defined by the abstract model structure. The accuracy of the domain model's overall structural level is represented by the Structural Conformity (SC) score.

[0158] b) Accuracy at the element level encompasses two aspects: element syntax accuracy and element accuracy. The accuracy of element syntax is determined by the degree of compliance with the element specifications defined by the abstract model structure, represented by the element conformance score (GC). The GC score is determined and calculated based on the constraints (including element or attribute attributes, element or attribute restrictions) defined for each element in the XSD (XML Schema Definition) file used to describe the abstract model structure. The alignment of each element's information with the requirements document determines the accuracy of the element's content. Given the high reliability requirements of business scenarios, this score is determined through a combination of automated assisted evaluation and manual review, represented by the element alignment score (IA).

[0159] Based on the four quantitative indicator scores IC, SC, GC, and IA, a comprehensive evaluation model based on the random forest regression model is constructed. This model randomly divides the training data set containing the four quantitative indicator features and randomly selects feature subsets to construct a decision tree. The final prediction result is obtained by averaging the multiple prediction results of several decision trees, that is, the comprehensive completeness ST of the domain model automatically generated by the large model.

[0160] In some embodiments, in order to further optimize and improve the quality and completeness of the generated results of the automatic model building, a human feedback (RLHF) method based on reinforcement learning can be used, and two reference signals, the completeness evaluation result ST and expert annotation feedback, can be introduced to design a reward model to achieve continuous self-optimization of the automatic model building. Figure 4 The figure shows the RLHF architecture of the large automatic model. At the bottom, preference data is first collected. Then, based on multiple metrics such as accuracy and completeness, the evaluation model and humans are asked to rank the output of the dataset. These two scoring datasets are used to train two reward models (one based on ST scores and one based on human scores). In the RLHF-PPO stage, the parameters of the large automatic model are optimized by iteratively learning from the experience gained from inference on the Prompt data.

[0161] Example 2

[0162] A domain modeling large model fine-tuning system, comprising:

[0163] a subtask construction module configured to construct a corresponding abstract model structure based on the elements and tasks of the target domain, divide the abstract model structure into a plurality of subtasks for extracting elements, and establish corresponding instruction patterns;

[0164] The annotation module is configured to annotate the elements and the relationships between the elements in the requirement text covering various scenarios in the target domain according to the established instruction model to form an instruction data set;

[0165] The fine-tuning module is configured to fine-tune the large model based on the instruction dataset to obtain a large model for domain automatic modeling that is suitable for various element extraction tasks;

[0166] The reasoning module is configured to construct prompt words for the target domain, which include task descriptions, requirement text for corresponding scenarios, and examples. Based on the prompt words, the module uses the domain automatic modeling model to extract elements of each subtask and obtain reasoning results.

[0167] The quantification and optimization module is configured to quantify the completeness and accuracy of the inference results, and optimize the parameters of the domain automatic modeling model based on the quantification results.

[0168] Example 3

[0169] A computer-readable storage medium is characterized in that it is used to store computer instructions, and when the computer instructions are executed by a processor, the steps in the method provided in embodiment 1 are completed.

[0170] Example 4

[0171] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the method provided in embodiment 1 are completed.

[0172] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0173] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0176] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made by those skilled in the art that fall within the spirit and principles of the present invention and do not require creative effort are intended to be within the scope of protection of the present invention.

Claims

1. A domain modeling large model fine-tuning method, characterized by: The following steps are involved: Based on the elements and tasks of the target domain, a corresponding abstract model structure is constructed, the abstract model structure is divided into multiple subtasks for extracting elements, and corresponding instruction patterns are established; Based on the established instruction model, the elements and relationships between the elements in the requirement text covering various scenarios in the target field are annotated to form an instruction dataset; Fine-tune the large model based on the instruction dataset to obtain a large model for domain automatic modeling that is suitable for various element extraction tasks; Constructing prompt words for the target domain, which include task descriptions, requirement text for the corresponding scenario, and examples. Based on the prompt words, the domain automatic modeling model is used to extract the elements of each subtask and obtain the inference results. The completeness and accuracy of the inference results are quantified, and the parameters of the large model for automatic domain modeling are optimized based on the quantified results.

2. A domain modeling large model fine-tuning method as claimed in claim 1, characterized in that: Based on the elements and tasks of the target domain, the process of constructing the corresponding abstract model structure includes: the abstract model structure includes multiple layers of elements, the parent structure of the entity is module and scene, the substructure of the entity is attribute, event and state, the substructure of the state is action, and the tasks of each layer of elements are defined as subtasks of the modeling task.

3. A domain modeling large model fine-tuning method as claimed in claim 1, characterized in that: The process of dividing the abstract model structure into multiple subtasks for extracting elements and establishing corresponding instruction patterns includes: According to the abstract model structure, seven subtasks are divided. Each subtask constructs a corresponding instruction data annotation mode. The description of each subtask and its instruction mode are as follows: For a given requirement text T, it is necessary to identify the domain scenario D to which the text belongs. The instruction annotation pattern of D is: D(T):{domain:D}, where domain is used to define the domain scenario; According to the different business and functions, the domain scenario D is divided into multiple modules C1, C2, C3, ..., C m , included in the module set C, the instruction pattern of C is: C(D):{context:[C i ],1≤i≤m}, where context is the configuration context module; Each module C i Contains at least one entity, according to the identified module C i Continue to identify the included entities E1, E2, E3, ..., E n , these entities are included in the entity set E(C i ), E(C i ) instruction mode is: E(C i ):{context:C i ,entity:[{name:E j ,namespace:DC i }],1≤j≤n}, where namespace is the representation of the domain scenario and module to which an entity belongs, entity is the entity representation, and name is the name; Each entity E j Contains multiple attributes A1, A2, A3, ..., A p , contained in the attribute set A(E j ), attribute A(E j ) instruction mode is: A(E j ): {entity:E j ,cattr:[A x ],1≤x≤p}, cattr is used to define attributes; Each entity E j Contains a preceding trigger event EV, the instruction mode of EV is: EV(E j ):{entity:E j ,event:EV}, event is used to define events; Each entity E j Contains a set of states S1, S2, S3, ... S q , contained in the state set S(E j ), S(E j ) instruction mode is: S(E j ):{entity:E j ,state:[S y ],1≤y≤q}, state is used to define the state; Each state S y Triggered by an event EV, a series of actions will be executed after the trigger, state S y Contains an EV and 0 to r actions AC1, AC2, AC3, ...AC r , the above actions are included in the action set AC(S y ), for state S y , the instruction pattern for triggering events and actions is: AC(S y ):{state:S y ,actions:[{action:AC k ,params:P},preEvent:EV,1≤k≤r], where P is the action AC k The parameter set P1, P2, P3, ... P t , action is used to define actions, and params is used to define parameters.

4. A domain modeling large model fine-tuning method as claimed in claim 1, characterized in that: Based on the established instruction model, each element and the relationship between elements in the requirement text covering each scenario in the target domain are annotated. The process of forming the instruction data set includes: annotating multiple data as seed data sets based on the instruction model, including annotating each element and the relationship between elements; Various sub-instruction data sets are formed according to the division of subtasks. Each data set contains three fields, namely instructions, input and output. The instructions are used to clarify the functions of the large model and the tasks it needs to complete. They are regulated by the instruction pattern and vary according to the requirements of each subtask in the instruction pattern. They also include a structured description of the output format defined by the instruction pattern, which is used to indicate how the large model presents its extracted model elements; the input is the input of the large model, that is, the requirement text of various domain scenarios; the output is the domain elements extracted by the corresponding subtask, and the final domain model is obtained through information mapping and combination.

5. A domain modeling large model fine-tuning method as claimed in claim 1, characterized in that: The process of fine-tuning the large model based on the instruction data set to obtain a large model for domain automatic modeling that is suitable for various element extraction tasks includes: labeling the elements of the input text based on the instruction labeling mode of the subtask to obtain an instruction data set containing the requirement text and domain model labeling information, using the instruction data set to fine-tune the parameters of the pre-trained model, incrementally performing fine-tuning on the instruction data corresponding to each subtask, optimizing the parameters by calculating the loss between the generated output and the standard output, and finally obtaining a large model for domain automatic modeling that is suitable for various element extraction tasks of the model.

6. A domain modeling large model fine-tuning method as claimed in claim 1, characterized in that: According to the prompt words, the domain automatic modeling model is used to extract the elements of each subtask, and the process of obtaining the inference result includes: constructing prompt words based on real-time input text, scheduling the automatic modeling model to extract model elements of each subtask in turn, the inference result contains the corresponding model elements, and the inference result is parsed, mapped and converted.

7. A domain modeling large model fine-tuning method as claimed in claim 1, characterized in that: The process of quantifying the completeness and accuracy of the inference results includes: quantifying the completeness and accuracy from four indicators: coverage of element information itself, coverage of information on relationships between elements, element syntax accuracy, and element accuracy; using a comprehensive evaluation model based on a random forest regression model, randomly dividing the training data set containing the four indicators and randomly selecting feature subsets to construct a decision tree; and obtaining the final prediction result by fusing multiple prediction results of several decision trees.

8. A domain modeling large model fine-tuning system, characterized by: include: a subtask construction module configured to construct a corresponding abstract model structure based on the elements and tasks of the target domain, divide the abstract model structure into a plurality of subtasks for extracting elements, and establish corresponding instruction patterns; The annotation module is configured to annotate the elements and the relationships between the elements in the requirement text covering various scenarios in the target domain according to the established instruction model to form an instruction data set; The fine-tuning module is configured to fine-tune the large model based on the instruction dataset to obtain a large model for domain automatic modeling that is suitable for various element extraction tasks; The reasoning module is configured to construct prompt words for the target domain, which include task descriptions, requirement text for corresponding scenarios, and examples. Based on the prompt words, the module uses the domain automatic modeling model to extract elements of each subtask and obtain reasoning results. The quantification and optimization module is configured to quantify the completeness and accuracy of the inference results, and optimize the parameters of the domain automatic modeling model based on the quantification results.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein the steps of the method according to any one of claims 1 to 7 are completed when the computer instructions are executed by the processor.

Images