Domain large model fine-tuning method, system, electronic device and readable medium

Abstract

The application provides a domain large model fine-tuning method, system, electronic equipment and readable medium, and relates to the technical field of artificial intelligence. The domain large model fine-tuning method comprises the following steps: obtaining an initial large language model and a first training data set; constructing a parameter fine-tuning search space corresponding to the initial large language model; inserting a plurality of candidate fine-tuning modules into the initial large language model to obtain an initial domain large model; training the initial domain large model according to the first training data set to obtain an intermediate domain large model containing at least one first fine-tuning module; obtaining a second training data set, a domain knowledge graph and a domain multi-modal database; performing data enhancement on the second training data set according to the domain knowledge graph and the domain multi-modal database to obtain an enhanced training data set; and fine-tuning training the intermediate domain large model according to the enhanced training data set to obtain a final domain large model.

Description

Technical Field

[0001] This application relates primarily to the field of artificial intelligence technology, and in particular to a method, system, electronic device, and readable medium for fine-tuning a large domain model. Background Technology

[0002] As the industrial energy sector and the power industry deepen their intelligent transformation, the industry has put forward higher requirements for the collection, processing and application capabilities of data resources. The value of data in vertical fields is gradually being valued, and manufacturing enterprises are continuously increasing their investment in building data-driven capabilities.

[0003] In the process of fine-tuning large language models in vertical domains, human experience is often relied upon, and there is a lack of effective automatic optimization mechanisms, which makes it impossible to efficiently fine-tune large language models in vertical domains.

[0004] Therefore, there is an urgent need for a domain-specific large model fine-tuning method that can efficiently fine-tune large language models in vertical domains. Summary of the Invention

[0005] The technical problem to be solved by this application is to provide a method, system, electronic device and readable medium for fine-tuning large language models in a domain, which can efficiently fine-tune large language models in a vertical domain.

[0006] To address the aforementioned technical problems, this application provides a method for fine-tuning a large domain model, comprising the following steps: obtaining an initial large language model and a first training dataset; constructing a parameter fine-tuning search space corresponding to the initial large language model, wherein the parameter fine-tuning search space includes multiple candidate fine-tuning modules; inserting the multiple candidate fine-tuning modules into the initial large language model to obtain an initial large domain model; training the initial large domain model based on the first training dataset to obtain an intermediate large domain model containing at least one first fine-tuning module, wherein the at least one first fine-tuning module is at least one candidate fine-tuning module trained from the multiple candidate fine-tuning modules; obtaining a second training dataset, a domain knowledge graph, and a domain multimodal database; performing data augmentation on the second training dataset based on the domain knowledge graph and the domain multimodal database to obtain an augmented training dataset; and fine-tuning the intermediate large domain model based on the augmented training dataset to obtain a final large domain model.

[0007] Optionally, the parameter fine-tuning search space also includes the type range, insertion position range, and corresponding hyperparameter value range of each candidate fine-tuning module. The step of training the initial domain large model based on the first training dataset to obtain an intermediate domain large model containing at least one first fine-tuning module further includes: progressively pruning multiple candidate fine-tuning modules in the initial domain large model based on the first training dataset, the type range, and insertion position range of each candidate fine-tuning module to obtain a domain large model to be trained containing at least one candidate fine-tuning module as a fine-tuning module to be trained; and performing Bayesian optimization on the domain large model to be trained based on the first training dataset and the value range of the hyperparameters corresponding to at least one fine-tuning module to be trained to obtain an intermediate domain large model containing at least one fine-tuning module with optimized corresponding hyperparameters as a first fine-tuning module.

[0008] Optionally, the step of progressively pruning multiple candidate fine-tuning modules in the initial domain model to obtain a training domain model containing at least one candidate fine-tuning module as a training fine-tuning module further includes: step a, determining whether the total number of parameters of multiple candidate fine-tuning modules is less than the total number of parameters threshold; if the determination result is yes, proceed to step b, otherwise proceed to step c; step b, using each candidate fine-tuning module as a training fine-tuning module; step c, calculating the resource consumption value of each candidate fine-tuning module, removing at least one candidate fine-tuning module with the largest resource consumption value from the initial domain model, and returning to step a.

[0009] Alternatively, the objective expression for Bayesian optimization is: In the formula To fine-tune the search space for parameters, In the current hyperparameter configuration and corresponding configurable optimal weights The performance metric function for the initial large-scale domain model. Configure the current hyperparameters The total number of parameters in the initial domain model is given, where the constraint expression for the objective function is: In the formula In the current hyperparameter configuration and corresponding configurable weights The training loss function for the initial domain-wide model. For the argmin operation, To be subject to.

[0010] Optionally, the domain multimodal database includes multiple modal data, and the domain knowledge graph includes multiple triples and multiple summary candidate subgraphs associated with the multiple triples.

[0011] Optionally, the second training dataset includes multiple second training input data. The step of augmenting the second training dataset according to the domain knowledge graph and the domain multimodal database to obtain the augmented training dataset further includes: for each second training input data, obtaining the corresponding modality data from the domain multimodal database through a semantic similarity method, and generating the first augmented data corresponding to the second training input data based on all the obtained modality data; for each second training input data, integrating the various summary candidate subgraphs associated with each triplet corresponding to the second training input data in the domain knowledge graph to obtain the second augmented data corresponding to the second training input data; and adding all the first augmented data and the second augmented data to the second training dataset to obtain the augmented training dataset.

[0012] Optionally, for each second training input data, the step of obtaining corresponding modality data from the domain multimodal database using a semantic similarity method, and generating first augmented data corresponding to the second training input data based on the modality data further includes: generating a retrieval statement based on the second training input data; obtaining a retrieval model, and enabling the retrieval model to retrieve the corresponding modality data from the domain multimodal database based on the retrieval statement and in combination with cosine similarity; and semantically encoding all the obtained modality data and the second training input data to obtain the first augmented data.

[0013] Optionally, the steps for obtaining a domain knowledge graph include: obtaining entity recognition prompt templates, relation extraction prompt templates, and existing large language models; constructing a domain knowledge graph containing multiple triples based on modal data from a domain multimodal database using entity recognition prompt templates, relation extraction prompt templates, and existing large language models; and dividing the domain knowledge graph into multiple subgraphs as candidate summary subgraphs using a graph partitioning algorithm, wherein each triple is associated with a candidate summary subgraph.

[0014] Optionally, the step of constructing a corresponding domain knowledge graph based on the modal data of the domain multimodal database using entity recognition prompt templates, relation extraction prompt templates, and an existing large language model further includes: inputting the modal data and entity recognition prompt templates into the existing large language model to obtain multiple entities; and inputting all the multiple entities and relation extraction prompt templates into the existing large language model to obtain multiple triples.

[0015] Optionally, the graph partitioning algorithm includes Leiden's algorithm, and the modularity of Leiden's algorithm is... The calculation expression is: In the formula This is the sum of the weights of all edges in the domain knowledge graph. For nodes in a domain knowledge graph and nodes The weight of the edges between them. For nodes The degree, For nodes The degree, For nodes and nodes The corresponding indicator function.

[0016] To address the aforementioned technical problems, this application provides a domain-wide large model fine-tuning system, comprising: a data acquisition module for acquiring an initial large language model, a first training dataset, a second training dataset, a domain knowledge graph, and a domain multimodal database; a search space construction module for constructing a parameter fine-tuning search space corresponding to the initial large language model, wherein the parameter fine-tuning search space includes multiple candidate fine-tuning modules; an initial domain-wide large model construction module for inserting multiple candidate fine-tuning modules into the initial large language model to obtain an initial domain-wide large model; a fine-tuning strategy optimization module for training the initial domain-wide large model based on the first training dataset to obtain an intermediate domain-wide large model containing at least one first fine-tuning module, wherein the at least one first fine-tuning module is at least one candidate fine-tuning module trained from multiple candidate fine-tuning modules; an augmented data construction module for augmenting the second training dataset based on the domain knowledge graph and the domain multimodal database to obtain an augmented training dataset; and a fusion training module for fine-tuning the intermediate domain-wide large model based on the augmented training dataset to obtain a final domain-wide large model.

[0017] Optionally, the parameter fine-tuning search space also includes the type range, insertion position range, and corresponding hyperparameter value range of each candidate fine-tuning module. The fine-tuning strategy optimization module includes: a pruning unit, which is used to progressively prune multiple candidate fine-tuning modules in the initial domain large model according to the first training dataset, the type range, and insertion position range of each candidate fine-tuning module, to obtain a domain large model to be trained that includes at least one candidate fine-tuning module as a fine-tuning module to be trained; and an optimization unit, which is used to perform Bayesian optimization on the domain large model to be trained according to the first training dataset and the value range of the hyperparameters corresponding to at least one fine-tuning module to be trained, to obtain an intermediate domain large model that includes at least one fine-tuning module with optimized corresponding hyperparameters as the first fine-tuning module.

[0018] Optionally, the domain multimodal database includes multiple modal data, the domain knowledge graph includes multiple triples and summary candidate subgraphs associated with the triples, the second training dataset includes multiple second training input data, and the augmented data construction module includes: a first data augmentation unit, which is used to obtain the corresponding modal data from the domain multimodal database for each second training input data through a semantic similarity method, and generate first augmented data corresponding to the second training input data based on all the obtained modal data; a second data augmentation unit, which is used to integrate the various summary candidate subgraphs associated with each triple corresponding to the second training input data in the domain knowledge graph for each second training input data to obtain the second augmented data corresponding to the second training input data; and a fusion unit, which is used to add all the first augmented data and the second augmented data to the second training dataset to obtain the augmented training dataset.

[0019] To address the aforementioned technical problems, this application proposes an electronic device comprising: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the aforementioned domain-wide model fine-tuning method.

[0020] To address the aforementioned technical problems, this application proposes a computer-readable medium storing computer program code, which, when executed by a processor, implements the aforementioned domain-wide model fine-tuning method.

[0021] Compared with existing technologies, this application has the following advantages: At least one first fine-tuning module capable of optimizing the performance of an initial large language model is selected and trained from the parameter fine-tuning search space using a first training dataset, thereby obtaining an intermediate domain-specific large model. Data augmentation is performed on the second training dataset using data from a domain knowledge graph and a domain multimodal database, resulting in an augmented training dataset with enhanced semantic information for a specific domain. Based on this, the intermediate domain-specific large model is fine-tuned using the augmented training dataset, resulting in a final domain-specific large model with stronger performance in that domain. Attached Figure Description

[0022] The accompanying drawings are included to provide a further understanding of this application; they are incorporated into and constitute a part of this application. The drawings illustrate embodiments of this application and, together with this specification, serve to explain the principles of this application. In the drawings: Figure 1 This is a flowchart illustrating a domain large model fine-tuning method according to an embodiment of this application; Figure 2 yes Figure 1 A flowchart illustrating the sub-steps of step S14; Figure 3 yes Figure 2A flowchart illustrating the sub-steps of step S141; Figure 4 yes Figure 1 A schematic diagram of the pruning and optimization stages in step S14; Figure 5 This is a flowchart illustrating the process of acquiring a domain knowledge graph; Figure 6 yes Figure 5 A flowchart illustrating the sub-steps of step S22; Figure 7 yes Figure 1 A flowchart illustrating the sub-steps of step S16; Figure 8 yes Figure 7 A flowchart illustrating the sub-steps of step S161; Figure 9 This is a flowchart illustrating the process of using the final domain-wide model; Figure 10 This is a block diagram of a domain large model fine-tuning system according to an embodiment of this application; and Figure 11 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this application. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0024] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0025] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0026] In the description of this application, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is usually based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this application and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this application; the directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0027] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0028] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, these terms have no special meaning and therefore should not be construed as limiting the scope of protection of this application. In addition, although the terminology used in this application is selected from commonly known and used terms, some terms mentioned in this application's specification may have been chosen by the applicant according to his or her judgment, and their detailed meanings are explained in the relevant sections of this description. Moreover, this application should be understood not only through the actual terms used, but also through the meaning implied by each term.

[0029] It should be understood that when a component is referred to as "on another component," "connected to another component," "coupled to another component," or "in contact with another component," it can be directly on, connected to, coupled to, or in contact with that other component, or there may be an intervening component. In contrast, when a component is referred to as "directly on another component," "directly connected to," "directly coupled to," or "directly in contact with" another component, there is no intervening component. Similarly, when a first component is referred to as "electrically contacting" or "electrically coupled to" a second component, there is an electrical path between the first and second components that allows current to flow. This electrical path may include capacitors, coupled inductors, and / or other components that allow current to flow, even if there is no direct contact between the conductive components.

[0030] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more steps may be removed from these processes.

[0031] Reference Figure 1 One embodiment of this application proposes a domain-specific large-scale model fine-tuning method 100. This domain-specific large-scale model fine-tuning method 100 is suitable for fine-tuning and training an existing initial large-scale language model to obtain a final domain-specific large-scale model. In this embodiment, the specific domain is the nuclear energy domain. It should be noted that this application does not limit the specific type of the specific domain. In some embodiments, the specific domain is other vertical domains, such as the power industry.

[0032] Continue to refer to Figure 1The domain-specific large-scale language model fine-tuning method 100 in this embodiment includes the following steps. Step S11 is to obtain an initial large-scale language model and a first training dataset. In this embodiment, the first training dataset is constructed based on corpus data from a specific domain. Step S12 is to construct a parameter fine-tuning search space corresponding to the initial large-scale language model. The parameter fine-tuning search space includes multiple candidate fine-tuning modules, as well as the type range, insertion position range, and corresponding hyperparameter value range for each candidate fine-tuning module. Specifically, in this embodiment, the type range of candidate fine-tuning modules includes LoRA, ParallelAdapter (PA), Serial Adapter (SA), Prefix Tuning (PA), and Bitfit; that is, in this embodiment, corresponding candidate fine-tuning modules are constructed according to the above-mentioned multiple parameter fine-tuning methods. Furthermore, in this embodiment, multiple candidate fine-tuning modules correspond to different levels and modules within the Transformer structure of the initial large-scale language model. That is, multiple candidate fine-tuning modules constructed by the same parameter fine-tuning method correspond to different levels and modules within the Transformer structure, and the same level or module within the Transformer structure can correspond to multiple candidate fine-tuning modules constructed by different parameter fine-tuning methods. It should be noted that in this embodiment, the corresponding candidate fine-tuning modules are inserted into the corresponding positions within the Transformer structure according to the parameters targeted by the above-described parameter fine-tuning method; this will not be elaborated upon here. The insertion position range of each type of candidate fine-tuning module in this embodiment corresponds to the various levels and modules that the corresponding parameter fine-tuning method can correspond to within the Transformer structure of the initial large language model. In this embodiment, the value range of hyperparameters includes rank, prefix length, and intermediate layer dimension control terms.

[0033] Continue to refer to Figure 1 Step S13 involves inserting multiple candidate fine-tuning modules into the initial large language model to obtain the initial domain large model. Specifically, in this embodiment, each candidate fine-tuning module is first inserted into the corresponding layer and module within the Transformer structure of the initial large language model, and the hyperparameters of each candidate fine-tuning module are set to preset empirical values. Step S14 involves training the initial domain large model based on the first training dataset to obtain an intermediate domain large model containing at least one first fine-tuning module. Here, at least one first fine-tuning module is at least one candidate fine-tuning module trained from among multiple candidate fine-tuning modules; that is, in step S14, at least one candidate fine-tuning module is selected and trained from among multiple candidate fine-tuning modules using the first training dataset, and the candidate fine-tuning module trained on the first training dataset is used as the first fine-tuning module. Further refer to... Figure 2Step S14 includes the following sub-steps. Step S141 involves progressively pruning multiple candidate fine-tuning modules in the initial domain large model based on the first training dataset, the type range of each candidate fine-tuning module, and the insertion position range, to obtain a training domain large model containing at least one candidate fine-tuning module as a training fine-tuning module. Specifically, in step S141 of this embodiment, the weights of each candidate fine-tuning module in the initial domain large model are initialized, and low-fidelity training is performed using the first training dataset. Then, the candidate fine-tuning modules are pruned according to a specific pruning strategy, and training and pruning are continuously performed iteratively until the number of parameters of the remaining candidate fine-tuning modules after pruning drops below a preset threshold. The remaining candidate fine-tuning modules are then used as the first fine-tuning modules. It can be understood that the main search objective of step S141 is to determine the type of the candidate fine-tuning modules and their optimal insertion layer position in the Transformer structure. That is, to determine the type range and insertion position range of multiple candidate fine-tuning modules suitable for insertion into the initial large language model in the parameter fine-tuning search space. Furthermore, by setting the hyperparameters of the candidate fine-tuning modules to empirical values ​​and keeping them frozen in the previous step S13, the influence of the hyperparameter values ​​on the selection of the optimal type and insertion position from the type range and insertion position range can be avoided. Thus, in step S141, the optimal type and optimal insertion position of each candidate fine-tuning module applicable to the initial domain large model can be obtained.

[0034] For the cyclic pruning process in step S141 above, further refer to Figure 3 Step S141 includes the following sub-steps. Step a is to determine whether the total number of parameters of all candidate fine-tuning modules is less than the total number of parameters threshold. If the determination result is yes, proceed to step b; otherwise, proceed to step c. Step b is to use each candidate fine-tuning module as a fine-tuning module to be trained. Step c is to calculate the resource consumption value of each candidate fine-tuning module, remove at least one candidate fine-tuning module with the largest resource consumption value from the initial domain large model, and return to step a. It should be noted that in this embodiment, in each loop pruning, i.e., in step c, after removing the candidate fine-tuning module, the weight values ​​of the candidate fine-tuning modules need to be re-initialized. Accordingly, in this embodiment, the resource consumption value is determined based on the activation value, weight value, and loss change corresponding to the candidate fine-tuning module. In one example, after training the candidate fine-tuning module based on the first training dataset, the corresponding resource consumption value is calculated from the activation value, weight value, and loss change corresponding to the candidate fine-tuning module.

[0035] Continue to refer to Figure 2Step S142 involves performing Bayesian optimization on the large-scale model to be trained based on the first training dataset and the range of hyperparameter values ​​corresponding to at least one fine-tuning module to be trained. This yields a large-scale model to be trained containing at least one well-optimized corresponding hyperparameter, serving as an intermediate large-scale model for the first fine-tuning module. Specifically, in step S142 of this embodiment, the Bayesian optimizer samples a set of hyperparameter configurations from the range of hyperparameter values ​​in the parameter fine-tuning search space and performs multi-fidelity training and evaluation on the large-scale model to be trained using the first training dataset. The evaluated fine-tuning performance metrics and total number of parameters are then fed back to the Bayesian optimizer to update the internal acquisition function. Next, hyperparameter configurations with greater potential are resampled based on the updated acquisition function. After a specified number of cyclic searches, a multi-objective Pareto optimal solution is finally obtained. Correspondingly, the objective expression for Bayesian optimization in this embodiment is: , In the formula To fine-tune the search space for parameters, In the current hyperparameter configuration and corresponding configurable optimal weights The performance metric function for the initial domain large model, i.e., the domain large model to be trained in step S142. Configure the current hyperparameters The total number of parameters in the initial domain model. Understandably, the result of the performance metric function calculation is the aforementioned fine-tuning performance index. Furthermore, the constraint expression of the objective expression in this embodiment is: , In the formula In the current hyperparameter configuration and corresponding configurable weights The training loss function for the initial domain-wide model. For the argmin operation, "Subject to" means "constrained by". This is understandable; it relates to the training loss function. Used to measure the current hyperparameter configuration With corresponding configurable optimal weights The loss value of the initial domain large model on the first training dataset.

[0036] Continue to refer to Figure 4In step S14 of this embodiment, during the pruning stage, after initializing the weights of each candidate fine-tuning module for the constructed parameter fine-tuning search space, low-fidelity training is performed using the first training dataset to determine the resource consumption value of each candidate fine-tuning module. Subsequently, a spatial pruner is used to sparsely prune candidate fine-tuning modules with high resource consumption values, thereby reducing the number of candidate fine-tuning modules in the parameter fine-tuning search space. This process is repeated until the total number of parameters of all candidate fine-tuning modules is less than a parameter total threshold. Then, the remaining candidate fine-tuning modules in the parameter fine-tuning search space are determined to be used in the large model to be trained, and the type and insertion position of each candidate fine-tuning module are determined. In the optimization stage following the pruning stage, after fixing the type and insertion position of each candidate fine-tuning module, multi-objective Bayesian optimization is used for the hyperparameters of each candidate fine-tuning module to obtain the optimal solution for the hyperparameter configuration of each candidate fine-tuning module, i.e., the better specific values ​​of the hyperparameters. The candidate fine-tuning module with the determined type, insertion position, and hyperparameters is then used as the first fine-tuning module in the intermediate domain large model. Understandably, step S14 of this embodiment achieves joint automatic search of candidate fine-tuning module types, insertion positions, and hyperparameter configurations in a large-scale search space by integrating progressive pruning and multi-objective Bayesian optimization. This effectively overcomes the problems of traditional fine-tuning strategy selection relying on experience settings and low combination efficiency, and achieves synergistic improvement in search accuracy, parameter control capability, and fine-tuning adaptation performance.

[0037] Continue to refer to Figure 1 Step S15 involves acquiring the second training dataset, the domain knowledge graph, and the domain multimodal database. The domain multimodal database includes multiple modalities of data. It is understood that in this embodiment, the domain multimodal database can be directly acquired or constructed based on existing domain data. The modal types of the modal data include text, images, audio, and video. Furthermore, in this embodiment, the domain knowledge graph includes multiple triples and multiple summary candidate subgraphs associated with the triples. For further details, please refer to... Figure 5 The process of obtaining a domain knowledge graph in this embodiment includes the following steps. Step S21 is to obtain entity recognition prompt templates, relation extraction prompt templates, and an existing large language model. In this embodiment, the existing large language model is any open-source or directly purchaseable large language model. Step S22 is to construct a domain knowledge graph containing multiple triples based on data from a domain multimodal database using the entity recognition prompt templates, relation extraction prompt templates, and the existing large language model. Specifically, in this embodiment, the entity recognition prompt templates and relation extraction prompt templates guide the existing large language model to automatically identify the relation triples between various entity sets and entity pairs in a zero-shot manner.

[0038] Further reference Figure 6Step S22 includes the following sub-steps. Step S221 involves inputting the modal data and entity recognition prompt template into an existing large language model to obtain multiple entities. That is, all modal data is input into the existing large language model to obtain multiple entities. Step S222 involves inputting the multiple entities and relation extraction prompt template into the existing large language model to obtain multiple triples. In this embodiment, the expressions for obtaining multiple entities and triples in steps S221 and S222 are as follows: , , In the formula For a collection of entities, It is a single entity and belongs to a set of entities. The elements in This indicates the entity recognition prompt template. Input is fed into an existing large language model and combined with the model parameters of the existing large language model. To generate entity sequences, For a set of triples, For entities and entity and the corresponding relationship The triples formed, and belong to the set of triples. The elements in This indicates a prompt template for extracting relationships. With two entities and After inputting them into the existing large language model, the model parameters of the existing large language model are used. The result obtained from entity relation identification. It can be understood that in this embodiment, the entities in the triples serve as nodes in the domain knowledge graph, and the relations in the triples serve as edges connecting the corresponding nodes, thus forming the domain knowledge graph, which is represented as follows: , In the formula Represents the structure of a domain knowledge graph. This represents the set of nodes in a domain knowledge graph. Represents the set of edges in a domain knowledge graph. This represents a node in the domain knowledge graph. This represents an edge in a domain knowledge graph.

[0039] Continue to refer to Figure 5Step S23 involves dividing the domain knowledge graph into multiple subgraphs as candidate summary subgraphs using a graph partitioning algorithm. Each triple is associated with a candidate summary subgraph. In this embodiment, the graph partitioning algorithm includes a community partitioning algorithm. Specifically, the community partitioning algorithm first treats each node in the obtained domain knowledge graph as a separate community, and then iteratively adjusts the community partitioning in the domain knowledge graph to improve the modularity of each community. Each iteration includes two main steps: one is moving nodes, where it is determined whether moving a node from one community to another can improve the modularity; if the result is yes, the node is moved to the other community. The other step is merging communities, where it is determined whether merging two other communities can improve the modularity; if the result is yes, the two communities are merged. In this embodiment, the community partitioning algorithm includes the Leiden algorithm, and the modularity of the Leiden algorithm... The calculation expression is: , In the formula This is the sum of the weights of all edges in the domain knowledge graph. For nodes in a domain knowledge graph and nodes The weight of the edges between them. For nodes The degree, For nodes The degree, For nodes and nodes The corresponding indicator function. Where, when the domain knowledge graph is an unweighted graph, it is... The total number of edges, and correspondingly, the number of edges corresponding to The value of is 1, corresponding to no edge. The value is 1. Furthermore, in this embodiment, when the node... and nodes Indicator function when belonging to the same community The value is 1 when the node and nodes Indicator function when not belonging to the same community The value of is 0.

[0040] Continue to refer to Figure 1 Step S16 involves augmenting the second training dataset using the domain knowledge graph and the domain multimodal database to obtain the augmented training dataset. Further refer to... Figure 7Step S16 includes the following sub-steps. Step S161 involves obtaining corresponding modality data from the domain multimodal database for each second training input data using a semantic similarity method, and generating first augmented data corresponding to the second training input data based on all obtained modality data. Further refer to... Figure 8 Step S161 includes the following sub-steps. Step S1611 generates a retrieval statement based on the second training input data. Specifically, in this embodiment, the corresponding retrieval statement is generated by combining the initial large language model with prompt words and the second training input data. Step S1612 obtains a retrieval model and enables the retrieval model to retrieve the corresponding modality data from the domain multimodal database based on the retrieval statement and cosine similarity. Specifically, in this embodiment, the retrieval model includes an existing embedding model, which can vectorize the retrieval statement, thereby enabling the retrieval model to perform vector retrieval of the retrieval statement in the domain multimodal database using cosine similarity to obtain the corresponding modality data. Step S1613 semantically encodes all the obtained modality data and the second training input data to obtain the first augmented data. Specifically, the expression for generating the first augmented data in step S16 is: , In the formula Enhance the cue vector for text-based retrieval. Enhance the cue vector for image modality retrieval. Enhance cue vectors for video modality retrieval. Enhance the cue vector for audio modality retrieval. , , and These are semantic encoders for the corresponding modalities, used to map the retrieved content into vectors. This is a retrieval model used to retrieve relevant semantic content from a domain multimodal database based on second training input data of different modalities. This is the second training input data for the text modality. The second training input data for the image modality. This serves as the second training input data for the video modality. This is the second training input data for the audio modality. This is a merging operation. Understandably, the first augmented data is the retrieval augmentation cue vector corresponding to the modality of the second training input data. Furthermore, when the second training input data contains multiple sub-data of different modalities, the first augmented data is the multi-modal cue vector resulting from the fusion of the retrieval augmentation cue vectors corresponding to each sub-data. .

[0041] Regarding the selection method of semantic content in the above process, the expression for obtaining relevant semantic content based on semantic similarity in this embodiment is as follows: , In the formula The most relevant knowledge fragments retrieved, i.e., the relevant semantic content, This represents a search query. This represents the complete set of all corresponding modal data in the domain multimodal database. For all retrieved candidate modal data, This is an operation to select the candidate modality data corresponding to the highest semantic similarity value as the relevant semantic content.

[0042] It should be noted that inputting the second training input data and the first augmented data generated through step S161 as prompts for content generation into the large language model enables multimodal input-driven large model output enhancement. Furthermore, since the nuclear energy field has a complex nuclear operating context, involving the collaboration of multiple different modal input data, this first augmented data is suitable for intelligent generation applications in complex nuclear operating contexts, thereby enhancing the performance of the final fine-tuned initial large language model.

[0043] Continue to refer to Figure 7 Step S162 involves integrating the candidate summary subgraphs associated with each triplet corresponding to the second training input data in the domain knowledge graph for each second training input data, thereby obtaining the second augmented data corresponding to the second training input data. Specifically, in this embodiment, multiple entities and relations are retrieved from the domain knowledge graph based on the second training input data, and then the candidate summary subgraphs corresponding to these entities and relations are integrated, i.e., graph summary fusion, to obtain globally relevant semantic content, which is then used as the second augmented data corresponding to the second training input data. Step S163 involves adding all the first and second augmented data to the second training dataset to obtain the augmented training dataset.

[0044] In this embodiment, step S16 introduces semantic similarity calculation, the Leiden community partitioning algorithm, multimodal cue vector generation, and graph summarization fusion mechanisms to achieve a technical approach that bidirectionally supplements the large model's generation capabilities from both the underlying semantic structure and graph semantic relationships. It features high automation and structural adaptability in cue construction and knowledge enhancement, overcoming the limitations of existing large models that rely on manual knowledge organization and cue design. By unifying semantic modeling and graph organization of heterogeneous data such as text, images, audio, and video, and combining graph neural network representation, implicit cue generation, and community structure optimization, it improves the large model's retrieval perception capabilities, structural understanding capabilities, and content generation accuracy in vertical domains.

[0045] Continue to refer to Figure 1 Step S17 involves fine-tuning the intermediate domain model using the augmented training dataset to obtain the final domain model. Specifically, in this embodiment, the expression for the output of the intermediate domain model during fine-tuning training is as follows: ,in, , In the formula The output of the intermediate domain large model, These are the trainable parameters for a large model in the intermediate domain. Extract prompt templates for the enhanced relationships after merging. For the second training input data, For the second training input data The corresponding first augmented data, For the second training input data The corresponding second augmented data, For including the second training input data First Enhanced Data Second Enhanced Data The prompt template. Understandably, when the second training input data only contains the corresponding first or second augmentation data, the corresponding augmentation data can also be used to enable the intermediate domain large model to generate more accurate output.

[0046] It should be noted that in step S17 of this embodiment, the parameters of each first fine-tuning module are adjusted by performing standard forward propagation, loss calculation, and backward gradient update on the enhanced training dataset, thereby obtaining a further optimized first fine-tuning module. That is, the final domain large model is an intermediate domain large model containing each optimized first fine-tuning module. It is understood that this embodiment does not adjust the parameters of the initial large language model itself, but only adjusts the hyperparameters of each candidate fine-tuning module and the parameters of each first fine-tuning module after pruning in different steps, thereby reducing the required training load and ensuring model performance. Furthermore, the process of adjusting only the parameters of the candidate fine-tuning modules and the first fine-tuning modules described above constitutes the fine-tuning training process of this embodiment.

[0047] It should be noted that in this embodiment, the data augmentation content generated in steps S15-S16 and the fine-tuning search content in steps S11-S14 are integrated through the domain large model fine-tuning method 100. Specifically, the first augmented data based on the domain multimodal database and the second augmented data based on the domain knowledge graph can collaboratively drive the fine-tuning training of the first fine-tuning module. Therefore, the domain large model fine-tuning method 100 constructs a domain large model fine-tuning call architecture with task adaptability, parameter efficiency, and prompt linkage, significantly improving its responsiveness and knowledge perception accuracy in multi-task scenarios while maintaining the stability of the initial large language model.

[0048] Reference Figure 9 In this embodiment, the trained final domain-wide model is used as follows: The user inputs the question (input data) into the industrial and intelligent manufacturing domain-wide model application task (final domain-wide model). The final domain-wide model then retrieves the corresponding semantic retrieval (first augmented data) from an external document library (domain multimodal database) and obtains the community summary (subgraph) (second augmented data) from a domain knowledge graph constructed based on the domain multimodal database through graph retrieval. Subsequently, through a preset prompt word template (a dual Prompt enhancement based on knowledge graph and semantic retrieval), the original question (input data), semantic retrieval (first augmented data), and community summary (second augmented data) are fused and input into the large model (LLM) (final domain-wide model), thereby generating the output data corresponding to the input data.

[0049] Reference Figure 10 One embodiment of this application proposes a domain large model fine-tuning system 200. For example... Figure 10As shown, the domain large model fine-tuning system 200 includes a data acquisition module 21, a search space construction module 22, an initial domain large model construction module 23, a fine-tuning strategy optimization module 24, an augmented data construction module 25, and a fusion training module 26. Specifically, the data acquisition module 21 is used to acquire an initial large language model, a first training dataset, a second training dataset, a domain knowledge graph, and a domain multimodal database. The domain multimodal database includes multiple modal data, the domain knowledge graph includes multiple triples and summary candidate subgraphs associated with the triples, and the second training dataset includes multiple second training input data. The search space construction module 22 is used to construct the parameter fine-tuning search space corresponding to the initial large language model. The parameter fine-tuning search space includes multiple candidate fine-tuning modules. The initial domain large model construction module 23 is used to insert multiple candidate fine-tuning modules into the initial large language model to obtain an initial domain large model. The fine-tuning strategy optimization module 24 is used to train the initial domain large model based on the first training dataset to obtain an intermediate domain large model containing at least one first fine-tuning module. In this embodiment, at least one first fine-tuning module is at least one candidate fine-tuning module trained from multiple candidate fine-tuning modules. The fine-tuning strategy optimization module 24 includes a pruning unit 241 and an optimization unit 242. The pruning unit 241 progressively prunes multiple candidate fine-tuning modules in the initial domain model based on the first training dataset, the type range of each candidate fine-tuning module, and the insertion position range, to obtain a domain model to be trained containing at least one candidate fine-tuning module as a fine-tuning module to be trained. The optimization unit 242 performs Bayesian optimization on the domain model to be trained based on the first training dataset and the value range of the hyperparameters corresponding to at least one fine-tuning module to be trained, to obtain an intermediate domain model containing at least one fine-tuning module with optimized corresponding hyperparameters as the first fine-tuning module.

[0050] Continue to refer to Figure 10 The augmented data construction module 25 is used to augment the second training dataset based on the domain knowledge graph and the domain multimodal database to obtain an augmented training dataset. The augmented data construction module 25 includes a first data augmentation unit 251, a second data augmentation unit 252, and a fusion unit 253. The first data augmentation unit 251 is used to obtain the corresponding modality data from the domain multimodal database for each second training input data using a semantic similarity method, and generate first augmented data corresponding to the second training input data based on all obtained modality data. The second data augmentation unit 252 is used to integrate the various summary candidate subgraphs associated with each triplet corresponding to the second training input data in the domain knowledge graph for each second training input data to obtain second augmented data corresponding to the second training input data. The fusion unit 253 is used to add all the first and second augmented data to the second training dataset to obtain the augmented training dataset. (Continue to refer to...) Figure 10 The fusion training module 26 is used to fine-tune the initial large language model with at least one first fine-tuning module inserted based on the augmented training dataset to obtain the final domain large model.

[0051] An embodiment of this application also proposes a method such as Figure 11 The electronic device 30 shown. According to Figure 11 The electronic device 30 may include an internal communication bus 31, a processor 32, a read-only memory (ROM) 33, a random access memory (RAM) 34, and a communication port 35. When used in a personal computer, the electronic device 30 may also include a hard disk 36.

[0052] The internal communication bus 31 enables data communication between components of the electronic device 30. The processor 32 can make judgments and issue prompts. In some embodiments, the processor 32 may consist of one or more processors. The communication port 35 enables data communication between the electronic device 30 and external devices. In some embodiments, the electronic device 30 can send and receive information and data from a network through the communication port 35.

[0053] Electronic device 30 may also include different forms of program storage units and data storage units, such as hard disk 36, read-only memory (ROM) 33, and random access memory (RAM) 34, capable of storing various data files used for computer processing and / or communication, as well as possible program instructions executed by processor 32. The processor executes these instructions to implement the main parts of the domain-wide model fine-tuning method. The results of processor processing are transmitted to the user device via a communication port and displayed on the user interface.

[0054] In addition, this application also proposes a computer-readable medium storing computer program code that implements the above-described domain large model fine-tuning method when executed by a processor.

[0055] The basic concepts have been described above. Obviously, for those skilled in the art, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore remain within the spirit and scope of the exemplary embodiments of this application.

[0056] Furthermore, this application uses specific terms to describe embodiments of the application. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of the application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of the application can be appropriately combined.

[0057] Similarly, it should be noted that, in order to simplify the description of the present application and thus aid in the understanding of one or more embodiments, the foregoing description of the embodiments of the present application sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this disclosure method does not imply that the subject matter of the present application requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of the single embodiments disclosed above.

[0058] Some aspects of this application can be executed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The aforementioned hardware or software may be referred to as a "data block," "module," "engine," "unit," "component," or "system." The processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or combinations thereof. Furthermore, aspects of this application may manifest as computer products residing in one or more computer-readable media, including computer-readable program code. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), optical discs (e.g., compressed CDs, digital multifunction DVDs, etc.), smart cards, and flash memory devices (e.g., cards, sticks, key drives, etc.).

[0059] A computer-readable medium may contain a propagated data signal containing computer program code, for example, on baseband or as part of a carrier wave. This propagated signal may take various forms, including electromagnetic, optical, and so on, or suitable combinations thereof. A computer-readable medium can be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, apparatus, or device to enable communication, propagation, or transmission of a program for use. The program code located on the computer-readable medium can be propagated through any suitable medium, including radio, cable, fiber optic cable, radio frequency signals, or similar media, or any combination of the above media.

[0060] Although this application has been described with reference to specific embodiments, those skilled in the art should recognize that the above embodiments are only used to illustrate this application, and various equivalent changes or substitutions can be made without departing from the spirit of this application. Therefore, any changes or modifications to the above embodiments within the essential spirit of this application will fall within the scope of the claims of this application.

Claims

1. A method for fine-tuning a large domain model, characterized in that, The domain-specific large model fine-tuning method is suitable for fine-tuning and training an existing initial large language model to obtain a final domain-specific large model associated with a specific domain, namely the nuclear energy domain. The domain-specific large model fine-tuning method includes the following steps: Obtain the initial large language model and the first training dataset, which is constructed based on corpus data in the nuclear energy field; Construct a parameter fine-tuning search space corresponding to the initial large language model, wherein the parameter fine-tuning search space includes multiple candidate fine-tuning modules; The candidate fine-tuning modules are inserted into the initial large language model to obtain the initial large domain model; The initial domain large model is trained based on the first training dataset to obtain an intermediate domain large model containing at least one first fine-tuning module, wherein the at least one first fine-tuning module is at least one candidate fine-tuning module trained among the plurality of candidate fine-tuning modules. Acquire a second training dataset, a domain knowledge graph, and a domain multimodal database. The second training dataset includes multiple second training input data, which include text modality, image modality, video modality, and audio modality. The second training dataset is augmented using the domain knowledge graph and the domain multimodal database to obtain an augmented training dataset. The intermediate domain model is fine-tuned and trained using the augmented training dataset to obtain the final domain model. The input data for the final domain-specific large model is the question, and the output data is the generated content. The final domain big model is adapted to obtain corresponding first augmented data from the domain multimodal database by retrieval, obtain second augmented data from the domain knowledge graph by graph retrieval, and generate the output data corresponding to the question based on the question, the first augmented data and the second augmented data.

2. The domain-wide model fine-tuning method as described in claim 1, characterized in that, The parameter fine-tuning search space also includes the type range, insertion position range, and corresponding hyperparameter value range of each candidate fine-tuning module. The step of training the initial domain large model based on the first training dataset to obtain an intermediate domain large model containing at least one first fine-tuning module further includes: Based on the first training dataset, the type range of each candidate fine-tuning module, and the insertion position range, progressive pruning is performed on the multiple candidate fine-tuning modules in the initial domain large model to obtain a training domain large model that includes at least one candidate fine-tuning module as a training fine-tuning module. Based on the first training dataset and the value range of the hyperparameters corresponding to at least one of the fine-tuning modules to be trained, Bayesian optimization is performed on the large model of the domain to be trained, resulting in the large model of the domain to be trained containing at least one well-optimized corresponding hyperparameter as the intermediate large model of the first fine-tuning module.

3. The domain-wide model fine-tuning method as described in claim 2, characterized in that, The step of progressively pruning the multiple candidate fine-tuning modules in the initial domain large model according to the first training dataset, the type range of each candidate fine-tuning module, and the insertion position range, to obtain a training domain large model containing at least one candidate fine-tuning module as a training fine-tuning module, further includes: Step a: Determine whether the total number of parameters of the multiple candidate fine-tuning modules is less than the total number of parameters threshold. If the determination result is yes, proceed to step b; otherwise, proceed to step c. Step b: Select each of the candidate fine-tuning modules as the fine-tuning module to be trained; Step c: Calculate the resource consumption value of each of the candidate fine-tuning modules, remove at least one of the candidate fine-tuning modules with the largest resource consumption value from the initial domain large model, and return to step a.

4. The domain-wide model fine-tuning method as described in claim 2, characterized in that, The objective expression for the Bayesian optimization is: ， In the formula Fine-tune the search space for the parameters. In the current hyperparameter configuration and corresponding configurable optimal weights The performance metric function for the initial domain large model is described below. Configure the current hyperparameters The total number of parameters in the initial domain large model described below, wherein the constraint expression of the objective expression is: ， In the formula In the current hyperparameter configuration and corresponding configurable weights The training loss function for the initial domain-wide model. For the argmin operation, To be subject to.

5. The domain-wide model fine-tuning method as described in claim 1, characterized in that, The domain multimodal database includes multiple modal data, and the domain knowledge graph includes multiple triples and multiple summary candidate subgraphs associated with the multiple triples.

6. The domain-wide model fine-tuning method as described in claim 5, characterized in that, The step of augmenting the second training dataset based on the domain knowledge graph and the domain multimodal database to obtain an augmented training dataset further includes: For each of the second training input data, the corresponding modality data is obtained from the domain multimodal database using a semantic similarity method, and the first augmented data corresponding to the second training input data is generated based on all the obtained modality data. For each of the second training input data, the summarization candidate subgraphs associated with each of the triples corresponding to the second training input data in the domain knowledge graph are integrated to obtain the second augmented data corresponding to the second training input data; Add all the first augmented data and the second augmented data to the second training dataset to obtain the augmented training dataset.

7. The domain-wide model fine-tuning method as described in claim 6, characterized in that, For each of the second training input data, the step of obtaining the corresponding modality data from the domain multimodal database using a semantic similarity method, and generating the first augmented data corresponding to the second training input data based on the modality data, further includes: Generate a search statement based on the second training input data; Obtain a retrieval model, and enable the retrieval model to retrieve the corresponding modality data from the domain multimodal database based on the retrieval statement and in combination with cosine similarity; The first augmented data is obtained by semantically encoding all the acquired modal data and the second training input data.

8. The domain-wide model fine-tuning method as described in claim 5, characterized in that, The steps for obtaining the domain knowledge graph include: Obtain entity recognition prompt templates, relation extraction prompt templates, and existing large language models; The domain knowledge graph, which contains multiple triples, is constructed based on the modal data of the domain multimodal database using the entity recognition prompt template, relation extraction prompt template, and existing large language model. The domain knowledge graph is divided into multiple subgraphs as candidate summary subgraphs using a graph partitioning algorithm, wherein each triple is associated with one of the candidate summary subgraphs.

9. The domain-wide model fine-tuning method as described in claim 8, characterized in that, The step of constructing the corresponding domain knowledge graph based on the modal data of the domain multimodal database using the entity recognition prompt template, relation extraction prompt template, and existing large language model further includes: The modal data and the entity recognition prompt template are input into the existing large language model to obtain multiple entities; The extraction prompt templates for the multiple entities and the relationships are input into the existing large language model to obtain multiple triples.

10. The domain-wide model fine-tuning method as described in claim 8, characterized in that, The graph partitioning algorithm includes the Leiden algorithm, and the modularity of the Leiden algorithm is... The calculation expression is: ， In the formula The sum of the weights of all edges in the domain knowledge graph. For nodes in the domain knowledge graph and nodes The weight of the edges between them. For nodes The degree, For nodes The degree, For nodes and nodes The corresponding indicator function.

11. A domain-wide large model fine-tuning system, characterized in that, A fine-tuning system suitable for fine-tuning existing initial large language models to obtain a final domain-specific large model associated with a specific domain, wherein the specific domain is the nuclear energy domain, and the domain-specific large model fine-tuning system includes: The data acquisition module is used to acquire the initial large language model, the first training dataset, the second training dataset, the domain knowledge graph, and the domain multimodal database. The first training dataset is constructed based on corpus data in the nuclear energy field. The second training dataset includes multiple second training input data, which include text modality, image modality, video modality, and audio modality. The search space construction module is used to construct the parameter fine-tuning search space corresponding to the initial large language model, wherein the parameter fine-tuning search space includes multiple candidate fine-tuning modules; An initial domain large model construction module is used to insert the multiple candidate fine-tuning modules into the initial large language model to obtain an initial domain large model. The fine-tuning strategy optimization module is used to train the initial domain large model based on the first training dataset to obtain an intermediate domain large model containing at least one first fine-tuning module, wherein the at least one first fine-tuning module is at least one candidate fine-tuning module trained among the plurality of candidate fine-tuning modules. An augmented data construction module is used to augment the second training dataset based on the domain knowledge graph and the domain multimodal database to obtain an augmented training dataset. The fusion training module is used to fine-tune the intermediate domain model based on the augmented training dataset to obtain the final domain model. The input data for the final domain-specific large model is the question, and the output data is the generated content. The final domain big model is adapted to obtain corresponding first augmented data from the domain multimodal database by retrieval, obtain second augmented data from the domain knowledge graph by graph retrieval, and generate the output data corresponding to the question based on the question, the first augmented data and the second augmented data.

12. The domain-wide large model fine-tuning system as described in claim 11, characterized in that, The parameter fine-tuning search space also includes the type range, insertion position range, and corresponding hyperparameter value range of each candidate fine-tuning module. The fine-tuning strategy optimization module includes: A pruning unit is configured to progressively prune the plurality of candidate fine-tuning modules in the initial domain model based on the first training dataset, the type range of each candidate fine-tuning module, and the insertion position range, to obtain a training domain model that includes at least one candidate fine-tuning module as a training fine-tuning module. An optimization unit is configured to perform Bayesian optimization on the large model of the training domain based on the first training dataset and the value range of the hyperparameters corresponding to at least one of the fine-tuning modules to be trained, so as to obtain the large model of the training domain containing at least one optimized corresponding hyperparameter as the intermediate large model of the first fine-tuning module.

13. The domain-wide model fine-tuning system as described in claim 11, characterized in that, The domain multimodal database includes multiple modal data, the domain knowledge graph includes multiple triples and summary candidate subgraphs associated with the triples, the second training dataset includes multiple second training input data, and the augmented data construction module includes: The first data augmentation unit is used to obtain the corresponding modality data from the domain multimodal database for each second training input data through a semantic similarity method, and generate the first augmented data corresponding to the second training input data based on all the obtained modality data; The second data augmentation unit is used to integrate the summarization candidate subgraphs associated with each triplet corresponding to the second training input data in the domain knowledge graph for each second training input data, so as to obtain the second augmented data corresponding to the second training input data. A fusion unit is used to add all the first augmented data and the second augmented data into the second training dataset to obtain the augmented training dataset.

14. An electronic device, comprising: Memory is used to store instructions that can be executed by the processor; And a processor for executing the instructions to implement the domain large model fine-tuning method as described in any one of claims 1-10.

15. A computer-readable medium storing computer program code that, when executed by a processor, implements the domain large model fine-tuning method as described in any one of claims 1-10.

Images