Fine tuning method and system for field large model, electronic equipment and readable medium

Abstract

The invention provides a field large model fine tuning method and system, electronic equipment and a readable medium, and relates to the technical field of artificial intelligence. The field large model fine tuning method comprises the steps of obtaining an initial large language model and a first training data set; constructing a parameter fine tuning search space corresponding to the initial large language model; inserting a plurality of candidate fine tuning modules into the initial large language model to obtain an initial field large model; training the initial domain large model according to the first training data set to obtain a middle domain large model comprising at least one first fine tuning module; obtaining a second training data set, a domain knowledge graph and a domain multi-modal database; performing data enhancement on the second training data set according to the domain knowledge graph and the domain multi-modal database to obtain an enhanced training data set; and performing fine tuning training on the intermediate field large model according to the enhanced training data set to obtain a final field large model.

Description

Technical Field

[0001] This application relates primarily to the field of artificial intelligence technology, and in particular to a method, system, electronic device, and readable medium for fine-tuning a large domain model. Background Technology

[0002] As the industrial energy sector and the power industry deepen their intelligent transformation, the industry has put forward higher requirements for the collection, processing and application capabilities of data resources. The value of data in vertical fields is gradually being valued, and manufacturing enterprises are continuously increasing their investment in building data-driven capabilities.

[0003] In the process of fine-tuning large language models in vertical domains, human experience is often relied upon, and there is a lack of effective automatic optimization mechanisms, which makes it impossible to efficiently fine-tune large language models in vertical domains.

[0004] Therefore, there is an urgent need for a domain-specific large model fine-tuning method that can efficiently fine-tune large language models in vertical domains. Summary of the Invention

[0005] The technical problem to be solved by this application is to provide a method, system, electronic device and readable medium for fine-tuning large language models in a domain, which can efficiently fine-tune large language models in a vertical domain.

[0006] To address the aforementioned technical problems, this application provides a method for fine-tuning a large domain model, comprising the following steps: obtaining an initial large language model and a first training dataset; constructing a parameter fine-tuning search space corresponding to the initial large language model, wherein the parameter fine-tuning search space includes multiple candidate fine-tuning modules; inserting the multiple candidate fine-tuning modules into the initial large language model to obtain an initial large domain model; training the initial large domain model based on the first training dataset to obtain an intermediate large domain model containing at least one first fine-tuning module, wherein the at least one first fine-tuning module is at least one candidate fine-tuning module trained from the multiple candidate fine-tuning modules; obtaining a second training dataset, a domain knowledge graph, and a domain multimodal database; performing data augmentation on the second training dataset based on the domain knowledge graph and the domain multimodal database to obtain an augmented training dataset; and fine-tuning the intermediate large domain model based on the augmented training dataset to obtain a final large domain model.

[0007] Optionally, the parameter fine-tuning search space also includes the type range, insertion position range, and corresponding hyperparameter value range of each candidate fine-tuning module. The step of training the initial domain large model based on the first training dataset to obtain an intermediate domain large model containing at least one first fine-tuning module further includes: progressively pruning multiple candidate fine-tuning modules in the initial domain large model based on the first training dataset, the type range, and insertion position range of each candidate fine-tuning module to obtain a domain large model to be trained containing at least one candidate fine-tuning module as a fine-tuning module to be trained; and performing Bayesian optimization on the domain large model to be trained based on the first training dataset and the value range of the hyperparameters corresponding to at least one fine-tuning module to be trained to obtain an intermediate domain large model containing at least one fine-tuning module with optimized corresponding hyperparameters as a first fine-tuning module.

[0008] Optionally, the step of progressively pruning multiple candidate fine-tuning modules in the initial domain model to obtain a training domain model containing at least one candidate fine-tuning module as a training fine-tuning module further includes: step a, determining whether the total number of parameters of multiple candidate fine-tuning modules is less than the total number of parameters threshold; if the determination result is yes, proceed to step b, otherwise proceed to step c; step b, using each candidate fine-tuning module as a training fine-tuning module; step c, calculating the resource consumption value of each candidate fine-tuning module, removing at least one candidate fine-tuning module with the largest resource consumption value from the initial domain model, and returning to step a.

[0009] Alternatively, the objective expression for Bayesian optimization is: In the formula To fine-tune the search space for parameters, In the current hyperparameter configuration and corresponding configurable optimal weights The performance metric function for the initial domain large model. Configure the current hyperparameters The total number of parameters in the initial domain model is given, where the constraint expression for the objective function is: In the formula In the current hyperparameter configuration and corresponding configurable weights The training loss function for the initial domain-wide model. For the argmin operation, To be subject to.

[0010] Optionally, the domain multimodal database includes multiple modal data, and the domain knowledge graph includes multiple triples and multiple summary candidate subgraphs associated with the multiple triples.

[0011] Optionally, the second training dataset includes multiple second training input data. The step of augmenting the second training dataset according to the domain knowledge graph and the domain multimodal database to obtain the augmented training dataset further includes: for each second training input data, obtaining the corresponding modality data from the domain multimodal database through a semantic similarity method, and generating the first augmented data corresponding to the second training input data based on all the obtained modality data; for each second training input data, integrating the various summary candidate subgraphs associated with each triplet corresponding to the second training input data in the domain knowledge graph to obtain the second augmented data corresponding to the second training input data; and adding all the first augmented data and the second augmented data to the second training dataset to obtain the augmented training dataset.

[0012] Optionally, for each second training input data, the step of obtaining corresponding modality data from the domain multimodal database using a semantic similarity method, and generating first augmented data corresponding to the second training input data based on the modality data further includes: generating a retrieval statement based on the second training input data; obtaining a retrieval model, and enabling the retrieval model to retrieve the corresponding modality data from the domain multimodal database based on the retrieval statement and in combination with cosine similarity; and semantically encoding all the obtained modality data and the second training input data to obtain the first augmented data.

[0013] Optionally, the steps for obtaining a domain knowledge graph include: obtaining entity recognition prompt templates, relation extraction prompt templates, and existing large language models; constructing a domain knowledge graph containing multiple triples based on modal data from a domain multimodal database using entity recognition prompt templates, relation extraction prompt templates, and existing large language models; and dividing the domain knowledge graph into multiple subgraphs as candidate summary subgraphs using a graph partitioning algorithm, wherein each triple is associated with a candidate summary subgraph.

[0014] Optionally, the step of constructing a corresponding domain knowledge graph based on the modal data of the domain multimodal database using entity recognition prompt templates, relation extraction prompt templates, and an existing large language model further includes: inputting the modal data and entity recognition prompt templates into the existing large language model to obtain multiple entities; and inputting all the multiple entities and relation extraction prompt templates into the existing large language model to obtain multiple triples.

[0015] Optionally, the graph partitioning algorithm includes Leiden's algorithm, and the modularity of Leiden's algorithm is... The calculation expression is: In the formula This is the sum of the weights of all edges in the domain knowledge graph. For nodes in a domain knowledge graph and nodes The weight of the edges between them. For nodes The degree, For nodes The degree, For nodes and nodes The corresponding indicator function.

[0016] To address the aforementioned technical problems, this application provides a domain-wide large model fine-tuning system, comprising: a data acquisition module for acquiring an initial large language model, a first training dataset, a second training dataset, a domain knowledge graph, and a domain multimodal database; a search space construction module for constructing a parameter fine-tuning search space corresponding to the initial large language model, wherein the parameter fine-tuning search space includes multiple candidate fine-tuning modules; an initial domain-wide large model construction module for inserting multiple candidate fine-tuning modules into the initial large language model to obtain an initial domain-wide large model; a fine-tuning strategy optimization module for training the initial domain-wide large model based on the first training dataset to obtain an intermediate domain-wide large model containing at least one first fine-tuning module, wherein the at least one first fine-tuning module is at least one candidate fine-tuning module trained from multiple candidate fine-tuning modules; an augmented data construction module for augmenting the second training dataset based on the domain knowledge graph and the domain multimodal database to obtain an augmented training dataset; and a fusion training module for fine-tuning the intermediate domain-wide large model based on the augmented training dataset to obtain a final domain-wide large model.

[0017] Optionally, the parameter fine-tuning search space also includes the type range, insertion position range, and corresponding hyperparameter value range of each candidate fine-tuning module. The fine-tuning strategy optimization module includes: a pruning unit, which is used to progressively prune multiple candidate fine-tuning modules in the initial domain large model according to the first training dataset, the type range, and insertion position range of each candidate fine-tuning module, to obtain a domain large model to be trained that includes at least one candidate fine-tuning module as a fine-tuning module to be trained; and an optimization unit, which is used to perform Bayesian optimization on the domain large model to be trained according to the first training dataset and the value range of the hyperparameters corresponding to at least one fine-tuning module to be trained, to obtain an intermediate domain large model that includes at least one fine-tuning module with optimized corresponding hyperparameters as the first fine-tuning module.

[0018] Optionally, the domain multimodal database includes multiple modal data, the domain knowledge graph includes multiple triples and summary candidate subgraphs associated with the triples, the second training dataset includes multiple second training input data, and the augmented data construction module includes: a first data augmentation unit, which is used to obtain the corresponding modal data from the domain multimodal database for each second training input data through a semantic similarity method, and generate first augmented data corresponding to the second training input data based on all the obtained modal data; a second data augmentation unit, which is used to integrate the various summary candidate subgraphs associated with each triple corresponding to the second training input data in the domain knowledge graph for each second training input data to obtain the second augmented data corresponding to the second training input data; and a fusion unit, which is used to add all the first augmented data and the second augmented data to the second training dataset to obtain the augmented training dataset.

[0019] To address the aforementioned technical problems, this application proposes an electronic device comprising: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the aforementioned domain-wide model fine-tuning method.

[0020] To address the aforementioned technical problems, this application proposes a computer-readable medium storing computer program code, which, when executed by a processor, implements the aforementioned domain-wide model fine-tuning method.

[0021] Compared with existing technologies, this application has the following advantages: At least one first fine-tuning module capable of optimizing the performance of an initial large language model is selected and trained from the parameter fine-tuning search space using a first training dataset, thereby obtaining an intermediate domain-specific large model. Data augmentation is performed on the second training dataset using data from a domain knowledge graph and a domain multimodal database, resulting in an augmented training dataset with enhanced semantic information for a specific domain. Based on this, the intermediate domain-specific large model is fine-tuned using the augmented training dataset, resulting in a final domain-specific large model with stronger performance in that domain. Attached Figure Description

[0022] The accompanying drawings are included to provide a further understanding of this application; they are incorporated into and constitute a part of this application. The drawings illustrate embodiments of this application and, together with this specification, serve to explain the principles of this application. In the drawings: Figure 1 This is a flowchart illustrating a domain large model fine-tuning method according to an embodiment of this application; Figure 2 yes Figure 1 A flowchart illustrating the sub-steps of step S14; Figure 3 yes Figure 2A flowchart illustrating the sub-steps of step S141; Figure 4 yes Figure 1 A schematic diagram of the pruning and optimization stages in step S14; Figure 5 This is a flowchart illustrating the process of acquiring a domain knowledge graph; Figure 6 yes Figure 5 A flowchart illustrating the sub-steps of step S22; Figure 7 yes Figure 1 A flowchart illustrating the sub-steps of step S16; Figure 8 yes Figure 7 A flowchart illustrating the sub-steps of step S161; Figure 9 This is a flowchart illustrating the process of using the final domain-wide model; Figure 10 This is a block diagram of a domain large model fine-tuning system according to an embodiment of this application; and Figure 11 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this application. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0024] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0025] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0026] In the description of this application, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is usually based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this application and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this application; the directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0027] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0028] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, these terms have no special meaning and therefore should not be construed as limiting the scope of protection of this application. In addition, although the terminology used in this application is selected from commonly known and used terms, some terms mentioned in this application's specification may have been chosen by the applicant according to his or her judgment, and their detailed meanings are explained in the relevant sections of this description. Moreover, this application should be understood not only through the actual terms used, but also through the meaning implied by each term.

[0029] It should be understood that when a component is referred to as "on another component," "connected to another component," "coupled to another component," or "in contact with another component," it can be directly on, connected to, coupled to, or in contact with that other component, or there may be an intervening component. In contrast, when a component is referred to as "directly on another component," "directly connected to," "directly coupled to," or "directly in contact with" another component, there is no intervening component. Similarly, when a first component is referred to as "electrically contacting" or "electrically coupled to" a second component, there is an electrical path between the first and second components that allows current to flow. This electrical path may include capacitors, coupled inductors, and / or other components that allow current to flow, even if there is no direct contact between the conductive components.

[0030] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more steps may be removed from these processes.

[0031] Reference Figure 1 One embodiment of this application proposes a domain-specific large-scale model fine-tuning method 100. This domain-specific large-scale model fine-tuning method 100 is suitable for fine-tuning and training an existing initial large-scale language model to obtain a final domain-specific large-scale model. In this embodiment, the specific domain is the nuclear energy domain. It should be noted that this application does not limit the specific type of the specific domain. In some embodiments, the specific domain is other vertical domains, such as the power industry.

[0032] Continue to refer to Figure 1The domain-specific large-scale language model fine-tuning method 100 in this embodiment includes the following steps. Step S11 is to obtain an initial large-scale language model and a first training dataset. In this embodiment, the first training dataset is constructed based on corpus data from a specific domain. Step S12 is to construct a parameter fine-tuning search space corresponding to the initial large-scale language model. The parameter fine-tuning search space includes multiple candidate fine-tuning modules, as well as the type range, insertion position range, and corresponding hyperparameter value range for each candidate fine-tuning module. Specifically, in this embodiment, the type range of candidate fine-tuning modules includes LoRA, ParallelAdapter (PA), Serial Adapter (SA), Prefix Tuning (PA), and Bitfit; that is, in this embodiment, corresponding candidate fine-tuning modules are constructed according to the above-mentioned multiple parameter fine-tuning methods. Furthermore, in this embodiment, multiple candidate fine-tuning modules correspond to different levels and modules within the Transformer structure of the initial large-scale language model. That is, multiple candidate fine-tuning modules constructed by the same parameter fine-tuning method correspond to different levels and modules within the Transformer structure, and the same level or module within the Transformer structure can correspond to multiple candidate fine-tuning modules constructed by different parameter fine-tuning methods. It should be noted that in this embodiment, the corresponding candidate fine-tuning modules are inserted into the corresponding positions within the Transformer structure according to the parameters targeted by the above-described parameter fine-tuning method; this will not be elaborated upon here. The insertion position range of each type of candidate fine-tuning module in this embodiment corresponds to the various levels and modules that the corresponding parameter fine-tuning method can correspond to within the Transformer structure of the initial large language model. In this embodiment, the value range of hyperparameters includes rank, prefix length, and intermediate layer dimension control terms.

[0033] Continue to refer to Figure 1 Step S13 involves inserting multiple candidate fine-tuning modules into the initial large language model to obtain the initial domain large model. Specifically, in this embodiment, each candidate fine-tuning module is first inserted into the corresponding layer and module within the Transformer structure of the initial large language model, and the hyperparameters of each candidate fine-tuning module are set to preset empirical values. Step S14 involves training the initial domain large model based on the first training dataset to obtain an intermediate domain large model containing at least one first fine-tuning module. Here, at least one first fine-tuning module is at least one candidate fine-tuning module trained from among multiple candidate fine-tuning modules; that is, in step S14, at least one candidate fine-tuning module is selected and trained from among multiple candidate fine-tuning modules using the first training dataset, and the candidate fine-tuning module trained on the first training dataset is used as the first fine-tuning module. Further refer to... Figure 2Step S14 includes the following sub-steps. Step S141 involves progressively pruning multiple candidate fine-tuning modules in the initial domain large model based on the first training dataset, the type range of each candidate fine-tuning module, and the insertion position range, to obtain a training domain large model containing at least one candidate fine-tuning module as a training fine-tuning module. Specifically, in step S141 of this embodiment, the weights of each candidate fine-tuning module in the initial domain large model are initialized, and low-fidelity training is performed using the first training dataset. Then, the candidate fine-tuning modules are pruned according to a specific pruning strategy, and training and pruning are continuously performed iteratively until the number of parameters of the remaining candidate fine-tuning modules after pruning drops below a preset threshold. The remaining candidate fine-tuning modules are then used as the first fine-tuning modules. It can be understood that the main search objective of step S141 is to determine the type of the candidate fine-tuning modules and their optimal insertion layer position in the Transformer structure. That is, to determine the type range and insertion position range of multiple candidate fine-tuning modules suitable for insertion into the initial large language model in the parameter fine-tuning search space. Furthermore, by setting the hyperparameters of the candidate fine-tuning modules to empirical values ​​and keeping them frozen in the previous step S13, the influence of the hyperparameter values ​​on the selection of the optimal type and insertion position from the type range and insertion position range can be avoided. Thus, in step S141, the optimal type and optimal insertion position of each candidate fine-tuning module applicable to the initial domain large model can be obtained.

[0034] For the cyclic pruning process in step S141 above, further refer to Figure 3 Step S141 includes the following sub-steps. Step a is to determine whether the total number of parameters of all candidate fine-tuning modules is less than the total number of parameters threshold. If the determination result is yes, proceed to step b; otherwise, proceed to step c. Step b is to use each candidate fine-tuning module as a fine-tuning module to be trained. Step c is to calculate the resource consumption value of each candidate fine-tuning module, remove at least one candidate fine-tuning module with the largest resource consumption value from the initial domain large model, and return to step a. It should be noted that in this embodiment, in each loop pruning, i.e., in step c, after removing the candidate fine-tuning module, the weight values ​​of the candidate fine-tuning modules need to be re-initialized. Accordingly, in this embodiment, the resource consumption value is determined based on the activation value, weight value, and loss change corresponding to the candidate fine-tuning module. In one example, after training the candidate fine-tuning module based on the first training dataset, the corresponding resource consumption value is calculated from the activation value, weight value, and loss change corresponding to the candidate fine-tuning module.

[0035] Continue to refer to Figure 2Step S142 involves performing Bayesian optimization on the large-scale model to be trained based on the first training dataset and the range of hyperparameter values ​​corresponding to at least one fine-tuning module to be trained. This yields a large-scale model to be trained containing at least one well-optimized corresponding hyperparameter, serving as an intermediate large-scale model for the first fine-tuning module. Specifically, in step S142 of this embodiment, the Bayesian optimizer samples a set of hyperparameter configurations from the range of hyperparameter values ​​in the parameter fine-tuning search space and performs multi-fidelity training and evaluation on the large-scale model to be trained using the first training dataset. The evaluated fine-tuning performance metrics and total number of parameters are then fed back to the Bayesian optimizer to update the internal acquisition function. Next, hyperparameter configurations with greater potential are resampled based on the updated acquisition function. After a specified number of cyclic searches, a multi-objective Pareto optimal solution is finally obtained. Correspondingly, the objective expression for Bayesian optimization in this embodiment is: , In the formula To fine-tune the search space for parameters, In the current hyperparameter configuration and corresponding configurable optimal weights The performance metric function for the initial domain large model, i.e., the domain large model to be trained in step S142. Configure the current hyperparameters The total number of parameters in the initial domain model. Understandably, the result of the performance metric function calculation is the aforementioned fine-tuning performance index. Furthermore, the constraint expression of the objective expression in this embodiment is: , In the formula In the current hyperparameter configuration and corresponding configurable weights The training loss function for the initial domain-wide model. For the argmin operation, "Subject to" means "constrained by". This is understandable; it relates to the training loss function. Used to measure the current hyperparameter configuration With corresponding configurable optimal weights The loss value of the initial domain large model on the first training dataset.

[0036] Continue to refer to Figure 4In step S14 of this embodiment, during the pruning stage, after initializing the weights of each candidate fine-tuning module for the constructed parameter fine-tuning search space, low-fidelity training is performed using the first training dataset to determine the resource consumption value of each candidate fine-tuning module. Subsequently, a spatial pruner is used to sparsely prune candidate fine-tuning modules with high resource consumption values, thereby reducing the number of candidate fine-tuning modules in the parameter fine-tuning search space. This process is repeated until the total number of parameters of all candidate fine-tuning modules is less than a parameter total threshold. Then, the remaining candidate fine-tuning modules in the parameter fine-tuning search space are determined to be used in the large model to be trained, and the type and insertion position of each candidate fine-tuning module are determined. In the optimization stage following the pruning stage, after fixing the type and insertion position of each candidate fine-tuning module, multi-objective Bayesian optimization is used for the hyperparameters of each candidate fine-tuning module to obtain the optimal solution for the hyperparameter configuration of each candidate fine-tuning module, i.e., the better specific values ​​of the hyperparameters. The candidate fine-tuning module with the determined type, insertion position, and hyperparameters is then used as the first fine-tuning module in the intermediate domain large model. Understandably, step S14 of this embodiment achieves joint automatic search of candidate fine-tuning module types, insertion positions, and hyperparameter configurations in a large-scale search space by integrating progressive pruning and multi-objective Bayesian optimization. This effectively overcomes the problems of traditional fine-tuning strategy selection relying on experience settings and low combination efficiency, and achieves synergistic improvement in search accuracy, parameter control capability, and fine-tuning adaptation performance.

[0037] Continue to refer to Figure 1 Step S15 involves acquiring the second training dataset, the domain knowledge graph, and the domain multimodal database. The domain multimodal database includes multiple modalities of data. It is understood that in this embodiment, the domain multimodal database can be directly acquired or constructed based on existing domain data. The modal types of the modal data include text, images, audio, and video. Furthermore, in this embodiment, the domain knowledge graph includes multiple triples and multiple summary candidate subgraphs associated with the triples. For further details, please refer to... Figure 5 The process of obtaining a domain knowledge graph in this embodiment includes the following steps. Step S21 is to obtain entity recognition prompt templates, relation extraction prompt templates, and an existing large language model. In this embodiment, the existing large language model is any open-source or directly purchaseable large language model. Step S22 is to construct a domain knowledge graph containing multiple triples based on data from a domain multimodal database using the entity recognition prompt templates, relation extraction prompt templates, and the existing large language model. Specifically, in this embodiment, the entity recognition prompt templates and relation extraction prompt templates guide the existing large language model to automatically identify the relation triples between various entity sets and entity pairs in a zero-shot manner.

[0038] Further reference Figure 6Step S22 includes the following sub-steps. Step S221 involves inputting the modal data and entity recognition prompt template into an existing large language model to obtain multiple entities. That is, all modal data is input into the existing large language model to obtain multiple entities. Step S222 involves inputting the multiple entities and relation extraction prompt template into the existing large language model to obtain multiple triples. In this embodiment, the expressions for obtaining multiple entities and triples in steps S221 and S222 are as follows: , , In the formula For a collection of entities, A single entity and belonging to a set of entities The elements in This indicates the entity recognition prompt template. Input is fed into an existing large language model and combined with the model parameters of the existing large language model. To generate entity sequences, For a set of triples, For entities and entity and the corresponding relationship The triples formed, and belong to the set of triples. The elements in This indicates a prompt template for extracting relationships. With two entities and After inputting them into the existing large language model, the model parameters of the existing large language model are used. The result obtained from entity relation identification. It can be understood that in this embodiment, the entities in the triples serve as nodes in the domain knowledge graph, and the relations in the triples serve as edges connecting the corresponding nodes, thus forming the domain knowledge graph, which is represented as follows: , In the formula Represents the structure of a domain knowledge graph. This represents the set of nodes in a domain knowledge graph. Represents the set of edges in a domain knowledge graph. This represents a node in the domain knowledge graph. This represents an edge in a domain knowledge graph.

[0039] Continue to refer to Figure 5Step S23 involves dividing the domain knowledge graph into multiple subgraphs as candidate summary subgraphs using a graph partitioning algorithm. Each triple is associated with a candidate summary subgraph. In this embodiment, the graph partitioning algorithm includes a community partitioning algorithm. Specifically, the community partitioning algorithm first treats each node in the obtained domain knowledge graph as a separate community, and then iteratively adjusts the community partitioning in the domain knowledge graph to improve the modularity of each community. Each iteration includes two main steps: one is moving nodes, where it is determined whether moving a node from one community to another can improve the modularity; if the result is yes, the node is moved to the other community. The other step is merging communities, where it is determined whether merging two other communities can improve the modularity; if the result is yes, the two communities are merged. In this embodiment, the community partitioning algorithm includes the Leiden algorithm, and the modularity of the Leiden algorithm... The calculation expression is: , In the formula This is the sum of the weights of all edges in the domain knowledge graph. For nodes in a domain knowledge graph and nodes The weight of the edges between them. For nodes The degree, For nodes The degree, For nodes and nodes The corresponding indicator function. Where, when the domain knowledge graph is an unweighted graph, it is... The total number of edges, and correspondingly, the number of edges corresponding to The value of is 1, corresponding to no edge. The value is 1. Furthermore, in this embodiment, when the node... and nodes Indicator function when belonging to the same community The value is 1 when the node and nodes Indicator function when not belonging to the same community The value of is 0.

[0040] Continue to refer to Figure 1 Step S16 involves augmenting the second training dataset using the domain knowledge graph and the domain multimodal database to obtain the augmented training dataset. Further refer to... Figure 7Step S16 includes the following sub-steps. Step S161 involves obtaining corresponding modality data from the domain multimodal database for each second training input data using a semantic similarity method, and generating first augmented data corresponding to the second training input data based on all obtained modality data. Further refer to... Figure 8 Step S161 includes the following sub-steps. Step S1611 generates a retrieval statement based on the second training input data. Specifically, in this embodiment, the corresponding retrieval statement is generated by combining the initial large language model with prompt words and the second training input data. Step S1612 obtains a retrieval model and enables the retrieval model to retrieve the corresponding modality data from the domain multimodal database based on the retrieval statement and cosine similarity. Specifically, in this embodiment, the retrieval model includes an existing embedding model, which can vectorize the retrieval statement, thereby enabling the retrieval model to perform vector retrieval of the retrieval statement in the domain multimodal database using cosine similarity to obtain the corresponding modality data. Step S1613 semantically encodes all the obtained modality data and the second training input data to obtain the first augmented data. Specifically, the expression for generating the first augmented data in step S16 is: , In the formula Enhance the cue vector for text-based retrieval. Enhance the cue vector for image modality retrieval. Enhance cue vectors for video modality retrieval. Enhance the cue vector for audio modality retrieval. , , and These are semantic encoders for the corresponding modalities, used to map the retrieved content into vectors. This is a retrieval model used to retrieve relevant semantic content from a domain multimodal database based on second training input data of different modalities. This is the second training input data for the text modality. The second training input data for the image modality. This serves as the second training input data for the video modality. This is the second training input data for the audio modality. This is a merging operation. Understandably, the first augmented data is the retrieval augmentation cue vector corresponding to the modality of the second training input data. Furthermore, when the second training input data contains multiple sub-data of different modalities, the first augmented data is the multi-modal cue vector resulting from the fusion of the retrieval augmentation cue vectors corresponding to each sub-data. .

[0041] Regarding the selection method of semantic content in the above process, the expression for obtaining relevant semantic content based on semantic similarity in this embodiment is as follows: , In the formula The most relevant knowledge fragments retrieved, i.e., the relevant semantic content, This represents a search query. This represents the complete set of all corresponding modal data in the domain multimodal database. For all retrieved candidate modal data, This is an operation to select the candidate modality data corresponding to the highest semantic similarity value as the relevant semantic content.

[0042] It should be noted that inputting the second training input data and the first augmented data generated through step S161 as prompts for content generation into the large language model enables multimodal input-driven large model output enhancement. Furthermore, since the nuclear energy field has a complex nuclear operating context, involving the collaboration of multiple different modal input data, this first augmented data is suitable for intelligent generation applications in complex nuclear operating contexts, thereby enhancing the performance of the final fine-tuned initial large language model.

[0043] Continue to refer to Figure 7 Step S162 involves integrating the candidate summary subgraphs associated with each triplet corresponding to the second training input data in the domain knowledge graph for each second training input data, thereby obtaining the second augmented data corresponding to the second training input data. Specifically, in this embodiment, multiple entities and relations are retrieved from the domain knowledge graph based on the second training input data, and then the candidate summary subgraphs corresponding to these entities and relations are integrated, i.e., graph summary fusion, to obtain globally relevant semantic content, which is then used as the second augmented data corresponding to the second training input data. Step S163 involves adding all the first and second augmented data to the second training dataset to obtain the augmented training dataset.

[0044] In this embodiment, step S16 introduces semantic similarity calculation, the Leiden community partitioning algorithm, multimodal cue vector generation, and graph summarization fusion mechanisms to achieve a technical approach that bidirectionally supplements the large model's generation capabilities from both the underlying semantic structure and graph semantic relationships. It features high automation and structural adaptability in cue construction and knowledge enhancement, overcoming the limitations of existing large models that rely on manual knowledge organization and cue design. By unifying semantic modeling and graph organization of heterogeneous data such as text, images, audio, and video, and combining graph neural network representation, implicit cue generation, and community structure optimization, it improves the large model's retrieval perception capabilities, structural understanding capabilities, and content generation accuracy in vertical domains.

[0045] Continue to refer to Figure 1 Step S17 involves fine-tuning the intermediate domain model using the augmented training dataset to obtain the final domain model. Specifically, in this embodiment, the expression for the output of the intermediate domain model during fine-tuning training is as follows: ,in, , In the formula The output of the intermediate domain large model, These are the trainable parameters for a large model in the intermediate domain. Extract prompt templates for the enhanced relationships after merging. For the second training input data, For the second training input data The corresponding first augmented data, For the second training input data The corresponding second augmented data, For including the second training input data First Enhanced Data Second Enhanced Data The prompt template. Understandably, when the second training input data only contains the corresponding first or second augmentation data, the corresponding augmentation data can also be used to enable the intermediate domain large model to generate more accurate output.

[0046] It should be noted that in step S17 of this embodiment, the parameters of each first fine-tuning module are adjusted by performing standard forward propagation, loss calculation, and backward gradient update on the enhanced training dataset, thereby obtaining a further optimized first fine-tuning module. That is, the final domain large model is an intermediate domain large model containing each optimized first fine-tuning module. It is understood that this embodiment does not adjust the parameters of the initial large language model itself, but only adjusts the hyperparameters of each candidate fine-tuning module and the parameters of each first fine-tuning module after pruning in different steps, thereby reducing the required training load and ensuring model performance. Furthermore, the process of adjusting only the parameters of the candidate fine-tuning modules and the first fine-tuning modules described above constitutes the fine-tuning training process of this embodiment.

[0047] It should be noted that in this embodiment, the data augmentation content generated in steps S15-S16 and the fine-tuning search content in steps S11-S14 are integrated through the domain large model fine-tuning method 100. Specifically, the first augmented data based on the domain multimodal database and the second augmented data based on the domain knowledge graph can collaboratively drive the fine-tuning training of the first fine-tuning module. Therefore, the domain large model fine-tuning method 100 constructs a domain large model fine-tuning call architecture with task adaptability, parameter efficiency, and prompt linkage, significantly improving its responsiveness and knowledge perception accuracy in multi-task scenarios while maintaining the stability of the initial large language model.

[0048] Reference Figure 9 In this embodiment, the trained final domain-wide model is used as follows: The user inputs the question (input data) into the industrial and intelligent manufacturing domain-wide model application task (final domain-wide model). The final domain-wide model then retrieves the corresponding semantic retrieval (first augmented data) from an external document library (domain multimodal database) and obtains the community summary (subgraph) (second augmented data) from a domain knowledge graph constructed based on the domain multimodal database through graph retrieval. Subsequently, through a preset prompt word template (a dual Prompt enhancement based on knowledge graph and semantic retrieval), the original question (input data), semantic retrieval (first augmented data), and community summary (second augmented data) are fused and input into the large model (LLM) (final domain-wide model), thereby generating the output data corresponding to the input data.

[0049] Reference Figure 10 One embodiment of this application proposes a domain large model fine-tuning system 200. For example... Figure 10As shown, the domain large model fine-tuning system 200 includes a data acquisition module 21, a search space construction module 22, an initial domain large model construction module 23, a fine-tuning strategy optimization module 24, an augmented data construction module 25, and a fusion training module 26. Specifically, the data acquisition module 21 is used to acquire an initial large language model, a first training dataset, a second training dataset, a domain knowledge graph, and a domain multimodal database. The domain multimodal database includes multiple modal data, the domain knowledge graph includes multiple triples and summary candidate subgraphs associated with the triples, and the second training dataset includes multiple second training input data. The search space construction module 22 is used to construct the parameter fine-tuning search space corresponding to the initial large language model. The parameter fine-tuning search space includes multiple candidate fine-tuning modules. The initial domain large model construction module 23 is used to insert multiple candidate fine-tuning modules into the initial large language model to obtain an initial domain large model. The fine-tuning strategy optimization module 24 is used to train the initial domain large model based on the first training dataset to obtain an intermediate domain large model containing at least one first fine-tuning module. In this embodiment, at least one first fine-tuning module is at least one candidate fine-tuning module trained from multiple candidate fine-tuning modules. The fine-tuning strategy optimization module 24 includes a pruning unit 241 and an optimization unit 242. The pruning unit 241 progressively prunes multiple candidate fine-tuning modules in the initial domain model based on the first training dataset, the type range of each candidate fine-tuning module, and the insertion position range, to obtain a domain model to be trained containing at least one candidate fine-tuning module as a fine-tuning module to be trained. The optimization unit 242 performs Bayesian optimization on the domain model to be trained based on the first training dataset and the value range of the hyperparameters corresponding to at least one fine-tuning module to be trained, to obtain an intermediate domain model containing at least one fine-tuning module with optimized corresponding hyperparameters as the first fine-tuning module.

[0050] Continue to refer to Figure 10 The augmented data construction module 25 is used to augment the second training dataset based on the domain knowledge graph and the domain multimodal database to obtain an augmented training dataset. The augmented data construction module 25 includes a first data augmentation unit 251, a second data augmentation unit 252, and a fusion unit 253. The first data augmentation unit 251 is used to obtain the corresponding modality data from the domain multimodal database for each second training input data using a semantic similarity method, and generate first augmented data corresponding to the second training input data based on all obtained modality data. The second data augmentation unit 252 is used to integrate the various summary candidate subgraphs associated with each triplet corresponding to the second training input data in the domain knowledge graph for each second training input data to obtain second augmented data corresponding to the second training input data. The fusion unit 253 is used to add all the first and second augmented data to the second training dataset to obtain the augmented training dataset. (Continue to refer to...) Figure 10 The fusion training module 26 is used to fine-tune the initial large language model with at least one first fine-tuning module inserted based on the augmented training dataset to obtain the final domain large model.

[0051] An embodiment of this application also proposes a method such as Figure 11 The electronic device 30 shown. According to Figure 11 The electronic device 30 may include an internal communication bus 31, a processor 32, a read-only memory (ROM) 33, a random access memory (RAM) 34, and a communication port 35. When used in a personal computer, the electronic device 30 may also include a hard disk 36.

[0052] The internal communication bus 31 enables data communication between components of the electronic device 30. The processor 32 can make judgments and issue prompts. In some embodiments, the processor 32 may consist of one or more processors. The communication port 35 enables data communication between the electronic device 30 and external devices. In some embodiments, the electronic device 30 can send and receive information and data from a network through the communication port 35.

[0053] Electronic device 30 may also include different forms of program storage units and data storage units, such as hard disk 36, read-only memory (ROM) 33, and random access memory (RAM) 34, capable of storing various data files used for computer processing and / or communication, as well as possible program instructions executed by processor 32. The processor executes these instructions to implement the main parts of the domain-wide model fine-tuning method. The results of processor processing are transmitted to the user device via a communication port and displayed on the user interface.

[0054] In addition, this application also proposes a computer-readable medium storing computer program code that implements the above-described domain large model fine-tuning method when executed by a processor.

[0055] The basic concepts have been described above. Obviously, for those skilled in the art, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore remain within the spirit and scope of the exemplary embodiments of this application.

[0056] Furthermore, this application uses specific terms to describe embodiments of the application. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of the application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of the application can be appropriately combined.

[0057] Similarly, it should be noted that, in order to simplify the description of the present application and thus aid in the understanding of one or more embodiments, the foregoing description of the embodiments of the present application sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this disclosure method does not imply that the subject matter of the present application requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of the single embodiments disclosed above.

[0058] Some aspects of this application can be executed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The aforementioned hardware or software may be referred to as a "data block," "module," "engine," "unit," "component," or "system." The processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or combinations thereof. Furthermore, aspects of this application may manifest as computer products residing in one or more computer-readable media, including computer-readable program code. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), optical discs (e.g., compressed CDs, digital multifunction DVDs, etc.), smart cards, and flash memory devices (e.g., cards, sticks, key drives, etc.).

[0059] A computer-readable medium may contain a propagated data signal containing computer program code, for example, on baseband or as part of a carrier wave. This propagated signal may take various forms, including electromagnetic, optical, and so on, or suitable combinations thereof. A computer-readable medium can be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, apparatus, or device to enable communication, propagation, or transmission of a program for use. The program code located on the computer-readable medium can be propagated through any suitable medium, including radio, cable, fiber optic cable, radio frequency signals, or similar media, or any combination of the above media.

[0060] Although this application has been described with reference to specific embodiments, those skilled in the art should recognize that the above embodiments are only used to illustrate this application, and various equivalent changes or substitutions can be made without departing from the spirit of this application. Therefore, any changes or modifications to the above embodiments within the essential spirit of this application will fall within the scope of the claims of this application.

Claims

1. A domain large model fine-tuning method, characterized in that, The method comprises the following steps: obtaining an initial large language model and a first training data set; constructing a parameter fine-tuning search space corresponding to the initial large language model, wherein the parameter fine-tuning search space comprises a plurality of candidate fine-tuning modules; inserting the plurality of candidate fine-tuning modules into the initial large language model to obtain an initial domain large model; training the initial domain large model according to the first training data set to obtain an intermediate domain large model comprising at least one first fine-tuning module, the at least one first fine-tuning module being at least one trained candidate fine-tuning module from the plurality of candidate fine-tuning modules; obtaining a second training data set, a domain knowledge graph, and a domain multi-modal database; performing data augmentation on the second training data set according to the domain knowledge graph and the domain multi-modal database to obtain an augmented training data set; performing fine-tuning training on the intermediate domain large model according to the augmented training data set to obtain a final domain large model.

2. The domain large model fine-tuning method of claim 1, wherein, The parameter fine-tuning search space further comprises a type range, an insertion position range, and a value range of corresponding hyperparameters of each candidate fine-tuning module, and the step of training the initial domain large model according to the first training data set to obtain an intermediate domain large model comprising at least one first fine-tuning module further comprises: performing progressive pruning on the plurality of candidate fine-tuning modules in the initial domain large model according to the first training data set, the type range, and the insertion position range of each candidate fine-tuning module to obtain a to-be-trained domain large model comprising at least one candidate fine-tuning module as a to-be-trained fine-tuning module; performing Bayesian optimization on the to-be-trained domain large model according to the first training data set and the value range of the corresponding hyperparameters of at least one to-be-trained fine-tuning module to obtain the intermediate domain large model comprising at least one to-be-trained fine-tuning module with the corresponding hyperparameters optimized as the first fine-tuning module.

3. The domain large model fine-tuning method of claim 2, wherein, The step of performing progressive pruning on the plurality of candidate fine-tuning modules in the initial domain large model according to the first training data set, the type range, and the insertion position range of each candidate fine-tuning module to obtain a to-be-trained domain large model comprising at least one candidate fine-tuning module as a to-be-trained fine-tuning module further comprises: Step a: determining whether the total amount of parameters of the plurality of candidate fine-tuning modules is less than a total parameter threshold, if the result is yes, proceeding to step b, otherwise proceeding to step c; Step b: regarding each candidate fine-tuning module as the to-be-trained fine-tuning module; Step c: calculating the resource consumption value of each candidate fine-tuning module, removing at least one candidate fine-tuning module with the largest resource consumption value from the initial domain large model, and returning to step a.

4. The domain large model fine-tuning method of claim 2, wherein, The target expression of the Bayesian optimization is: ， In the formula is the parameter fine-tuning search space, is the performance metric function of the initial domain large model under the current hyperparameter configuration and the corresponding configurable optimal weight is the performance metric function of the initial domain large model under the current hyperparameter configuration is the total parameter quantity of the initial domain large model under the current hyperparameter configuration is the total parameter quantity of the initial domain large model under the current hyperparameter configuration ， wherein is the current hyperparameter configuration and the corresponding configurable weights the training loss function of the initial domain large model, is the argmin operation, is constrained to.

5. The domain large model fine-tuning method of claim 1, wherein, The domain multi-modal database comprises a plurality of modal data, and the domain knowledge graph comprises a plurality of triples and a plurality of abstract candidate subgraphs associated with the plurality of triples.

6. The domain large model fine-tuning method of claim 5, wherein, The second training data set includes a plurality of second training input data, and the step of performing data augmentation on the second training data set according to the domain knowledge graph and the domain multi-modal database to obtain an augmented training data set further includes: For each second training input data, the corresponding modal data is obtained from the domain multi-modal database by a semantic similarity method, and the first augmented data corresponding to the second training input data is generated according to all the obtained modal data; For each second training input data, the second augmented data corresponding to the second training input data is obtained by integrating each of the summary candidate subgraphs associated with each of the triples in the domain knowledge graph corresponding to the second training input data; All the first augmented data and the second augmented data are added to the second training data set to obtain the augmented training data set.

7. The domain large model fine-tuning method of claim 6, wherein, For each second training input data, the corresponding modal data is obtained from the domain multi-modal database by a semantic similarity method, and the first augmented data corresponding to the second training input data is generated according to the modal data. Generate a search statement according to the second training input data; Obtain a search model, and make the search model retrieve the corresponding modal data from the domain multi-modal database according to the search statement and in combination with cosine similarity; Semantically encode all the obtained modal data and the second training input data to obtain the first augmented data.

8. The domain large model fine-tuning method of claim 5, wherein, The step of obtaining the domain knowledge graph includes: Obtain an entity recognition prompt template, a relationship extraction prompt template, and an existing large language model; Construct the domain knowledge graph containing a plurality of triples according to the modal data of the domain multi-modal database by the entity recognition prompt template, the relationship extraction prompt template, and the existing large language model; Divide the domain knowledge graph into a plurality of subgraphs as summary candidate subgraphs by a graph partitioning algorithm, wherein each triple is associated with a summary candidate subgraph.

9. The domain large model fine-tuning method of claim 8, wherein, The step of constructing the corresponding domain knowledge graph according to the modal data of the domain multi-modal database by the entity recognition prompt template, the relationship extraction prompt template, and the existing large language model further includes: Input the modal data and the entity recognition prompt template into the existing large language model to obtain a plurality of entities; Input the plurality of entities and the relationship extraction prompt template into the existing large language model to obtain a plurality of triples.

10. The domain large model fine-tuning method of claim 8, wherein, The graph partitioning algorithm includes a Leiden algorithm, a modularity The calculation expression is: ， wherein is the sum of the weights of all edges in the domain knowledge graph, is the weight of the edge between node and node in the domain knowledge graph, is the degree of node , is the degree of node , is the indicator function corresponding to node and node .

11. A domain large model fine-tuning system, characterized in that, It includes: A data acquisition module, the data acquisition module is used for obtaining initial large language model, first training data set, second training data set, domain knowledge graph and domain multi-modal database; A search space construction module, the search space construction module is used for constructing the parameter fine-tuning search space corresponding to the initial large language model, wherein the parameter fine-tuning search space includes a plurality of candidate fine-tuning modules; An initial domain large model construction module, the initial domain large model construction module is used for inserting the plurality of candidate fine-tuning modules into the initial large language model to obtain an initial domain large model; a fine-tuning strategy optimization module, configured to train the initial domain large model according to the first training dataset, to obtain an intermediate domain large model containing at least one first fine-tuning module, the at least one first fine-tuning module being at least one trained candidate fine-tuning module in the plurality of candidate fine-tuning modules; an enhanced data construction module, configured to perform data enhancement on the second training dataset according to the domain knowledge graph and the domain multi-modal database, to obtain an enhanced training dataset; a fusion training module, configured to fine-tune train the intermediate domain large model according to the enhanced training dataset, to obtain a final domain large model.

12. The domain large model fine-tuning system of claim 11, wherein, The parameter fine-tuning search space further includes a type range, an insertion position range and a value range of corresponding hyperparameters of each candidate fine-tuning module, and the fine-tuning strategy optimization module includes: a pruning unit, configured to perform progressive pruning on the plurality of candidate fine-tuning modules in the initial domain large model according to the first training dataset, the type range and the insertion position range of each candidate fine-tuning module, to obtain a to-be-trained domain large model containing at least one candidate fine-tuning module as a to-be-trained fine-tuning module; an optimization unit, configured to perform Bayesian optimization on the to-be-trained domain large model according to the first training dataset and a value range of the hyperparameters corresponding to at least one to-be-trained fine-tuning module, to obtain the intermediate domain large model containing at least one to-be-trained fine-tuning module with the corresponding hyperparameters optimized as the first fine-tuning module.

13. The domain large model fine-tuning system of claim 11, wherein, The domain multi-modal database includes a plurality of modal data, the domain knowledge graph includes a plurality of triples and a summary candidate subgraph associated with the triples, the second training dataset includes a plurality of second training input data, and the enhanced data construction module includes: a first data enhancement unit, configured to, for each second training input data, obtain corresponding modal data from the domain multi-modal database through a semantic similarity method, and generate first enhanced data corresponding to the second training input data according to all the obtained modal data; a second data enhancement unit, configured to, for each second training input data, integrate each summary candidate subgraph associated with each triple corresponding to the second training input data in the domain knowledge graph, to obtain second enhanced data corresponding to the second training input data; a fusion unit, configured to add all the first enhanced data and the second enhanced data to the second training dataset to obtain the enhanced training dataset.

14. An electronic device, comprising: a memory configured to store instructions executable by a processor; and the processor is configured to execute the instructions to implement the domain large model fine-tuning method according to any one of claims 1-10.

15. A computer readable medium storing computer program code which, when executed by a processor, implements the domain large model fine-tuning method of any one of claims 1-10.

Images