Deep learning pre-training model fine-tuning method and device, electronic equipment and medium
By constructing a tree-shaped label category system and a graph neural network structure, and embedding a deep learning pre-trained model, the problems of difficult data collection and large number of parameters in TCM disease classification are solved, and efficient optimization training is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2023-02-14
- Publication Date
- 2026-05-15
AI Technical Summary
Existing technologies for classifying diseases in Traditional Chinese Medicine (TCM) present difficulties in data collection due to the large number of categories and subtle differences. Furthermore, existing optimization methods require building a model for each category, resulting in a large number of parameters, high computational cost, and low optimization efficiency.
The target domain's tag category system is constructed as a tree structure. The tag category data of the subtree structure is determined based on the number of rows in which node words appear in the corpus. A graph neural network is used to construct the structure and embed a deep learning pre-trained model. The parameters of the graph neural network structure are determined as the tuning parameters and then tuned and trained.
The number of tuning parameters was reduced, tuning efficiency was improved, and the deep learning pre-trained model was optimized while ensuring accuracy.
Smart Images
Figure CN116049734B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device and storage medium for tuning deep learning pre-trained models. Background Technology
[0002] Large-scale pre-trained models have garnered widespread attention due to their powerful generalization capabilities and efficient utilization of large-scale data. Data imbalance caused by an excessive number of classification categories is often a challenge in training intelligent classification models and a bottleneck for improving model accuracy. While data sampling strategies (upsampling / downsampling, SMOTE sampling, etc.) can theoretically achieve certain results, in practical applications, due to the complexity of classification tasks, they often fail to meet expectations.
[0003] Taking traditional Chinese medicine (TCM) disease classification as an example, there are over 1600 known disease types (syndromes) in TCM, with very subtle differences between them. The best-performing method for tuning large-scale pre-trained models currently available requires building a classification model for 1600 classes, and its effectiveness depends on the quality of the training data for each of those 1600 classes. In reality, collecting high-quality data for all 1600 classes is extremely difficult. Furthermore, building a separate model for each of the 1600 classes would mean tuning the original large-scale pre-trained model 1600 times, with the total number of parameters in all models being 1600 times that of the original model. This would result in a massive amount of work and an enormous computational burden on the parameters during tuning. Summary of the Invention
[0004] The main objective of this application is to provide a method, apparatus, electronic device, and storage medium for tuning deep learning pre-trained models. The aim is to reduce the number of tuning parameters and improve the tuning efficiency of deep learning pre-trained models while ensuring their accuracy.
[0005] To achieve the above objectives, a first aspect of this application proposes a method for tuning a deep learning pre-trained model, the method comprising:
[0006] Collect linguistic data from the target domain to obtain a corpus;
[0007] Construct a tag category system for the target domain, wherein the tag category system is a tree structure;
[0008] Based on the number of rows in which the node words in the tag category system appear in the corpus, the tag category data with a subtree structure is determined in the tag category system;
[0009] Based on the tag category data of the subtree structure, a graph neural network structure is constructed.
[0010] The graph neural network structure is embedded into a deep learning pre-trained model, and the parameters of the graph neural network structure are determined as tuning parameters.
[0011] The deep learning pre-trained model is tuned and trained based on the tuning parameters.
[0012] In some embodiments, determining the subtree structure of tag category data in the tag category system based on the number of lines in which node words in the tag category system appear in the corpus includes:
[0013] The node words in the tag category system are matched in the corpus, and the number of rows in which the node words of each node appear in the corpus is counted.
[0014] Select nodes whose number of occurrences in the corpus is greater than a preset threshold.
[0015] The subtree structure of the label category data is determined based on the selected nodes.
[0016] In some embodiments, determining the tag category data based on the selected nodes for the subtree structure includes:
[0017] Determine the target leaf node from the selected nodes;
[0018] Based on the target leaf node, determine the target path; select the target parent node in the target path, and determine the target sub-path based on the target parent node;
[0019] Based on the target path and the target sub-path, determine the tag category data of the subtree structure.
[0020] In some embodiments, constructing a graph neural network structure based on the tag category data of the subtree structure includes:
[0021] Obtain the nodes from the tag category data of the subtree structure;
[0022] The parameters of the graph neural network are determined based on the number of nodes in the label category data of the subtree structure.
[0023] Based on the determined parameters, a graph neural network structure is constructed.
[0024] In some embodiments, the method further includes, before embedding the graph neural network structure into a deep learning pre-trained model:
[0025] Obtain the target corpus, which is the corpus corresponding to the tag category data of the subtree structure;
[0026] The first corpus is obtained based on the corpus and the target corpus;
[0027] Based on the first corpus, the deep learning network structure is trained to obtain the deep learning pre-trained model.
[0028] In some embodiments, embedding the graph neural network structure into a deep learning pre-trained model and determining the parameters of the graph neural network structure as tuning parameters includes:
[0029] Obtain the text vector of the target corpus;
[0030] Obtain the output vector of the graph neural network structure;
[0031] The text vector is concatenated with the output vector as a prefix and then embedded into a deep learning pre-trained model. The parameters of the graph neural network structure are then determined as tuning parameters.
[0032] In some embodiments, if the tag category data of the subtree structure is changed, the method further includes:
[0033] Based on the label category data of the modified subtree structure, determine the parameters of the modified graph neural network structure;
[0034] The deep learning pre-trained model is tuned and trained using the parameters of the modified graph neural network structure as tuning parameters.
[0035] To achieve the above objectives, a second aspect of this application provides a deep learning pre-trained model tuning apparatus, the apparatus comprising:
[0036] The data acquisition module is used to collect corpus data in the target domain to obtain a corpus.
[0037] The first construction module is used to construct a tag category system for the target domain, wherein the tag category system is a tree structure;
[0038] The first determining module is used to determine the subtree structure of tag category data in the tag category system based on the number of rows in which the node words in the tag category system appear in the corpus;
[0039] The second construction module is used to construct a graph neural network structure based on the tag category data of the subtree structure;
[0040] The second determining module is used to embed the graph neural network structure into a deep learning pre-trained model and determine the parameters of the graph neural network structure as tuning parameters.
[0041] The tuning module is used to tune and train the deep learning pre-trained model based on the tuning parameters.
[0042] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.
[0043] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the method described in the first aspect.
[0044] This application proposes a method, apparatus, electronic device, and storage medium for tuning a deep learning pre-trained model. The method includes: collecting corpus data from a target domain to obtain a corpus; constructing a tag category system for the target domain; determining subtree-structured tag category data within the tag category system based on the number of rows in the corpus where node words in the tag category system appear; constructing a graph neural network structure based on the subtree-structured tag category data; embedding the graph neural network structure into a deep learning pre-trained model and determining the parameters of the graph neural network structure as tuning parameters; and tuning the deep learning pre-trained model based on the determined tuning parameters. By constructing a graph neural network structure, embedding it into the deep learning pre-trained model, and then determining the parameters of the graph neural network structure as the tuning parameters for the deep learning pre-trained model, the number of tuning parameters can be reduced, and the tuning efficiency of the deep learning pre-trained model can be improved while ensuring its accuracy. Attached Figure Description
[0045] Figure 1 This is a flowchart of the deep learning pre-trained model tuning method provided in the embodiments of this application;
[0046] Figure 2 This is a schematic diagram of the tag category system provided in the embodiments of this application;
[0047] Figure 3 This is a schematic diagram of the disease label category system provided in the embodiments of this application;
[0048] Figure 4 A flowchart illustrating the steps for determining subtree structured tag category data in the tag category system based on the number of rows in the corpus where node words in the tag category system appear in the tag category system, as provided in this application embodiment;
[0049] Figure 5 This is a flowchart illustrating the steps for determining tag category data based on selected nodes in an embodiment of this application.
[0050] Figure 6 This is a schematic diagram of the subtree structure of tag category data provided in the embodiments of this application;
[0051] Figure 7 This is a flowchart illustrating the steps of constructing a graph neural network structure based on subtree-structured tag category data provided in this application embodiment.
[0052] Figure 8 This is a flowchart illustrating the steps involved in training a deep learning pre-trained model, as provided in an embodiment of this application.
[0053] Figure 9 This application provides a schematic diagram illustrating the embedding of a graph neural network structure into a deep learning pre-trained model.
[0054] Figure 10 This is a schematic diagram of the deep learning pre-trained model tuning device provided in the embodiments of this application;
[0055] Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0057] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0059] First, let's analyze some of the terms used in this application:
[0060] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.
[0061] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.
[0062] Graph Neural Networks (GNNs) are a general term for algorithms that use neural networks to learn graph-structured data, extract and discover features and patterns within it, and meet the needs of graph learning tasks such as clustering, classification, prediction, segmentation, and generation. GNNs are a framework that has emerged in recent years, utilizing deep learning to directly learn from graph-structured data. Their superior performance has attracted significant attention and in-depth exploration by researchers. By formulating certain strategies on nodes and edges in the graph, GNNs transform graph-structured data into a standardized representation, which is then fed into various neural networks for training, achieving excellent results in tasks such as node classification, edge information propagation, and graph clustering.
[0063] Graph Attention Network (GAT): The attention mechanism allows a neural network to focus only on the information needed for task learning; it can select specific inputs. Introducing the attention mechanism into GNNs allows the neural network to focus on nodes and edges that are more relevant to the task, improving training effectiveness and testing accuracy, thus forming the Graph Attention Network (GAT).
[0064] The Transformer is an advanced deep learning model. Compared to traditional deep learning networks (e.g., multilayer perceptrons, convolutional neural networks, recurrent neural networks), the Transformer architecture is easier to deepen and has smaller model biases, making it more suitable for the pre-training-fine-tuning paradigm. A typical pre-training-fine-tuning paradigm involves first training the model (usually in a self-supervised manner) on a large amount of training data, and then fine-tuning it on a smaller dataset for downstream tasks. Pre-training helps the model learn general representations, which in turn benefit downstream tasks.
[0065] Transformer-based pre-training methods were first proposed for natural language processing tasks and achieved significant performance improvements. For Transformer-based models, pre-training and fine-tuning has become a typical learning paradigm, which avoids training new models from scratch for different tasks or datasets.
[0066] After training a deep learning pre-trained model on a large amount of data, it is applied to specific downstream tasks. At this point, new training data needs to be acquired for the specific downstream task to optimize the deep learning pre-trained model. However, some downstream tasks have massive amounts of label category data, and high-quality label category data is difficult to collect in reality, making it difficult to optimize the deep learning pre-trained model, or the optimization effect may be poor. Taking the disease classification task in the field of Traditional Chinese Medicine as an example, there are more than 1,600 known disease classifications (syndromes) in TCM, and the differences between these 1,600 categories are very subtle. If the optimal method for optimizing the deep learning pre-trained model is adopted, it is necessary to build a classification model of 1,600 categories and collect training data for each of the 1,600 categories to train the 1,600 categories. The training effect depends on the quality of the training data for each of the 1,600 categories. In reality, collecting high-quality training data for 1,600 categories is very difficult. Building a separate model for each of the 1600 classes would require tuning the original large-scale pre-trained model 1600 times, and the total number of parameters in all models would be 1600 times that of the original model. This would result in a massive amount of work, and the computational load on the parameters during tuning would be enormous. For resource-constrained real-world production, this approach is impractical.
[0067] Based on this, this application proposes a method for tuning a deep learning pre-trained model. By constructing a graph neural network structure and embedding the graph neural network structure into a deep learning pre-trained model, the parameters of the graph neural network structure are determined as the tuning parameters of the deep learning pre-trained model. This method can reduce the number of tuning parameters and improve the tuning efficiency of the deep learning pre-trained model while ensuring its accuracy.
[0068] Reference Figure 1 , Figure 1 This is a flowchart of the deep learning pre-trained model tuning method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.
[0069] Step S101: Collect corpus data of the target domain to obtain a corpus.
[0070] In this step, corpus data of the target domain can be collected from books such as classics, textbooks, and guides, or relevant corpus data can be crawled from the Internet. After collection, it is saved in text format. The sources of corpus are rich and diverse, and there are many methods to obtain corpus. In addition to the commonly used search-download-format conversion method, computer technology can also be used for information crawling. For example, you can first search for web pages containing corpus data of the target domain by keyword, and then use web crawling software to crawl the relevant corpus data. After collection, the data is cleaned to remove non-standard formats, symbols, content, etc., and then a corpus is built based on the collected corpus data.
[0071] Step S102: Construct a tag category system for the target domain. The tag category system is a tree structure.
[0072] Designing a tag category system requires starting with the "root directory." The data granularity corresponding to the "root directory" is "objects." Objects are divided into two main types: entity objects (people, things) and relational objects (strong relationships, weak relationships). Therefore, there are two main types of tag category trees: entity trees and relational trees. The branches of the tree correspond to the categories in the tag category system, i.e., tag classifications. Categories are classifications of "tags," not "objects." The leaves / flowers of the tree correspond to the various attributes of the objects, i.e., tags. Tags are mapped to fields in the database table, representing the most suitable data asset granularity verified through extensive data application practice. The tag category system can be likened to a tree; it first has roots, from which multiple branches grow, and from these branches, corresponding leaves and flowers form—this tree-like structure is the tag category system.
[0073] The tag category system consists of a root directory, a category hierarchy, and attribute tags. The category hierarchy is a tree structure: the first-level branches growing from the root directory are called first-level categories; the second-level branches growing from the first-level branches are called second-level categories; the third-level branches growing from the second-level branches are called tertiary categories, and so on. The hierarchical setting of the category structure depends on the actual situation. Categories without a parent category are called first-level categories, and categories without a subcategory are called leaf categories. The specific leaves attached to leaf categories are tags. Categories with subcategories are the parent categories of those subcategories, and categories with parent categories are the child categories of those parent categories.
[0074] In this step, a tree-structured tag category system for the target domain is constructed based on the existing tag category system. Specifically, refer to... Figure 2 For example, based on the target domain D, a system was constructed such as... Figure 2 The label category system shown. Figure 2In this diagram, A, B, and C are first-level branches emanating from the root directory. First-level category A then spawns second-level branches A1, A2, and A3; first-level category B spawns second-level branches B1 and B2; and first-level category C spawns second-level branches C1, C2, and C3. Second-level category A2 spawns third-level branches A21, A22, and A23, with A22 itself spawning a fourth-level branch A221. A221 has no subcategories; therefore, it is a leaf category, and its content consists of tags. Similarly, A21 and A23 also have no subcategories; therefore, they are both leaf categories, and their content consists of tags. A22, however, has a subcategory; it is the parent category of the leaf category A221 and also a child category of the second-level category A2. From the perspective of the growth path, each category level corresponds to a specific node. For example, A221 is a leaf node, A22 is the parent node of A221, and also a child node of A2.
[0075] This application uses the field of Traditional Chinese Medicine as an example to construct a disease category labeling system as follows: Figure 3 As shown, Figure 3 In this tree, diseases are first classified into categories, including the eight cardinal categories and etiological categories. The eight cardinal categories are further divided into Yin syndrome, Yang syndrome, and superficial syndrome. Superficial syndromes are further divided into superficial excess syndrome and superficial deficiency syndrome, with superficial excess syndrome including superficial water retention syndrome. Superficial water retention syndrome is a leaf node, and superficial excess syndrome is its parent node and a child node of superficial syndrome. All nodes in the tree, except for the classification names (eight cardinal categories and etiological categories), can be used as outputs of the classification results. The path depth from the parent node to the leaf node varies; the deeper the node, the finer the granularity of the classification. For example, "superficial water retention syndrome" is a specific manifestation of "superficial excess syndrome."
[0076] Step S103: Based on the number of rows in which node words in the tag category system appear in the corpus, determine the tag category data with subtree structure in the tag category system.
[0077] In this embodiment of the application, after constructing the tag category system, it is necessary to further match each node word in the tag category system with the corpus and count the number of rows in the corpus where the node word appears. Then, based on the counted number of rows, a subtree structure of tag category data is determined.
[0078] Reference Figure 4 , Figure 4 This is a flowchart of the steps to determine the subtree structure of tag category data in the tag category system based on the number of rows in which node words in the tag category system appear in the corpus, as provided in the embodiments of this application, including but not limited to steps S401 to S403.
[0079] Step S401: Match the node words in the tag category system in the corpus and count the number of rows in the corpus where the node words of each node appear.
[0080] Step S402: Select nodes whose number of occurrences in the corpus is greater than a preset threshold;
[0081] Step S403: Determine the tag category data of the subtree structure based on the selected nodes.
[0082] In this embodiment, each node corresponds to a node word. The node words need to be matched in a corpus first, and then the number of lines in which the node word appears in the corpus is counted. It should be noted that the corpus consists of multiple text datasets containing linguistic data from the target domain. Counting the number of lines in which a node word appears in the corpus essentially counts the probability that the node word is used; the more lines in the corpus where a node word appears, the higher the probability that the node word is used.
[0083] For example, with Figure 3 Taking the disease category labeling system shown as an example, statistics show that the number of rows for "Tablet of Water Retention Syndrome" in the corpus is 10,000, the number of rows for "Tablet of Concrete Syndrome" is 10,000, the number of rows for "Tablet of Deceptive Syndrome" is 2,000, and the number of rows for "Tablet of Concrete Syndrome" is 12,000. It should be noted that during the statistical process, if a node word is not a leaf node word, the number of rows for the node word of its various child nodes needs to be accumulated to obtain the number of rows for that node word in the corpus. For example, Figure 3 In the corpus, the "closed-water stop" clause is a leaf node. Therefore, the count of 10,000 occurrences of "closed-water stop" in the corpus is taken as the statistical result for this leaf node. However, "evidence" is not a leaf node; it is the parent node of both "positive evidence" and "negative evidence." Therefore, the count of "positive evidence" in the corpus is obtained by adding the counts of "negative evidence" to the counts of "positive evidence." Thus, the statistical result for "evidence" is 12,000.
[0084] After counting the number of lines in the corpus where each node's word appears, nodes whose word appears in the corpus at a number greater than a preset threshold are selected. For example, if the preset threshold is set to 8,000, then... Figure 3 In the tree-structured tag category system shown, the statistical results for "Table Closed Water Certificate", "Table Verified Certificate", and "Table Certificate" are all greater than 8,000. Then, a subtree-structured tag category data is determined from the selected nodes.
[0085] Reference Figure 5 , Figure 5This is a flowchart of the steps for determining the tag category data of the subtree structure based on the selected nodes provided in the embodiments of this application, including but not limited to steps S501 to S504.
[0086] Step S501: Determine the target leaf node from the selected nodes;
[0087] Step S502: Determine the target path based on the target leaf node;
[0088] Step S503: Select the target parent node in the target path, and determine the target sub-path based on the target parent node;
[0089] Step S504: Based on the target path and target sub-path, determine the tag category data of the subtree structure.
[0090] In this embodiment, since the number of rows in the corpus corresponding to the node selected in step S402 is greater than a preset threshold, and the selected nodes include multiple leaf nodes, multiple child nodes, and multiple parent nodes, it is necessary to first arbitrarily select one of the selected leaf nodes as the target leaf node, and then find the target path corresponding to the target leaf node upwards based on the target leaf node; arbitrarily select one of the parent nodes in the target path as the target parent node, and find the target sub-path corresponding to the target parent node downwards based on the target parent node; then, based on the determined target path and target sub-path, determine the tag category data of the subtree structure. That is, backtracking upwards from the leaf node to the path that meets the threshold condition until the end node of the path, and then starting from the nodes along the way, finding the sub-path that meets the threshold condition downwards to form a subtree.
[0091] It should be understood that the embodiments of this application only exemplify one method for determining the tag category data of the subtree structure based on the selected nodes. The embodiments of this application do not specifically limit the method of determining the tag category data of the subtree structure. For example, one can first arbitrarily select a parent node from all the selected nodes, and then find the path that meets the threshold condition through the parent node. After selecting the parent node, another parent node is selected, and the path that meets the threshold condition is found through the parent node, until all the paths that meet the threshold condition are found to form a subtree, thereby obtaining the tag category data of the subtree structure.
[0092] Reference Figure 6 , Figure 6 This is a schematic diagram of the subtree structure of tag category data provided in the embodiments of this application. Figure 6 Is Figure 2 This diagram illustrates the tag category data of one of the subtree structures defined in the tag category system. It can be understood that... Figure 6This is merely an illustrative diagram of one possible structure for subtree-structured tag category data.
[0093] For example, also with Figure 3 Taking the tree-structured tag category system shown as an example, firstly, from the selected nodes "Table Closed Water Certificate", "Table Real Certificate", and "Table Certificate" whose statistical results are greater than a preset threshold, the leaf node "Table Closed Water Certificate" is determined as the target leaf node. Then, the path that satisfies the threshold condition is traced back upwards until the terminal node "Table Certificate" is reached. That is, the determined target path is "Table Closed Water Certificate - Table Real Certificate - Table Certificate". Then, starting from the nodes along the way, such as "Table Virtual Certificate", the sub-paths that satisfy the threshold condition are found downwards, forming a subtree.
[0094] Step S104: Construct a graph neural network structure based on the subtree structure of the tag category data.
[0095] In this embodiment of the application, after obtaining the label category data of the subtree structure, the graph neural network structure is constructed by using the graph attention network (GAT) in the graph neural network.
[0096] Specifically, refer to Figure 7 , Figure 7 This is a flowchart of the steps for constructing a graph neural network structure based on subtree-structured tag category data provided in the embodiments of this application, including but not limited to steps S701 to S703.
[0097] Step S701: Obtain the nodes in the tag category data of the subtree structure;
[0098] Step S702: Determine the parameters of the graph neural network based on the number of nodes in the label category data of the subtree structure;
[0099] Step S703: Based on the determined parameters, construct the graph neural network structure.
[0100] In this embodiment of the application, let the tree-structured tag category system of the target domain D be T, and the tag category data of the subtree structure selected from the tree-structured tag category system T be t(λ), where λ represents the node in the subtree. Using the structure of the Graph Attention Network (GAT), the nodes in t(λ) are input, and the embedding parameter is set to W. t(λ) ∈R T*d Where, t(λ)={n1,n2,Λn L}, where L is the number of nodes, W t(λ)Let t(λ) be a node embedding parameter matrix of d dimensions representing the total number of nodes (equivalent to the number of words) in the subtree-structured tag category data. Then the graph neural network structure is: G(t(λ), W(t(λ)))={h1,h2,Λh L} t(λ) Its output is L d-dimensional node vectors.
[0101] In this embodiment of the application, after constructing the graph neural network structure, it is necessary to obtain corpus of the target domain and train a deep learning pre-trained model based on the Transformer network structure. Specifically, refer to... Figure 8 , Figure 8 This is a flowchart of the steps for training a deep learning pre-trained model according to the embodiments of this application, including but not limited to steps S801 to S803.
[0102] Step S801: Obtain the target corpus, which is the corpus corresponding to the tag category data of the subtree structure;
[0103] Step S802: Obtain the first corpus based on the corpus and the target corpus;
[0104] Step S803: Based on the first corpus, train the deep learning network structure to obtain a deep learning pre-trained model.
[0105] In this embodiment, the target corpus is first obtained, which is the corpus corresponding to the tag category data of the subtree structure; then, the first corpus is obtained based on the corpus and the target corpus; and then, the first corpus is used to train a deep learning network structure to obtain a deep learning pre-trained model. In this embodiment, the deep learning pre-trained model is obtained by training based on the Transformer network structure using the first corpus, which can avoid the overfitting problem in the training process to a certain extent.
[0106] Step S105: Embed the graph neural network structure into the deep learning pre-trained model and determine the parameters of the graph neural network structure as tuning parameters.
[0107] In this embodiment of the application, after obtaining the graph neural network structure from the subtree-based tag category system, the graph neural network structure is embedded into a pre-trained deep learning model. Specifically, as shown... Figure 9 As shown, the text vector of the target corpus is obtained, which is the corpus corresponding to the tag category data of the subtree structure; the output vector of the graph neural network structure is obtained; the text vector is concatenated with the output vector as the prefix and then embedded into the deep learning pre-trained model, and the parameters of the graph neural network structure are determined as the tuning parameters. Figure 9 middle, The output structure vector represents the structure of the graph neural network. This represents the text vector of the target corpus. Since the parameters of the pre-trained deep learning model are fixed, it is equivalent to reusing the original model. The only parameter that needs to be tuned is the node embedding parameter W. t(λ) And the graph neural network structure parameters W G This greatly reduces the overall parameter size.
[0108] Step S106: Optimize and train the deep learning pre-trained model based on the optimization parameters.
[0109] In this embodiment, when applying the pre-trained deep learning model to a specific downstream task, it is necessary to fine-tune the deep learning model for that specific task. This requires first identifying the target domain and collecting corpus data for that downstream task to construct a corpus, then building a tree-structured tag category system for the target domain, and finally, using a graph neural network to construct a graph neural network structure based on that system. This graph neural network structure is then embedded as a prefix into the deep learning pre-trained model, allowing the original parameters of the model to remain unchanged during fine-tuning, while only the node embedding parameter W needs to be tuned. t(λ) And the graph neural network structure parameters W G Finally, by optimizing and training the pre-trained deep learning model based on the determined optimization parameters, the efficiency of optimizing the pre-trained deep learning model can be greatly improved and the workload reduced.
[0110] In this embodiment, for tag category data with different subtree structures, only the parameters of the corresponding prefix part need to be replaced. This embodiment can perform optimization training on any node λ in the hierarchical structure of the tag category data in the subtree structure, without restricting it to a leaf node or an intermediate node. That is, if tag category data with another subtree structure is obtained through a tree-structured tag category system, similarly, a graph neural network structure is first obtained based on the tag category data of that subtree structure using a graph neural network. Then, using this graph neural network structure as a prefix and the text vector of the subtree node corpus corresponding to the tag category data of that subtree structure as a suffix, the concatenation is embedded into the deep learning pre-trained model. This allows for optimization training of the deep learning pre-trained model by only adjusting the graph neural network structure parameters W of the prefix part. G Perform updated training.
[0111] For example, if the trained deep learning pre-trained model is applied to a specific downstream task, such as a disease classification task or a retrieval task, different strategies can be adopted depending on the different task requirements and resource conditions. For example, in the disease classification task applied to the field of traditional Chinese medicine, although there are more than 1600 classification tasks, it is only necessary to select to optimize only the prefix part, where W t(λ) Embedding parameters are shared across categories and do not need to be created separately. The original parameters of the pre-trained deep learning model also do not need to be updated; only the graph neural network structure parameters W need to be created. G That's it. Since nodes at the same level share the same parent node, when optimizing training for any node λ in the hierarchical structure of the subtree-structured tag category data, the parameters of the same subtree-structured tag category data can be shared, and the actual number of parameter copies will be less than 1600. This allows for flexible determination of the target node granularity used in downstream training.
[0112] It should be noted that the deep learning pre-trained model obtained from the training can be applied to downstream tasks of TCM disease classification, as well as to downstream tasks of other categories, such as product classification and personality classification.
[0113] Please see Figure 10 This application embodiment also provides a deep learning pre-trained model tuning device 10, which can implement the above-mentioned deep learning pre-trained model tuning method. The device includes:
[0114] The acquisition module 110 is used to acquire corpus data in the target domain to obtain a corpus;
[0115] The first construction module 120 is used to construct a tag category system for the target domain. The tag category system is a tree structure.
[0116] The first determining module 130 is used to determine the tag category data of the subtree structure in the tag category system based on the number of rows in which the node words in the tag category system appear in the corpus;
[0117] The second construction module 140 is used to construct a graph neural network structure based on the subtree-structured tag category data;
[0118] The second determining module 150 is used to embed the graph neural network structure into the deep learning pre-trained model and determine the parameters of the graph neural network structure as tuning parameters.
[0119] The tuning module 160 is used to tune and train a deep learning pre-trained model based on determined tuning parameters.
[0120] The specific implementation of this deep learning pre-trained model tuning device is basically the same as the specific implementation of the deep learning pre-trained model tuning method described above, and will not be repeated here.
[0121] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned deep learning pre-trained model tuning method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.
[0122] Please see Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:
[0123] The processor 1101 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.
[0124] The memory 1102 can be implemented in the form of read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1102 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1102 and is called and executed by the processor 1101 to execute the deep learning pre-trained model tuning method of the embodiments of this application.
[0125] Input / output interface 1103 is used to implement information input and output;
[0126] The communication interface 1104 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0127] Bus 1105 transmits information between various components of the device (e.g., processor 1101, memory 1102, input / output interface 1103, and communication interface 1104);
[0128] The processor 1101, memory 1102, input / output interface 1103 and communication interface 1104 are connected to each other within the device via bus 1105.
[0129] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described deep learning pre-trained model tuning method.
[0130] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0131] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0132] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0133] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0134] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0135] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0136] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0137] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0138] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0139] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0140] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0141] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A method for tuning a deep learning pre-trained model, characterized in that, The method includes: Collect linguistic data from the target domain to obtain a corpus; Construct a tag category system for the target domain, wherein the tag category system is a tree structure; Based on the number of rows in which the node words in the tag category system appear in the corpus, the tag category data with a subtree structure is determined in the tag category system; Based on the tag category data of the subtree structure, a graph neural network structure is constructed. The graph neural network structure is embedded into a deep learning pre-trained model, and the parameters of the graph neural network structure are determined as tuning parameters. The deep learning pre-trained model is tuned and trained based on the tuning parameters. The step of determining the subtree structure of tag category data in the tag category system based on the number of rows in which node words in the tag category system appear in the corpus includes: The node words in the tag category system are matched in the corpus, and the number of rows in which the node words of each node appear in the corpus is counted. Select nodes whose number of occurrences in the corpus is greater than a preset threshold. The subtree structure of the tag category data is determined based on the selected nodes; The graph neural network structure is constructed based on the tag category data of the subtree structure, including: Obtain the nodes from the tag category data of the subtree structure; The parameters of the graph neural network are determined based on the number of nodes in the label category data of the subtree structure. Based on the determined parameters, a graph neural network structure is constructed. The step of embedding the graph neural network structure into a deep learning pre-trained model and determining the parameters of the graph neural network structure as tuning parameters includes: Obtain the text vector of the target corpus, wherein the target corpus is the corpus corresponding to the tag category data of the subtree structure; Obtain the output vector of the graph neural network structure; The text vector is concatenated with the output vector as a prefix and then embedded into a deep learning pre-trained model. The parameters of the graph neural network structure are then determined as tuning parameters.
2. The method according to claim 1, characterized in that, The tag category data for determining the subtree structure based on selected nodes includes: Determine the target leaf node from the selected nodes; Determine the target path based on the target leaf node; Select a target parent node in the target path, and determine the target sub-path based on the target parent node; Based on the target path and the target sub-path, determine the tag category data of the subtree structure.
3. The method according to claim 1, characterized in that, Before embedding the graph neural network structure into a deep learning pre-trained model, the method further includes: Obtain the target corpus; The first corpus is obtained based on the corpus and the target corpus; The deep learning network structure is trained based on the first corpus to obtain the deep learning pre-trained model.
4. The method according to claim 1, characterized in that, If the tag category data of the subtree structure is changed, the method further includes: Based on the label category data of the modified subtree structure, determine the parameters of the modified graph neural network structure; The deep learning pre-trained model is tuned and trained using the parameters of the modified graph neural network structure as tuning parameters.
5. A deep learning pre-trained model tuning device, characterized in that, The device includes: The data acquisition module is used to collect corpus data in the target domain to obtain a corpus. The first construction module is used to construct a tag category system for the target domain, wherein the tag category system is a tree structure; The first determining module is used to determine the subtree structure of tag category data in the tag category system based on the number of rows in which the node words in the tag category system appear in the corpus; The second construction module is used to construct a graph neural network structure based on the tag category data of the subtree structure; The second determining module is used to embed the graph neural network structure into a deep learning pre-trained model and determine the parameters of the graph neural network structure as tuning parameters. The tuning module is used to tune and train the deep learning pre-trained model based on the tuning parameters. The first determining module is further configured to: The node words in the tag category system are matched in the corpus, and the number of rows in which the node words of each node appear in the corpus is counted. Select nodes whose number of occurrences in the corpus is greater than a preset threshold. The subtree structure of the tag category data is determined based on the selected nodes; The second building module is also used for: Obtain the nodes from the tag category data of the subtree structure; The parameters of the graph neural network are determined based on the number of nodes in the label category data of the subtree structure. Based on the determined parameters, a graph neural network structure is constructed. The second determining module is also used for: Obtain the text vector of the target corpus, wherein the target corpus is the corpus corresponding to the tag category data of the subtree structure; Obtain the output vector of the graph neural network structure; The text vector is concatenated with the output vector as a prefix and then embedded into a deep learning pre-trained model. The parameters of the graph neural network structure are then determined as tuning parameters.
6. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 4.
7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.