A data matching method, device and equipment for pre-training of vertical large-scale models
Through the data proportioning method for pre-training of vertical large-scale models, the dynamic mixing of attention features and data is solved, and the problem of low efficiency and poor accuracy of training data in large language models is improved and the model performance is improved and the application is widely used. .
Patent Information
- Application Number
- CN202411054262.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-02
AI Technical Summary
The existing large language model training data ratio methods have problems such as low efficiency and difficulty in accurately reflecting the importance and contribution of data, especially in continuous pre-training scenarios, which lead to unstable model performance.
Using a data proportioning method for pre-training of vertical large model, the mixing weight of the training data set is determined by combining attention characteristics and dynamic data mixing, and dynamically adjusting the proportion of the data set, and using computer clusters and modular devices to achieve accurate Data ratio.
It improves the efficiency and accuracy of model training, is suitable for different application fields and basic models, realizes accurate and effective data ratios, and improves model performance and practical application value.
Smart Images

Figure CN118569338B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and more specifically to a data matching method, device and equipment for pre-training of vertical large-scale models. Background Art
[0002] In the training process of large language models (LLMs), the source and composition of training data are crucial factors. The training data of large language models usually contains multiple data sources (such as Wikipedia, GitHub, etc.). The mixed ratio of these data has a significant impact on the performance of large language models, especially in the scenario of continuous pre-training. In order to continuously use new data to update and optimize model parameters, it is necessary to consider the impact of factors such as the target domain and the basic model, making the continuous pre-training process difficult and complicated.
[0003] The existing large model training data methods, such as determining the proportions of different training corpora based on heuristic rules or enumerating a large number of possible mixed weights, can solve the data matching problem to a certain extent, but they also have obvious limitations. Heuristic rules are usually based on some experience or intuition, lack scientificity and systematicity, and it is difficult to guarantee a better data matching solution. The enumeration rule requires trying a large number of combinations and weight allocations, which has high computational costs and is prone to falling into local optimal solutions. These training methods are not only inefficient, but may also lead to unstable model performance. Due to the differences and complexity between different data sources, simple proportional allocation is difficult to accurately reflect the importance and contribution of data.
[0004] Therefore, how to design a more accurate and effective data matching method to meet the needs of pre-training of large-scale language models is an urgent problem to be solved. Summary of the invention
[0005] The purpose of the present invention is to provide a data matching method, device and equipment for pre-training of large vertical models, which uses a method combining attention features and dynamic data mixing to achieve accurate and effective data matching of data sets and improve model training efficiency and accuracy.
[0006] To achieve the above object, the present invention discloses the following technical solution:
[0007] On one hand, the present invention provides a data matching method for pre-training of a vertical large model, which is applied to a computer cluster, wherein the computer cluster includes a plurality of computer devices, and the method includes the following steps:
[0008] Determine the target domain, input the domain subset and the universal subset into the training model, calculate the attention features, and obtain the attention distribution of the domain subset and the universal subset;
[0009] Using the training model to perform knowledge detection on the domain subset and the general subset to obtain knowledge weights of the domain subset and the general subset;
[0010] Determining a mixing weight of a training data set based on the attention distribution and knowledge weight of the domain subset and the general subset;
[0011] The mixing ratio of the training data set is dynamically adjusted according to the mixing weight of the training data set, and the training model is further trained until the training model reaches a preset standard.
[0012] Furthermore, the above data matching method, before inputting the domain subset and the general subset into the training model, also includes:
[0013] A plurality of training data sets from different sources are obtained, and the training data sets are classified to obtain the domain subset and the general subset.
[0014] Furthermore, in the above-mentioned data matching method, the domain subset and the universal subset are input into the training model, the attention features are calculated, and the attention distribution of the domain subset and the universal subset is obtained, including:
[0015] From the first domain subset (X d1 ,X d2 ,……,X dn ) and the universal subset (X g1 ,X g2 ,……,X gm ) randomly select X i ′ as the source dataset, where n represents the number of domain subsets, m represents the number of general subsets, i∈[d1,…,dn,g1,…, gm];
[0016] From the second domain subset Randomly select X t ′ as the target data set, where t∈[d1,…,dn], the first domain subset and the second domain subset are different data sets in the domain subset;
[0017] The source dataset X i ′ and the target dataset X t ′ Input the training model M and calculate the corresponding attention matrix A i and A t , where A i and A t The dimension is is a multidimensional array of |B|, which represents the batch size of the data, L×H represents the number of transformer layers and the number of multi-head self-attention, S represents the maximum length of the sequence, and S 2 Represents the pairwise attention relationship between sequences.
[0018] Furthermore, in the above data matching method, the use of the training model to perform knowledge detection on the domain subset and the general subset to obtain the knowledge weights of the domain subset and the general subset includes:
[0019] Calculate the training model M on the source dataset X i The perplexity p(i, M) on ′;
[0020] Based on the perplexity p(i, M), the knowledge weight is obtained as K(i)=1 / p(i, M), where p(i, M) represents the perplexity and K(i) represents the knowledge weight.
[0021] Furthermore, in the above data matching method, the training model M is calculated based on the source data set X. i The perplexity p(i, M) on ′ includes:
[0022] For the source dataset X i 'Perform preprocessing and input the preprocessed data into the training model M to obtain the output probability distribution of the training model M;
[0023] Calculate the negative value of the logarithmic probability of each word in the data set and accumulate them to get the total cross entropy loss;
[0024] Dividing the total cross entropy loss by the number of words in the data set to obtain an average cross entropy loss;
[0025] The average cross entropy loss is exponentially calculated to obtain the perplexity p(i, M).
[0026] Furthermore, in the above data matching method, the mixing weight of the training data set is determined based on the attention distribution and knowledge weight of the domain subset and the general subset, including:
[0027] Based on the attention matrix and , calculate the source data set X i The dataset weights of ′ are as follows:
[0028] in, Represents the attention matrix The average value over the |B| dimension, Represents the attention matrix Average value over |B| dimension, dataset weight Indicates the similarity between the source data and the target data in terms of attention. This helps the training model automatically assign weights to different data sources. The more similar the source data is to the target data in terms of attention, the higher the weight it gets.
[0029] According to the source dataset X i 'The confusion , calculate the penalty parameter as follows:
[0030] Based on the penalty parameter , the mixed weight of the training data set is obtained as:
[0031] in, Represents the mixed weight, penalty parameter Used to balance the relationship between training model and attention distribution.
[0032] Furthermore, in the above data matching method, the mixing ratio of the training data set is dynamically adjusted according to the mixing weight of the training data set, and the adjusted training data is obtained as follows:
[0033] in, , Indicates uniform sampling from the source dataset.
[0034] Another aspect of the present invention provides a data matching device for pre-training of a vertical large model, which is applied to a computer cluster, wherein the computer cluster includes a plurality of computer devices, and the device includes:
[0035] An attention analysis module, used to determine the target domain, input the domain subset and the universal subset into the training model, calculate the attention features, and obtain the attention distribution of the domain subset and the universal subset;
[0036] A knowledge weight calculation module, used to perform knowledge detection on the domain subset and the general subset using the training model to obtain the knowledge weights of the domain subset and the general subset;
[0037] A hybrid weight determination module, configured to determine a hybrid weight of a training data set based on the attention distribution and knowledge weight of the domain subset and the general subset;
[0038] The model optimization module is used to dynamically adjust the mixing ratio of the training data set according to the mixing weight of the training data set, and further train the training model until the training model reaches a preset standard.
[0039] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data matching method for pre-training of a vertical domain large model as described above is implemented.
[0040] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the data matching method for pre-training of a vertical domain large model as described above is implemented.
[0041] The effects provided in the content of the invention are only the effects of the embodiments, not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects:
[0042] The technical solution provided by the embodiment of the present disclosure integrates the vertical domain characteristics and general domain characteristics of the data, determines the target domain and data classification, determines the mixed weight of the training data set according to the attention distribution and knowledge weight of the domain subset and the general subset, dynamically adjusts the mixed ratio of the training data set, continuously optimizes the model performance, achieves accurate and effective data matching, and improves the effect and performance of model training. At the same time, the solution is applicable to different application fields and basic models, and can adjust the data matching according to the characteristics and needs of different fields, and has high practical application value. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0044] Figure 1 A schematic flow chart of a data matching method for vertical domain large model pre-training provided in one embodiment of the present application;
[0045] Figure 2 A schematic flow chart of a data matching method for vertical domain large model pre-training provided in another embodiment of the present application;
[0046] Figure 3 for Figure 2 Schematic diagram of the implementation process of the data matching method for pre-training of large vertical models;
[0047] Figure 4 A schematic diagram of the structure of a data matching device for vertical domain large model pre-training provided in one embodiment of the present application;
[0048] Figure 5 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application;
[0049] Reference numerals:
[0050] 410-attention analysis module, 420-knowledge weight calculation module, 430-mixed weight determination module, 440-model optimization module;
[0051] 510 - input unit, 520 - memory, 530 - processor, 540 - output unit. DETAILED DESCRIPTION
[0052] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0053] It should be noted that references to "one embodiment", "embodiment", "example embodiment", etc. in this specification refer to the embodiment being described which may include specific features, structures or characteristics, but not every embodiment must include these specific features, structures or characteristics. In addition, such expressions do not refer to the same embodiment. Furthermore, when describing specific features, structures or characteristics in conjunction with an embodiment, whether or not there is an explicit description, it is indicated that incorporating such features, structures or characteristics into other embodiments is within the knowledge of those skilled in the art.
[0054] In addition, certain words are used in the specification and subsequent claims to refer to specific components or parts. Those with ordinary knowledge in the relevant field should understand that manufacturers can use different nouns or terms to refer to the same component or part. This specification and subsequent claims do not use differences in names as a way to distinguish components or parts, but use differences in the functions of components or parts as the criteria for distinction. "Including" and "including" mentioned throughout the specification and subsequent claims are open-ended terms and should be interpreted as "including but not limited to". In addition, the word "connected" here includes any direct and indirect electrical connection means. Indirect electrical connection means include connection through other devices.
[0055] Due to the existing large model training data methods, there are great limitations in solving the data matching problem. Not only is the training efficiency low, but it is also difficult to accurately reflect the importance and contribution of the data. In order to solve the shortcomings of the existing technology, the following embodiments provide a data matching method, device and equipment for vertical large model pre-training, which is suitable for different application fields (such as medical, finance) and basic models (such as Qwen, Yi). By comprehensively considering the vertical field characteristics and versatility of the data, and using attention features and model internal knowledge to determine data weights, accurate and effective data matching is achieved, and the effect and performance of model training are improved.
[0056] refer to Figure 1 , Figure 1 A schematic flow chart of a data matching method for vertical domain large model pre-training provided by an embodiment of the present application is shown. The method is applied to a computer cluster, and the computer cluster includes multiple computer devices. The method includes the following steps:
[0057] S110, determine the target domain, input the domain subset and the general subset into the training model, calculate the attention features, and obtain the attention distribution of the domain subset and the general subset.
[0058] In some embodiments, before inputting the domain subset and the universal subset into the training model, the method further includes obtaining training data sets from multiple different sources and classifying the training data sets to obtain the domain subset and the universal subset.
[0059] S120. Use the training model to perform knowledge detection on the domain subset and the general subset to obtain knowledge weights of the domain subset and the general subset.
[0060] In some embodiments, a source data set and a target data set may be extracted from a domain subset and a general subset, the perplexity of the training model on the source data set may be calculated, and then the knowledge weight may be obtained based on the perplexity.
[0061] S130. Determine a mixing weight of the training data set based on the attention distribution and knowledge weight of the domain subset and the general subset.
[0062] In some embodiments, a penalty parameter can be used to balance the relationship between the attention distribution and the knowledge weight of the training model to obtain a mixed weight of the training data set.
[0063] S140. Dynamically adjust the mixing ratio of the training data set according to the mixing weight of the training data set, and further train the training model until the training model reaches a preset standard.
[0064] As mentioned above, the existing large model pre-training data method has low training efficiency and is difficult to accurately reflect the importance and contribution of the data. In this embodiment, the vertical field characteristics and general field characteristics of the data are comprehensively considered to ensure the pertinence and diversity of the model training data. By dynamically adjusting the mixing ratio of the training data set according to the attention distribution and knowledge weight, the model performance is continuously optimized, and accurate and effective data matching is achieved, which improves the effect and performance of model training. In addition, this solution is applicable to different application fields and basic models, and has high practical application value.
[0065] In order to describe the technical solution of the present invention in more detail, Figure 2 , Figure 3 The data matching process for pre-training of a large vertical model in an embodiment of the present application is described in detail.
[0066] refer to Figure 2 , Figure 2 A schematic flow chart of a data matching method for vertical domain large model pre-training provided by another embodiment of the present application is shown. The method is applied to a computer cluster, and the computer cluster includes multiple computer devices. The method includes the following steps:
[0067] S210, determining the target domain, obtaining training data sets from multiple different sources, and classifying the training data sets to obtain domain subsets and general subsets.
[0068] Specifically, for example, taking the medical field as the target field, medical-related training data sets can be collected from a variety of sources (such as medical literature databases, electronic medical record data, and general corpora, etc.). Then the training data sets are classified to obtain domain subsets (such as medical literature databases, electronic medical record data, etc.) and general subsets (such as Wikipedia, Common Crawl, etc.).
[0069] S220, input the domain subset and the general subset into the training model, calculate the attention features, and obtain the attention distribution of the domain subset and the general subset.
[0070] Specifically, in some embodiments, step S220 includes the following steps:
[0071] From the first domain subset (X d1 ,X d2 ,……,X dn ) and the universal subset (X g1 ,X g2 ,……,X gm ) randomly select X i ′ as the source dataset, where n represents the number of domain subsets, m represents the number of general subsets, i∈[d1,…,dn,g1,…, gm];
[0072] From the second domain subset Randomly select X t ′ as the target data set, where t∈[d1,…,dn], the first domain subset and the second domain subset are different data sets in the domain subset;
[0073] The source dataset X i ′ and target dataset X t ′ Input the training model M and calculate the corresponding attention matrix A i and A t , where A i and A t yes The multidimensional array in |B| represents the batch size of the data, L×H represents the number of transformer layers and the number of multi-head self-attention, S represents the maximum length of the sequence, S 2 Represents the pairwise attention relationship between sequences. Attention matrix A i and A t contains the information of attention distribution.
[0074] S230. Use the training model to perform knowledge detection on the domain subset and the general subset to obtain knowledge weights of the domain subset and the general subset.
[0075] Specifically, in some embodiments, step S230 includes the following steps:
[0076] First, calculate the training model M on the source dataset X i The perplexity p(i, M) on ′ includes:
[0077] For the source dataset X i ′ is preprocessed, and the preprocessed data is input into the training model M to obtain the output probability distribution of the training model M as P=(P 1 ,P 2 ,……,P n );
[0078] Calculate the negative of the log probability of each word in the dataset , the total cross entropy loss is accumulated, the total cross entropy loss The calculation formula is as follows:
[0079] in, Represents the total number of words in the dataset, Indicates The predicted probability of a word;
[0080] Divide the total cross entropy loss by the number of words in the dataset to get the average cross entropy loss ;
[0081] Exponentially take the average cross entropy loss to get the perplexity , perplexity The calculation formula is as follows:
[0082] Next, based on the perplexity The knowledge weight is ,in, Indicates confusion, Represents the knowledge weight.
[0083] S240. Determine a mixing weight of the training data set based on the attention distribution and knowledge weight of the domain subset and the general subset.
[0084] In some embodiments, step S230 includes the following steps:
[0085] Based on the attention matrix and , calculate the source data set X i The dataset weights of ′ are as follows:
[0086] in, Represents the attention matrix The average value over the |B| dimension, Represents the attention matrix Average value over |B| dimension, dataset weight Indicates the similarity between the source data and the target data in terms of attention. This helps the training model automatically assign weights to different data sources. The more similar the source data is to the target data in terms of attention, the higher the weight it gets.
[0087] According to the source dataset X i 'The confusion , calculate the penalty parameter as follows:
[0088] Based on the penalty parameter , the mixed weight of the training data set is obtained as:
[0089] in, Represents the mixed weight, penalty parameter Used to balance the relationship between training model and attention distribution.
[0090] S250, dynamically adjusting the mixing ratio of the training data set according to the mixing weight of the training data set to obtain adjusted training data, and determining whether the training data meets a preset standard,
[0091] If yes, execute step S260;
[0092] If not, the model is further trained and steps S220 to S250 are re-executed until the training model reaches a preset standard or reaches a maximum number of iterations.
[0093] In some embodiments, the mixing ratio of the training data set is dynamically adjusted according to the mixing weight of the training data set, and the adjusted training data is obtained as follows:
[0094] in, , Indicates uniform sampling from the source dataset.
[0095] S260, end model training.
[0096] In this embodiment, first, by dividing the data set of the target domain into domain subsets and general subsets, and calculating their attention distribution and knowledge weights respectively, the characteristics of the target domain can be modeled more accurately, and the performance of the model in the target domain can be improved. Secondly, the mixed weights are calculated based on the attention distribution and perplexity, and the mixing ratio of the training data is dynamically adjusted, so that the model can automatically optimize the weight distribution of the data source during the training process, thereby improving the training efficiency and effect. In addition, the introduction of perplexity to calculate the knowledge weights and balancing the relationship between the two through the penalty parameter can help the model better understand the potential correlation between the data and enhance the generalization ability of the model. In summary, this solution significantly improves the training efficiency and effect of the model by accurately dividing the data set, dynamically adjusting the data mixing ratio, optimizing the data weight distribution, and introducing the penalty parameter to balance the relationship between the model and the attention distribution, and enhances the prediction accuracy and robustness of the model, and has broad application prospects and practical value.
[0097] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.
[0098] refer to Figure 4 , Figure 4The schematic diagram of the structure of a data matching device for pre-training a large vertical model provided by an embodiment of the present application is shown. The data matching device can be used to achieve data matching for pre-training a large vertical model. The data matching device described below can correspond to the data matching method described above. Specifically, the data matching device is applied to a computer cluster, which includes multiple computer devices, and the device includes:
[0099] An attention analysis module 410 is used to determine a target domain, input a domain subset and a common subset into a training model, calculate attention features, and obtain attention distributions of the domain subset and the common subset;
[0100] A knowledge weight calculation module 420 is used to perform knowledge detection on the domain subset and the general subset using the training model to obtain knowledge weights of the domain subset and the general subset;
[0101] A hybrid weight determination module 430, for determining a hybrid weight of a training data set based on the attention distribution and knowledge weight of the domain subset and the general subset;
[0102] The model optimization module 440 is used to dynamically adjust the mixing ratio of the training data set according to the mixing weight of the training data set, and further train the training model until the training model reaches a preset standard.
[0103] In some embodiments, the attention analysis module 410 is also used to obtain training data sets from multiple different sources, and classify the training data sets to obtain domain subsets and general subsets.
[0104] Specifically, in some embodiments, the knowledge weight calculation module 420 may extract a source data set and a target data set from a domain subset and a general subset, calculate the perplexity of the training model on the source data set, and then obtain the knowledge weight based on the perplexity.
[0105] Specifically, in some embodiments, the hybrid weight determination module 430 balances the relationship between the attention distribution and the knowledge weight of the training model through a penalty parameter to obtain the hybrid weight of the training data set.
[0106] Regarding the processes not described in detail in the data matching device for pre-training of large vertical models in this embodiment, reference can be made to the relevant parts in the above-mentioned data matching method embodiment for pre-training of large vertical models, which will not be repeated here.
[0107] The present application also provides an electronic device, such as Figure 5As shown, the electronic device may include an input unit 510, a memory 520, a processor 530 and an output unit 540. The memory 520 stores program instructions that can be run on the processor 530, and the processor 530 calls the program instructions to execute the methods and / or technical solutions based on the aforementioned multiple embodiments. The electronic device may be a mobile terminal device such as a mobile phone or a computer. For the processes not described in detail in the electronic device in this embodiment, reference may be made to the relevant parts in the above-mentioned embodiment of the riser curve optimization method, which will not be repeated here.
[0108] In addition, the embodiment of the present application also provides a computer-readable storage medium for storing execution Figure 1 or Figure 2 Embodiment Any computer program of a data matching method for pre-training of a large vertical model. For example, a computer program instruction, when executed by a computer, can call or provide a method and / or technical solution according to the present application through the operation of the computer. The program instructions for calling the method of the present application may be stored in a fixed or removable storage medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium and / or stored in a storage medium that runs according to the program instructions.
[0109] The above is a detailed introduction to the data matching method, device and equipment for pre-training of large vertical models provided in the embodiments of the present application. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can refer to each other. For the equipment disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.
[0110] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the present application is not limited to any specific combination of hardware and software.
[0111] Although the present application has been described in detail above with general descriptions and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements may be made to the present application. Therefore, these modifications or improvements made without departing from the spirit of the present application all fall within the scope of protection claimed in the present application.
Claims
1. A data matching method for pre-training of a vertical large model, characterized in that: Applied to a computer cluster, the computer cluster includes a plurality of computer devices, and the method includes the following steps: Determine a target domain, input a domain subset and a universal subset into a training model, calculate attention features, and obtain attention distributions of the domain subset and the universal subset; the target domain includes a medical field, the domain subset includes a medical literature database, and the universal subset includes Wikipedia; including: From the first domain subset (X d1 ,X d2 ,……,X dn ) and the universal subset (X g1 ,X g2 ,……,X gm ) randomly select X i ′ as the source dataset, where n represents the number of domain subsets, m represents the number of general subsets, i∈[d1,…,dn,g1,…,gm]; From the second domain subset (X * d1 ,X * d2 ,……,X * dn ) randomly select X t ′ as the target data set, where t∈[d1,…,dn], the first domain subset and the second domain subset are different data sets in the domain subset; The source dataset X i ′ and the target dataset X t ′ Input the training model M and calculate the corresponding attention matrix A i and A t , where A i and A t The dimension is is a multidimensional array of |B|, which represents the batch size of the data, L×H represents the number of transformer layers and the number of multi-head self-attention, and S 2 Represents the pairwise attention relationship between sequences; Using the training model to perform knowledge detection on the domain subset and the general subset to obtain knowledge weights of the domain subset and the general subset; including: Calculate the training model M on the source dataset X i The perplexity p(i, M) on ′; Based on the perplexity p(i, M), the knowledge weight is obtained as K(i)=1 / p(i, M), where p(i, M) represents perplexity and K(i) represents knowledge weight; Determining a mixing weight of a training data set based on the attention distribution and knowledge weight of the domain subset and the general subset; comprising: Based on the attention matrix A i and A t , calculate the source data set X i The dataset weights of ′ are as follows: in, Represents the attention matrix A i The average value over the |B| dimension, Represents the attention matrix A t The average value over |B| dimensions, the dataset weight w i Indicates the similarity between the source data and the target data in terms of attention; According to the source dataset X i ′’s confusion p(i, M), calculate the penalty parameter Δ(P i )as follows: Based on the penalty parameter Δ(P i ), the mixed weight of the training data set is: w′ i =w i +Δ(P i ) (3) Among them, w′ i represents the mixed weight, the penalty parameter Δ(P i ) is used to balance the relationship between the training model and the attention distribution; The mixing ratio of the training data set is dynamically adjusted according to the mixing weight of the training data set, and the training model is trained until the training model reaches a preset standard.
2. The data matching method according to claim 1, characterized in that: Before inputting the domain subset and the general subset into the training model, the method further includes: A plurality of training data sets from different sources are obtained, and the training data sets are classified to obtain the domain subset and the general subset.
3. The data matching method according to claim 1, characterized in that: The calculation of the training model M on the source data set X i The perplexity p(i, M) on ′ includes: For the source dataset X i 'Perform preprocessing and input the preprocessed data into the training model M to obtain the output probability distribution of the training model M; Calculate the negative value of the logarithmic probability of each word in the data set and accumulate them to get the total cross entropy loss; Dividing the total cross entropy loss by the number of words in the data set to obtain an average cross entropy loss; The average cross entropy loss is exponentially calculated to obtain the perplexity p(i, M).
4. The data matching method according to claim 3, characterized in that: The mixing ratio of the training data set is dynamically adjusted according to the mixing weight of the training data set, and the adjusted training data is obtained as follows: Among them, |X i |=w′ i |X|,U(X i ) indicates uniform sampling from the source dataset.
5. A data matching device for pre-training of vertical large-scale models, characterized in that: Applied to a computer cluster, the computer cluster includes a plurality of computer devices, and the apparatus includes: The attention analysis module is used to determine the target domain, input the domain subset and the universal subset into the training model, calculate the attention features, and obtain the attention distribution of the domain subset and the universal subset; the target domain includes the medical field, the domain subset includes the medical literature database, and the universal subset includes Wikipedia; including: From the first domain subset (X d1 ,X d2 ,……,X dn ) and the universal subset (X g1 ,X g2 ,……,X gm ) randomly select X i ′ as the source dataset, where n represents the number of domain subsets, m represents the number of general subsets, i∈[d1,…,dn,g1,…,gm]; From the second domain subset (X * d1 ,X * d2 ,……,X * dn ) randomly select X t ′ as the target data set, where t∈[d1,…,dn], the first domain subset and the second domain subset are different data sets in the domain subset; The source dataset X i ′ and the target dataset X t ′ Input the training model M and calculate the corresponding attention matrix A i and A t , where A i and A t The dimension is is a multidimensional array of |B|, which represents the batch size of the data, L×H represents the number of transformer layers and the number of multi-head self-attention, and S 2 Represents the pairwise attention relationship between sequences; A knowledge weight calculation module, used to perform knowledge detection on the domain subset and the general subset using the training model to obtain the knowledge weights of the domain subset and the general subset; comprising: Calculate the training model M on the source dataset X i The perplexity p(i, M) on ′; Based on the perplexity p(i, M), the knowledge weight is obtained as K(i)=1 / p(i, M), where p(i, M) represents perplexity and K(i) represents knowledge weight; A hybrid weight determination module, used to determine the hybrid weight of the training data set based on the attention distribution and knowledge weight of the domain subset and the general subset; comprising: Based on the attention matrix A i and A t , calculate the source data set X i The dataset weights of ′ are as follows: in, Represents the attention matrix A i The average value over the |B| dimension, Represents the attention matrix A t The average value over |B| dimensions, the dataset weight w i Indicates the similarity between the source data and the target data in terms of attention; According to the source dataset X i ′’s confusion p(i, M), calculate the penalty parameter Δ(P i )as follows: Based on the penalty parameter Δ(P i ), the mixed weight of the training data set is: w′ i =w i +Δ(P i ) (3) Among them, w′ i represents the mixed weight, the penalty parameter Δ(P i ) is used to balance the relationship between the training model and the attention distribution; The model optimization module is used to dynamically adjust the mixing ratio of the training data set according to the mixing weight of the training data set, and train the training model until the training model reaches a preset standard.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the data matching method for pre-training of vertical large models as described in any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the data matching method for pre-training of a vertical large model as described in any one of claims 1 to 4.
Citation Information
Patent Citations
Multi-round dialogue reply generation method based on dual-channel semantic enhancement and terminal equipment
CN115495552A
Multi-label classification model training method, multi-label classification method and related device
CN118378148A
