Automatic construction method and system for large chemical scene model and medium
By combining a complete training with fine-tuning and distillation, the operational complexity and information loss problems of large chemical scene models are solved, and efficient and simple model construction and quality improvement are achieved.
Patent Information
- Application Number
- CN202510720355.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-30
- Publication Date
- 2025-09-16
AI Technical Summary
In the existing technology, the model fine-tuning and distillation processes are separated into two independent steps. The operation process is complicated, the quality of the data set is not guaranteed, there is loss in information transmission, and it is difficult to efficiently build a high-quality large model of chemical scene.
A method of one-time complete training combined with fine-tuning and distillation is adopted. The open source base model with a large number of model parameters is used for training on the chemical industry scenario fine-tuning dataset. The attenuation parameters and strategies are set, and knowledge flow is achieved through backpropagation and gradient update. Finally, a chemical industry scenario model with a small number of parameters is pruned.
It simplifies the operation process, maximizes the use of fine-tuning data sets, improves model quality, achieves smooth transition, shortens training time, and improves model convergence speed and effect.
Smart Images
Figure CN120653735A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of data processing technology, in particular to big data processing in the chemical industry, and specifically to a method, system and medium for automatically constructing a large model of a chemical scene. Background Art
[0002] The chemical industry scenario big model refers to a big model with chemical industry knowledge, which can help chemical industry practitioners to conduct chemical-related knowledge questions and answers, parameter optimization, etc., and significantly improve the work efficiency of chemical industry personnel.
[0003] A large model refers to a machine learning model with large-scale parameters and complex computing structure. It is usually built by a deep neural network and has billions or even hundreds of billions of parameters. It can process massive data and complete various complex tasks such as natural language processing, computer vision, and speech recognition.
[0004] Model fine-tuning refers to the process of using data from a specific domain or task to perform additional training on a pre-trained large model to adapt the model to the new task. Pre-trained models are usually trained on large-scale general data, learning a wide range of knowledge and patterns, but this knowledge is often not targeted. Through fine-tuning, the model can learn the characteristics and patterns of a specific domain, thereby showing higher professionalism and accuracy on a specific task.
[0005] Model distillation is a technique that transfers knowledge from a large, complex model (the teacher model) to a small, simple model (the student model). The core idea is to significantly reduce the size and computational complexity of the student model while maintaining high performance by mimicking the output or intermediate features of the teacher model.
[0006] In the process of developing large-scale models for chemical industry scenarios, the following technical solutions are usually adopted for fine-tuning and distillation of the model: use an open source base model with a large number of parameters (such as Qwen-72B-base) to perform fine-tuning training on the fine-tuning dataset 1 of the chemical industry scenario to obtain a large chemical industry model Qwen-72B-field with chemical industry scenario knowledge; prepare a certain number of questions in the chemical industry field, and use the above-mentioned model Qwen-72B-field to generate answers to form chemical industry scenario question and answer pairs 2; at this time, Qwen-72B-field serves as a teacher model, which generates and solidifies chemical industry field knowledge; prepare an open source base model with a small number of parameters (such as Qwen-7B-base) as a student model, train it on the above-mentioned chemical industry scenario question and answer pairs 2, learn the output of the teacher model, and thus obtain the distilled model Qwen-7B-field; deploy Qwen-7B-field to the production environment, develop large-scale model applications, and use them for users in chemical industry scenarios.
[0007] However, the prior art has at least the following defects:
[0008] Existing models separate fine-tuning and distillation into two independent steps, which makes the operation process complicated;
[0009] The dataset generated by the fine-tuned teacher model depends on the quality of the questions, and the quality of the dataset is not guaranteed;
[0010] After the teacher model is fine-tuned, the domain knowledge is transferred to the student model, and there is information loss in the middle, making it difficult to obtain a high-quality distillation model. Summary of the Invention
[0011] In view of this, on one hand, some embodiments disclose a method for automatically constructing a large model of a chemical industry scene, including:
[0012] Obtain a chemical industry scene fine-tuning dataset;
[0013] Use the open source base model with large model parameters and perform a complete training using the chemical industry scenario fine-tuning dataset to complete the fine-tuning process and obtain the open source base model with large model parameters for chemical industry scenarios.
[0014] According to the relevant configuration of the small parameter open source base model, set the attenuation parameter in the model structure node of the large parameter open source base model of the chemical scene;
[0015] Set the attenuation strategy for the set attenuation parameters, fine-tune the training steps of the data set according to the chemical scene, and train the open source base model of the chemical scene large model parameters to complete a complete training;
[0016] The trained open source base model with large parameters of chemical scene model is pruned to obtain a chemical scene model with small parameters.
[0017] Furthermore, in some embodiments of the disclosed method for automatically constructing a large model of a chemical scene, the principle of setting the attenuation parameter is to obtain the complete structure of the small model after removing the large model parameter quantity according to the attenuation parameter to open the structure of the original model.
[0018] In some embodiments of the disclosed method for automatically constructing a large model of a chemical industry scene, the distribution of attenuation parameters remains uniform.
[0019] Some embodiments disclose an automated construction method for a large model of a chemical scene, in which the open source base model with large model parameters is the Qwen-72B-base model with 80 model layers, and the open source base model with small model parameters is the Qwen-7B-Base model with 28 model layers; a layer attenuation coefficient is added to the 52 model layers, and the network is assembled after each attenuation layer is multiplied by the layer attenuation coefficient.
[0020] Some embodiments disclose an automated construction method for a large chemical scene model, and the obtained open source base model for the large chemical scene model parameters is the Qwen-72B-field model. During the training process of the Qwen-72B-field model, the knowledge of the model flows through back propagation and gradient update, gradually concentrating from the attenuated nodes to the non-attenuated nodes. When the training is completed, all the attenuation parameters become 0.
[0021] In some embodiments of the disclosed method for automatically constructing a large chemical scene model, the attenuation strategy of the attenuation parameter includes a linear attenuation strategy and a cosine attenuation strategy.
[0022] In the method for automatically constructing a large chemical scene model disclosed in some embodiments, for a linear attenuation strategy, the total number of training steps is n total , the current step number is n, then the attenuation coefficient of the next update is 1-n / n total ;
[0023] For the cosine decay strategy, the total number of training steps is n total , the current step number is n, then the attenuation coefficient for the next update is 0.5×(1+cos(n×π / n total ).
[0024] Some embodiments disclose an automated construction method for a large chemical scene model, which trains the parameters of the large chemical scene model according to the training steps of the chemical scene fine-tuning data set. During the training process of the open source base model, all attenuation parameters are reduced from 1 to 0 to complete a complete training.
[0025] On the other hand, some embodiments disclose an automated construction system for a large chemical industry scene model, including:
[0026] A data acquisition module, configured to acquire a chemical industry scenario fine-tuning dataset;
[0027] The large model training module is configured to use the open source base model with large model parameters, perform a complete training using the chemical industry scenario fine-tuning dataset, complete the fine-tuning process, and obtain the open source base model with large model parameters for the chemical industry scenario;
[0028] The chemical scene large model parameter open source base model training module is configured to set the attenuation parameter in the model structure node of the chemical scene large model parameter open source base model according to the relevant configuration of the small parameter Kaiyuan base model; according to the set attenuation parameter, the attenuation strategy is set, and the chemical scene large model parameter open source base model is trained according to the training steps of the chemical scene fine-tuning data set to complete a complete training;
[0029] The pruning module is configured to prune the trained open source base model with large parameters of the chemical scene model to obtain a chemical scene model with small parameters.
[0030] On the other hand, some embodiments disclose a computer-readable medium having executable code stored thereon. When the executable code is read, the method for automatically constructing a large model of a chemical industry scene disclosed in an embodiment of the present invention is implemented.
[0031] The automated construction method of a large chemical scene model disclosed in some embodiments of the present invention has at least the following beneficial technical effects:
[0032] The fine-tuning and distillation processes are organically combined to form a unified end-to-end process, which simplifies the training process.
[0033] The distillation stage uses a fine-tuning dataset, which eliminates the need to regenerate the dataset and simplifies the process. It can also maximize the use of the fine-tuning dataset and improve model quality.
[0034] The two tasks from the fine-tuning stage to the distillation stage achieve a smooth transition without obvious parameter distribution jumps, avoiding parameter optimization oscillations, accelerating model convergence, and improving model performance.
[0035] The attenuation parameter is attenuated by the set attenuation coefficient to achieve the distillation operation from the teacher model to the student model. The double-shoulder parameter is attenuated to 0 to achieve the pruning operation from the teacher model to the student model. The attenuation parameter slowly decays from 1 to 0, which helps the model to adaptively realize the flow of knowledge between nodes. After the training is completed, the corresponding nodes can be directly removed to achieve the purpose of one-click pruning. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 Example 1 Flowchart of the automated construction method of a large chemical industry scene model. DETAILED DESCRIPTION
[0037] The term "embodiment" is used herein specifically to describe any embodiment as "exemplary," and should not be construed as superior or preferable to other embodiments. Performance indicators in the embodiments of the present invention were tested using conventional testing methods in the art, unless otherwise specified. It should be understood that the terms used in the embodiments of the present invention are intended solely to describe specific implementations and are not intended to limit the disclosure of the embodiments of the present invention.
[0038] Unless otherwise specified, the technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which the embodiments of the present invention pertain; any experimental methods and technical means not otherwise specified in the embodiments of the present invention refer to experimental methods and technical means commonly used by those skilled in the art.
[0039] As used herein, the terms "substantially" and "approximately" are used to describe small fluctuations. For example, they can refer to less than or equal to ±5%, such as less than or equal to ±2%, such as less than or equal to ±1%, such as less than or equal to ±0.5%, such as less than or equal to ±0.2%, such as less than or equal to ±0.1%, such as less than or equal to ±0.05%. Numerical data expressed or presented in range format herein are used for convenience and brevity only and should therefore be interpreted flexibly to include not only the values explicitly listed as the limits of the range, but also all independent values or subranges contained within the range. For example, a numerical range of "1-5%" should be interpreted to include not only the explicitly listed values of 1% to 5%, but also the independent values and subranges within the indicated range. Thus, included in this numerical range are independent values such as 2%, 3.5%, and 4%, and subranges such as 1% to 3%, 2% to 4%, and 3% to 5%, etc. This principle also applies to ranges that only list a single value. Furthermore, this interpretation applies regardless of the width of the range or the characteristics described.
[0040] Throughout this document, including in the claims, transitional terms such as "comprises," "includes," "with," "having," "contains," "involving," and "accommodating" are understood to be open-ended, meaning "including but not limited to." Only the transitional terms "consisting of" and "composed of" are closed transitional terms.
[0041] In order to better illustrate the present invention, numerous specific details are provided in the following specific examples. It should be understood by those skilled in the art that the present invention can be practiced without certain specific details. In the examples, some methods, means, instruments, and equipment well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present invention.
[0042] Under the premise of no conflict, the technical features disclosed in the embodiments of the present invention can be arbitrarily combined, and the resulting technical solutions belong to the contents disclosed in the embodiments of the present invention.
[0043] The technical details are further illustrated below with reference to embodiments.
[0044] In some embodiments, as Figure 1 As shown in the figure, the automated construction method of the chemical industry scene large model includes:
[0045] Obtain a chemical industry scenario fine-tuning dataset; typically, the chemical industry scenario fine-tuning dataset can be in the form of question-answer pairs, reflecting data on relevant technical content in the field of chemical industry scenario technology;
[0046] Use the open source base model with large model parameters and perform a complete training using the chemical industry scenario fine-tuning dataset to complete the fine-tuning process and obtain the open source base model with large model parameters for chemical industry scenarios.
[0047] According to the relevant configuration of the small parameter open source base model, the attenuation parameter is set in the model structure node of the chemical scene large model parameter open source base model; the principle of setting the attenuation parameter disclosed in some embodiments is to obtain the complete structure of the small model after removing the structure of the large model parameter open source model according to the attenuation parameter; generally, the distribution of the attenuation parameter should be kept as uniform as possible;
[0048] Set the attenuation strategy for the set attenuation parameters, fine-tune the training steps of the data set according to the chemical scene, and train the open source base model of the chemical scene large model parameters to complete a complete training;
[0049] The trained open source base model with large parameters of chemical scene model is pruned to obtain a chemical scene model with small parameters.
[0050] In some embodiments, the method for automatically constructing a large chemical scene model is disclosed. The open-source base model for the large model parameters is the Qwen-72B-base model. The open-source base model for the large chemical scene model parameters obtained after a complete training on the chemical scene fine-tuning dataset is the Qwen-72B-field model. The Qwen-72B-field model and the Qwen-72B-base model have completely identical model structures.
[0051] In some embodiments, the open source base model with a large number of model parameters is the Qwen-72B-base model, and the number of model layers is 80; the open source base model with a small number of parameters is the Qwen-7B-Base model, and the number of model layers is 28; therefore, attenuation parameters can be added to 80-28=52 model layers; according to the principle of uniform distribution, 52 of the 80 model layers are selected as attenuation layers, and each attenuation layer is multiplied by the layer attenuation coefficient a to assemble the network, and the remaining 28 model layers are kept as layers to be retained without any changes;
[0052] In some embodiments, the layers to be attenuated in the middle hidden layer are multiplied by the attenuation coefficient b, the Querry, key, and value matrix parameters to be attenuated in the Attention module are multiplied by the attenuation coefficients c, d, and e, respectively, and the matrix parameters of the linear layer MLP are multiplied by the attenuation coefficient f;
[0053] For the layer attenuation coefficient a, hidden layer dimension attenuation coefficient b, Querry matrix attenuation coefficient c, Key matrix attenuation coefficient d, Value matrix attenuation coefficient e, and MLP matrix attenuation coefficient f, you can set a linear attenuation strategy or a cosine attenuation strategy;
[0054] For the linear decay strategy, the decay coefficients a, b, c, d, e, and f are updated at a rate of 1-n / n. total , where the total number of training steps is ntotal , the current step number is n;
[0055] For the cosine attenuation strategy, the attenuation coefficients a, b, c, d, e, and f are updated as 0.5×(1+cos(n×π / n total )), where the total number of training steps is n total , the current step number is n;
[0056] The resulting open-source base model for the large chemical scene model is the Qwen-72B-field model. During the training process of the Qwen-72B-field model, the model's knowledge flows through backpropagation and gradient updates, gradually concentrating from attenuated nodes to unattenuated nodes. At the end of training, all attenuated parameters become 0. The corresponding model structure is then pruned, and the corresponding parameters in the model are discarded to complete distillation, resulting in the small-parameter scene model Qwen-7B-field. The resulting small-parameter scene model Qwen-7B-field is the target model structure with 28 model layers, namely the large chemical scene model.
[0057] Some embodiments disclose an automated construction method for a large chemical scene model, which trains the parameters of the large chemical scene model according to the training steps of the chemical scene fine-tuning data set. During the training process of the open source base model, all attenuation parameters are reduced from 1 to 0 to complete a complete training.
[0058] In some embodiments of the disclosed method for automatically constructing a large chemical scene model, the attenuation strategy of the attenuation parameter includes a linear attenuation strategy and a cosine attenuation strategy.
[0059] In some embodiments, the automated construction system of a large chemical industry scene model includes:
[0060] A data acquisition module, configured to acquire a chemical industry scenario fine-tuning dataset;
[0061] The large model training module is configured to use the open source base model with large model parameters, perform a complete training using the chemical industry scenario fine-tuning dataset, complete the fine-tuning process, and obtain the open source base model with large model parameters for the chemical industry scenario;
[0062] The chemical scene large model parameter open source base model training module is configured to set the attenuation parameter in the model structure node of the chemical scene large model parameter open source base model according to the relevant configuration of the small parameter Kaiyuan base model; according to the set attenuation parameter, the attenuation strategy is set, and the chemical scene large model parameter open source base model is trained according to the training steps of the chemical scene fine-tuning data set to complete a complete training;
[0063] The pruning module is configured to prune the trained open source base model with large parameters of the chemical scene model to obtain a chemical scene model with small parameters.
[0064] Some embodiments disclose computer-readable media, on which executable code is stored. When the executable code is read, the method for automatically constructing a large chemical scene model disclosed in the embodiments of the present invention is implemented.
[0065] The automated construction method for a large chemical industry scenario model disclosed in an embodiment of the present invention organically combines the fine-tuning and distillation processes into a unified end-to-end process, simplifying the training process.
[0066] The distillation stage uses a fine-tuning dataset, eliminating the need to regenerate the dataset and simplifying the process. This also maximizes the use of the fine-tuning dataset, improving model quality. The two tasks from the fine-tuning stage to the distillation stage achieve a smooth transition without noticeable parameter distribution jumps, avoiding parameter optimization oscillations, accelerating model convergence, and improving model performance. The decay parameter slowly decays from 1 to 0, helping the model to adaptively realize the flow of knowledge between nodes. After training, the corresponding nodes can be directly removed, achieving one-click pruning.
[0067] The technical solutions and technical details disclosed in the embodiments of the present invention are merely illustrative of the inventive concept of the present invention and do not constitute a limitation on the technical solutions of the embodiments of the present invention. Any conventional changes, replacements or combinations of the technical details disclosed in the embodiments of the present invention have the same inventive concept as the present invention and are within the scope of protection of the claims of the present invention.
Claims
1. A method for automatically constructing a large chemical scene model, characterized in that: include: Obtain a chemical industry scene fine-tuning dataset; Use the open source base model with large model parameters and perform a complete training using the chemical industry scenario fine-tuning dataset to complete the fine-tuning process and obtain the open source base model with large model parameters for chemical industry scenarios. According to the relevant configuration of the small parameter open source base model, set the attenuation parameter in the model structure node of the large parameter open source base model of the chemical scene; Set the attenuation strategy for the set attenuation parameters, fine-tune the training steps of the data set according to the chemical scene, and train the open source base model of the chemical scene large model parameters to complete a complete training; The trained open source base model with large parameters of chemical scene model is pruned to obtain a chemical scene model with small parameters.
2. The method for automatically constructing a large chemical scene model according to claim 1, characterized in that: The principle of setting the attenuation parameter is to obtain the complete structure of the small model after removing the large model parameters according to the attenuation parameter to open the structure of the original model.
3. The method for automatically constructing a large chemical scene model according to claim 1, characterized in that: The distribution of the decay parameters remains uniform.
4. The method for automatically constructing a large chemical scene model according to claim 1, characterized in that: The open source base model with large model parameters is the Qwen-72B-base model, with 80 model layers, and the open source base model with small parameters is the Qwen-7B-Base model, with 28 model layers. A layer attenuation coefficient is added to the 52 model layers, and the network is assembled after each attenuation layer is multiplied by the layer attenuation coefficient.
5. The method for automatically constructing a large chemical scene model according to claim 4, characterized in that: The open source base model for the large model parameters of chemical scene is the Qwen-72B-field model. During the training process of the Qwen-72B-field model, the knowledge of the model flows through back propagation and gradient update, and gradually concentrates from the attenuated nodes to the non-attenuated nodes. When the training is completed, all the attenuated parameters become 0.
6. The method for automatically constructing a large chemical scene model according to claim 1, characterized in that: The attenuation strategies of the attenuation parameters include linear attenuation and cosine attenuation.
7. The method for automatically constructing a large chemical scene model according to claim 6, characterized in that: For the linear decay strategy, the total number of training steps is n total , the current step number is n, then the attenuation coefficient of the next update is 1-n / n total ; For the cosine decay strategy, the total number of training steps is n total , the current step number is n, then the attenuation coefficient of the next update is 0.5×(1+cos(n×π / n total ).
8. The method for automatically constructing a large chemical scene model according to claim 1, characterized in that: According to the training steps of the chemical scene fine-tuning dataset, the parameters of the chemical scene large model are trained. During the training process of the open source base model, all attenuation parameters are reduced from 1 to 0 to complete a complete training.
9. The automated construction system of a large chemical scene model is characterized by: include: A data acquisition module, configured to acquire a chemical industry scenario fine-tuning dataset; The large model training module is configured to use the open source base model with large model parameters, perform a complete training using the chemical industry scenario fine-tuning dataset, complete the fine-tuning process, and obtain the open source base model with large model parameters for the chemical industry scenario; The chemical scene large model parameter open source base model training module is configured to set the attenuation parameter in the model structure node of the large model parameter open source base model according to the relevant configuration of the small parameter Kaiyuan base model; according to the set attenuation parameter, the attenuation strategy is set, and the chemical scene large model parameter open source base model is trained according to the training steps of the chemical scene fine-tuning data set to complete a complete training; The pruning module is configured to prune the trained open source base model with large parameters of the chemical scene model to obtain a chemical scene model with small parameters.
10. A computer-readable medium, characterized in that The computer-readable medium stores executable code, and when the executable code is read, the automated construction method of the chemical industry scene large model according to any one of claims 1 to 9 is implemented.