Text question answering method and device based on miniaturized large language model

Abstract

The present invention provides a text question-answering method and device based on a miniaturized large language model, which relates to the field of artificial intelligence technology. The method includes: obtaining input text; inputting the input text into the miniaturized large language model to obtain a predicted answer output by the miniaturized large language model; wherein the miniaturized large language model is obtained by fine-tuning the large language model using limited domain supervised fine-tuning SFT data, and pruning the fine-tuned model in combination with the model fine-tuning result, and fine-tuning the pruned model using general SFT data, and pruning the fine-tuned and pruned model in combination with the pruning model fine-tuning result. The present invention uses limited domain SFT data and general SFT data to fine-tune and prune the large language model respectively, so that the model forgets important parameters in general knowledge while adapting to the language characteristics and knowledge requirements of a specific field, thereby reducing the number of parameters and computational complexity of the model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a text question answering method and device based on a miniaturized large language model. Background Art

[0002] On the one hand, language models based on ultra-large-scale parameters demonstrate exciting general and specialized emergent learning capabilities. On the other hand, the computing power required to train and infer a large language model also makes the threshold for large language models high. The performance of a large language model is directly related to its model size, and the scale effect is very significant. The larger the model parameters and the more training data, the better the model effect. This makes cost and effect a difficult trade-off, and the large language models that have been put into practical use also face huge computing power and energy consumption. Therefore, academia and industry have been working hard to miniaturize large language models so that small-scale language models can retain capabilities close to those of large language models.

[0003] Currently, most miniaturized language models focus on fine-tuning parameters on the model to forget some sensitive information that does not need to be known, thereby retaining the general knowledge capabilities of the large language model and enabling the small model to achieve better scores on various benchmarks.

[0004] However, in actual applications, there are many scenarios that do not require the general knowledge capabilities of large language models, but only the knowledge capabilities of specific fields, such as customer service, medical consultation, consultation, navigation, recommendation, etc., and the large language models in these subdivided fields are mostly obtained by fine-tuning the general large language model, so that it has the general capabilities of the large language model on the one hand, and the knowledge capabilities specific to the field on the other hand. This requires the proprietary large language model to retain the parameter volume of the large language model while retaining the knowledge capabilities that are not required for a large number of tasks. In addition, fine-tuning updates the model parameters, and does not change the parameter volume, nor the size and structure of the model.

[0005] Based on this, we propose a method to forget unnecessary multi-task solving capabilities as much as possible while training and fine-tuning the model's proprietary knowledge capabilities, and only retain the general language capabilities related to specific tasks. We propose a pruning to forget method, which uses pruning to force the large language model to forget unimportant knowledge, but retain important neural network pathways related to the current specific task. Summary of the invention

[0006] The present invention provides a text question answering method and device based on a miniaturized large language model, which is used to solve the defect in the prior art that fine-tuning and updating model parameters causes excessive invalid knowledge of a specific large language model to affect deployment, thereby reducing the number of parameters of the large language model while retaining specific task capabilities and reducing deployment costs.

[0007] The present invention provides a text question answering method based on a miniaturized large language model, comprising: obtaining input text; inputting the input text into the miniaturized large language model to obtain a predicted answer output by the miniaturized large language model; wherein the miniaturized large language model is obtained by fine-tuning a large language model using limited domain supervised fine-tuning SFT data, and pruning the fine-tuned model in combination with the model fine-tuning result, and by fine-tuning the pruned model using general SFT data, and pruning the fine-tuned pruned model in combination with the pruned model fine-tuning result.

[0008] According to a text question-answering method based on a miniaturized large language model provided by the present invention, before inputting the input text into the miniaturized large language model, the method includes: obtaining general SFT data and limited domain SFT data; using the limited domain SFT data to train the previously selected large language model to obtain a model fine-tuning result; according to the model fine-tuning result, determining the first output value of each model parameter and the first error signal of each model parameter corresponding to reverse regression; according to the first output value of each model parameter and the first error signal of each model parameter corresponding to reverse regression, combined with a first preset weight allocation strategy, determining a first pruning weight matrix; wherein the first preset weight allocation strategy is used to limit the allocation weights of each first output value and the first error signal corresponding to each model parameter; according to the first pruning weight matrix, the values ​​are arranged from small to large. In order, model parameters corresponding to the first preset ratio are selected for pruning to obtain a pruned model; the pruned model is trained using general SFT data to obtain a pruned model fine-tuning result; according to the pruned model fine-tuning result, a second output value of each pruned model parameter and a second error signal corresponding to each pruned model parameter during reverse regression are determined; according to the second output value of each pruned model parameter and the second error signal corresponding to each pruned model parameter during reverse regression, a second pruning weight matrix is ​​determined in combination with a second preset weight allocation strategy; wherein the second preset weight allocation strategy is used to limit the allocation weights of each second output value and each model parameter corresponding to the second error signal; according to the second pruning weight matrix, pruned model parameters corresponding to the second preset ratio are selected for pruning in a descending order of values ​​to obtain a miniaturized large language model.

[0009] According to a text question-answering method based on a miniaturized large language model provided by the present invention, a first pruning weight matrix is ​​determined according to the first output value of each model parameter and the first error signal of each model parameter corresponding to the reverse regression, combined with a first preset weight allocation strategy, including: determining the first average value of each layer of model parameters according to the first output value of each model parameter, and determining the first signal average value of each layer of model parameters according to the first error signal of each model parameter corresponding to the reverse regression; determining the corresponding first threshold value based on the first preset forgetting parameter and the first average value of the model parameter of the corresponding layer, and determining the first signal average value of each layer of model parameters based on the first preset forgetting parameter and the first signal average value of the model parameter of the corresponding layer. The average value is determined to correspond to the second threshold; for each iteration, whether the first output value of each model parameter is less than the first threshold, and whether the first error signal of the corresponding layer model parameter is less than the second threshold; if the first output value of the model parameter is less than the corresponding first threshold, and the first error signal of the corresponding layer model parameter is less than the corresponding second threshold, then based on the first preset weight allocation strategy, the weight of the model parameter is determined to be the sum of the weight determined in the previous iteration and the preset value; otherwise, based on the first preset weight allocation strategy, the weight of the model parameter is determined to be the weight determined in the previous iteration; according to the determined weights of the model parameters, a first pruning weight matrix is ​​obtained.

[0010] According to a text question answering method based on a miniaturized large language model provided by the present invention, according to the fine-tuning result of the pruning model, the second output value of each pruning model parameter and the second error signal of each pruning model parameter corresponding to the reverse regression are determined, including: determining the second average value of each layer of pruning model parameters according to the second output value of each pruning model parameter, and determining the second signal average value of each layer of pruning model parameters according to the second error signal of each pruning model parameter corresponding to the reverse regression; determining the corresponding third threshold value based on the second preset forgetting parameter and the second average value of the pruning model parameters of the corresponding layer, and determining the corresponding third threshold value based on the second preset forgetting parameter and the second signal average value of the pruning model parameters of the corresponding layer. Corresponding to the fourth threshold; for each iterative training, determine whether the second output value of each pruning model parameter is greater than the corresponding third threshold, and determine whether the second error signal of the corresponding layer pruning model parameter is greater than the corresponding fourth threshold; if the second output value of the pruning model parameter is greater than the third threshold, and the second error signal of the corresponding layer pruning model parameter is greater than the fourth threshold, then based on the second preset weight allocation strategy, determine the weight of the pruning model parameter as the sum of the weight determined in the previous iteration and the preset value; otherwise, based on the second preset weight allocation strategy, determine the weight of the pruning model parameter as the weight determined in the previous iteration; according to the determined weights of the pruning model parameters, obtain the second pruning weight matrix.

[0011] According to a text question-answering method based on a miniaturized large language model provided by the present invention, limited domain SFT data is obtained, including: obtaining domain-limited text; generating domain-limited questions based on the domain-limited text and combining preset question generation rules; predicting answers to the domain-limited questions to obtain predicted answers; and obtaining limited domain SFT data based on the domain-limited questions and corresponding predicted answers.

[0012] According to a text question-answering method based on a miniaturized large language model provided by the present invention, limited-domain SFT data is obtained according to domain-limited questions and corresponding predicted answers, including: obtaining limited-domain training data according to domain-limited questions and corresponding predicted answers; and screening the limited-domain training data to obtain limited-domain SFT data.

[0013] According to a text question-answering method based on a miniaturized large language model provided by the present invention, limited domain training data is screened to obtain limited domain SFT data, including: inputting the limited domain training data into a screening model to obtain a screening label output by the screening model; wherein the screening model is trained based on question-answering data and the screening labels corresponding to the question-answering data; and according to the screening labels, the limited domain training data is screened to obtain the limited domain SFT data.

[0014] According to a text question-answering method based on a miniaturized large language model provided by the present invention, domain-limited questions are generated based on domain-limited text and in combination with preset question generation rules, including: extracting entities in the domain-limited text to obtain screening entities; generating domain-limited questions based on the screening entities and preset question generation rules; wherein the preset question generation rules are used to limit question instance templates or prompt words.

[0015] According to a text question answering method based on a miniaturized large language model provided by the present invention, entities in a domain-limited text are extracted to obtain filtered entities, including: performing word segmentation on the domain-limited text and deleting stop words in each word segmentation; performing named entity recognition on each word segmentation to obtain filtered entities.

[0016] According to a text question-answering method based on a miniaturized large language model provided by the present invention, responses to domain-limited questions are predicted to obtain predicted responses, including: based on a program programming API interface, at least one external large language model is accessed, and the accessed external large language model is used to predict responses to the domain-limited questions respectively to obtain corresponding predicted responses.

[0017] According to a text question-answering method based on a miniaturized large language model provided by the present invention, general SFT data is obtained, including: collecting text data from a preset information source, or collecting text data using a search engine; cleaning and screening the collected text data; extracting entities from the cleaned and screened text data, and generating questions based on the extracted entities, and marking the answers to the corresponding questions to obtain general SFT data.

[0018] The present invention also provides a text question and answer device based on a miniaturized large language model, comprising: a question acquisition module, which acquires input text; a question and answer prediction module, which inputs the input text into the miniaturized large language model to obtain a predicted answer output by the miniaturized large language model; wherein the miniaturized large language model is obtained by fine-tuning the large language model using limited domain supervised fine-tuning SFT data, and pruning the fine-tuned model in combination with the model fine-tuning results, and by fine-tuning the pruned model using general SFT data, and pruning the fine-tuned and pruned model in combination with the pruned model fine-tuning results.

[0019] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the text question answering method based on the miniaturized large language model as described above is implemented.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described text question-answering methods based on a miniaturized large language model.

[0021] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned text question answering methods based on a miniaturized large language model.

[0022] The text question answering method and device based on a miniaturized large language model provided by the present invention fine-tune the large language model by using limited domain supervised fine-tuning SFT data, so that the model can better adapt to the language characteristics and knowledge requirements of a specific domain, and prune the fine-tuned model in combination with the model fine-tuning results, thereby removing unnecessary parameters and calculations, making the model more concise and efficient; the pruned model is fine-tuned again using general SFT data, while maintaining adaptability to a specific domain, a certain degree of universality is introduced, and the fine-tuned pruned model is pruned in combination with the pruned model fine-tuning results, so that the model forgets important parameters in general knowledge and reduces the interference of general knowledge on specific tasks, thereby reducing the number of parameters and calculation complexity of the model, balancing the specific domain requirements and general language processing capabilities, so that the model can perform well in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 It is one of the flow charts of the text question answering method based on the miniaturized large language model provided by the present invention;

[0025] Figure 2 This is the second flow chart of the text question answering method based on the miniaturized large language model provided by the present invention;

[0026] Figure 3 It is a schematic diagram of a process for obtaining supervised fine-tuning SFT data in a limited domain provided by the present invention;

[0027] Figure 4 It is a structural schematic diagram of a text question-answering device based on a miniaturized large language model provided by the present invention;

[0028] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Figure 1-Figure 3 is a schematic diagram of the text question answering method based on a miniaturized large language model provided by the present invention, with reference to Figure 1 , the method comprising:

[0031] S11, obtaining input text;

[0032] S12, inputting the input text into the miniaturized large language model to obtain the predicted response output by the miniaturized large language model; wherein the miniaturized large language model is obtained by fine-tuning the large language model using limited domain supervised fine-tuning SFT data, and pruning the fine-tuned model in combination with the model fine-tuning results, and fine-tuning the pruned model using general SFT data, and pruning the fine-tuned pruned model in combination with the pruned model fine-tuning results.

[0033] refer to Figure 2 In an optional embodiment, before inputting the input text into the miniaturized large language model, the method includes:

[0034] S21, obtaining general SFT data and limited domain SFT data.

[0035] In this embodiment, reference Figure 3 , obtain domain-limited supervised fine-tuning (SFT) data, including: obtaining domain-limited text; generating domain-limited questions based on the domain-limited text and combining preset question generation rules; predicting answers to domain-limited questions to obtain predicted answers; obtaining domain-limited SFT data based on domain-limited questions and corresponding predicted answers.

[0036] It should be noted that the acquired field-limited text can obtain corresponding text materials according to the required limited field. The text materials can be screened based on expert screening or text screening strategies formulated based on expert experience. Specifically, they can be textbooks, papers, forum articles, journals, etc. in the corresponding limited field. No further limitation is made here.

[0037] Specifically, based on the domain-limited text and combined with preset question generation rules, domain-limited questions are generated, including: extracting entities in the domain-limited text to obtain screening entities; generating domain-limited questions based on the screening entities and preset question generation rules; wherein the preset question generation rules are used to limit question instance templates or prompt words.

[0038] It should be noted that the preset question generation rules may be "Please explain [named entity] in detail.", "What are the applications of [named entity]?", "Explain the theoretical basis of [named entity].", etc. The preset question generation rules are set according to the actual fields involved and design requirements. For example, the field of artificial intelligence may include "Please write a piece of code to implement [named entity] and explain the code logic", etc., which are not further limited here.

[0039] Specifically, extracting entities from the domain-restricted text to obtain the filtered entities includes: segmenting the domain-restricted text and deleting stop words in each segmented word; performing named entity recognition on each segmented word to obtain the filtered entities. It should be added that the segmentation can be performed using a tokenizer, etc., and the named entity recognition can use NER (Named Entity Recognition) technology, etc., to ensure that entities related to the restricted domain are screened out.

[0040] In an optional embodiment, after performing named entity recognition on each word segment to obtain the screened entities, it also includes: based on a preset expert entity screening strategy, further screening the screened entities to further eliminate named entities that are not related to the limited field to ensure the accuracy of the extracted entities.

[0041] In addition, answer prediction is performed for domain-limited questions to obtain predicted answers, including: based on a program programming API interface, accessing at least one external large language model, and using the accessed external large language model to predict answers to the domain-limited questions respectively to obtain corresponding predicted answers.

[0042] It is worth noting that in order to ensure the quality of the generated replies, the existing large language model with high prediction accuracy can be used for reply prediction. At the same time, multiple external large language models can be used to answer the same question. Since different large language models may be based on different training data and algorithms, they have different knowledge systems and perspectives, which makes it easier to answer the same question through multiple external large language models, avoiding the knowledge bottleneck and cognitive bias of a single large language model, increasing the diversity of answers, and facilitating subsequent screening and verification of data based on the consistency of answers to ensure the accuracy and reliability of the data.

[0043] In addition, based on the domain-limited questions and the corresponding predicted answers, limited domain SFT data is obtained, including: based on the domain-limited questions and the corresponding predicted answers, limited domain training data is obtained; the limited domain training data is screened to obtain limited domain SFT data. It should be noted that the screening of limited domain training data can be performed by experts manually, or by a screening mechanism based on expert experience, so as to further screen out unreasonable questions and wrong answers, and avoid the external large language model being affected by the "hallucination" problem and affecting the accuracy of the answer.

[0044] Furthermore, since the amount of limited domain training data may be large, manual screening is costly or even completely infeasible. Therefore, the limited domain training data is screened to obtain limited domain SFT data, including: inputting the limited domain training data into the screening model to obtain the screening labels output by the screening model; wherein the screening model is trained based on the question and answer data and the screening labels corresponding to the question and answer data; according to the screening labels, the limited domain training data is screened to obtain the limited domain SFT data.

[0045] It should be noted that the screening model can adopt a binary classification model based on the Transformer architecture to better capture the semantic information and contextual relationships in the question and answer data, so as to perform more accurate classification and screening of question and answer pairs; the screening labels corresponding to the question and answer data can be annotated based on expert experience to accurately judge the quality, accuracy and applicability of the question and answer data, and provide high-quality annotated data for the screening model, so as to facilitate the use of question and answer data and the screening labels corresponding to the question and answer data for training, while effectively removing low-quality, inaccurate or inapplicable question and answer data, improving the adaptability of the model, and being able to adapt to the ever-changing question and answer data and application scenarios, and maintaining good screening performance; the screening labels include pass and fail or other symbolic representations of the meaning of "pass" and "fail".

[0046] In an optional embodiment, obtaining general SFT data includes: collecting text data from a preset information source, or using a search engine to collect text data; cleaning and filtering the collected text data; extracting entities from the cleaned and filtered text data, and generating questions based on the extracted entities, and marking the answers to the corresponding questions to obtain general SFT data.

[0047] It should be noted that the preset information source can be the source of an open source text data set, such as various Internet information interaction platforms or databases. In addition, generating questions based on the extracted entities and marking the answers to the corresponding questions to obtain general SFT data can also refer to the above-mentioned method of generating domain-limited questions based on domain-limited texts and combining preset question generation rules, predicting answers to domain-limited questions, and obtaining predicted answers, as well as obtaining domain-limited SFT data based on domain-limited questions and corresponding predicted answers, which will not be repeated here.

[0048] S22, using the limited domain SFT data, train the previously selected large language model to obtain the model fine-tuning result. It should be noted that in the limited domain learning stage, the limited domain SFT data is used to fine-tune the selected large language model to fully learn the relevant knowledge of the limited domain corresponding to the limited domain in the limited domain SFT data, so that the large language model's answers in the limited domain are more accurate, professional and in-depth, and then it is convenient to prune according to the importance of the current model parameter weight in fine-tuning, so as to reduce the number of parameters of the current model and facilitate subsequent deployment on edge devices.

[0049] S23, determining the first output value of each model parameter and the first error signal corresponding to the reverse regression of each model parameter according to the model fine-tuning result.

[0050] S24, determining a first pruning weight matrix according to the first output value of each model parameter and the first error signal corresponding to each model parameter during reverse regression, combined with a first preset weight allocation strategy; wherein the first preset weight allocation strategy is used to limit the allocation weights of each first output value and the first error signal corresponding to each model parameter.

[0051] In this embodiment, according to the first output value of each model parameter and the first error signal of each model parameter corresponding to the reverse regression, in combination with the first preset weight allocation strategy, a first pruning weight matrix is ​​determined, including: determining the first average value of each layer of model parameters according to the first output value of each model parameter, and determining the first signal average value of each layer of model parameters according to the first error signal of each model parameter corresponding to the reverse regression; determining the corresponding first threshold value based on the first preset forgetting parameter and the first average value of the model parameter of the corresponding layer, and determining the corresponding second threshold value based on the first preset forgetting parameter and the first signal average value of the model parameter of the corresponding layer; for each iteration, determining whether the first output value of each model parameter is less than the corresponding first threshold value, and determining whether the first error signal of the model parameter of the corresponding layer is less than the corresponding second threshold value; if the first output value of the model parameter is less than the first threshold value, and the first error signal of the model parameter of the corresponding layer is less than the second threshold value, then based on the first preset weight allocation strategy, the weight of the model parameter is determined to be the sum of the weight determined in the previous iteration and the preset value; otherwise, based on the first preset weight allocation strategy, the weight of the model parameter is determined to be the weight determined in the previous iteration; according to the determined weights of the model parameters, the first pruning weight matrix is ​​obtained.

[0052] It should be noted that since the parameters activated by each sample or each generated word may be different even in a domain-limited dataset, the first pruning weight matrix can be determined in the above manner to count the importance of each model parameter in the training process of the limited domain SFT data, thereby facilitating the subsequent pruning of the parameters with too small first output values ​​of each model parameter and the first error signal corresponding to each model parameter during reverse regression according to the determined importance of each model parameter, so as to reduce the number of model parameters and facilitate subsequent deployment on edge devices.

[0053] In addition, the model parameter weights are determined, expressed as:

[0054] ;

[0055] in, Represents the weight of the jth model parameter of the i-th layer in the current iteration; Indicates that the weight of the current iteration model parameter is the sum of the weight determined in the previous iteration and 1. The preset value can be set according to actual needs. Here, 1 is only an example; Represents the first output value of the jth model parameter of the i-th layer in the current iteration; represents the first average value of all model parameters in the i-th layer; Represents the first error signal of the backward regression of the j-th model parameter of the i-th layer in the current iteration; Represents the first signal average of all model parameters in the i-th layer; k represents the first preset forgetting parameter, which can be adjusted based on actual design requirements or prior experience, and is generally 0.01. k determines the unimportance of the pruned model parameters. The larger the k value, the lower the pruning threshold, otherwise; Compared with the current iteration model parameters The weights determined in the previous iteration.

[0056] S25, according to the first pruning weight matrix, in order of values ​​from small to large, select model parameters corresponding to the first preset ratio for pruning to obtain a pruned model.

[0057] It should be noted that by using the first pruning weight matrix to determine the importance of each model parameter, the model parameters of the first preset ratio with the smallest value are selected for pruning, thereby reducing the number of model parameters, making the model more concise and efficient, and reducing the storage requirements and computing costs of the model. In addition, the first preset ratio can be set according to prior experience or actual design requirements, and is not further limited here.

[0058] S26, using the general SFT data, trains the pruned model to obtain the pruned model fine-tuning result.

[0059] It should be noted that in the general knowledge forgetting stage, the general SFT data is used to fine-tune the pruned model to forget the important parameters in the general knowledge, so that the model can focus more on learning and application in specific fields, reduce the interference of general knowledge on specific tasks, and prevent the model from over-relying on general knowledge, thereby avoiding overfitting in specific tasks.

[0060] S27, determining, according to the pruning model fine-tuning result, a second output value of each pruning model parameter and a second error signal corresponding to each pruning model parameter during reverse regression.

[0061] S28, determining a second pruning weight matrix according to the second output value of each pruning model parameter and the second error signal corresponding to each pruning model parameter during reverse regression, combined with a second preset weight allocation strategy; wherein the second preset weight allocation strategy is used to limit the allocation weights of each second output value and the second error signal corresponding to each pruning model parameter.

[0062] In this embodiment, according to the pruning model fine-tuning result, the second output value of each pruning model parameter and the second error signal of each pruning model parameter corresponding to the reverse regression are determined, including: determining the second average value of each layer of pruning model parameters according to the second output value of each pruning model parameter, and determining the second signal average value of each layer of pruning model parameters according to the second error signal of each pruning model parameter corresponding to the reverse regression; determining the corresponding third threshold value based on the second preset forgetting parameter and the second average value of the pruning model parameter of the corresponding layer, and determining the corresponding fourth threshold value based on the second preset forgetting parameter and the second signal average value of the pruning model parameter of the corresponding layer; for each Iterate training to determine whether the second output value of each pruning model parameter is greater than the corresponding third threshold, and determine whether the second error signal of the corresponding layer pruning model parameter is greater than the corresponding fourth threshold; if the second output value of the pruning model parameter is greater than the third threshold, and the second error signal of the corresponding layer pruning model parameter is greater than the fourth threshold, then based on the second preset weight allocation strategy, determine the weight of the pruning model parameter as the sum of the weight determined in the previous iteration and the preset value; otherwise, based on the second preset weight allocation strategy, determine the weight of the pruning model parameter as the weight determined in the previous iteration; obtain the second pruning weight matrix according to the determined weights of the pruning model parameters.

[0063] It should be noted that, according to the fine-tuning results of the pruning model, the second output value of each pruning model parameter and the second error signal of each pruning model parameter corresponding to the reverse regression can be determined by referring to the first output value of each model parameter and the first error signal of each model parameter corresponding to the reverse regression, combined with the first preset weight allocation strategy, to determine the first pruning weight matrix, which will not be repeated here.

[0064] In addition, the determined pruning model parameter weights are expressed as:

[0065] ;

[0066] in, Represents the weight of the jth pruning model parameter of the i-th layer in the current iteration; Indicates that the weight of the current iteration pruning model parameter is the sum of the weight determined in the previous iteration and 1. The preset value can be set according to actual needs. Here, 1 is only an example; Represents the second output value of the jth simple model parameter of the i-th layer in the current iteration; Represents the second average value of all pruned model parameters in the i-th layer; Represents the second error signal during the backward regression of the j-th pruning model parameter of the i-th layer in the current iteration; represents the second signal average value of all pruned model parameters of the i-th layer; k represents the second preset forgetting parameter, which can be adjusted based on actual design requirements or prior experience, and is generally a positive integer greater than 2. k determines the unimportance of the pruned model parameters. The larger the k value, the lower the pruning threshold, otherwise it is the opposite; Compared with the current iteration model parameters The weights determined in the previous iteration.

[0067] S29, according to the second pruning weight matrix, in descending order of values, select pruning model parameters corresponding to the second preset ratio for pruning to obtain a miniaturized large language model.

[0068] It should be noted that by using the second pruning weight matrix, the importance of each pruning model parameter is determined to select the pruning model parameters of the second preset ratio with the largest value for pruning, so that the model forgets the important parameters in the general knowledge, reduces the interference of the general knowledge on the specific task, reduces the number of parameters and the computational complexity of the model, and thus reduces the computing resource requirements. In addition, the second preset ratio can be set according to prior experience or actual design requirements, and is not further limited here.

[0069] It should be noted that during the training process, the specific domain learning stage and the general knowledge forgetting stage are alternately performed multiple times, so that the model learns the relevant knowledge of the limited domain, forgets the important parameters in the general knowledge, and then learns the relevant knowledge of the limited domain in a cycle. In addition, the end of training can be determined based on a preset number of iterations, model convergence, or pruning weight matrix convergence. The convergence of the pruning weight matrix can be determined by determining that the smallest value in the matrix is ​​greater than 1 / 10 of the mean of all values. The specific setting can be based on actual design requirements and is not further limited here.

[0070] In summary, the embodiment of the present invention fine-tunes the large language model by using limited domain supervised fine-tuning SFT data, so that the model can better adapt to the language characteristics and knowledge requirements of a specific domain, and prunes the fine-tuned model in combination with the model fine-tuning results, removes unnecessary parameters and calculations, and makes the model more concise and efficient; uses general SFT data to fine-tune the pruned model again, while maintaining adaptability to a specific domain, it also introduces a certain degree of universality, and prunes the fine-tuned pruned model in combination with the pruned model fine-tuning results, so that the model forgets important parameters in general knowledge and reduces the interference of general knowledge on specific tasks, thereby reducing the number of parameters and computational complexity of the model, balancing specific domain requirements and general language processing capabilities, so that the model can perform well in different scenarios.

[0071] The text question and answer device based on a miniaturized large language model provided by the present invention is described below. The text question and answer device based on a miniaturized large language model described below and the text question and answer method based on a miniaturized large language model described above can be referred to each other.

[0072] Figure 4 A structural schematic diagram of a text question-answering device based on a miniaturized large language model is shown, the device comprising:

[0073] The question acquisition module 41 acquires input text;

[0074] The question-answer prediction module 42 inputs the input text into the miniaturized large language model to obtain the predicted answer output by the miniaturized large language model; wherein the miniaturized large language model is obtained by fine-tuning the large language model using limited domain supervised fine-tuning SFT data, and pruning the fine-tuned model in combination with the model fine-tuning results, and by fine-tuning the pruned model using general SFT data, and pruning the fine-tuned and pruned model in combination with the pruned model fine-tuning results.

[0075] In an optional embodiment, the device further includes: a data acquisition module, which acquires general SFT data and limited domain SFT data before inputting the input text into the miniaturized large language model; a first fine-tuning module, which uses the limited domain SFT data to train the previously selected large language model to obtain a model fine-tuning result; a first statistical module, which determines the first output value of each model parameter and the first error signal of each model parameter corresponding to reverse regression according to the model fine-tuning result; a first matrix determination module, which determines the first pruning weight matrix according to the first output value of each model parameter and the first error signal of each model parameter corresponding to reverse regression, combined with a first preset weight allocation strategy; wherein the first preset weight allocation strategy is used to limit the allocation weights of each first output value and the first error signal corresponding to each model parameter; the first pruning module, according to the first pruning weight matrix, in order from small to large values, The model parameters corresponding to the first preset ratio are selected for pruning to obtain a pruned model; the second fine-tuning module uses the general SFT data to train the pruned model to obtain the fine-tuning result of the pruned model; the second statistical module determines the second output value of each pruned model parameter and the second error signal of each pruned model parameter corresponding to the reverse regression according to the fine-tuning result of the pruned model; the second matrix determination module determines the second pruning weight matrix according to the second output value of each pruned model parameter and the second error signal of each pruned model parameter corresponding to the reverse regression, combined with the second preset weight allocation strategy; wherein the second preset weight allocation strategy is used to limit the allocation weights of each second output value and each pruned model parameter corresponding to the second error signal; the second pruning module selects the pruned model parameters corresponding to the second preset ratio according to the second pruning weight matrix and in descending order of values ​​to perform pruning to obtain a miniaturized large language model.

[0076] Specifically, the data acquisition module includes: a text acquisition unit, which acquires domain-limited text; a question generation unit, which generates domain-limited questions based on the domain-limited text and preset question generation rules; a reply prediction unit, which predicts the reply to the domain-limited question and obtains the predicted reply; a data acquisition unit, which obtains domain-limited SFT data based on the domain-limited question and the corresponding predicted reply.

[0077] Furthermore, the question generation unit includes: an entity extraction subunit, which extracts entities in the domain-limited text to obtain filtered entities; a question generation subunit, which generates domain-limited questions based on the filtered entities and preset question generation rules; wherein the preset question generation rules are used to limit question instance templates or prompt words.

[0078] Specifically, the entity extraction subunit includes: a word segmentation processing subunit, which performs word segmentation on the domain-restricted text and deletes stop words in each word segmentation; and an entity recognition subunit, which performs named entity recognition on each word segmentation to obtain a screening entity.

[0079] In an optional embodiment, the data acquisition module also includes: a screening unit, which, after performing named entity recognition on each word segment and obtaining the screened entities, further screens the screened entities based on a preset expert entity screening strategy, and further eliminates named entities that are not related to the limited field to ensure the accuracy of the extracted entities.

[0080] In addition, the answer prediction unit includes: an access subunit, which accesses at least one external large language model based on a program programming API interface; a prediction subunit, which uses the accessed external large language model to predict answers to domain-limited questions and obtain corresponding predicted answers.

[0081] The data acquisition unit includes: a data acquisition subunit, which obtains limited domain training data according to domain-limited questions and corresponding predicted answers; and a screening subunit, which screens the limited domain training data to obtain limited domain SFT data.

[0082] Furthermore, the screening sub-unit includes: a screening prediction grandson unit, which inputs the limited domain training data into the screening model to obtain the screening label output by the screening model; wherein the screening model is trained based on the question and answer data and the screening labels corresponding to the question and answer data; and a screening grandson unit, which screens the limited domain training data according to the screening labels to obtain the limited domain SFT data.

[0083] In an optional embodiment, the data acquisition module further includes: a data acquisition unit, which collects text data from a preset information source or uses a search engine to collect text data; a data processing unit, which cleans and filters the collected text data; a data generation unit, which extracts entities from the cleaned and filtered text data, generates questions based on the extracted entities, and annotates the answers to the corresponding questions to obtain general SFT data. It should be noted that the data generation unit can be set in combination with the above-mentioned question generation unit and answer prediction unit, and will not be repeated here.

[0084] In an optional embodiment, the first matrix determination module includes: a first data determination unit, which determines the first average value of each layer of model parameters according to the first output value of each model parameter, and determines the first signal average value of each layer of model parameters according to the first error signal corresponding to each model parameter during reverse regression; a first threshold determination unit, which determines the corresponding first threshold based on the first preset forgetting parameter and the first average value of the corresponding layer model parameter, and determines the corresponding second threshold based on the first preset forgetting parameter and the first signal average value of the corresponding layer model parameter; a first judgment unit, which determines, for each iteration, whether the first output value of each model parameter is less than the corresponding first threshold, and determines whether the first error signal of the corresponding layer model parameter is less than the corresponding second threshold; a first weight determination unit, which determines the weight of the model parameter as the sum of the weight determined in the previous iteration and the preset value based on the first preset weight allocation strategy if the first output value of the model parameter is less than the first threshold and the first error signal of the corresponding layer model parameter is less than the second threshold; otherwise, the weight of the model parameter is determined as the weight determined in the previous iteration based on the first preset weight allocation strategy; a first matrix determination unit, which obtains a first pruning weight matrix based on the determined weights of the model parameters.

[0085] In an optional embodiment, the second matrix determination module includes: a second data determination unit, which determines the second average value of each layer of pruning model parameters according to the second output value of each pruning model parameter, and determines the second signal average value of each layer of pruning model parameters according to the second error signal corresponding to each pruning model parameter during reverse regression; a second threshold determination unit, which determines the corresponding third threshold based on the second preset forgetting parameter and the second average value of the pruning model parameter of the corresponding layer, and determines the corresponding fourth threshold based on the second preset forgetting parameter and the second signal average value of the pruning model parameter of the corresponding layer; a second judgment unit, which determines the value of each pruning model parameter for each iterative training. Whether the second output value is greater than the corresponding third threshold, and whether the second error signal of the corresponding layer pruning model parameter is greater than the corresponding fourth threshold; a second weight determination unit, if the second output value of the pruning model parameter is greater than the third threshold, and the second error signal of the corresponding layer pruning model parameter is greater than the fourth threshold, then based on the second preset weight allocation strategy, the weight of the pruning model parameter is determined to be the sum of the weight determined in the previous iteration and the preset value; otherwise, based on the second preset weight allocation strategy, the weight of the pruning model parameter is determined to be the weight determined in the previous iteration; a second matrix determination unit, according to the determined weights of the pruning model parameters, obtains a second pruning weight matrix.

[0086] In summary, the embodiment of the present invention fine-tunes the large language model by using limited domain supervised fine-tuning SFT data, so that the model can better adapt to the language characteristics and knowledge requirements of a specific domain, and prunes the fine-tuned model in combination with the model fine-tuning results, removes unnecessary parameters and calculations, and makes the model more concise and efficient; uses general SFT data to fine-tune the pruned model again, while maintaining adaptability to a specific domain, it also introduces a certain degree of universality, and prunes the fine-tuned pruned model in combination with the pruned model fine-tuning results, so that the model forgets important parameters in general knowledge and reduces the interference of general knowledge on specific tasks, thereby reducing the number of parameters and computational complexity of the model, balancing specific domain requirements and general language processing capabilities, so that the model can perform well in different scenarios.

[0087] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a text question answering method based on a miniaturized large language model, the method comprising: obtaining an input text; inputting the input text into the miniaturized large language model to obtain a predicted answer output by the miniaturized large language model; wherein the miniaturized large language model is obtained by fine-tuning the large language model using limited domain supervised fine-tuning SFT data, and pruning the fine-tuned model in combination with the model fine-tuning result, and fine-tuning the pruned model using general SFT data, and pruning the fine-tuned pruned model in combination with the pruning model fine-tuning result.

[0088] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0089] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the text question answering method based on the miniaturized large language model provided by the above methods, the method including: obtaining input text; inputting the input text into the miniaturized large language model to obtain a predicted answer output by the miniaturized large language model; wherein the miniaturized large language model is obtained by fine-tuning the large language model using limited domain supervised fine-tuning SFT data, and pruning the fine-tuned model in combination with the model fine-tuning results, and fine-tuning the pruned model using general SFT data, and pruning the fine-tuned pruned model in combination with the pruned model fine-tuning results.

[0090] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the text question-answering method based on a miniaturized large language model provided by the above-mentioned methods, the method comprising: obtaining input text; inputting the input text into the miniaturized large language model to obtain a predicted answer output by the miniaturized large language model; wherein the miniaturized large language model is obtained by fine-tuning the large language model using limited domain supervised fine-tuning SFT data, and pruning the fine-tuned model in combination with the model fine-tuning results, and by fine-tuning the pruned model using general SFT data, and pruning the fine-tuned and pruned model in combination with the pruned model fine-tuning results.

[0091] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text question answering method based on a miniaturized large language model, characterized in that: include: Get input text; Inputting the input text into the miniaturized large language model to obtain a predicted answer output by the miniaturized large language model; wherein the miniaturized large language model is obtained by fine-tuning the large language model using limited domain supervised fine-tuning SFT data, and pruning the fine-tuned model in combination with the model fine-tuning result, and fine-tuning the pruned model using general SFT data, and pruning the fine-tuned pruned model in combination with the pruned model fine-tuning result; Before inputting the input text into the miniaturized large language model, the large language model includes: Obtain general SFT data and limited domain SFT data; Using the limited domain SFT data, the previously selected large language model is trained to obtain a model fine-tuning result; Determine, according to the model fine-tuning result, a first output value of each model parameter and a first error signal corresponding to each model parameter during reverse regression; Determine a first pruning weight matrix according to the first output value of each model parameter and the first error signal corresponding to each model parameter during reverse regression, in combination with a first preset weight allocation strategy; wherein the first preset weight allocation strategy is used to limit the allocation weights of each first output value and the first error signal corresponding to each model parameter; According to the first pruning weight matrix, in order of values ​​from small to large, model parameters corresponding to a first preset ratio are selected for pruning to obtain a pruned model; Using the general SFT data, the pruning model is trained to obtain a pruning model fine-tuning result; Determine, according to the pruning model fine-tuning result, a second output value of each pruning model parameter and a second error signal corresponding to each pruning model parameter during reverse regression; Determine a second pruning weight matrix according to the second output value of each pruning model parameter and the second error signal corresponding to each pruning model parameter during reverse regression, in combination with a second preset weight allocation strategy; wherein the second preset weight allocation strategy is used to limit the allocation weights of each second output value and the second error signal corresponding to each model parameter; According to the second pruning weight matrix, pruning model parameters corresponding to a second preset ratio are selected in descending order of values ​​for pruning to obtain a miniaturized large language model.

2. The text question answering method based on a miniaturized large language model according to claim 1, characterized in that: According to the first output value of each model parameter and the first error signal corresponding to each model parameter during reverse regression, combined with a first preset weight allocation strategy, a first pruning weight matrix is ​​determined, including: Determine a first average value of each layer of model parameters according to the first output value of each model parameter, and determine a first signal average value of each layer of model parameters according to the first error signal corresponding to each model parameter during reverse regression; Determine a corresponding first threshold based on a first preset forgetting parameter and a first average value of a corresponding layer model parameter, and determine a corresponding second threshold based on a first preset forgetting parameter and a first signal average value of the corresponding layer model parameter; For each iteration, determining whether the first output value of each of the model parameters is less than the first threshold, and determining whether the first error signal of the corresponding layer model parameter is less than the second threshold; If the first output value of the model parameter is less than the corresponding first threshold value, and the first error signal of the corresponding layer model parameter is less than the corresponding second threshold value, then based on the first preset weight allocation strategy, the weight of the model parameter is determined to be the sum of the weight determined in the previous iteration and the preset value; otherwise, based on the first preset weight allocation strategy, the weight of the model parameter is determined to be the weight determined in the previous iteration; According to the determined weights of the model parameters, a first pruning weight matrix is ​​obtained.

3. The text question answering method based on a miniaturized large language model according to claim 1, characterized in that: Determining, according to the pruning model fine-tuning result, a second output value of each pruning model parameter and a second error signal corresponding to each pruning model parameter during reverse regression, including: Determine a second average value of each layer of pruning model parameters according to the second output value of each pruning model parameter, and determine a second signal average value of each layer of pruning model parameters according to a second error signal corresponding to each pruning model parameter during reverse regression; Determine a corresponding third threshold based on a second preset forgetting parameter and a second average value of the corresponding layer pruning model parameter, and determine a corresponding fourth threshold based on a second preset forgetting parameter and a second signal average value of the corresponding layer pruning model parameter; For each iterative training, determining whether the second output value of each pruning model parameter is greater than the corresponding third threshold, and determining whether the second error signal of the corresponding layer pruning model parameter is greater than the corresponding fourth threshold; If the second output value of the pruning model parameter is greater than the third threshold, and the second error signal of the corresponding layer pruning model parameter is greater than the fourth threshold, then based on the second preset weight allocation strategy, the weight of the pruning model parameter is determined to be the sum of the weight determined in the previous iteration and the preset value; otherwise, based on the second preset weight allocation strategy, the weight of the pruning model parameter is determined to be the weight determined in the previous iteration; According to the determined weights of the pruning model parameters, a second pruning weight matrix is ​​obtained.

4. The text question answering method based on a miniaturized large language model according to claim 1, characterized in that: Acquiring the limited domain SFT data includes: Get domain-qualified text; Based on the domain-limited text and in combination with preset question generation rules, generate domain-limited questions; Predicting answers to the domain-limited questions and obtaining predicted answers; According to the domain-limited questions and the corresponding predicted answers, limited domain SFT data is obtained.

5. The text question answering method based on a miniaturized large language model according to claim 4, characterized in that: According to the domain-limited questions and the corresponding predicted answers, the limited domain SFT data is obtained, including: Obtaining domain-limited training data according to the domain-limited questions and corresponding predicted answers; The limited domain training data is screened to obtain limited domain SFT data.

6. The text question answering method based on a miniaturized large language model according to claim 5, characterized in that: The limited domain training data is screened to obtain limited domain SFT data, including: Inputting the limited domain training data into a screening model to obtain a screening label output by the screening model; wherein the screening model is trained based on the question and answer data and the screening labels corresponding to the question and answer data; The limited domain training data is filtered according to the filtering label to obtain limited domain SFT data.

7. The text question answering method based on a miniaturized large language model according to claim 4, characterized in that: Based on the domain-limited text and in combination with preset question generation rules, domain-limited questions are generated, including: Extracting entities from the domain-restricted text to obtain screening entities; A domain-limited question is generated according to the screening entity and the preset question generation rule; wherein the preset question generation rule is used to limit a question instance template or prompt word.

8. The text question answering method based on a miniaturized large language model according to claim 7, characterized in that: Entities in the domain-restricted text are extracted to obtain screening entities, including: Segmenting the domain-restricted text, and deleting stop words in each segmentation; Perform named entity recognition on each of the word segments to obtain a screening entity.

9. The text question answering method based on a miniaturized large language model according to claim 4, characterized in that: The predicted answers to the domain-limited questions are predicted, and the predicted answers include: Based on a program programming API interface, at least one external large language model is accessed, and the accessed external large language model is used to predict answers to the domain-limited questions respectively to obtain corresponding predicted answers.

10. The text question answering method based on a miniaturized large language model according to claim 1, characterized in that: The obtaining of general SFT data includes: Collect text data from a preset information source or use a search engine to collect text data; Clean and filter the collected text data; Entities are extracted from the cleaned and filtered text data, and questions are generated based on the extracted entities, and the answers to the corresponding questions are annotated to obtain general SFT data.

11. A text question-answering device based on a miniaturized large language model, characterized in that: include: Question acquisition module, obtains input text; A question-answer prediction module inputs the input text into a miniaturized large language model to obtain a predicted answer output by the miniaturized large language model; wherein the miniaturized large language model is obtained by fine-tuning the large language model using limited domain supervised fine-tuning SFT data, and pruning the fine-tuned model in combination with the model fine-tuning result, and fine-tuning the pruned model using general SFT data, and pruning the fine-tuned pruned model in combination with the pruned model fine-tuning result; The device further comprises: A data acquisition module, which acquires general SFT data and limited domain SFT data before inputting the input text into the miniaturized large language model; A first fine-tuning module uses the limited domain SFT data to train the previously selected large language model to obtain a model fine-tuning result; A first statistical module, which determines, according to the model fine-tuning result, a first output value of each model parameter and a first error signal corresponding to each model parameter during reverse regression; A first matrix determination module determines a first pruning weight matrix according to the first output value of each model parameter and the first error signal corresponding to each model parameter during reverse regression, combined with a first preset weight allocation strategy; wherein the first preset weight allocation strategy is used to limit the allocation weights of each first output value and the first error signal corresponding to each model parameter; A first pruning module, according to the first pruning weight matrix, selects model parameters corresponding to a first preset ratio in order of values ​​from small to large for pruning to obtain a pruning model; A second fine-tuning module, using the general SFT data to train the pruning model to obtain a pruning model fine-tuning result; A second statistical module determines, according to the pruning model fine-tuning result, a second output value of each pruning model parameter and a second error signal corresponding to each pruning model parameter during reverse regression; A second matrix determination module determines a second pruning weight matrix according to the second output value of each pruning model parameter and the second error signal corresponding to each pruning model parameter during reverse regression, in combination with a second preset weight allocation strategy; wherein the second preset weight allocation strategy is used to limit the allocation weights of each second output value and the second error signal corresponding to each model parameter; The second pruning module selects pruning model parameters corresponding to a second preset ratio in descending order of values ​​according to the second pruning weight matrix to perform pruning to obtain a miniaturized large language model.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the text question answering method based on the miniaturized large language model is implemented as described in any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text question answering method based on a miniaturized large language model as described in any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the text question answering method based on a miniaturized large language model as described in any one of claims 1 to 10 is implemented.

Images