Large model archive question and answer system based on two-parameter sparse regularization parameter fine-tuning

Abstract

The application is suitable for the technical field of archive management query, and particularly relates to a large model archive question and answer system based on double-parameter sparse regularization parameter fine-tuning, which comprises: a query generation module, which is used for converting the question input by a user into semantic query data through natural language processing to generate a semantic vector; a knowledge base retrieval module, which is used for screening archive content corresponding to the user query; a generation enhancement module, which is used for outputting an answer; and an optimization module, which is used for expanding the semantic coverage of a knowledge base, analyzing picture content, pre-processing an OCR document, and fine-tuning a large model. The large model archive question and answer system based on double-parameter sparse regularization parameter fine-tuning provided by the application adopts a qwen2-7b-instruct model for intelligent preprocessing in view of common problems such as random codes, information loss and structure disorder in an OCR document, and through the model, the system can automatically clean up redundant line feeds and reconstruct paragraph structures, so as to improve the logic and continuity of the text.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of archive management query, and particularly relates to a large model archive question and answer system based on double-parameter sparse regularization parameter fine-tuning. BACKGROUND

[0002] Under the impetus of the digital wave, archive management is undergoing unprecedented changes. Traditional archive management methods, which rely on manual processing and paper storage, not only have problems such as slow retrieval speed, high storage cost, and easy damage, but also are difficult to meet the demand for efficient and accurate information retrieval in modern society.

[0003] With the rapid development of artificial intelligence technology, especially the rise of large model technology represented by large language models (LLM), a new solution is provided for the digital upgrade of archive management. These models can extract valuable information from large-scale data through powerful language understanding and generation capabilities, greatly improving the intelligent level of archive management.

[0004] However, general large models still face the following challenges in specific applications: processing of unstructured text: Many archives contain mixed information of text and pictures, and the picture part may involve important charts, handwritten records or scanned copies. The processing ability of general large models for picture information is limited. Existing models cannot directly extract high-quality text or semantic information from pictures and effectively associate them with text content, resulting in the neglect of picture-related content in retrieval and question answering. SUMMARY

[0005] The purpose of the present application is to provide a large model archive question and answer system based on double-parameter sparse regularization parameter fine-tuning, which aims to solve the problem that existing models cannot directly extract high-quality text or semantic information from pictures and effectively associate them with text content, resulting in the neglect of picture-related content in retrieval and question answering.

[0006] The present application is realized in this way, a large model archive question and answer system based on double-parameter sparse regularization parameter fine-tuning, the system comprises:

[0007] A query generation module for converting user input questions into semantic query data through natural language processing, generating semantic vectors;

[0008] A knowledge base retrieval module for extracting corresponding documents from a local knowledge base, retrieving based on semantic matching, and filtering out archive content matching user queries;

[0009] A generation enhancement module for taking the retrieved documents as context information, combining context semantics with user query content, and outputting answers;

[0010] An optimization module is configured to expand semantic coverage of the knowledge base, parse picture content, pre-process OCR documents, and fine-tune a large model.

[0011] Preferably, in the process of expanding semantic coverage of the knowledge base, a domain-specific synonym library is constructed, synonym mapping relationships are generated through natural language processing tools, professional dictionaries and industry term tables, user query inputs are replaced by synonyms and expanded in semantics, and an expanded query set is generated.

[0012] Preferably, the expansion function is defined as: Expand(Q)={Q,Q (1) ,Q (2) ,…,Q (k)}

[0013] A series of semantic variants Q (k) of the original query Q are obtained.

[0014] Preferably, in the step of parsing picture content, in the picture text extraction link, a picture large model is used to extract text information and identify scene content of the archive picture, in the semantic analysis and mapping stage, the extracted text content and picture labels are further mapped to nodes and relationships in the knowledge graph to form complete structured data representation, and in the multi-modal fusion stage, the picture information and related text content in the archive are embedded into a unified semantic space.

[0015] Preferably, the knowledge graph is represented by G=(V, E), where V={v1, v2, …} represents knowledge nodes, and E={e i,j} represents the relationship between nodes.

[0016] Preferably, in the process of pre-processing the OCR document, in the model preprocessing stage, the original OCR text is intelligently repaired through the qwen2-7b-instruct model; in the redundancy deletion and structuring stage, the formatting processing of the text is optimized, and meaningless line breaks, blank lines or repeated content generated in the OCR extraction process are removed; in the knowledge base optimization stage, the index system of the knowledge base is reconstructed based on the pre-processed text content.

[0017] Preferably, in the step of fine-tuning the large model, sparse LoRA fine-tuning is adopted, and the mathematical model of sparse LoRA is:

[0018] Suppose the weight matrix of the model that needs to be fine-tuned is W∈R d×k , the fine-tuned weight is represented as: W LR =W+ΔW,ΔW=AB where A∈R d×r , B∈R r×kare real matrices of size d x r and r x k, respectively, d, r and k are integers, representing the number of rows and columns of the matrices, r << min(d, k), and AW is a low-rank fine-tuning matrix.

[0019] Preferably, a two-parameter L2,1 norm regularization is introduced to constrain the sparsity in AW, and the regularization objective function is:

[0020]

[0021] where λ A and λ B are regularization coefficients, controlling the sparsity strength, A j,j and B i,j are the i-th row j-th column elements of matrix A and matrix B, respectively.

[0022] Preferably, the task objective loss and the regularization term are integrated, and the overall optimization objective is:

[0023] L = Ltask + Lreg

[0024] where Ltask is the cross-entropy loss function of the archive question and answer task.

[0025] Preferably, the implementation steps of sparse LoRA fine-tuning include:

[0026] Initialize the low-rank matrix: randomly initialize matrices A and B, and set the sparse regularization coefficients λ A and λ B to a preset value;

[0027] Sparse matrix update: through iterative optimization, gradually sparsify the preset parameters of the matrix, and each round of iteration includes the following steps:

[0028] Calculate the gradient: according to the task loss function L, calculate the gradients of the elements A i,j and B i,j in matrices A and B and

[0029] Element update of matrices A and B:

[0030]

[0031] where η is the learning rate;

[0032] Sparse matrix truncation: apply the sparsification rule to the updated matrix AW, and set the elements smaller than the threshold ∈ to zero: A i,j = 0 if |A i,j | < ∈; B i,j = 0 if |B i,j | < ∈;

[0033] Normalization operation: ensure that the row norm of the matrix AW remains within a preset range;

[0034] Convergence detection: repeated until the loss function L converges.

[0035] The large model archive question and answer system based on double-parameter sparse regularization parameter fine-tuning provided by the present application adopts the qwen2-7b-instruct model for intelligent preprocessing in view of the common problems of garbled code, missing information and structure confusion in OCR documents. Through the model, the system can automatically clean up redundant line feeds and reconstruct the paragraph structure, thereby improving the logicality and continuity of the text. At the same time, the model can also identify and repair the garbled code problem in the text and automatically fill in the missing content. This preprocessing process significantly improves the quality of unstructured documents in the knowledge base, making the subsequent retrieval and generation more accurate. In addition, the method also optimizes the segmentation and index construction of the document, ensuring the integrity of the retrieval content and overcoming the problems of manual intervention and low efficiency in the traditional OCR preprocessing process. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The working flow chart of the large model archive question and answer system based on double-parameter sparse regularization parameter fine-tuning provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical scheme and advantages of the present application clearer and more understandable, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0038] As shown in the figure, the working flow chart of the large model archive question and answer system based on double-parameter sparse regularization parameter fine-tuning provided by the present application is shown in the figure. The system includes: Figure 1 A query generation module for converting user input questions into semantic query data through natural language processing and generating semantic vectors.

[0039] In the present system, the query generation module is responsible for converting user input questions into semantic queries. Through natural language processing technology, the system can accurately analyze the user's query intention and generate high-quality semantic vectors. These vectors provide support for subsequent knowledge base retrieval, ensuring the relevance and accuracy of the retrieval.

[0040] A knowledge base retrieval module for extracting corresponding documents from the local knowledge base based on semantic matching and filtering out the archive content matching the user query.

[0041]

[0042] ​In the system, the knowledge base retrieval module is used to extract high-relevance documents from the local knowledge base. Based on the semantic matching retrieval method, the system can quickly filter the archive content closely related to the user query, optimizing the accuracy and coverage of the results.

[0043] The generation enhancement module is used to combine the retrieved documents as context information with the user's query content and output answers based on context semantics.

[0044] In the system, the generation enhancement module uses the retrieved relevant documents as context information to further improve the accuracy and coherence of the large model in generating answers. By combining the context semantics with the user's query content, the system can output answers that better meet the actual needs.

[0045] The optimization module is used to expand the semantic coverage of the knowledge base, analyze picture content, preprocess OCR documents, and fine-tune the large model.

[0046] In the system, the optimization module further improves the system performance through various technical means, including: through synonym replacement, expanding the semantic coverage of the knowledge base, increasing the flexibility of query matching; through the picture large model to analyze the picture content in the archive and convert it into usable text information and knowledge graph; intelligent preprocessing of OCR documents in the archive, repairing and optimizing unstructured text, improving the integrity and quality of the knowledge base; through the pre-training large model fine-tuned by sparse LoRA, effectively reducing the computational cost and storage requirements of the model.

[0047] In the embodiment of the present application, synonym library construction and synonym replacement optimization: to improve the semantic coverage of the knowledge base, the present application constructs a domain-specific synonym library to optimize the expansion ability of the query. First, using natural language processing tools such as NLTK and spaCy, combined with professional dictionaries and industry term tables, a set of synonym mapping relationships containing domain-specific vocabulary is generated. This synonym library provides basic support for subsequent semantic expansion. In specific applications, the user's query input will be processed through synonym replacement and semantic expansion to generate an expanded query set, for example, by defining an expansion function

[0048] Expand(Q)={Q,Q (1) ,Q (2) ,…,Q (k)},

[0049] A series of semantic variants Q (k)These variants achieve multi-dimensional semantic expression of the query through synonym replacement. To ensure the effectiveness and relevance of the expanded results, further rounds of optimization are conducted on the expanded query in combination with the domain corpus, and a semantic similarity filtering method is used to remove invalid expanded variants, thereby ensuring the accuracy and applicability of the final query results. This method significantly improves the semantic coverage of the knowledge base and provides users with more comprehensive and accurate query services.

[0050] In the embodiment of the application, the working process of the system includes picture content processing and knowledge graph construction: for picture data in the archives, a picture content processing and knowledge graph generation process is added. For picture data in the archives, a systematic picture content processing and knowledge graph generation process is constructed to realize effective structuring and semanticization of picture information. First, in the picture text extraction link, an advanced picture large model (such as BLIP) is used to extract text information and identify scene content of the archive picture. For example, for an organizational chart, the model can extract the names of departments at all levels, position information, and hierarchical relationships, and output these information in text form. This process not only extracts explicit text information in the picture, but also captures implicit semantics in the image in combination with scene recognition technology, such as the relationship between an icon and its associated relationship.

[0051] Then, in the semantic analysis and mapping stage, the extracted text content and picture labels are further mapped to nodes and relationships in the knowledge graph to form a complete structured data representation. The knowledge graph can be represented as G=(V,E), where V={v1,v2,…} represents knowledge nodes, and E={e i,j} represents the relationship between nodes. For example, for the information extracted from the organizational chart, “Sales Department” and “Technical Department” can be mapped to nodes v1 and v2, and “Sales Department reports to Technical Department” can be represented as the relationship e 1,2 between nodes. This mapping can convert picture information into structured data that can be understood by a computer, laying the foundation for subsequent analysis and processing.

[0052] Finally, in the multi-modal fusion stage, the picture information and related text content in the archives are embedded into a unified semantic space, and the accuracy of retrieval and question answering is further improved through the unification of semantic representation. For example, combining the information extracted from the organizational chart with the description in the archive text can answer questions such as “What is the direct superior department of the Sales Department” or “Who is the head of the Technical Department”. Multi-modal fusion makes full use of the complementarity of picture and text information sources, providing higher accuracy and reliability for intelligent processing and knowledge services of archives.

[0053] In one embodiment of the invention, to address OCR document quality issues, an intelligent preprocessing process is added to effectively address text structure disorder, garbled characters, and missing content, ultimately generating formatted documents and optimizing knowledge base content.

[0054] First, in the model preprocessing stage, advanced qwen2-7b-instruct models are used to intelligently repair the original OCR text. This model can identify and repair garbled characters and missing content caused by OCR errors. For example, the original OCR document may extract such content: "Th1s is a t3st d0cum3nt w1th err0rs", after model repair, it can be restored to the correct text: "This is a

[0055] test document with errors". In addition, for text misplacement or logically incoherent paragraphs caused by typesetting errors, the model can also make reasonable adjustments through context reasoning, so that the text restores normal logical order and semantic integrity.

[0056] Next, in the redundancy deletion and structuring stage, the text formatting process is further optimized. A large number of meaningless line breaks, blank lines or repeated content generated in the OCR extraction process are removed. For example, a paragraph of text may be incorrectly divided into multiple lines: "This is\na text\nwith broken lines.", after preprocessing it will be reconstructed as: "This is a text with broken lines." At the same time, for lists, tables or hierarchical structures in complex documents, regularized paragraph division and typesetting are adopted to make the document more readable and logical.

[0057] Finally, in the knowledge base optimization stage, the index system of the knowledge base is reconstructed based on the preprocessed text content, thereby improving the accuracy of document retrieval. For example, a conference record document after OCR extraction due to garbled characters and paragraph disorder, the key information such as conference theme, participant list and key decisions cannot be effectively indexed. After preprocessing, these information will be accurately extracted and structured stored in the knowledge base, so that users can quickly retrieve the target content through keyword or natural language query.

[0058] Through this preprocessing process, not only the text quality of OCR documents is significantly improved, but also the foundation for subsequent knowledge base application is laid, providing an efficient solution for intelligent document processing.

[0059] In a preferred embodiment of the present application, in order to further improve the answer quality of the archive large model, the present application proposes a sparse LoRA fine-tuning technology on the basis of the above processing. This technology is a lightweight fine-tuning method for large models, aiming to reduce the computational cost and storage demand of model fine-tuning, while improving the adaptability to specific tasks. Specifically, the present application introduces double-parameter L2,1 norm regularization on the basis of the traditional LoRA method, which is used to generate sparse low-rank matrices, thereby improving the efficiency and sparsity of fine-tuning.

[0060] In fine-tuning large language models (such as GPT), the traditional method usually updates the large-scale full model parameters, which leads to the following problems: high computational overhead: the model parameter scale is large, and full model fine-tuning requires huge computational resources and storage space; redundant update: part of the weights have little effect on specific tasks, but are still updated, causing resource waste; insufficient generalization ability: dense weight update may cause overfitting of the model on specific tasks. Therefore, the present application combines LoRA technology to limit the sparsity of the update matrix, making the fine-tuning process more efficient while maintaining the performance and generalization ability of the model.

[0061] The mathematical model of sparse LoRA is as follows:

[0062] Assume that the weight matrix of the model is W ∈ R d×k In the LoRA method, the fine-tuned weight is represented as:

[0063] W LR = W + ΔW, ΔW = AB

[0064] Where A ∈ R d×r , B ∈ R r×k are real matrices of d × r and r × k respectively, d, r and k are integers representing the number of rows and columns of the matrix, r << min(d, k). ΔW is a low-rank fine-tuning matrix that greatly reduces the number of updated parameters.

[0065] To reduce unnecessary updates, the present application introduces double-parameter L2,1 norm regularization to constrain the sparsity in ΔW. The regularization objective function is:

[0066]

[0067] Where λ A and λ B are regularization coefficients that control the sparsity strength, A i,j and B i,j are the i-th row j-th column elements of matrix A and matrix B respectively. This regularization term sparsifies the L2 norm of each row, ensuring that only non-zero values are retained in the key feature dimensions.

[0068] The overall optimization objective is a combination of the task objective loss and the regularization term:

[0069] L = Ltask + Lreg

[0070] where Ltask is the cross-entropy loss function for the archive question and answer task, which ensures the accuracy of the model's answers.

[0071] The implementation steps of sparse LoRA fine-tuning are as follows:

[0072] (1) Initialize low-rank matrices

[0073] Randomly initialize matrices A and B, and set the sparse regularization coefficients λ A and λ B to a small value to balance model performance and sparsity;

[0074] (2) Sparse matrix update

[0075] Through iterative optimization, gradually sparsify the non-important parameters of the matrix. Each iteration includes the following steps:

[0076] Calculate the gradient: calculate the gradient according to the task loss function

[0077] Low-rank matrix update:

[0078]

[0079] where η is the learning rate.

[0080] Sparse matrix truncation: apply the sparsification rule to the updated matrix ΔW, and set the elements less than the threshold ∈ to zero: A i,j = 0 if |A i,j | < ∈; B i,j = 0 if |B i,j | < ∈.

[0081] Normalization operation: ensure that the row norm of the matrix ΔW remains within a certain range to avoid gradient explosion or disappearance problems.

[0082] (3) Convergence detection: repeat the above steps until the loss function L converges, i.e., the change in the loss value is less than the pre-set threshold.

[0083] Through sparse LoRA fine-tuning, the following performance improvements are achieved: 1. Computational efficiency is improved: compared to full model fine-tuning, sparse LoRA only updates about 1% of the parameters, significantly improving training speed; 2. Memory usage is reduced: due to the sparsification of parameter matrices, storage requirements are reduced, allowing the model to run on devices with limited resources. 3. Performance is not lost: in the knowledge base retrieval and enhanced generation task, the question and answer performance of the sparse LoRA fine-tuned model is comparable to that of the full parameter fine-tuned model, but the efficiency is higher.

[0084] The present application realizes comprehensive coverage of multi-modal archives by combining synonym replacement expansion, picture information extraction and knowledge graph construction. By using synonym replacement technology, the semantic coverage of the knowledge base is significantly expanded, thereby improving the diversity and accuracy of the search results. By using a picture large model (such as BLIP) to extract key information in the picture, such as charts, handwritten records, etc., and converting it into text embedding, the text content is uniformly processed. Further, by constructing a knowledge graph, the semantic correlation of the archive information is made explicit, so that the system can accurately understand the context relationship in the archive content. This multi-modal information fusion method overcomes the problem of insufficient coverage of single-modal data in traditional methods, making the coverage of the search more extensive and the results more accurate.

[0085] In view of the common problems of garbled code, information missing and structure confusion in OCR documents, the present application uses the qwen2-7b-instruct model for intelligent preprocessing. Through this model, the system can automatically clean up redundant line breaks and reconstruct the paragraph structure, thereby improving the logicality and continuity of the text. At the same time, the model can also identify and repair garbled code problems and automatically fill in missing content. This preprocessing process significantly improves the quality of unstructured documents in the knowledge base, making the subsequent search and generation more accurate. In addition, this method also optimizes the segmentation and index construction of the document, ensuring the integrity of the search content and overcoming the problem of manual intervention and low efficiency in traditional OCR preprocessing.

[0086] Based on the LoRA (Low-Rank Adaptation) technology, the present application introduces double-parameter L2,1-norm regularization, further optimizing the fine-tuning process. Traditional full-parameter fine-tuning method has high computational cost and is prone to overfitting, while the present application sparsifies the low-rank matrix, so that only a small amount of parameters need to be updated to realize the localization adaptation of the large model. Specifically, by regularizing the weight matrix with L2,1 norm, the sparsity of the parameters can be effectively controlled, important features can be preserved and redundant features can be suppressed, thereby reducing the storage demand and computational complexity of the model. For example, for high-frequency domain-specific terms, the sparse LoRA method can assign a lower regularization strength to ensure that these key features are preserved, while for noise data, a higher regularization strength can be assigned to significantly weaken its impact. This method not only greatly reduces the parameter update amount, but also has comparable performance to the full-parameter fine-tuning method in the question and answer generation task, overcoming the problems of high cost and low efficiency in traditional fine-tuning.

[0087] The above embodiments only express several implementation manners of the present application, which are described in a more specific and detailed manner, but should not be understood as a limitation on the patent scope of the present application. It should be noted that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

[0088] The above merely describes the preferred embodiments of the present application and should not be used to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A large model archive question and answer system based on two-parameter sparse regularization parameter fine-tuning, characterized by, The system comprises: a query generation module for converting a user input question into semantic query data through natural language processing to generate a semantic vector; a knowledge base retrieval module for extracting corresponding documents from a local knowledge base, performing retrieval based on semantic matching, and screening out content corresponding to the user query profile; a generation enhancement module for taking the retrieved documents as context information, combining the context semantics with the content corresponding to the user query profile, and outputting an answer; an optimization module for expanding the semantic coverage of the knowledge base, analyzing picture content, preprocessing OCR documents, and fine-tuning a large model; In the process of preprocessing the OCR document, in the model preprocessing stage, the original OCR text is intelligently repaired through the qwen2-7b-instruct model; in the redundancy deletion and structuring stage, the formatting processing of the text is optimized, and meaningless line breaks, blank lines or repeated content generated in the OCR extraction process are removed; in the knowledge base optimization stage, the index system of the knowledge base is reconstructed based on the preprocessed text content; In the step of fine-tuning the large model, sparse LoRA fine-tuning is adopted, and the mathematical model of sparse LoRA is: Suppose we need to adjust the model weight matrix as follows: Fine-tuning is performed, and the adjusted weights are expressed as follows: ， wherein, is a low-rank real matrix, d, r and k are integers, representing the number of rows and columns of the matrix, is a low-rank fine-tuning matrix; The double-parameter L2,1 norm regularization is introduced to constrain the sparsity of The regularization objective function is defined as: wherein and are regularization coefficients for controlling the sparsity strength, and are the i-th row j-th column elements of matrix A and matrix B, respectively.

2. The large model archive question and answer system based on two-parameter sparse regularization parameter fine-tuning according to claim 1, characterized in that, In the process of expanding the semantic coverage of the knowledge base, a synonym library of a domain is constructed, the synonym mapping relationship is generated through natural language processing tools, professional dictionaries and industry term tables, the user's query input is replaced by synonyms and expanded in semantics to generate an expanded query set.

3. The large model archive question and answer system based on two-parameter sparse regularization parameter fine-tuning according to claim 2, characterized in that, Define the expansion function: obtaining a set of semantic variants of the original query Q .

4. The large model archive question and answer system based on two-parameter sparse regularization parameter fine-tuning according to claim 1, characterized in that, In the step of analyzing picture content, in the picture text extraction link, the picture large model is used to extract text information and identify scene content from the archive picture, in the semantic analysis and mapping stage, the extracted text content and picture labels are further mapped to the nodes and relationships in the knowledge graph to form complete structured data representation, and in the multi-modal fusion stage, the picture information and related text content in the archive are embedded into a unified semantic space.

5. The large model archive question and answer system based on two-parameter sparse regularization parameter fine-tuning according to claim 3, characterized in that, Knowledge graph uses to represent, wherein represents a knowledge node, represents the relationship between nodes.

6. The large model archive question and answer system based on two-parameter sparse regularization parameter fine-tuning according to claim 1, characterized in that, The overall optimization target is: wherein: is the cross-entropy loss function for the archive question answering task.

7. The large model archive question and answer system based on two-parameter sparse regularization parameter fine-tuning according to claim 6, characterized in that, The implementation steps of sparse LoRA fine-tuning include: Initialize low-rank matrices: randomly initialize matrices A and B, and set the regularization coefficients and to a preset value; Sparse matrix update: through iterative optimization, the preset parameters of the sparse matrix are gradually sparsified, and each iteration includes the following steps: Compute gradients: Compute gradients of elements in matrices A and B with respect to overall loss function L and and ;​ Element update of matrices A and B: wherein, is a learning rate; Sparse matrix truncation: set elements of the updated matrix to zero if they are smaller than a threshold value set to zero: ; Normalization operation: Ensures that the row norms of the matrix stay within a pre-set range; Convergence detection: repeat until the loss function L converges.

Images