Text analysis method based on LoRA algorithm

Abstract

The invention discloses a text analysis method based on a LoRA algorithm, and relates to the technical field of text analysis, and the method comprises the following steps: 1, obtaining annual report text data; step 2, constructing an annual report preprocessing model, converting unstructured data in annual report text data into structured information, and performing fine tuning based on an LoRA algorithm; step 3, constructing a classification and scoring model, performing digital transformation on the structured information in the converted annual report text data, and classifying and scoring a digital transformation result; and 4, constructing a doctor diagnosis model, and carrying out comprehensive evaluation and analysis on the digital transformation of the annual report text data. According to the text analysis method based on the LoRA algorithm, an enterprise has more comprehensive and systematic analysis on decision support, and the enterprise is assisted to make accurate strategies and decisions.

Description

Technical Field

[0001] The present invention relates to the technical field of text analysis, and in particular to a text analysis method based on a LoRA algorithm. Background Art

[0002] In today's information society, corporate annual reports, as detailed records of the company's annual operating conditions, carry a lot of important information. These annual report text data are usually unstructured, which means that they exist in the form of natural language and contain a large amount of text, tables, graphics and other data. The existence of these unstructured data brings great challenges to the extraction and analysis of information. In the current field of text analysis technology, how to effectively extract valuable information from documents such as annual reports has become an important research direction. Traditional information extraction technology usually relies on natural language processing (NLP) technology to analyze text by building rule-based methods or using machine learning models. These traditional methods may perform well when processing data in specific fields, but their effects are often unsatisfactory when faced with a large amount of unstructured text. In particular, as the content of corporate annual reports becomes increasingly complex, the retraining and adaptability of traditional methods are insufficient, resulting in low information extraction efficiency. With the rapid development of deep learning, many researchers have begun to try to use deep learning models for text analysis. Among them, pre-trained language models (such as BERT, GPT, etc.) have been proven to have excellent performance in tasks such as text classification and sentiment analysis. However, these pre-trained models usually require a lot of computing resources and training data, which is costly for small and medium-sized enterprises to implement. In addition, fine-tuning these models on specific tasks also requires high professional knowledge and skills, which limits their promotion and application. In summary, existing methods often have the following shortcomings: first, the text data preprocessing process is complex and time-consuming, and it is difficult to ensure the integrity of the information; second, the existing classification and scoring models have limited ability to interpret annual report information, and it is difficult to fully reflect the actual situation of the enterprise; third, the lack of a comprehensive evaluation mechanism leads to insufficient accuracy and reliability of the final analysis results. In order to solve the above problems, the LoRA (Low-Rank Adaptation) algorithm, as a new model fine-tuning method, has begun to attract attention from academia and industry. The LoRA algorithm significantly reduces the computational cost and data requirements of fine-tuning by applying low-rank matrix decomposition to certain weights in the pre-trained model and only adjusting some parameters. This method not only improves the performance of the model on specific tasks, but also greatly reduces the required computing resources, and is suitable for enterprises of all sizes. Summary of the invention

[0003] The purpose of the present invention is to provide a text analysis method based on the LoRA algorithm to solve the problems existing in the prior art that the text data preprocessing process is complicated and time-consuming, it is difficult to ensure the integrity of information, and the ability to interpret annual report information is limited, it is difficult to fully reflect the actual situation of the enterprise, and there is a lack of a comprehensive evaluation mechanism, resulting in insufficient accuracy and reliability of the final analysis results.

[0004] To achieve the above object, the present invention provides a text analysis method based on the LoRA algorithm, comprising the following steps:

[0005] Step 1: Obtain annual report text data;

[0006] Step 2: Build an annual report preprocessing model to convert the unstructured data in the annual report text data into structured information and perform fine-tuning based on the LoRA algorithm;

[0007] Step 3: Build a classification and scoring model to digitally transform the structured information in the converted annual report text data, and classify and score the results of the digital transformation;

[0008] Step 4: Build a doctor diagnosis model to conduct a comprehensive assessment and analysis of the digital transformation of annual report text data.

[0009] Preferably, the annual report text data in step 1 is obtained from the official website of the enterprise or the securities market, including company introduction, financial statements, and management discussion and analysis.

[0010] Preferably, the process of constructing the annual report preprocessing model in step 2 is as follows:

[0011] S21. Clean the acquired annual report text data to remove non-text information including special characters;

[0012] S22, dividing the annual report text data cleaned by S21 into blocks, removing stop words, and performing structural processing;

[0013] S23, researchers check and correct the data processed by S22;

[0014] S24. According to S23, fine-tune the annual report preprocessing model based on the LoRA algorithm, and further extract features from the annual report text data.

[0015] Preferably, the process of S22 is as follows:

[0016] S221. Based on the structure of the annual report, divide the text into paragraphs, chapters or specific formats, and use the text processing tool Python to divide the text into blocks;

[0017] S222. Design the corresponding data model in combination with the required structured format (such as CSV, JSON, database table, etc.), clarify the field names and data types, such as "company name", "transaction amount", "date", etc., map the text obtained by segmentation in S221 according to the preset data model, and convert the unstructured information into a structured format. For example, fill the identified attributes such as company name, amount and date into the corresponding fields to form a structured record.

[0018] Preferably, the process of S24 is as follows:

[0019] S241. Annotate the content in the annual report text to form an annotated data set, and divide the annotated data set into a training set and a validation set;

[0020] S242. Add a LoRA adapter based on the BERT pre-trained model, and simulate the change of the original weight parameters through low-rank matrix decomposition. Specifically, introduce a bypass structure in the original network. The bypass structure consists of a low-rank matrix A and a low-rank matrix B. The increment of the product of the low-rank matrix A and the low-rank matrix B is used as a reference for updating the original weight W. While updating the low-rank matrix A and the low-rank matrix B during the training process, the increment of their product is also used to dynamically update the original training weight W. The updating process is as follows:

[0021]

[0022] Among them, W t is the original weight at the tth time step, A t ,B t are two low-rank matrices at the tth time step, λ is the update ratio of the weight matrix W, h represents the height of the convolution kernel, and if the gradient descent method is used to update W, then W t+1 =W t -ηg(W), where η is the learning rate, g(W) is the gradient of the matrix, and the weight increment of the pre-trained matrix is ​​ΔW = -ηg(W);

[0023] S243. Update the LoRA adapter through the back propagation algorithm, fine-tune the annual report preprocessing model, and further extract features from the annual report text data.

[0024] Preferably, the process of constructing the classification and scoring model in step 3 is as follows:

[0025] S31. Based on ChatGLM 3, the pre-processed annual report text is preliminarily classified and the corresponding content is mapped to different technical fields;

[0026] S32, preliminarily scoring the classified text fragments, and dividing the scores into five levels according to the description depth and accuracy of the corresponding categories;

[0027] S33, researchers review and provide feedback on the initial classification and scoring;

[0028] S34. According to the feedback from S33, the classification and scoring models are trained and fine-tuned based on the LoRA algorithm.

[0029] Preferably, the technical fields include artificial intelligence technology, big data technology, cloud computing technology and blockchain technology.

[0030] Preferably, in S32, the classified text segments are preliminarily scored based on BLEU (Bilingual Evaluation Understudy, a bilingual evaluation auxiliary tool), and the calculation expression is as follows:

[0031]

[0032] Where P n Indicates the precision of n-gram, W n represents the weight of n-gram, BP represents the penalty factor, lc represents the length of machine translation, lr represents the length of the shortest classified text segment, C represents the machine translation, i represents the n-gram in C, h i (C) represents the number of times n_gram i appears in C, h i (S j ) represents the number of times n_gram i appears in the jth reference translation S.

[0033] Preferably, the five rating levels are:

[0034] First level: 1 point: extremely poor, irrelevant or with major errors;

[0035] Second level: 2 points: poor, low relevance, incomplete content;

[0036] Level 3: 3 points: Moderate, moderately relevant, and partially complete information;

[0037] Level 4: 4 points: Good, complete and relevant;

[0038] Level 5: 5 points: Excellent, the information is complete and highly relevant.

[0039] Preferably, the process of constructing the doctor diagnosis model in step 4 is as follows:

[0040] S41, regularly updating the database of the doctor diagnosis model according to the annual report preprocessing model constructed in step 2 and the classification and scoring model constructed in step 3;

[0041] S42. Input the company's annual report into the doctor's diagnosis model after updating the database to identify the company's strengths and weaknesses;

[0042] S43. Generate an evaluation and diagnosis report, including the scores of the enterprise in various technical fields, and put forward improvement suggestions and advantage development strategies.

[0043] Therefore, the present invention adopts the above-mentioned text analysis method based on the LoRA algorithm, which has the following beneficial effects:

[0044] (1) Improve analysis efficiency: Through automated information extraction and processing, the efficiency of annual report analysis can be significantly improved, reducing the time and cost of manual intervention;

[0045] (2) Optimize data processing: Use the fine-tuning technology of the LoRA algorithm to improve its adaptability to specific industries and companies, ensuring that the extracted information is more accurate and in line with industry characteristics;

[0046] (3) Enhance data analysis capabilities: Establish classification and scoring models to provide more comprehensive and systematic analysis in decision support, helping enterprises to formulate accurate strategies and make decisions;

[0047] (4) Reduce complexity: complex financial information is simplified into a digital format that is easy to understand and apply through structured processing and model analysis, which enhances decision makers' ability to understand financial indicators;

[0048] (5) Strong adaptability: This method is not only applicable to the analysis of specific corporate annual reports, but can also be extended to data analysis in different fields and industries, and has good versatility and scalability.

[0049] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is an overall flow chart of a text analysis method based on the LoRA algorithm of the present invention. DETAILED DESCRIPTION

[0051] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] See also Figure 1 , a text analysis method based on LoRA algorithm, comprising the following steps:

[0053] Step 1: Obtain annual report text data; wherein the annual report text data is obtained from the official website of the enterprise or the securities market, including company introduction, financial statements, and management discussion and analysis.

[0054] Step 2: Build an annual report preprocessing model to convert the unstructured data in the annual report text data into structured information and fine-tune it based on the LoRA algorithm; this model is designed to process the unstructured data in the company's annual report and convert it into structured information that can be analyzed by subsequent models. The original large language model already has basic text processing capabilities, but in order to adapt to the specific annual report content and format, customized fine-tuning is required. The process of building the annual report preprocessing model is as follows:

[0055] S21. Clean the acquired annual report text data to remove non-text information such as special characters. Specifically, use the model to identify and remove various interference items in the annual report text, such as page numbers, advertisements, legal statements, irrelevant chart descriptions, etc. This step is crucial because if these non-critical information are not removed, it may affect the accuracy and efficiency of subsequent models;

[0056] S22: After the annual report text data has been cleaned in S21, it is divided into blocks, stop words are removed, and structured. In this process, the model divides the annual report text into logically coherent paragraphs and parts, such as distinguishing the company overview, financial statements, management discussion and analysis, technology development and future forecasts. In addition, the model converts unstructured text into structured data by identifying key entities in the text, such as dates, amounts, technical terms, etc., laying the foundation for subsequent in-depth analysis. The specific process is as follows:

[0057] S221. Based on the structure of the annual report, divide the text into paragraphs, chapters or specific formats, and use the text processing tool Python to divide the text into blocks;

[0058] S222. Design the corresponding data model in combination with the required structured format (such as CSV, JSON, database table, etc.), clarify the field names and data types, such as "company name", "transaction amount", "date", etc., map the text obtained by segmentation in S221 according to the preset data model, and convert the unstructured information into a structured format. For example, fill the identified attributes such as company name, amount and date into the corresponding fields to form a structured record.

[0059] S23. Researchers check and correct the data processed by S22; although large language models perform well in text processing, human intervention may still be required in specific application scenarios to ensure the highest data quality. Researchers will check the text after the model is processed to ensure that all key information is correctly identified and retained, and correct the content that the model may have ignored or processed incorrectly. This feedback will be directly used for retraining and fine-tuning of the model to improve its automation capabilities and accuracy when processing similar texts in the future.

[0060] S24: Fine-tune the annual report preprocessing model based on the LoRA algorithm according to S23, and further extract features from the annual report text data. The specific process is as follows:

[0061] S241. Annotate the content in the annual report text to form an annotated data set, and divide the annotated data set into a training set and a validation set;

[0062] S242. Add a LoRA adapter based on the BERT pre-trained model, and simulate the change of the original weight parameters through low-rank matrix decomposition. Specifically, introduce a bypass structure in the original network. The bypass structure consists of a low-rank matrix A and a low-rank matrix B. The increment of the product of the low-rank matrix A and the low-rank matrix B is used as a reference for updating the original weight W. While updating the low-rank matrix A and the low-rank matrix B during the training process, the increment of their product is also used to dynamically update the original training weight W. The updating process is as follows:

[0063]

[0064] Among them, W t is the original weight at the tth time step, A t ,B t are two low-rank matrices at the tth time step, λ is the update ratio of the weight matrix W, h represents the height of the convolution kernel, and if the gradient descent method is used to update W, then W t+1 =W t -ηg(W), where η is the learning rate, g(W) is the gradient of the matrix, and the weight increment of the pre-trained matrix is ​​ΔW = -ηg(W);

[0065] S243. Update the LoRA adapter through the back-propagation algorithm, fine-tune the annual report pre-processing model, and further extract features from the annual report text data. The goal of fine-tuning is to reduce the intervention of researchers, improve the model's automated processing capabilities, and ensure that the output of each processing is highly consistent and accurate.

[0066] Step 3: Build a classification and scoring model to digitally transform the structured information in the converted annual report text data, and classify and score the results of the digital transformation; the process of building the classification and scoring model is as follows:

[0067] S31. Based on ChatGLM 3, the pre-processed annual report text is preliminarily classified, and the corresponding content is mapped to different technical fields; among them, the technical fields include artificial intelligence technology, big data technology, cloud computing technology and blockchain technology.

[0068] S32. Perform a preliminary rating on the classified text fragments, and divide the rating into five levels according to the description depth and accuracy of the corresponding categories; the five rating levels are specifically:

[0069] First level: 1 point: extremely poor, irrelevant or with major errors;

[0070] Second level: 2 points: poor, low relevance, incomplete content;

[0071] Level 3: 3 points: Moderate, moderately relevant, and partially complete information;

[0072] Level 4: 4 points: Good, complete and relevant;

[0073] Level 5: 5 points: Excellent, the information is complete and highly relevant.

[0074] Among them, the classified text fragments are preliminarily scored based on BLEU (Bilingual Evaluation Understudy, a bilingual evaluation auxiliary tool), and the calculation expression is as follows:

[0075]

[0076]

[0077] Where P n Indicates the precision of n-gram, W n represents the weight of n-gram, BP represents the penalty factor, lc represents the length of machine translation, lr represents the length of the shortest classified text segment, C represents the machine translation, i represents the n-gram in C, h i (C) represents the number of times n_gram i appears in C, h i (S j ) represents the number of times n_gram i appears in the jth reference translation S.

[0078] S33. Researchers review and provide feedback on the preliminary classification and scoring. In this stage, researchers not only check the accuracy of the classification, but also evaluate whether the score of each sentence is appropriate to ensure that the score reflects the true contribution of the sentence to the technical category. Researchers will adjust the classification and scoring according to actual conditions, and these adjusted data will be fed back to the model for subsequent training and fine-tuning.

[0079] S34. According to the feedback from S33, the classification and scoring models are trained and fine-tuned based on the LoRA algorithm.

[0080] Step 4: Build a doctor diagnosis model to comprehensively evaluate and analyze the digital transformation of annual report text data. The process of building a doctor diagnosis model is as follows:

[0081] S41. Regularly update the database of the doctor diagnosis model based on the annual report preprocessing model constructed in step 2 and the classification and scoring model constructed in step 3 to reflect the average performance and emerging trends in the industry;

[0082] S42. Input the company's annual report into the doctor's diagnosis model after updating the database to identify the company's strengths and weaknesses. The diagnosis model evaluates the newly input company's annual report and compares its scores in various technical fields with the industry average in the database. The model will identify the company's strengths and weaknesses compared to the industry standard. This comparison can be accurate to the application and maturity of specific technologies.

[0083] S43. Generate evaluation and diagnosis reports, including the scores of enterprises in various technical fields, and put forward improvement suggestions and advantage development strategies. These reports are crucial for guiding the adjustment of enterprises' digital strategies.

[0084] Therefore, the present invention adopts the above-mentioned text analysis method based on the LoRA algorithm. By constructing an annual report preprocessing model, a classification and scoring model, a doctor diagnosis model and fine-tuning technology based on the LoRA algorithm, the enterprise can have a more comprehensive and systematic analysis in decision support, helping the enterprise to formulate accurate strategies and decisions.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A text analysis method based on the LoRA algorithm, characterized in that: The following steps are involved: Step 1: Obtain annual report text data; Step 2: Build an annual report preprocessing model to convert the unstructured data in the annual report text data into structured information and perform fine-tuning based on the LoRA algorithm; Step 3: Build a classification and scoring model to digitally transform the structured information in the converted annual report text data, and classify and score the results of the digital transformation; Step 4: Build a doctor diagnosis model to conduct a comprehensive assessment and analysis of the digital transformation of annual report text data.

2. A text analysis method based on the LoRA algorithm according to claim 1, characterized in that: The annual report text data in step 1 is obtained from the official website of the enterprise or the securities market, including company introduction, financial statements, and management discussion and analysis.

3. A text analysis method based on LoRA algorithm according to claim 2, characterized in that: The process of building the annual report preprocessing model in step 2 is as follows: S21. Clean the acquired annual report text data to remove non-text information including special characters; S22, dividing the annual report text data cleaned by S21 into blocks, removing stop words, and performing structural processing; S23, researchers check and correct the data processed by S22; S24. According to S23, fine-tune the annual report preprocessing model based on the LoRA algorithm, and further extract features from the annual report text data.

4. A text analysis method based on LoRA algorithm according to claim 3, characterized in that: The process of S22 is as follows: S221. Based on the structure of the annual report, divide the text into paragraphs, chapters or specific formats, and use the text processing tool Python to divide the text into blocks; S222. Design a corresponding data model in combination with the required structured format, map the text obtained through segmentation in S221 according to a preset data model, and convert the unstructured information into a structured format.

5. A text analysis method based on LoRA algorithm according to claim 4, characterized in that: The process of S24 is as follows: S241. Annotate the content in the annual report text to form an annotated data set, and divide the annotated data set into a training set and a validation set; S242. Add a LoRA adapter based on the BERT pre-trained model, and simulate the change of the original weight parameters through low-rank matrix decomposition. Specifically, introduce a bypass structure in the original network. The bypass structure consists of a low-rank matrix A and a low-rank matrix B. The increment of the product of the low-rank matrix A and the low-rank matrix B is used as a reference for updating the original weight W. While updating the low-rank matrix A and the low-rank matrix B during the training process, the increment of their product is also used to dynamically update the original training weight W. The updating process is as follows: Among them, W t is the original weight at the tth time step, A t ,B t are two low-rank matrices at the tth time step, λ is the update ratio of the weight matrix W, h represents the height of the convolution kernel, and if the gradient descent method is used to update W, then W t+1 =W t -ηg(W), where η is the learning rate, g(W) is the gradient of the matrix, and the weight increment of the pre-trained matrix is ​​ΔW = -ηg(W); S243. Update the LoRA adapter through the back propagation algorithm, fine-tune the annual report preprocessing model, and further extract features from the annual report text data.

6. A text analysis method based on LoRA algorithm according to claim 5, characterized in that: The process of building the classification and scoring model in step 3 is as follows: S31. Based on ChatGLM 3, the pre-processed annual report text is preliminarily classified and the corresponding content is mapped to different technical fields; S32, preliminarily scoring the classified text fragments, and dividing the scores into five levels according to the description depth and accuracy of the corresponding categories; S33, researchers review and provide feedback on the initial classification and scoring; S34. According to the feedback from S33, the classification and scoring models are trained and fine-tuned based on the LoRA algorithm.

7. A text analysis method based on LoRA algorithm according to claim 6, characterized in that: Technical fields include artificial intelligence technology, big data technology, cloud computing technology and blockchain technology.

8. A text analysis method based on LoRA algorithm according to claim 7, characterized in that: In S32, the classified text fragments are preliminarily scored based on BLEU, and the calculation expression is as follows: Where P n Indicates the precision of n-gram, W n represents the weight of n-gram, BP represents the penalty factor, lc represents the length of machine translation, lr represents the length of the shortest classified text segment, C represents the machine translation, i represents the n-gram in C, h i (C) represents the number of times n_grami appears in C, h i (S j ) represents the number of times n_grami appears in the jth reference translation S.

9. A text analysis method based on LoRA algorithm according to claim 8, characterized in that: The five rating levels are: First level: 1 point: extremely poor, irrelevant or with major errors; Second level: 2 points: poor, low relevance, incomplete content; Level 3: 3 points: Moderate, moderately relevant, and partially complete information; Level 4: 4 points: Good, complete and relevant; Level 5: 5 points: Excellent, the information is complete and highly relevant.

10. A text analysis method based on LoRA algorithm according to claim 9, characterized in that: The process of building the doctor diagnosis model in step 4 is as follows: S41, regularly updating the database of the doctor diagnosis model according to the annual report preprocessing model constructed in step 2 and the classification and scoring model constructed in step 3; S42. Input the company's annual report into the doctor's diagnosis model after updating the database to identify the company's strengths and weaknesses; S43. Generate an evaluation and diagnosis report, including the scores of the enterprise in various technical fields, and put forward improvement suggestions and advantage development strategies.

Images