Document tag generation method based on large language model

Through technologies such as large language models and word embedding models, the labels of massive engineering documents are generated and evaluated, and the problems of low manual extraction efficiency and poor automatic extraction accuracy are solved, high-quality and automated document label generation is achieved, and data middle platform management is supported. .

CN119089873APending Publication Date: 2024-12-06CHINA STATE CONSTRUCTION ACADEMY CORPERATION LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410858528.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-28
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing technology has low efficiency in manual label extraction and poor accuracy in automatic extraction in the management of massive engineering documents, and the results of large language model generation cannot be evaluated, which cannot meet the high-quality and efficient label operation and maintenance needs.

Method used

The document label generation method based on the large language model is adopted, including document preprocessing, full text word segmentation processing and large language model tuning generation. Text content is extracted through word embedding model and deep learning OCR technology, and combined with prompt word templates and user-defined content generation Tags and filters through tag frequency and position evaluation algorithms.

Benefits of technology

It realizes automated and high-quality label generation of massive documents, improves label extraction efficiency and accuracy, solves the problems of time-consuming and labor-intensive manual extraction and poor automatic extraction accuracy, and forms a complete technical path to support data middle platform management. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089873A_ABST
    Figure CN119089873A_ABST
Patent Text Reader

Abstract

The invention provides a document label generation method based on a large language model, aiming at the problem of label generation of different types of massive documents in the building industry. Comprising the steps of document preprocessing, full text word segmentation processing and big language model tuning and generating. Wherein the big language model tuning generation comprises big language model label generation and comprehensive screening based on generated labels; in the step of tag generation of the large language model, a full-text vector composed of M paragraphs subjected to word segmentation processing is input into the large language model, and in combination with a prompt word template, input content defined by a user, K tags generated by each paragraph and M * K tags generated by a full text; according to the comprehensive screening based on the generated labels, the occurrence frequency of the labels in the full text and the occurrence positions of the labels are comprehensively evaluated, and finally high-quality screening of the labels is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of building information technology, and in particular relates to a document tag production method assisted by a large language model. Background Art

[0002] my country's construction industry is huge. By the end of 2023, there will be 157,939 construction companies in the country, with about 3,500 new projects added each year. With the continuous deepening of BIM applications, drawings, models, and documents carrying engineering information are generated and circulated in large quantities. From planning and design to completion and delivery, an engineering project will generate more than 10,000 documents of various types. There are tens of thousands of projects under construction in the construction industry every year, and millions of engineering data are generated every day. However, due to the lack of a data accumulation tool platform, most engineering data exists in the form of scattered files, forgotten in filing cabinets and hard drives, and the actual engineering data utilization rate is less than 0.4%.

[0003] These data are in various forms, including numbers, text, images, etc., but they are often difficult to understand and use directly. In order to effectively organize, store, and utilize massive engineering big data resources, it is necessary to build a labeling system. As a method of simplifying data into an easy-to-understand form, labels can be used to identify, classify, and describe things, and play an important role in data processing and analysis. By selecting appropriate labels, automating the labeling process, and establishing a sound labeling management system, the efficiency of data processing and analysis can be improved.

[0004] Advantages of tags:

[0005] ① Improve readability: Labeling makes data easier to understand, especially for non-professionals. By using short labels instead of complex raw data, information can be conveyed more quickly.

[0006] ② Enhanced usability: Labeling helps improve the usability of data. By organizing data into labels, it can be compared and summarized on a larger scale, thus better understanding the distribution and trends of the data.

[0007] ③ Reduce complexity: Labeling can reduce the complexity of data, making the data analysis process simpler and more efficient. By simplifying raw data into labels, the difficulty of processing and analysis can be reduced.

[0008] Artificial intelligence technologies such as machine learning algorithms and natural language processing can automatically identify and extract data features and generate corresponding labels, improving the efficiency of data processing and analysis. However, methods based on machine learning and natural language processing lack certain precision and accuracy in actual effects. The understanding and generation capabilities of large language models can just solve the above problems. Therefore, how to use large language models to automatically generate efficient and accurate document labels has become a problem worth studying. Summary of the invention

[0009] In view of this, the present invention proposes a document tag generation method based on a large language model. The problems that the present invention can solve are as follows:

[0010] 1. Low efficiency of manual extraction: Manually reading documents and writing labels is the most primitive and basic method. People with certain professional knowledge read the document content and define the document labels according to their own cognitive habits and rules. This method is only applicable to label extraction tasks with a small number of files and fixed domain knowledge. When the number of documents is large and the content covers a wide range of fields, the accuracy and speed of manual extraction will be limited. In the face of today's massive engineering document management, manual work cannot meet the needs of high-quality and efficient label operation and maintenance.

[0011] 2. Automated extraction lacks accuracy: When processing documents using intelligent methods such as machine learning and natural language processing, a large amount of semantic annotation and long-term model training are required, and the scope of application is limited; for engineering data with poor reusability, large quantity, many types, and complex content, the effect of automated extraction is not ideal.

[0012] 3. The generated results cannot be evaluated: Relying on a large language model, document tags can be generated quickly and efficiently, but the large language model also has certain uncertainties, and the generated results cannot be estimated directly using the large language model.

[0013] The specific technical solution of the present invention includes three steps:

[0014] Document preprocessing: used to process massive documents of different types in the construction industry, extract text content from the documents, and obtain the full text; full text word segmentation processing: used to segment the full text, and at the same time, to facilitate label quality screening, use the word embedding model to convert the full text into a full-text vector; large language model tuning and generation: including large language model label generation and comprehensive screening based on generated labels; wherein, the large language model label generation, inputs the full-text vector composed of M paragraphs after word segmentation processing into the large language model, combines the prompt word template and user-defined input content, generates K labels for each paragraph, and generates M×K labels for the full text; the comprehensive screening based on the generated labels, realizes label screening by comprehensively evaluating the number of times the label appears in the full text and the position where the label appears.

[0015] Furthermore, for text-type documents, the preprocessing process includes directly obtaining text information, cleaning the entire text to ensure the text correctness, and grouping the entire text into a large paragraph;

[0016] For non-text documents, document type conversion is required during the preprocessing process, as follows:

[0017] ①Conversion of non-text document types into single-page images: read non-text documents page by page, convert the page contents into images for temporary storage, and perform tilt correction, binarization and denoising on the temporarily stored images;

[0018] ② Obtaining the single-page text of a single-page image: For the processed single-page image, use the deep learning OCR technology to extract characters and obtain the single-page text;

[0019] ③ Full text acquisition: Get all single-page texts to get the full text of non-text documents.

[0020] Furthermore, the full text segmentation processing includes:

[0021] ① Paragraph division: The upper limit of the number of words in a paragraph is set to N, and finally M paragraphs are formed with N words and relatively close numbers of words in each paragraph;

[0022] ② Paragraph vector extraction: Use the word embedding model to extract features from M paragraphs one by one, convert the paragraph text into a paragraph vector for label vector matching; temporarily store based on "document name + paragraph number m + paragraph text + paragraph vector", 1≤m≤M;

[0023] The word embedding model needs to be pre-trained in advance.

[0024] Furthermore, the comprehensive screening method based on generated labels includes a vector-based label matching method and a label comprehensive evaluation method;

[0025] The vector-based label matching method is used to ensure that the label is consistent with the full text content, as follows:

[0026] ① Label vectorization: remove duplicates from the generated KxM labels, obtain L labels, and convert them into label vector X;

[0027] ② Label vector matching: Brute force matching is performed on the label vectors X of L labels in the vector Y of the full text one by one, and the vector similarity is calculated. For vectors with a value higher than the threshold, the label corresponding to the vector is considered to be a label related to the document.

[0028] The label comprehensive evaluation method is as follows:

[0029] ① Tag frequency evaluation: Calculate the relevance between tags and texts by constructing a tag frequency evaluation function, which is the ratio of the total number of times each tag appears to the number of all tags;

[0030] ② Label position evaluation: Based on the Euclidean distance, a label position evaluation function is constructed to calculate the correlation between the position distribution of each label and the text. The formula is as follows:

[0031]

[0032] Among them, D t represents the position evaluation value of the t-th label, and its value range is [0, 1).

[0033] T t Indicates the total number of times the t-th label appears in the entire text, 0≤t≤L,

[0034] dis is the Euclidean distance formula,

[0035] j is a natural integer used to represent the order of appearance of a single label, j∈[1, T t ],

[0036] P t is the position code set of the t-th tag. The position code P of the tag represents the position of the tag in the full text each time it appears. The expression is as follows:

[0037]

[0038] Where N is the number of characters in a single segment, and n is the character number that appears in the mth segment;

[0039] ③ All labels are scored through a comprehensive evaluation function. The higher the score, the more accurate the corresponding label is, thus completing label optimization.

[0040] Furthermore, the comprehensive evaluation function is obtained by weighting the label frequency evaluation function and the label position evaluation function.

[0041] Beneficial Effects

[0042] 1. In view of the problem that the generation of massive document tags is time-consuming, labor-intensive and ineffective, the present invention proposes an automatic tag generation method based on a large language model, which can automatically generate high-quality tags for documents, solving the problem that manual extraction is time-consuming and labor-intensive, and automatic extraction has poor accuracy and effect.

[0043] 2. In order to solve the problem of randomness and uncertainty in the generation of labels by large language models, the present invention designs a label evaluation algorithm to evaluate the generated labels from two perspectives: the frequency of label occurrence and the location of the label.

[0044] 3. The present invention forms a complete technical path and solution, which facilitates the realization of automated document processing and provides strong support for the management of the data center. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 , Flow chart of the method of the present invention Specific implementation plan

[0046] 1. Document tag generation method based on large language model

[0047] Aiming at the label generation scheme for massive documents in the construction industry, the present invention is implemented in three steps: document preprocessing, full text word segmentation processing, and large language model tuning and generation.

[0048] (1) Document preprocessing

[0049] There are many types of data in the construction industry. For label management, the focus is on the text information in the document. Therefore, it is necessary to process different types of documents and extract the text content in the document to obtain the full text.

[0050] The specific steps are as follows

[0051] ①Conversion of different document types. Since text documents (doc, docx, txt, md) can directly obtain text information, this step mainly processes non-text documents (ppt, excel, pdf, dwg, png, jpg, etc.). First, read the non-text document page by page, and convert the page content into pictures for temporary storage. The temporarily stored pictures are tilted, binarized and denoised.

[0052] ② Single-page text acquisition. Since the full text of text documents can be directly obtained in ①, the main purpose of this step is to obtain the full text of non-text documents. For the single-page image processed in ①, use deep learning OCR technology to extract characters, obtain the single-page text, add it to the full text list, and then delete the temporarily stored image.

[0053] ③ Obtain the full text. Repeat the previous step for the next page of text until the last page of the document. At this point, the full text of the text and non-text documents can be obtained.

[0054] ④ Text cleaning. After step 1-(1)-② is executed, the entire text is cleaned, line breaks and links in the text are deleted, spelling errors, grammatical errors and other text errors are corrected to ensure the correctness of the text, and the entire text is grouped into a large paragraph.

[0055] (2) Full text word segmentation processing

[0056] The entire text cannot be input into a large language model at one time, so it is necessary to segment the entire text. At the same time, to facilitate label quality screening, a word embedding model is used to convert the entire text into a full-text vector. The specific steps are as follows

[0057] ① Paragraph division. Based on the large paragraphs of text after document preprocessing, the text is segmented according to the upper limit of the number of words in the paragraph. The upper limit of the number of words in a paragraph is set to N, which is set according to the text processing capability of the large language model and does not exceed the maximum input length. Finally, M paragraphs with about N words and relatively close word counts are formed.

[0058] ② Paragraph vector extraction. Use the word embedding model to extract features from M paragraphs one by one, and convert the paragraph text into a paragraph vector (for label vector matching). Temporary storage is performed based on "document name + paragraph number m (1≤m≤M) + paragraph text + paragraph vector".

[0059] The word embedding model used in the previous step needs to be pre-trained in advance. The training model can be a Chinese model such as Word2Vec, GloVe, FastText, etc. It needs to be trained on the basis of the pre-trained model with the construction industry materials as the corpus. The word segmentation effect is positively correlated with the training effect.

[0060] (3) Large language model label generation

[0061] This process mainly generates labels in batches based on the generation capability of the large language model. It is necessary to input the full text consisting of M paragraphs after word segmentation into the large language model, and then input M paragraphs in sequence based on the prompt word template and user-defined input content. K labels are generated for each paragraph, and a total of M×K labels are generated for the full text.

[0062] ① Construct the input format of the large language model. The input format is mainly formed by combining the Promote word template and user-defined content. The customized content includes "construction stage", "user-defined keywords", "number of generated labels for a single segment K", "mth paragraph text Context m ", the input format is as follows:

[0063]

[0064] Among them, "building stage" is divided into design stage, construction stage, and operation and maintenance stage; "user-defined keywords" are input by the user to describe the subject of the document, and there are no more than 3 keywords; "number of tags generated K" is defined by the user, K is an integer ranging from 0 to 10, and the default value of K is 3.

[0065] ② Input the large language model to generate labels. Input the input format constructed in the previous step into the large language model. The large language models that can be used in this step include GLM2-6B, GLM3-6B, GLM4-9B, Qwen-7B, Qwen2-7B, Qwen-14B and other open source large language models, and the processing text length is not less than 2K (the parameter max_sequence_length in the large language model is not less than 2048). Based on context learning, few-sample learning, and prompt word engineering technology, make full use of the understanding and generation capabilities of the large language model to input the full text segment by segment, and generate K labels segment by segment, and finally obtain KxM labels. The label generation logic is as follows:

[0066]

[0067]

[0068] Among them, the relevant principles in the generation process are as follows:

[0069] Contextual learning: The large language model understands the relationship between the meaning of the prompt word and the label to be generated by learning from the context in the full text. This involves the n-gram model and word embedding representation, so that the large language model can capture the probabilistic relationship between words and further understand the contextual relationship in the text.

[0070] Few-shot learning: Large language models use a small number of examples to quickly learn new task patterns without the need for large amounts of data training. This is mainly based on the principles of transfer learning and meta-learning. Transfer learning enables the model to transfer knowledge learned in one task to another; meta-learning is based on machine learning technology and enables the model to quickly adapt to new tasks by forming meta-knowledge.

[0071] Prompt engineering: Prompt engineering is a method that can be used in large language models to design or construct natural language prompts that are input into the model to guide the model to generate specific types of responses or perform specific tasks.

[0072] 2. Comprehensive screening method based on generated tags

[0073] The tags generated by the large language model are random and unstable in content and quality. Therefore, the tags generated by the large language model need to be further processed and screened. The present invention achieves the screening of high-quality tags by comprehensively evaluating the number of times the tags appear in the full text and the quality of the tags.

[0074] (1) Vector-based label matching method

[0075] Large language models will produce "hallucinations" with a certain probability, that is, they will generate labels beyond the given full text. Therefore, it is necessary to screen the generated labels to ensure that the labels are consistent with the full text content. The present invention designs a vector-based label matching method, which matches the label with the full text in the word vector state to improve the matching speed and accuracy, and obtains labels with higher similarity to the text by screening the cosine similarity measurement factor above the threshold. The specific steps of the process are as follows:

[0076] ① Label vectorization. This step requires converting the generated KxM labels into label necklaces. First, all KxM labels are deduplicated to obtain L labels. Similar to step 1-(2)-②, the text of L labels is converted into label vector X using a word embedding model.

[0077] ② Label vector matching. Perform brute force matching on the label vectors X of L labels in the vector Y of the full text one by one, calculate the vector similarity, and measure it with the cosine similarity operator. The formula is as follows:

[0078]

[0079] Where X is the vector of a single label and Y is the vector of the entire text.

[0080] By setting a threshold ρ (usually 0.7), we can determine whether the match is successful. For vectors above the threshold, we consider the label corresponding to the vector to be a label related to the document. At this time, we need to record the total number of times the t-th label appears in the entire text. t ,0≤t≤L,T t Used for tag frequency evaluation, see 2-(2)-② for details; at the same time, the position code P of a single tag in the full text is recorded each time it appears, and the expression is as follows:

[0081]

[0082] Where N is the number of characters in a single segment, and n is the sequence number of the characters appearing in the mth segment.

[0083] The position code of the t-th label is composed into a set P t , used for label position evaluation, see step 2-(2) for details.

[0084] (2) Label comprehensive evaluation method

[0085] During the training process of a large language model, factors such as data compression, model tuning methods, and model transition generalization may make the generated labels accidental and uncertain. Therefore, it is necessary to set up a label evaluation algorithm to evaluate the labels generated by the large language. The present invention conducts a comprehensive evaluation based on the frequency of label occurrence and the position in the paragraph.

[0086] ① Tag frequency evaluation. The quality of a tag is not only related to the content of the text, but also to the frequency of its appearance. The more times a tag appears in a document, the more times the tag is mentioned in the text, and the tag is one of the topics of the document. In this case, the tag is considered to have a higher relevance to the text. In order to evaluate the tag frequency, the present invention provides a tag frequency evaluation method to count the proportion of a single tag in all tags. Specifically, it mainly counts the total number of times T the t-th tag appears counted in step 2-(1)-②. t Compare with the number of all tags KxM to get the frequency score F of a single tag t , the formula is as follows:

[0087]

[0088] F t Occurrence frequency evaluation function, whose value range is [0, 1).

[0089] ② Tag position evaluation. This step mainly evaluates the position distribution of tags in the text. Without considering the content of the tags, if the tags are densely distributed in the text, then the tags may be the focus of a certain chapter; if the tags are evenly distributed in the text, then it means that the tags run through the entire text and may be one of the main themes of the article. In order to extract tags that fit the main theme of the full text, the present invention constructs a tag position evaluation function D based on the Euclidean distance. t The position distribution of the labels is evaluated using the following formula:

[0090]

[0091] Where dis is the Euclidean distance formula, j is a natural integer used to represent the order of appearance of a single label, j∈[1, T t ], D t The value range of is [0, 1).

[0092] ③ Comprehensive evaluation function. The present invention comprehensively integrates frequency evaluation and position evaluation to construct a comprehensive evaluation function, and uses weight coefficients α and β to weigh the proportion of the two functions. The comprehensive evaluation function is as follows:

[0093] score (Ft,Dt) =αF t +βD t

[0094] Among them, α and β are coefficients α∈(0,1), β∈(0,1) and α+β=1.

[0095] After comprehensive evaluation of L labels, L score values ​​are obtained, which are arranged in descending order, and the values ​​with higher scores are selected as high-quality labels according to user needs.

Claims

1. A document tag generation method based on a large language model, characterized in that: include, Document preprocessing: used to process different types of massive documents in the construction industry, extract text content from the documents, and obtain the full text; Full text word segmentation: used to segment the full text and convert the full text into full text vectors using a word embedding model to facilitate label quality screening. Large language model tuning and generation: including large language model label generation and comprehensive screening based on generated labels; wherein, the large language model label generation inputs the full-text vector consisting of M paragraphs after word segmentation processing into the large language model, combines the prompt word template and user-defined input content, generates K labels for each paragraph, and generates M×K labels for the full text; the comprehensive screening based on generated labels implements label screening by comprehensively evaluating the number of times the label appears in the full text and the position where the label appears.

2. The document tag generation method based on a large language model according to claim 1, characterized in that: For text-type documents, the preprocessing process includes directly obtaining text information, cleaning the entire text, ensuring the correctness of the text, and grouping the entire text into a large paragraph. For non-text-type documents, the preprocessing process requires document type conversion, as follows: ①Conversion of non-text document types into single-page images: read non-text documents page by page, convert the page contents into images for temporary storage, and perform tilt correction, binarization and denoising on the temporarily stored images; ② Obtaining the single-page text of a single-page image: For the processed single-page image, use the deep learning OCR technology to extract characters and obtain the single-page text; ③ Full text acquisition: Get all single-page texts to get the full text of non-text documents.

3. The document tag generation method based on a large language model according to claim 2, characterized in that: The full text segmentation processing includes: ① Paragraph division: The upper limit of the number of words in a paragraph is set to N, and finally M paragraphs are formed with about N words and relatively close numbers of words in each paragraph; ② Paragraph vector extraction: Use the word embedding model to extract features from M paragraphs one by one, convert the paragraph text into a paragraph vector for label vector matching; temporarily store based on "document name + paragraph number m + paragraph text + paragraph vector", 1≤m≤M; The word embedding model needs to be pre-trained in advance.

4. The document tag generation method based on a large language model according to any one of claims 1 to 3, characterized in that: Comprehensive screening methods based on generated labels include vector-based label matching methods and label comprehensive evaluation methods; The vector-based label matching method is used to ensure that the label is consistent with the full text content, as follows: ① Label vectorization: remove duplicates from the generated KxM labels, obtain L labels, and convert them into label vector X; ② Label vector matching: Brute force matching is performed on the label vectors X of L labels in the vector Y of the full text one by one, and the vector similarity is calculated. For vectors with a value higher than the threshold, the label corresponding to the vector is considered to be a label related to the document. The label comprehensive evaluation method is as follows: ① Tag frequency evaluation: Calculate the relevance between tags and texts by constructing a tag frequency evaluation function, which is the ratio of the total number of times each tag appears to the number of all tags; ② Label position evaluation: Based on the Euclidean distance, a label position evaluation function is constructed to calculate the correlation between the position distribution of each label and the text. The formula is as follows: Among them, D t represents the position evaluation value of the t-th label, and its value range is [0, 1). t Indicates the total number of times the t-th label appears in the entire text, 0≤t≤L, dis is the Euclidean distance formula, j is a natural integer used to represent the order of appearance of a single label, j∈[1, T t ], P t is the position code set of the t-th tag. The position code P of the tag represents the position of the tag in the full text each time it appears. The expression is as follows: Where N is the number of characters in a single segment, and n is the character number that appears in the mth segment; ③ All labels are scored through a comprehensive evaluation function. The higher the score, the more accurate the corresponding label is, thus completing label optimization.

5. The document tag generation method based on a large language model according to any one of claim 4, characterized in that: The comprehensive evaluation function is obtained by weighting the label frequency evaluation function and the label position evaluation function.

Citation Information

Cited By

  • Electric power business work order information intelligent identification method based on OCR technology

    CN120182979A

  • Offshore wind power corpus processing method and device

    CN121074911A

  • An offshore wind power corpus processing method and device

    CN121074911B