Semantic understanding-based end-to-end text matching auxiliary labeling method
Through the end-to-end text matching auxiliary labeling method based on semantic understanding, the problems of low efficiency and high cost of PDF file information extraction and manual labeling are solved, efficient and accurate text matching and labeling are achieved, and end-to-end intelligent auxiliary is provided. The labeling system generates high-quality labeling data sets and improves the ease of use and security of the system.
Patent Information
- Application Number
- CN202410947826.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-15
AI Technical Summary
The prior art is inefficient and costly when extracting information from complex PDF files and performing manual labeling, and lacks end-to-end system design, resulting in complex and insecure information processing and making it difficult to generate high-quality labeling data sets.
Using end-to-end text matching assisted annotation method based on semantic understanding, the key PDF information is extracted automatically through semantic matching and unsupervised algorithms, and combined with multi-expert model recommendations, it can achieve efficient and accurate text matching and annotation, and integrate it into one end-to-end in the system.
It greatly improves the efficiency and accuracy of text matching and labeling, reduces labor costs, realizes a complete process from PDF files to database, provides an efficient, safe and easy-to-use intelligent auxiliary labeling system, which can quickly generate high-quality Label the dataset.
Smart Images

Figure CN118964611A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of text matching auxiliary annotation, and in particular to an end-to-end text matching auxiliary annotation method based on semantic understanding. Background Art
[0002] (I) Extraction of PDF file information
[0003] With the rapid development of information technology, PDF (Portable Document Format) has become the main way to store and transmit digital information. Due to its high degree of structure and compatibility, PDF is widely used in various business scenarios, such as scientific research papers, government reports, corporate manuals, etc., which contain rich information such as text, charts, formulas, etc. However, the complex structure of PDF files poses a great challenge to information extraction. PDF files have a complex structure and rich content. On the one hand, PDF files not only include plain text content, but may also contain multiple elements such as tables, images, hypertext links, etc. The existence of these elements makes the content of PDF files extremely rich, but also makes it difficult to extract useful information from them. On the other hand, the information contained in PDF files is very rich, and the effective information is often sparsely distributed. How to accurately locate and extract effective information in such a complex file structure, significantly reduce the waste of computing resources, and improve processing efficiency has become a huge challenge.
[0004] 2. Manual labeling
[0005] Artificial intelligence (AI) is rapidly becoming a major force in the modern technology world, driving innovation in a variety of industries, from automated office work to advanced medical diagnostics. As these technologies continue to advance, the demand for labeled datasets is also growing. These datasets are the basis for training machine learning models, especially in the field of deep learning, where high-quality labeled data is key to the accuracy of the algorithms.
[0006] First, labeled data sets provide a blueprint for artificial intelligence systems to learn. By analyzing labeled data, machine learning models can identify patterns and associations and learn how to perform specific tasks. This is like giving AI an answer key and letting it learn the correct way to judge and make decisions through examples. Whether it is image recognition, natural language processing or predictive analysis, all of these applications require a large amount of labeled data to "train" the algorithm so that it can run accurately in real applications.
[0007] However, it is not only the demand for data volume that is growing, but also the demand for data quality and diversity. The performance of AI models is directly affected by the quality of input data. If there are biases in the training data, the model may develop the same biases and make wrong decisions in real applications. Therefore, high-quality labeled data is needed, which must be accurate, consistent, and contain enough variables to ensure that the model can generalize to new, unseen data.
[0008] In addition, as AI applications expand, the demand for specific types of data is also growing. For example, in the medical industry, medical images and case reports with clinical relevance need to be annotated, and such annotation work requires a high degree of expertise and precision. In the field of self-driving cars, a large number of street views and traffic scenes need to be annotated to train vehicles to correctly identify various objects and conditions.
[0009] As the size and complexity of data increases, manual annotation becomes increasingly impractical. Not only is it time-consuming and costly, but it is also difficult to maintain the consistency and quality of annotations. This has led to the development of AI-assisted annotation techniques, such as semi-automatic annotation tools and active learning systems, which can reduce manpower requirements and improve annotation efficiency and accuracy.
[0010] In this context, the demand for AI-friendly annotation tools and platforms is also growing. These platforms can support large-scale data processing and annotation, provide quality control mechanisms, and allow multi-user collaboration to improve the efficiency of data processing. At the same time, as AI technology expands to edge computing and mobile devices, there is a greater demand for lightweight and efficient data sets.
[0011] With the rapid development of deep learning technology, annotated datasets for specific tasks have gradually become a rigid demand for training domain-specific models. However, existing text matching and annotation methods mainly rely on manual work, which is time-consuming, inefficient, and prone to errors. Manual annotation is an important part of many data-driven tasks, especially in text matching and annotation tasks. However, in real applications, this task is often very time-consuming. On the one hand, the annotation work requires manual traversal of all possible text matching pairs, and the workload of this task is proportional to the number of retrieval texts to be annotated (n) and the number of knowledge texts (m), respectively (the complexity is O(nm)), which makes the annotation task more and more difficult as the data scale grows. On the other hand, since the results of manual annotation are affected by the annotator's professional knowledge, experience, and subjectivity, the annotation quality often has a certain instability in the case of large-scale manual annotation. These factors together lead to extremely high labor costs for text matching and annotation tasks, and it is difficult to ensure the accuracy and consistency of the annotation results.
[0012] 3. End-to-end system
[0013] "End-to-end systems" play a vital role in modern technology and computing. The design concept of an end-to-end (E2E) system means that in a complex system, data processing or information transmission does not go through any intermediate branches from the starting point to the end point. This design approach has the following key advantages: 1) Reduce data processing links: The end-to-end system reduces the delay and complexity of the data transmission process by minimizing the data processing links; 2) Improve the efficiency and accuracy of the system: The end-to-end system processes data directly from input to output, reducing the risk of information loss and improving overall efficiency and accuracy; 3) Simplify system design and maintenance: The end-to-end system reduces the dependencies between system components and simplifies the overall system architecture, which not only reduces the complexity of design and implementation, but also makes system maintenance and updates easier; 4) Improve data security: Due to the reduction of links in the data transmission process, the end-to-end system can better protect data from tampering or illegal access. Especially in encrypted communications and data security-sensitive applications, end-to-end encryption technology ensures data privacy and security. The importance of end-to-end systems lies in that they provide an efficient, accurate, secure and easy-to-maintain solution, which is particularly critical when dealing with complex and high-speed data processing tasks. As technology continues to advance, the scope and impact of end-to-end systems will continue to expand.
[0014] Therefore, how to accurately extract effective information from complex PDF files, how to reduce the workload of manual annotation tasks, and integrate them into an end-to-end system have become urgent problems to be solved in modern information processing.
[0015] The existing technical solutions are mainly divided into PDF file information extraction and manual annotation process.
[0016] (1) Extraction of PDF file information.
[0017] Information extraction is the first and most challenging step to understand and process PDF files. Existing solutions mainly include two steps: parsing PDF files and subsequent information extraction. Among them, PDF files are usually parsed in the following two ways: one is optical character recognition (OCR); the other is to directly extract information from PDF files.
[0018] Optical character recognition (OCR) technology is mainly used to convert image data in PDF files into recognizable text data. This technology recognizes characters in images and converts them into text that can be processed by computers. In this process, OCR technology not only needs to be able to recognize text in various fonts, fonts, sizes and colors, but also needs to be able to handle various noises and interferences, such as background color, image quality, etc.
[0019] There are also methods in the prior art to directly extract information from PDF documents, which avoids the information loss and errors that may be caused by the imaging and OCR processes. A PDF file is a binary file that contains the structured information and content data of the document. Parsing PDF files usually requires a dedicated PDF parser, such as Adobe's PDF Library, PDFBox, PyPDF2, etc. These libraries can parse the internal structure of PDF files and extract information such as text, images, links, metadata, etc. However, the structure of PDF files is very complex and may contain many nested objects and references, which makes parsing PDF files a very complex task. In addition, because the PDF standard allows a lot of flexibility, different PDF files may use completely different ways to represent the same content, which increases the complexity of the parsing work.
[0020] 2. Manual labeling
[0021] Artificial intelligence (AI) is rapidly becoming a major force in the modern technology landscape, driving innovation in multiple industries, from automated office work to advanced medical diagnostics. As these technologies continue to advance, the demand for labeled datasets is also growing. With the rapid development of deep learning technology, labeled datasets for specific tasks have gradually become a necessity for training domain-specific models. The training of deep learning models relies on large amounts of labeled data. For specific tasks, these datasets often require expert knowledge to accurately label them, and the labeling process also needs to consider various possible situations and boundary conditions. Therefore, the process of generating these datasets is both complex and time-consuming.
[0022] In existing technical solutions, although the manual labeling process is indispensable, it consumes a lot of manpower and time. Especially when the scale of labeled data increases, the complexity of manual labeling will increase with the square of the scale of labeled data, which will greatly increase the labor cost. In addition, as the complexity and scale of the model increase, the requirements for the quantity and quality of labeled data sets are also gradually increasing. Training larger and more complex models requires more data, and the quality of the data directly affects the performance of the model. Therefore, how to quickly, accurately and efficiently generate a large number of high-quality labeled data sets has become a major challenge in the field of deep learning.
[0023] Although existing technical solutions have made certain progress in some aspects, there are still some important shortcomings.
[0024] 1. Difficulty in extracting information from PDF files: Although there are methods to extract information directly from PDF files, the process of extracting information directly from PDF files still faces many challenges due to the complexity of PDF files. First, the structure of PDF files is very complex and may contain a large number of nested objects and references, which makes parsing PDF files a technical challenge. Secondly, different PDF files may use completely different ways to represent the same content, which increases the complexity of extracting information. In addition, the content in PDF files is not limited to text, but may also include images, tables, formulas, etc. The extraction and parsing of these non-text content often requires more complex technologies.
[0025] 2. The manual labeling process is inefficient and costly: Although manual labeling has its irreplaceable advantages in ensuring data quality, its disadvantages are also very obvious. First, the manual labeling process is very time-consuming, especially on large-scale data sets, the labeling process may take months or even years. Secondly, the accuracy of manual labeling is often limited by the professional level and experience of the labelers, and these professional labelers are usually costly. Finally, as the amount of data increases, the complexity of manual labeling increases exponentially, which makes the manual labeling process more difficult.
[0026] 3. Lack of end-to-end system design: First, many technical solutions tend to focus only on a specific single link, while ignoring the collaborative work of the entire system. In this case, the transmission and conversion of data between different components is time-consuming and error-prone, especially when the components are developed independently by different teams. For example, data collection, processing, analysis, and presentation are often separate links. This segmentation leads to duplication of work and consistency issues, which ultimately affects the accuracy and credibility of the data. Secondly, the lack of end-to-end design means that automation and optimization are difficult to achieve. In a fully integrated system, algorithms and machine learning models can be used to automate data processing and decision-making processes to improve efficiency and accuracy. However, in a segmented system, manual intervention is required for each link, which not only increases labor costs but also increases the possibility of errors. Furthermore, security and privacy protection are often overlooked in existing technical solutions. In an end-to-end system design, security measures can be comprehensively considered and implemented to ensure the security of data throughout the entire processing process. In contrast, data in existing multi-component systems may be exposed to more risks during the transfer process because different system components may have different security standards and protocols. In addition, another disadvantage of existing technical solutions is the lack of flexibility and scalability. End-to-end system design often allows for easier addition of new features and upgrades, as the entire system is considered as a whole. However, in a solution made up of multiple independent parts, any change may require large-scale modifications to multiple components, which is not only costly but may also cause downtime for the entire system.
[0027] Therefore, it has become an urgent need to seek a new technical solution that can not only quickly and efficiently extract PDF file information, but also accurately and effectively generate large-scale high-quality annotated data sets, and integrate them into an end-to-end system. Summary of the invention
[0028] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an end-to-end text matching assisted annotation method based on semantic understanding, so as to realize a new technical solution that can not only quickly and efficiently extract PDF file information, but also accurately and effectively generate large-scale high-quality annotation data sets, and integrate them into an end-to-end system.
[0029] In order to achieve the above purpose, an end-to-end text matching auxiliary annotation method based on semantic understanding is designed. The method includes S1. Extracting PDF key information based on semantic understanding, including: S1.1 According to the meta information of the PDF format file, obtaining the title information of each level of the input PDF; S1.2 Selecting the candidate title T, judging whether its level matches the target level, if yes, proceeding to the next step, otherwise not matching; S1.3 Given a keyword list {k i |i∈[0,1,…,n]}, if there is a keyword k i Appears in the candidate title T, proceed to the next step, otherwise it does not match; S1.4: Given a standard title list {t i |i∈[0,1,…,m]}, if there exists a standard title t i If one of the following fuzzy matching criteria is met, the next step is performed, otherwise no match is made; fuzzy matching criteria: (1) keyword similarity ≥ 0.9, keyword overlap ≥ 0.5, and title overlap ≥ 0.5; or (2) keyword overlap ≥ 0.9 and title overlap ≥ 0.5; (3) keyword overlap ≥ 0.5 and title overlap ≥ 0.9; (4) keyword overlap ≥ 0.75 and title overlap ≥ 0.75; S1.5 Given a list of blocked words {b i|i∈[0,1,…,p]}, if the candidate title T contains blocked words, it will not match; otherwise, the candidate title T is successfully matched, and the interval where the corresponding ordinary text content is located is the matching interval; S1.6 loops over the remaining candidate titles, and if there is no matching title, the returned result is empty; otherwise, if there is an overlap in intervals, the overlapping intervals are merged into one interval, and the next step is performed; S1.7 intercepts the PDF document according to the matching interval, retaining only the page where the matching content is located; S1.8 extracts text from the intercepted page and saves the result as a text file; S2. Multi-expert model recommendation based on unsupervised algorithms, combined with unsupervised learning algorithms and pre-trained language models, fuses the outputs of multiple expert models into an integrated model for large-scale comprehensive scoring. For each document data, several retrieval data with the highest final score are selected as recommendations, and finally annotated by human experts.
[0030] Preferably, step S1 of the present invention further includes
[0031] S1.9 repeats step S1.4, matches the title information in the extracted text again, and extracts the corresponding text information under the matching title; S1.10 repeats step S1.5, if the extracted text contains blocked words, it is not matched, otherwise the extracted text is output.
[0032] Preferably, the step S2 of the present invention includes S2.1 firstly combining the data in the search database Q and the knowledge database K in pairs to form a search-knowledge text pair (q i ,k j ); S2.2 Two-way search-knowledge text pair (q i ,k j ) to perform similarity scoring. The first path uses the pre-trained language model to directly perform similarity scoring and obtains the similarity score s 1 ,The second path first uses the pre-trained language model to extract entities and manually screens the entity types. Then all the screened entities in the corpus are used to train the word vector model and generate (q i ,k j )’s word embedding (E q ,E k ), and finally calculate the cosine similarity to get the similarity score s 2 , the specific formula is as follows:
[0033]
[0034] The final similarity score s is s 1 and 2The weighted sum of , and the weight is adjusted according to the actual situation; S2.3 determines whether the similarity score s is higher than the preset threshold. If it is higher, it enters the manual labeling stage, otherwise it is treated as a negative sample; S2.4 For the samples entering the manual labeling stage, multiple experts manually label them. Each labeling expert needs to choose between "yes", "no", and "uncertain", and select the answer label according to the principle of majority vote. If there is a tie or the label is "uncertain", step 2.4 needs to be repeated until the label is "yes" or "no".
[0035] Compared with the prior art, the present invention has the following advantages:
[0036] 1. The present invention greatly improves the efficiency and accuracy of text matching and annotation through an automated semantic matching and annotation method. At the same time, the technical method adopted by the present invention is widely used in the fields of text processing and information retrieval, and has high practical value.
[0037] 2. The present invention proposes an end-to-end intelligent auxiliary annotation system that can complete the entire process from the PDF file end to the database end, thereby improving the usability and integrity of the system.
[0038] 3. An intelligent auxiliary annotation method provides intelligent alternative option recommendations to annotators through semantic understanding of the text, greatly reducing the cost of manual annotation.
[0039] 4. Extract PDF key information based on semantic understanding, starting from the title information of the file, through semantic-based fuzzy matching and keyword-based precise matching mode, improve the ability to locate and extract key information, greatly reduce the resource overhead of text conversion, and reduce the noise of text extraction results. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 , is a schematic diagram of an end-to-end text matching assisted annotation method based on semantic recommendation;
[0041] Figure 2 , is a schematic diagram of PDF key information extraction based on semantic understanding;
[0042] Figure 3 , is a schematic diagram of the manual labeling assistance system based on unsupervised algorithm. DETAILED DESCRIPTION
[0043] In order to make the purpose, principle and structure of the present invention more clear, it is further described below in conjunction with the drawings and specific embodiments.
[0044] The present invention proposes an end-to-end text matching auxiliary annotation method based on semantic matching (such as Figure 1As shown in Figure 1), it can effectively extract key information, perform semantic matching, and assist in manual annotation. The specific steps are as follows:
[0045] Phase 1: Extraction of PDF key information based on semantic understanding (see Figure 2 )
[0046] The following steps are involved:
[0047] Step 1.1: Get the input PDF title information at all levels according to the metadata of the PDF file;
[0048] Step 1.2: For the candidate title T, determine whether its level matches the target level. If yes, proceed to the next step; otherwise, it does not match.
[0049] Step 1.3: Given a keyword list {k i |i∈[0,1,…,n]}, if there is a keyword k i If it appears in the alternative title T, proceed to the next step, otherwise it does not match;
[0050] Step 1.4: Given a list of standard headers {t i |i∈[0,1,…,m]}, if there exists a standard title t i If one of the following fuzzy matching criteria is met, the next step will be taken, otherwise no match will be made; fuzzy matching criteria: (1) keyword similarity ≥ 0.9, and keyword overlap (the ratio of the number of overlapping words between keywords and standard titles to the number of keywords) ≥ 0.5, and title overlap (the ratio of the number of overlapping words between keywords and standard titles to the number of words in the standard title) ≥ 0.5; (2) keyword overlap ≥ 0.9 and title overlap ≥ 0.5; (3) keyword overlap ≥ 0.5 and title overlap ≥ 0.9; (4) keyword overlap ≥ 0.75 and title overlap ≥ 0.75;
[0051] Step 1.5: Given a list of blocked words {b i |i∈[0,1,…,p]}, if the candidate title T contains blocked words, it will not match; otherwise, the candidate title T is successfully matched, and the interval where its corresponding ordinary text content is located is the matching interval;
[0052] Step 1.6: Loop through the candidate titles. If there is no matching title, the returned result is empty; otherwise, if there is an overlap in the intervals, the overlapping intervals are merged into one interval and the next step is performed;
[0053] Step 1.7: Extract the PDF document according to the matching interval and only keep the pages where the matching content is located;
[0054] Step 1.8: Use Adobe Cloud Service API to extract text from the captured page and save the result as a text file;
[0055] Step 1.9: Repeat step 1.4, match the title information in the extracted text again, and extract the corresponding text information under the matching title;
[0056] Step 1.10: Repeat step 1.5. If the extracted text contains blocked words, it will not match. Otherwise, output the extracted text.
[0057] Phase 2: Multi-expert model recommendation based on unsupervised algorithm (see Figure 3 )
[0058] This phase combines unsupervised learning algorithms and industry-leading pre-trained language models, such as Figure 3 As shown in the figure, the outputs of multiple expert models are merged into an integrated model for large-scale comprehensive scoring. For each document 1 data, several retrieval data with the highest final score are selected as recommendations, and finally annotated by human experts, which includes the following steps:
[0059] Step 2.1: First, the data in the search database Q and the knowledge database K are combined in pairs to form a search-knowledge text pair (q i ,k j ).
[0060] Step 2.2: Retrieve knowledge text pairs in two ways (q i ,k j ) to perform similarity scoring. The first path uses the pre-trained language model to directly perform similarity scoring and obtains the similarity score s 1 The second path first uses the pre-trained language model to extract entities and manually screens the entity types. Then all the screened entities in the corpus are used to train a word vector model (such as word2vec) and generate (q i ,k j )’s word embedding (E q ,E k ), and finally calculate the cosine similarity to get the similarity score s 2 , the specific formula is as follows:
[0061]
[0062] The final similarity score s is s 1 and 2 The weights can be adjusted according to actual conditions.
[0063] Step 2.3: Determine whether the similarity score s is higher than the preset threshold. If so, enter the manual labeling stage, otherwise it is treated as a negative sample.
[0064] Step 2.4: For samples that enter the manual labeling stage, multiple experts will manually label them. Each labeling expert needs to choose between "yes", "no", and "uncertain", and select the answer label based on the majority vote principle. If there is a tie or the label is "uncertain", step 2.4 needs to be repeated until the label is "yes" or "no".
[0065] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent substitutions or changes according to the technical solutions and novel concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. An end-to-end text matching auxiliary annotation method based on semantic understanding, characterized in that The method comprises S1. Extraction of PDF key information based on semantic understanding, including: S1.1 Obtaining the input PDF title information at all levels according to the metadata of the PDF format file; S1.2 Select the candidate title T and determine whether its level matches the target level. If yes, proceed to the next step; otherwise, it does not match. S1.3 Given a keyword list {k i |i∈[0,1,…,n]}, if there is a keyword k i If it appears in the alternative title T, proceed to the next step, otherwise it does not match; S1.4: Given a list of standard titles {t i |i∈[0,1,…,m]}, if there exists a standard title t i If one of the following fuzzy matching criteria is met, the next step is performed, otherwise no match is made; fuzzy matching criteria: (1) keyword similarity ≥ 0.9, keyword overlap ≥ 0.5, and title overlap ≥ 0.5; or (2) keyword overlap ≥ 0.9 and title overlap ≥ 0.5; (3) keyword overlap ≥ 0.5 and title overlap ≥ 0.9; (4) keyword overlap ≥ 0.75 and title overlap ≥ 0.75; S1.5 Given a list of blocked words {b i |i∈[0,1,…,p]}, if the candidate title T contains blocked words, it will not match; otherwise, the candidate title T is successfully matched, and the interval where its corresponding ordinary text content is located is the matching interval; S1.6 loops through the remaining candidate titles. If there is no matching title, the returned result is empty; otherwise, if there is an overlap in the intervals, the overlapping intervals are merged into one interval and the next step is performed; S1.7 intercepts the PDF document according to the matching interval and only retains the page where the matching content is located; S1.8 extract text from the intercepted page and save the result as a text file; S2. Multi-expert model recommendation based on unsupervised algorithms combines unsupervised learning algorithms and pre-trained language models to fuse the outputs of multiple expert models into an integrated model for large-scale comprehensive scoring. For each document data, several retrieval data with the highest final scores are selected as recommendations, and finally labeled by human experts.
2. The end-to-end text matching auxiliary annotation method based on semantic understanding as claimed in claim 1, characterized in that The step S1 also includes S1.9 repeats step S1.4, matches the title information in the extracted text again, and extracts the corresponding text information under the matching title; S1.10 repeats step S1.
5. If the extracted text contains blocked words, it is not matched, otherwise the extracted text is output.
3. The end-to-end text matching auxiliary annotation method based on semantic understanding as claimed in claim 1, characterized in that The step S2 includes S2.1 First, the data in the retrieval database Q and the knowledge database K are combined in pairs to form a retrieval-knowledge text pair (q i ,k j ); S2.2 Two-way retrieval - knowledge text pair (q i ,k j ) to score similarity. The first path uses the pre-trained language model to directly score similarity and obtain the similarity score s1. The second path first uses the pre-trained language model to extract entities and manually screen the entity types. Then all the screened entities in the corpus are used to train the word vector model and generate (q i ,k j )’s word embedding (E q ,E k ), and finally calculate the cosine similarity to get the similarity score s2. The specific formula is as follows: The final similarity score s is the weighted sum of s1 and s2, and the weight is adjusted according to the actual situation; S2.3 determines whether the similarity score s is higher than the preset threshold. If so, it enters the manual labeling stage, otherwise it is treated as a negative sample; S2.4 For samples that enter the manual labeling stage, multiple experts will manually label them. Each labeling expert needs to choose between "yes", "no", and "uncertain", and select the answer label based on the majority vote principle. If there is a tie or the label is "uncertain", step 2.4 needs to be repeated until the label is "yes" or "no".
