Visualization method and system for large model training corpus in scientific field

Through visualization methods of large-model training corpus for scientific fields, segmented streaming and rich text analytical rendering, comparison diagrams are generated and metadata watermarks are attached, which solves the problems of low efficiency and security evaluation of large-model training corpus and improves data quality and evaluate accuracy, reducing the risk of browser lag.

CN119047458AActive Publication Date: 2024-11-29ZHEJIANG LAB

Patent Information

Application Number
CN202411525964.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-30
Publication Date
2024-11-29
Estimated Expiration
2044-10-30

AI Technical Summary

Technical Problem

The data quality evaluation of large-model training corpus in the scientific field is low efficiency and poor accuracy, and OCR parsing errors and rich text content complexity, resulting in browser lag and security problems, making it difficult to achieve efficient and accurate corpus quality evaluation.

Method used

Using a visualization method of large-scale model training corpus for scientific fields, the markdown format corpus is streamed to the client in segments through server-side, and the rich text content is analyzed and rendered, and a comparison diagram of PDF original text and analysis results is generated, and corpus elements are attached. Watermark of data and user information, supporting external sharing and evaluation.

Benefits of technology

It realizes efficient and secure corpus data transmission and parsing, reduces evaluation costs, improves the quality of large model training data, avoids the risks of browser lag and XSS attacks, supports independently selecting the scores of areas of interest, and ensures data confidentiality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119047458A_ABST
    Figure CN119047458A_ABST
Patent Text Reader

Abstract

The invention discloses a visualization method and system for large-scale model training corpora in the scientific field. The method comprises the steps that the large-scale model training corpora in the scientific field and in a Markdown format obtained from an object storage server are transmitted to a client side in a segmented streaming mode at a server side; the rich text content in the received markdown text is analyzed and rendered at the client to obtain a reduction result; and generating a comparison graph of the PDF original text content of the Markdown text and the reduction result at the client, and adding corpus metadata, user information watermarks and corpus quality scores on the comparison graph. The method can efficiently, accurately and safely transmit, analyze, render and share the large model training corpus in the markdown format in the scientific field, and share the corpus quality evaluation result to internal and external experts, thereby helping large model training in the scientific field to provide high-quality corpus data, and improving the accuracy of the large model in the scientific field.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of data visualization, and particularly relates to a visualization method and system for large model training corpora in the scientific field. Background Art

[0002] In recent years, the application of large language models in the scientific field (AI4Science, covering geology, astronomy, biology, etc.) has rapidly emerged and become a key force driving scientific research and discovery. Their capabilities mainly depend on three major elements: data, computing power, and algorithms. Among them, data is the core and fundamental factor, and the quality and quantity of data are crucial for improving the capabilities of large models.

[0003] The training data of large models in the scientific field usually comes from PDF documents containing rich text content (such as text, tables, formulas, pictures, lists, references, etc.). However, large models cannot directly interpret the content of PDF documents and need to rely on OCR technology to convert them into markdown format corpora that are easy for large model training. This conversion process needs to ensure the quality and accuracy of the markdown format corpora to maximize the improvement of model performance and avoid negative impacts.

[0004] Currently, the evaluation of markdown format training corpora mainly relies on manual comparison, but this method is inefficient because markdown format is not friendly to human reading, especially the markdown text in large model training corpora in the scientific field, which is parsed from PDF by OCR technologies such as Grobid and Nougat and has two major characteristics: one is the existence of OCR parsing errors, including additional auxiliary and useless content; the other is the inclusion of a large amount of rich text content such as tables and formulas, and the parsing rules are very complex. In response to these characteristics, post-processing is required after parsing to delete the identification content added by OCR and design an efficient and stable large file transmission method to avoid client-side lag. At the same time, since it is unrealistic to manually configure parsing rules, and there are no requirements for personalized rendering styles during evaluation, and high accuracy is required, it is also necessary to ensure the security of markdown parsing results and filter out unsafe content that may cause XSS attacks.

[0005] In addition, the accuracy of rich texts such as text, references, tables, formulas, etc. in the recognized, parsed, and rendered markdown content compared with the original PDF needs to be fed back by external personnel such as scientific experts and large model algorithm engineers to continuously improve the parsing and recognition algorithms. However, due to data confidentiality requirements, external personnel usually cannot directly access the data content. Therefore, it is necessary to compare the PDF original and the markdown parsing results, extract key areas of interest, and attach user information watermarks and corpus metadata information such as corpus title and corpus author to generate a corpus quality assessment result diagram, which can be shared externally in the form of a link.

[0006] In summary, large model training corpora in the scientific field in markdown format face many challenges in the data quality assessment stage: (a) The content is rich and large in size, which takes a long time to download, parse and render, and may cause browser lag and affect user experience; (b) Contains identifying content added by the OCR algorithm, which helps to identify rich text such as tables and formulas. However, it needs to be deleted after parsing and rendering, which increases the difficulty of post-processing. (c) Rich text content such as tables, formulas, lists, and references are not human-friendly and need to be accurately and securely restored to help evaluate the quality of large model training corpora; (d) The PDF original text and markdown parsing results of the corpus need to support the selection of the target area, generate the corpus quality assessment result map, and share it with external review. At the same time, data confidentiality must be guaranteed. Generating comparison maps for the entire area is slow and unnecessary, and it is easy to cause browser crashes.

[0007] Therefore, there is an urgent need for a complete visualization method and system for large model training corpus in the scientific field to solve the above problems and improve the efficiency and accuracy of data quality assessment. Summary of the invention

[0008] In view of the above, the purpose of the present invention is to provide a visualization method and system for large model training corpus in the scientific field, which can provide a complete, efficient and safe corpus visualization implementation solution for corpus quality assessment workers, reduce the cost and burden of corpus quality assessment, provide high-quality corpus data for large model training in the scientific field, and thus improve the accuracy of large models in the scientific field.

[0009] In the first aspect, in order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows: An embodiment of the present invention provides a visualization method for a large model training corpus in a scientific field, comprising the following steps: On the server side, the markdown format of the scientific field large model training corpus obtained from the object storage server is streamed to the client in segments; The client parses and renders the rich text content in the received markdown format of the scientific field large model training corpus to obtain the restoration result; Generate a comparison chart of the original PDF content and the restored results of the scientific field large model training corpus in markdown format on the client, and attach corpus metadata, user information watermark and corpus quality score to the comparison chart.

[0010] Specifically, the server side streams the large model training corpus in the markdown format obtained from the object storage server to the client in segments, including: Establish a persistent connection for sending data from the server to the client. The server downloads the markdown format of the scientific field large model training corpus from the object storage server in a streaming manner. Whenever the server receives a streamed content, it sends a new message event based on the text / event stream protocol to the client. The client listens to the new message event and identifies the event type as a message, then merges and saves the newly received streamed content into the historical received content. When the server encounters an error in receiving streaming content from the object storage server, it sends a new message event based on the text / event stream protocol to the client. The client listens to the new message event and identifies the event type as an error. Then, the client executes the transmission retry mechanism and re-executes the streaming content transmission. After the server receives the entire content of the large scientific model training corpus in markdown format, it sends a new message event based on the text / event stream protocol to the client. The client listens to the new message event and identifies the event type as end, closes the persistent connection, and releases resources.

[0011] Specifically, the client parses and renders the rich text content in the received markdown format scientific field large model training corpus to obtain the restoration result, including: Preprocess the received markdown format scientific large model training corpus; Based on the markdown parser, the preprocessed markdown text is parsed into a markdown abstract syntax tree represented by a JSON object, and the rich text content in the preprocessed markdown text is parsed into multiple sub-markdown abstract syntax trees. The position of each sub-markdown abstract syntax tree in the markdown abstract syntax tree is found and the content of the corresponding position node is replaced to obtain the updated markdown abstract syntax tree; Convert the updated markdown abstract syntax tree to an HTML abstract syntax tree based on the HTML parser, render the rich text content in the markdown text, build a child HTML abstract syntax tree for each rich text content, find the position of each child HTML abstract syntax tree in the HTML abstract syntax tree and replace the content of the corresponding position node to obtain the updated HTML abstract syntax tree; The updated HTML abstract syntax tree is converted into an HTML string, and the HTML string is rendered to the client to obtain a restored result.

[0012] Specifically, each node in the markdown abstract syntax tree contains four attributes: type, value, depth, and child node list. The relationships between nodes in the tree include sibling relationships represented by arrays in JSON and parent-child relationships represented by parent-child key-value pairs in JSON.

[0013] Specifically, the preprocessing includes: filtering and deleting the identification content added by the OCR algorithm when parsing the PDF original text into markdown corpus.

[0014] Specifically, the step of converting the updated markdown abstract syntax tree into an HTML abstract syntax tree based on an HTML parser includes: The entire updated markdown abstract syntax tree is traversed recursively, and each node in the updated markdown abstract syntax tree is converted into an HTML abstract syntax tree containing type, tag name, value, attribute, and child node list.

[0015] Specifically, the method of converting the updated markdown abstract syntax tree into an HTML abstract syntax tree based on an HTML parser further includes: Filter and delete tags and attributes in the HTML abstract syntax tree that may lead to cross-site scripting attacks.

[0016] Specifically, the method generates a comparison chart of the original PDF content of the scientific field large model training corpus in markdown format and the restored result on the client, and adds corpus metadata, user information watermark and corpus quality score to the comparison chart, including: Render the currently logged-in user information into the canvas to form a user information watermark, and regenerate the watermark when it is detected that the user information watermark is hidden or deleted; Recursively traverse the DOM tree of the entire HTML document of the restored result and parse it to obtain the DOM node position information; The PDF original is obtained and rendered to the client, and the user selects the area of ​​interest in the restored result and the PDF original; According to the DOM node position information and the area of ​​interest obtained by parsing, all DOM nodes whose DOM node positions are within the area of ​​interest are filtered out; Render all the filtered DOM nodes into the canvas; Query the corpus metadata information and render it to the canvas; The user draws a corpus quality score for the area of ​​interest, and converts the canvas including the selected restoration result and the area of ​​interest in the PDF original, user information watermark, corpus metadata information and corpus quality score into a comparison chart, and supports exporting the comparison chart for external viewing.

[0017] In a second aspect, to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a visualization system for a large model training corpus in a scientific field, which is implemented by the above-mentioned visualization method for a large model training corpus in a scientific field, and includes: a corpus data transmission module, a content analysis and rendering module, and a comparison chart generation module; The corpus data transmission module is used to stream the large model training corpus in the scientific field in markdown format obtained from the object storage server to the client in segments on the server side; The content parsing and rendering module is used to parse and render the rich text content in the received markdown format scientific field large model training corpus on the client to obtain a restoration result; The comparison chart generation module is used to generate a comparison chart of the PDF original content and the restored result of the scientific field large model training corpus in markdown format on the client, and add corpus metadata, user information watermark and corpus quality score to the comparison chart.

[0018] In the third aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention also provides a visualization device for large-model training corpus in the scientific field, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-mentioned visualization method for large-model training corpus in the scientific field when executing the computer program.

[0019] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention takes into account the characteristics of large-scale model training corpus in the scientific field, which is rich in content and large in volume. Based on the method that the server downloads in segments from the object storage server and transmits it to the client in real time in one direction, the markdown corpus data is efficiently downloaded and transmitted, avoiding the problem of manual operation required for paging loading and the problem of browser freeze caused by non-paged loading. In addition, an automatic retry mechanism for errors is added to achieve efficient and accurate data transmission.

[0020] (2) The present invention aims to solve the problem that large model training corpora in the scientific field contain a large amount of rich text and the markdown format is not friendly to human workers. By parsing and restoring the rich text in the markdown, and finally filtering and deleting the identification content and noise added by the OCR algorithm, it is convenient for corpus quality assessment workers to conduct online evaluation and comparison. In addition, the present invention ensures that the parsing results are safe as much as possible, avoids XSS attacks on the website, and ensures efficient and secure transmission and parsing of data.

[0021] (3) In view of the high confidentiality of large model training corpora in the scientific field, the present invention supports autonomous selection of regions of interest, autonomous drawing of corpus quality scores, automatic addition of corpus metadata information and user watermark information, and ensures that watermarks will not be manually deleted, making it easy for corpus quality assessors to share the assessment results with internal and external experts for review and feedback, and then continuously improve the OCR recognition algorithm based on the feedback, thereby generating higher quality, markdown-formatted large model training corpora, ultimately helping to improve the performance of large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 It is a flowchart of a visualization method for a large model training corpus in a scientific field provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of the process of corpus segment streaming transmission provided by an embodiment of the present invention; Figure 3 It is a flowchart of corpus recognition, parsing and rendering provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of a process for generating a corpus quality assessment result comparison chart provided by an embodiment of the present invention; Figure 5 It is a structural diagram of a visualization system for large-scale model training corpus in the scientific field provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0025] The inventive concept of the present invention is: in response to the many challenges faced by the markdown format scientific field large model training corpus in the data quality assessment stage in the prior art, such as low accuracy and efficiency, the embodiment of the present invention provides a visualization method and system for the scientific field large model training corpus. First, the markdown format scientific field large model training corpus is transmitted to the client in segments, and then the rich text in the markdown is recognized, parsed and rendered. Finally, a comparison chart of the original corpus and the parsed result is automatically generated, and the corpus metadata, user information watermark and corpus quality score are attached. It supports sharing the comparison chart with external experts to report and review the corpus quality. It is suitable for visualization and quality assessment application scenarios of various large model training corpora in the scientific field, such as geological large models, astronomical large models, biological large models and general large models that can empower scientific research fields.

[0026] Figure 1 Schematic diagram of the flow of the visualization method for large-scale model training corpus in the scientific field provided by the embodiment of the present invention. Figure 1 As shown, the embodiment provides a visualization method for large model training corpus in the scientific field, including the following steps: S1, on the server side, the markdown format of the large scientific model training corpus obtained from the object storage server is streamed to the client in segments.

[0027] The large model training corpus in the scientific field needs to be converted into a markdown format that is friendly to large models after data acquisition and data analysis steps, and the massive markdown corpus is usually stored in an object storage server. In the embodiment, a simplex streaming transmission method based on SSE (Server-sent events) is designed and implemented, which only allows the server to transmit data one-way to the client, such as Figure 2 As shown, the method downloads in segments from the object storage server and transmits the markdown file to the client in real time. The specific steps are as follows.

[0028] S1.1, build a segmented streaming server, start listening for the connection request event initiated by the segmented streaming client. Build a segmented streaming client, create multiple listeners to listen for different types of new message events. The client instantiates the EventSource interface (event source interface) and establishes a persistent connection with the server, so that the server can continuously send data to the client unilaterally.

[0029] S1.2, the segmented streaming client establishes a connection request to the segmented streaming client and starts communication. The segmented streaming server starts to request the scientific field large model training corpus in markdown format from the object storage server in a streaming download manner. The transmission data stream is a binary file stream.

[0030] S1.3, whenever the server receives a stream of content, it sends a new message event to the client. The response header is text / event-stream. The content of the data packet sent includes: (1) eventType, which is the agreed message type, indicating that text content will be sent in this message, and the client should save the content of this message sent by the server; (2) eventContent (event content), which is the content after the streaming content is converted into a string format.

[0031] S1.4, the client listens to new message events, identifies that eventType is message, merges and saves the newly received eventContent content into the historical received content, and renders it to the page.

[0032] S1.5, when the server encounters an error in receiving streaming content from the object storage server, it sends a new message event to the client. The response header is text / event-stream. The content of the data packet sent includes: eventType, which is a fixed error type, indicating that the segmented streaming client should execute the error retry mechanism.

[0033] S1.6, the client listens to the new message event, identifies eventType as error, then executes the transmission retry mechanism, clears the markdown data content that the client has saved, re-executes the current streaming content transmission, and displays a pop-up window on the page to prompt the retry. The pop-up window is closed when the retry succeeds or the maximum number of retries is reached. After waiting for the retry to succeed, save the new markdown data content resent by the segmented streaming server.

[0034] S1.7, after the server receives the entire content of the scientific field large model training corpus in markdown format, it sends a new message event to the client. The response header is text / event-stream, and the content of the data packet sent is: eventType event type, which is a fixed end type, indicating that the transmission is completed, and the segmented streaming client should disconnect.

[0035] S1.8, the client listens for new message events, identifies eventType as end, closes the persistent connection, and releases resources.

[0036] S2, parses and renders the rich text content in the received markdown format of the scientific field large model training corpus on the client to obtain the restoration result.

[0037] After receiving the scientific model training corpus in markdown format, the markdown text is accurately parsed and rendered into a restoration result consistent with the original PDF text using parsing and serialization technology, such as Figure 3 As shown, the specific steps are as follows.

[0038] S2.1, preprocess the received scientific field large model training corpus in markdown format. The preprocessing includes: filtering and deleting the identification content added by the OCR algorithm when parsing the PDF original text into markdown corpus. The identification content includes useless content, content that does not conform to the standard (such as content that does not conform to Latex writing rules), etc.

[0039] S2.2, based on the markdown parser (the embodiment uses the remark-parse plug-in), the preprocessed markdown text is parsed, including: reading the markdown text line by line, identifying identifiers such as titles, emphasis, hyperlinks, serial numbers, etc. in the markdown, and constructing a markdown abstract syntax tree represented by a JSON object layer by layer. Each node in the markdown abstract syntax tree contains four attributes: type, value, depth, and child node list. The relationship between nodes in the tree includes a sibling relationship represented by an array in JSON and a parent-child relationship represented by a parent-child key-value pair in JSON.

[0040] S2.3, based on multiple types of remark plug-ins (including remark-math plug-in for parsing mathematical formulas, remark-gfm plug-in for parsing tables, etc.), parse the rich text content such as tables, formulas, lists, pictures, references, etc. in the preprocessed markdown text, including: reading the markdown text line by line, identifying all rich text content such as inline formulas, interline formulas, tables, etc., parsing into multiple sub-markdown abstract syntax trees, each sub-markdown abstract syntax tree is consistent with the markdown abstract syntax tree representation. Traverse all sub-markdown abstract syntax trees, find the position of each sub-markdown abstract syntax tree in the markdown abstract syntax tree and replace the content of the corresponding position node to obtain the updated markdown abstract syntax tree.

[0041] S2.4, based on the HTML parser (the remark-rehype plug-in is used in the embodiment), the updated markdown abstract syntax tree is converted into an HTML abstract syntax tree, including: recursively traversing the entire updated markdown abstract syntax tree, and converting each node in the updated markdown abstract syntax tree into an HTML abstract syntax tree containing a type (type), a tag name (tagName), a value (value), a property (property) and a child node list (children).

[0042] S2.5, based on the rehype-sanitize plug-in, the tags and attributes in the HTML abstract syntax tree that may cause cross-site scripting (commonly known as XSS) are filtered and deleted to ensure that the generated HTML abstract syntax tree is safe. The tags that should be filtered include: <iframe>, tags such as <script> that may execute JavaScript scripts. Attributes that should be filtered include: event listener attributes of all tags (such as onmouseover (triggered when the mouse hovers), onclick (triggered when the mouse clicks), etc.), href attributes containing JavaScript scripts in tags, onerror (triggered when an error occurs) event listener attributes containing JavaScript scripts in tags, etc.

[0043] S2.6, Render the rich text content in the markdown text based on multiple types of rehype plugins (including the rehype-ketax plugin to render mathematical formulas, the rehype-toc plugin to render tables, the rehype-citation plugin to render cited references, the rehype-figure plugin to render images, etc.), including: Traverse the HTML abstract syntax tree in a recursive manner, detect whether the value of each node contains rich text content such as mathematical formulas, tables, chart annotations, etc. If so, construct a sub-HTML abstract syntax tree for each rich text. Traverse all sub-HTML abstract syntax trees, find the positions of each sub-HTML abstract syntax tree in the HTML abstract syntax tree and replace the content of the corresponding position nodes to obtain an updated HTML abstract syntax tree.

[0044] S2.7, Convert the updated HTML abstract syntax tree into an HTML string based on the rehype-stringify plugin.

[0045] S2.8, Render the HTML string as DOM elements on the client to obtain the restored result.

[0046] It should be noted that since the markdown content in the large model training corpus usually contains a large amount of rich text content such as formulas, tables, and charts, the parsing and rendering process of the corpus rich text content is very time-consuming. Frequent execution of this operation will cause the browser client to freeze and cannot respond to user operations in a timely manner. Therefore, in the method proposed and implemented in the embodiments of the present invention, after waiting for all markdown files to be streamed and transferred completely, the rich text content parsing and rendering operations are performed. This avoids the problem of browser freezing caused by parsing and rendering while receiving during the streaming transfer process.

[0047] S3, Generate a comparison graph of the PDF original content and the restored result of the scientific field large model training corpus in markdown format on the client, and attach corpus metadata, user information watermarks, and corpus quality scores to the comparison graph.

[0048] Due to the requirements of data confidentiality, non-internal relevant personnel are usually not allowed to directly access the massive large model training corpus data. It is necessary to combine the comparison screenshots of the original PDF text of the corpus and the markdown parsing results, as well as the corpus quality score, corpus metadata information, and user information evaluating the corpus, into a corpus quality evaluation result graph, and support sharing it with non-relevant personnel such as external experts. The implementation method of automatically generating the comparison graph of corpus quality evaluation results is shown in Figure 4, and the specific steps are as follows.

[0049] S3.1, Render the currently logged-in user information onto the canvas to form a user information watermark, and regenerate the watermark when it is detected that the user information watermark is hidden or deleted, including: obtaining user information, including user id, user name, and user unit; creating a canvas node covering the entire page size; rendering the user information onto the canvas node; adding a listening mechanism to regenerate the watermark when it is detected that the watermark is hidden or deleted.

[0050] S3.2, Recursively traverse the DOM tree of the entire HTML document of the restoration result and parse to obtain the DOM node position information.

[0051] S3.3, According to the method in step S1, download and transfer the original PDF text of the corpus from the object storage server to the client, and render the original PDF text of the corpus onto the page. The user uses the mouse to select the region of interest (ROI) in the restoration result and the original PDF text. This region usually contains rich text content such as a certain formula in the original PDF text of the corpus, as well as the corresponding restoration result of the markdown corpus of this rich text content such as the formula.

[0052] S3.4, Filter out all DOM nodes whose DOM node positions are within the region of interest according to the parsed DOM node position information and the region of interest.

[0053] S3.5, Recursively render all the filtered DOM nodes onto the canvas in the stacking order.

[0054] S3.6, Query the corpus metadata information including corpus title, corpus type, corpus source, etc. and render it onto the canvas.

[0055] S3.7. The user draws the corpus quality score for the region of interest, and converts the canvas including the restored result of the selected area, the region of interest in the PDF original text, the user information watermark, the corpus metadata information, and the corpus quality score into a comparison graph. It supports uploading the comparison graph to the object storage server, obtaining the comparison graph preview link from the object storage server, and sharing it with external users to view the corpus quality assessment result.

[0056] In summary, a visualization method for scientific domain large model training corpus provided by an embodiment of the present invention segments and streams the scientific domain large model training corpus in markdown format to the client, recognizes, parses, and renders the rich text in markdown, automatically generates a comparison graph of the corpus original text and the markdown parsing result, and attaches corpus metadata, user information watermark, and corpus quality score. It can efficiently, accurately, and securely transmit, parse, render, and share the scientific domain large model training corpus in markdown format, help large model workers quickly evaluate the quality of the training corpus online, share the corpus quality assessment result with internal and external experts, continuously collect expert feedback, and then help generate higher quality training corpus, ultimately providing a large amount of high-quality corpus for large model training and improving the performance of scientific domain large models.

[0057] Based on the same inventive concept, as shown in Figure 5, an embodiment of the present invention also provides a visualization system 500 for scientific domain large model training corpus, including: a corpus data transmission module 510, a content parsing and rendering module 520, and a comparison graph generation module 530.

[0058] The corpus data transmission module 510 is used to segment and stream the scientific domain large model training corpus in markdown format obtained from the object storage server to the client on the server side.

[0059] The content parsing and rendering module 520 is used to parse and render the rich text content in the scientific domain large model training corpus in markdown format received on the client to obtain the restored result.

[0060] The comparison graph generation module 530 is used to generate a comparison graph of the PDF original text content and the restored result of the scientific domain large model training corpus in markdown format on the client, and attach corpus metadata, user information watermark, and corpus quality score to the comparison graph.

[0061] Based on the same inventive concept, an embodiment of the present invention further provides a visualization device for the training corpus of a large model in the scientific field, including a memory and one or more processors. The memory is used to store a computer program, and the processor is used to implement the above-mentioned visualization method for the training corpus of a large model in the scientific field when executing the computer program.

[0062] It should be noted that the visualization system for the training corpus of a large model in the scientific field and the visualization device for the training corpus of a large model in the scientific field provided in the above embodiments both belong to the same inventive concept as the visualization method for the training corpus of a large model in the scientific field. For the specific implementation process, please refer to the embodiments of the visualization method for the training corpus of a large model in the scientific field, which will not be elaborated here.

[0063] The above-mentioned specific implementation manners have elaborated in detail the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, supplements, equivalent replacements, etc. made within the scope of the principles of the present invention shall be included in the protection scope of the present invention.< / iframe>

Claims

1. A visualization method for large model training corpus in the scientific field, characterized in that: The following steps are involved: On the server side, the markdown format of the scientific field large model training corpus obtained from the object storage server is streamed to the client in segments; The client parses and renders the rich text content in the received markdown format of the scientific field large model training corpus to obtain the restoration result; Generate a comparison chart of the original PDF content and the restored results of the scientific field large model training corpus in markdown format on the client, and attach corpus metadata, user information watermark and corpus quality score to the comparison chart.

2. The visualization method for large-scale model training corpus in the scientific field according to claim 1, characterized in that: The server side streams the large model training corpus in the markdown format obtained from the object storage server to the client in segments, including: Establish a persistent connection for sending data from the server to the client. The server downloads the markdown format of the scientific field large model training corpus from the object storage server in a streaming manner. Whenever the server receives a streamed content, it sends a new message event based on the text / event stream protocol to the client. The client listens to the new message event and identifies the event type as a message, then merges and saves the newly received streamed content into the historical received content. When the server encounters an error in receiving streaming content from the object storage server, it sends a new message event based on the text / event stream protocol to the client. The client listens to the new message event and identifies the event type as an error. Then, the client executes the transmission retry mechanism and re-executes the streaming content transmission. After the server receives the entire content of the large scientific model training corpus in markdown format, it sends a new message event based on the text / event stream protocol to the client. The client listens to the new message event and identifies the event type as end, closes the persistent connection, and releases resources.

3. The visualization method for large-scale model training corpus in the scientific field according to claim 1, characterized in that: The client parses and renders the rich text content in the received markdown format scientific field large model training corpus to obtain the restoration result, including: Preprocess the received markdown format scientific large model training corpus; Based on the markdown parser, the preprocessed markdown text is parsed into a markdown abstract syntax tree represented by a JSON object, and the rich text content in the preprocessed markdown text is parsed into multiple sub-markdown abstract syntax trees. The position of each sub-markdown abstract syntax tree in the markdown abstract syntax tree is found and the content of the corresponding position node is replaced to obtain the updated markdown abstract syntax tree; Convert the updated markdown abstract syntax tree to an HTML abstract syntax tree based on the HTML parser, render the rich text content in the markdown text, build a child HTML abstract syntax tree for each rich text content, find the position of each child HTML abstract syntax tree in the HTML abstract syntax tree and replace the content of the corresponding position node to obtain the updated HTML abstract syntax tree; The updated HTML abstract syntax tree is converted into an HTML string, and the HTML string is rendered to the client to obtain a restored result.

4. The visualization method for large-scale model training corpus in the scientific field according to claim 3 is characterized in that: Each node in the markdown abstract syntax tree contains four attributes: type, value, depth, and child node list. The relationships between nodes in the tree include sibling relationships represented by arrays in JSON and parent-child relationships represented by parent-child key-value pairs in JSON.

5. The visualization method for large-scale model training corpus in the scientific field according to claim 3 is characterized in that: Preprocessing includes: Filter and remove the identifying content added by the OCR algorithm when parsing the PDF source into markdown corpus.

6. The visualization method for large-scale model training corpus in scientific fields according to claim 3, characterized in that: The HTML parser-based method converts the updated markdown abstract syntax tree into an HTML abstract syntax tree, including: The entire updated markdown abstract syntax tree is traversed recursively, and each node in the updated markdown abstract syntax tree is converted into an HTML abstract syntax tree containing type, tag name, value, attribute, and child node list.

7. The visualization method for large model training corpus in scientific fields according to claim 3 or 6, characterized in that: The method of converting the updated markdown abstract syntax tree into an HTML abstract syntax tree based on an HTML parser further includes: Filter and delete tags and attributes in the HTML abstract syntax tree that may lead to cross-site scripting attacks.

8. The visualization method for large-scale model training corpus in the scientific field according to claim 1, characterized in that: The method generates a comparison chart of the original PDF content of the scientific field large model training corpus in markdown format and the restored result on the client, and adds corpus metadata, user information watermark and corpus quality score to the comparison chart, including: Render the currently logged-in user information into the canvas to form a user information watermark, and regenerate the watermark when it is detected that the user information watermark is hidden or deleted; Recursively traverse the DOM tree of the entire HTML document of the restored result and parse it to obtain the DOM node position information; The original PDF is obtained and rendered to the client, and the user selects the area of ​​interest in the restored result and the original PDF; According to the DOM node position information and the area of ​​interest obtained by parsing, all DOM nodes whose DOM node positions are within the area of ​​interest are filtered out; Render all the filtered DOM nodes into the canvas; Query the corpus metadata information and render it to the canvas; The user draws a corpus quality score for the area of ​​interest, and converts the canvas including the selected restoration result and the area of ​​interest in the PDF original, user information watermark, corpus metadata information and corpus quality score into a comparison chart, and supports exporting the comparison chart for external viewing.

9. A visualization system for large model training corpus in the scientific field, implemented by the visualization method for large model training corpus in the scientific field according to any one of claims 1 to 8, characterized in that: include: Corpus data transmission module, content parsing and rendering module, and comparison chart generation module; The corpus data transmission module is used to stream the large model training corpus in the scientific field in markdown format obtained from the object storage server to the client in segments on the server side; The content parsing and rendering module is used to parse and render the rich text content in the received markdown format scientific field large model training corpus on the client to obtain a restoration result; The comparison chart generation module is used to generate a comparison chart of the PDF original content and the restored result of the scientific field large model training corpus in markdown format on the client, and add corpus metadata, user information watermark and corpus quality score to the comparison chart.

10. A visualization device for large model training corpus in the scientific field, comprising a memory and one or more processors, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the visualization method for large model training corpus in the scientific field as described in any one of claims 1 to 8 when executing the computer program.

Citation Information

Patent Citations

  • Project document review system and method based on artificial intelligence technology

    CN116703337A

  • Scientific field-oriented multi-modal corpus data construction method and device

    CN118170933A

  • Automatic flow chart generation system and method based on text analysis

    CN118643821A

  • Data visualization analysis processing method, chip and terminal

    CN118733662A

  • Vital text analytics system for the enhancement of requirements engineering documents and other documents

    US20140172417A1

Cited By

  • Data rendering method, related device and computer program product

    CN120874812A

  • Document analysis result evaluation method and device and electronic equipment

    CN121435963A