A visualization method and system for large model training corpus in scientific fields

Abstract

The present invention discloses a visualization method and system for training corpus of large models in the scientific field, including: streaming the training corpus of large models in the scientific field in markdown format obtained from an object storage server to a client in segments at a server end; parsing and rendering the rich text content in the received markdown text at a client end to obtain a restoration result; generating a comparison chart of the PDF original content of the markdown text and the restoration result at a client end, and attaching corpus metadata, user information watermark and corpus quality score to the comparison chart. The present invention can efficiently, accurately and safely transmit, parse, render and share the training corpus of large models in the scientific field in markdown format, and share the corpus quality assessment results to internal and external experts, thereby helping the training of large models in the scientific field to provide high-quality corpus data and improve the accuracy of large models in the scientific field.

Description

Technical Field

[0001] The present invention belongs to the technical field of data visualization, and in particular relates to a visualization method and system for large-scale model training corpus in the scientific field. Background Art

[0002] In recent years, the application of large language models in the scientific field (AI4Science, covering geology, astronomy, biology, etc.) has rapidly emerged, becoming a key force in promoting scientific research and discovery. Its capabilities mainly depend on three major factors: data, computing power, and algorithms. Among them, data is the core and basic factor. The quality and quantity of data are crucial to improving the capabilities of large models.

[0003] The training data for large models in the scientific field usually comes from PDF documents containing rich text content (such as text, tables, formulas, pictures, lists, references, etc.). However, large models cannot directly interpret the content of PDF documents and need to rely on OCR technology to convert them into markdown format corpus that is easy for large model training. This conversion process needs to ensure the quality and accuracy of the markdown format corpus to maximize the performance of the model and avoid negative effects.

[0004] At present, the evaluation of markdown format training corpus mainly relies on manual comparison, but this method is inefficient because the markdown format is not friendly to human reading, especially the markdown text in the large model training corpus in the scientific field. They are parsed from PDF by OCR technologies such as Grobid and Nougat, and have two major characteristics: one is that there are OCR parsing errors, including additional auxiliary and useless content; the other is that they contain a large number of rich text contents such as tables and formulas, and the parsing rules are very complex. In view of these characteristics, post-processing is required after the parsing is completed to delete the identification content added by OCR, and to design an efficient and stable large file transmission method to avoid client jams. At the same time, since it is unrealistic to manually configure the parsing rules, and there is no requirement for personalized rendering style during evaluation, the accuracy requirement is high, and it is also necessary to ensure the security of the markdown parsing results and filter out unsafe content that may trigger XSS attacks.

[0005] In addition, the accuracy of rich texts such as text, references, tables, formulas, etc. in the recognized, parsed, and rendered markdown content compared with the original PDF needs to be fed back by external personnel such as scientific experts and large model algorithm engineers to continuously improve the parsing and recognition algorithms. However, due to data confidentiality requirements, external personnel usually cannot directly access the data content. Therefore, it is necessary to compare the PDF original and the markdown parsing results, extract key areas of interest, and attach user information watermarks and corpus metadata information such as corpus title and corpus author to generate a corpus quality assessment result diagram, which can be shared externally in the form of a link.

[0006] In summary, large model training corpora in the scientific field in markdown format face many challenges in the data quality assessment stage:

[0007] (a) The content is rich and large in size, which takes a long time to download, parse and render, and may cause browser lag and affect user experience;

[0008] (b) Contains identifying content added by the OCR algorithm, which helps to identify rich text such as tables and formulas. However, it needs to be deleted after parsing and rendering, which increases the difficulty of post-processing.

[0009] (c) Rich text content such as tables, formulas, lists, and references are not human-friendly and need to be accurately and securely restored to help evaluate the quality of large model training corpora;

[0010] (d) The PDF original text and markdown parsing results of the corpus need to support the selection of the target area, generate the corpus quality assessment result map, and share it with external review. At the same time, data confidentiality must be guaranteed. Generating comparison maps for the entire area is slow and unnecessary, and it is easy to cause browser crashes.

[0011] Therefore, there is an urgent need for a complete visualization method and system for large model training corpus in the scientific field to solve the above problems and improve the efficiency and accuracy of data quality assessment. Summary of the invention

[0012] In view of the above, the purpose of the present invention is to provide a visualization method and system for large model training corpus in the scientific field, which can provide a complete, efficient and safe corpus visualization implementation solution for corpus quality assessment workers, reduce the cost and burden of corpus quality assessment, provide high-quality corpus data for large model training in the scientific field, and thus improve the accuracy of large models in the scientific field.

[0013] In the first aspect, in order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0014] An embodiment of the present invention provides a visualization method for a large model training corpus in a scientific field, comprising the following steps:

[0015] On the server side, the markdown format of the scientific field large model training corpus obtained from the object storage server is streamed to the client in segments;

[0016] The client parses and renders the rich text content in the received markdown format of the scientific field large model training corpus to obtain the restoration result;

[0017] Generate a comparison chart of the original PDF content and the restored results of the scientific field large model training corpus in markdown format on the client, and attach corpus metadata, user information watermark and corpus quality score to the comparison chart.

[0018] Specifically, the server side streams the large model training corpus in the markdown format obtained from the object storage server to the client in segments, including:

[0019] Establish a persistent connection for sending data from the server to the client. The server downloads the markdown format of the scientific field large model training corpus from the object storage server in a streaming manner.

[0020] Whenever the server receives a streamed content, it sends a new message event based on the text / event stream protocol to the client. The client listens to the new message event and identifies the event type as a message, then merges and saves the newly received streamed content into the historical received content.

[0021] When the server encounters an error in receiving streaming content from the object storage server, it sends a new message event based on the text / event stream protocol to the client. The client listens to the new message event and identifies the event type as an error. Then, the client executes the transmission retry mechanism and re-executes the streaming content transmission.

[0022] After the server receives the entire content of the large scientific model training corpus in markdown format, it sends a new message event based on the text / event stream protocol to the client. The client listens to the new message event and identifies the event type as end, closes the persistent connection, and releases resources.

[0023] Specifically, the client parses and renders the rich text content in the received markdown format scientific field large model training corpus to obtain the restoration result, including:

[0024] Preprocess the received markdown format scientific large model training corpus;

[0025] Based on the markdown parser, the preprocessed markdown text is parsed into a markdown abstract syntax tree represented by a JSON object, and the rich text content in the preprocessed markdown text is parsed into multiple sub-markdown abstract syntax trees. The position of each sub-markdown abstract syntax tree in the markdown abstract syntax tree is found and the content of the corresponding position node is replaced to obtain the updated markdown abstract syntax tree;

[0026] Convert the updated markdown abstract syntax tree to an HTML abstract syntax tree based on the HTML parser, render the rich text content in the markdown text, build a child HTML abstract syntax tree for each rich text content, find the position of each child HTML abstract syntax tree in the HTML abstract syntax tree and replace the content of the corresponding position node to obtain the updated HTML abstract syntax tree;

[0027] The updated HTML abstract syntax tree is converted into an HTML string, and the HTML string is rendered to the client to obtain a restored result.

[0028] Specifically, each node in the markdown abstract syntax tree contains four attributes: type, value, depth, and child node list. The relationships between nodes in the tree include sibling relationships represented by arrays in JSON and parent-child relationships represented by parent-child key-value pairs in JSON.

[0029] Specifically, the preprocessing includes: filtering and deleting the identification content added by the OCR algorithm in the process of parsing the PDF original text into markdown corpus.

[0030] Specifically, the step of converting the updated markdown abstract syntax tree into an HTML abstract syntax tree based on an HTML parser includes:

[0031] The entire updated markdown abstract syntax tree is traversed recursively, and each node in the updated markdown abstract syntax tree is converted into an HTML abstract syntax tree containing type, tag name, value, attribute, and child node list.

[0032] Specifically, the method of converting the updated markdown abstract syntax tree into an HTML abstract syntax tree based on an HTML parser further includes:

[0033] Filter and delete tags and attributes in the HTML abstract syntax tree that may lead to cross-site scripting attacks.

[0034] Specifically, the method generates a comparison chart of the original PDF content of the scientific field large model training corpus in markdown format and the restored result on the client, and adds corpus metadata, user information watermark and corpus quality score to the comparison chart, including:

[0035] Render the currently logged-in user information into the canvas to form a user information watermark, and regenerate the watermark when it is detected that the user information watermark is hidden or deleted;

[0036] Recursively traverse the DOM tree of the entire HTML document of the restored result and parse it to obtain the DOM node position information;

[0037] The original PDF is obtained and rendered to the client, and the user selects the area of ​​interest in the restored result and the original PDF;

[0038] According to the DOM node position information and the area of ​​interest obtained by parsing, all DOM nodes whose DOM node positions are within the area of ​​interest are filtered out;

[0039] Render all the filtered DOM nodes into the canvas;

[0040] Query the corpus metadata information and render it to the canvas;

[0041] The user draws a corpus quality score for the area of ​​interest, and converts the canvas including the selected restoration result and the area of ​​interest in the PDF original, user information watermark, corpus metadata information and corpus quality score into a comparison chart, and supports exporting the comparison chart for external viewing.

[0042] In a second aspect, in order to achieve the above-mentioned invention object, an embodiment of the present invention further provides a visualization system for a large model training corpus in a scientific field, which is implemented by the above-mentioned visualization method for a large model training corpus in a scientific field, and includes: a corpus data transmission module, a content analysis and rendering module, and a comparison chart generation module;

[0043] The corpus data transmission module is used to stream the large model training corpus in the scientific field in markdown format obtained from the object storage server to the client in segments on the server side;

[0044] The content parsing and rendering module is used to parse and render the rich text content in the received markdown format scientific field large model training corpus on the client to obtain a restoration result;

[0045] The comparison chart generation module is used to generate a comparison chart of the PDF original content and the restored result of the scientific field large model training corpus in markdown format on the client, and add corpus metadata, user information watermark and corpus quality score to the comparison chart.

[0046] In the third aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention also provides a visualization device for large-model training corpus in the scientific field, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-mentioned visualization method for large-model training corpus in the scientific field when executing the computer program.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) The present invention takes into account the characteristics of large-scale model training corpus in the scientific field, which is rich in content and large in volume. Based on the method that the server downloads in segments from the object storage server and transmits it to the client in real time in one direction, the markdown corpus data is efficiently downloaded and transmitted, avoiding the problem of manual operation required for paging loading and the problem of browser freeze caused by non-paged loading. In addition, an automatic retry mechanism for errors is added to achieve efficient and accurate data transmission.

[0049] (2) The present invention aims to solve the problem that large model training corpora in the scientific field contain a large amount of rich text and the markdown format is not friendly to human workers. By parsing and restoring the rich text in the markdown, and finally filtering and deleting the identification content and noise added by the OCR algorithm, it is convenient for corpus quality assessment workers to conduct online evaluation and comparison. In addition, the present invention ensures that the parsing results are safe as much as possible, avoids XSS attacks on the website, and ensures efficient and secure transmission and parsing of data.

[0050] (3) In view of the high confidentiality of large model training corpora in the scientific field, the present invention supports autonomous selection of regions of interest, autonomous drawing of corpus quality scores, automatic addition of corpus metadata information and user watermark information, and ensures that watermarks will not be manually deleted, making it easy for corpus quality assessors to share the assessment results with internal and external experts for review and feedback, and then continuously improve the OCR recognition algorithm based on the feedback, thereby generating higher quality, markdown-formatted large model training corpora, ultimately helping to improve the performance of large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 It is a flowchart of a visualization method for a large model training corpus in a scientific field provided by an embodiment of the present invention;

[0053] Figure 2 It is a schematic diagram of the process of corpus segment streaming transmission provided by an embodiment of the present invention;

[0054] Figure 3 It is a flowchart of corpus recognition, parsing and rendering provided by an embodiment of the present invention;

[0055] Figure 4 It is a schematic diagram of a process for generating a corpus quality assessment result comparison chart provided by an embodiment of the present invention;

[0056] Figure 5 It is a structural diagram of a visualization system for large-scale model training corpus in the scientific field provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0058] The inventive concept of the present invention is: in response to the many challenges faced by the markdown format scientific field large model training corpus in the data quality assessment stage in the prior art, such as low accuracy and efficiency, the embodiment of the present invention provides a visualization method and system for the scientific field large model training corpus. First, the markdown format scientific field large model training corpus is transmitted to the client in segments, and then the rich text in the markdown is recognized, parsed and rendered. Finally, a comparison chart of the original corpus and the parsed result is automatically generated, and the corpus metadata, user information watermark and corpus quality score are attached. It supports sharing the comparison chart with external experts to report and review the corpus quality. It is suitable for visualization and quality assessment application scenarios of various large model training corpora in the scientific field, such as geological large models, astronomical large models, biological large models and general large models that can empower scientific research fields.

[0059] Figure 1 Schematic diagram of the flow of the visualization method for large-scale model training corpus in the scientific field provided by the embodiment of the present invention. Figure 1 As shown, the embodiment provides a visualization method for large model training corpus in the scientific field, including the following steps:

[0060] S1, on the server side, the markdown format of the large scientific model training corpus obtained from the object storage server is streamed to the client in segments.

[0061] The large model training corpus in the scientific field needs to be converted into a markdown format that is friendly to large models after data acquisition and data analysis steps, and the massive markdown corpus is usually stored in an object storage server. In the embodiment, a simplex streaming transmission method based on SSE (Server-sent events) is designed and implemented, which only allows the server to transmit data one-way to the client, such as Figure 2 As shown, the method downloads in segments from the object storage server and transmits the markdown file to the client in real time. The specific steps are as follows.

[0062] S1.1, build a segmented streaming server, start listening for the connection request event initiated by the segmented streaming client. Build a segmented streaming client, create multiple listeners to listen for different types of new message events. The client instantiates the EventSource interface (event source interface) and establishes a persistent connection with the server, so that the server can continuously send data to the client unilaterally.

[0063] S1.2, the segmented streaming client establishes a connection request to the segmented streaming client and starts communication. The segmented streaming server starts to request the scientific field large model training corpus in markdown format from the object storage server in a streaming download manner. The transmission data stream is a binary file stream.

[0064] S1.3, whenever the server receives a stream of content, it sends a new message event to the client. The response header is text / event-stream. The content of the data packet sent includes:

[0065] (1) eventType, which is the agreed message type, indicating that text content will be sent in this message, and the client should save the content of this message sent by the server;

[0066] (2) eventContent (event content), which is the content after the streaming content is converted into a string format.

[0067] S1.4, the client listens to new message events, identifies that eventType is message, merges and saves the newly received eventContent content into the historical received content, and renders it to the page.

[0068] S1.5, when the server encounters an error in receiving streaming content from the object storage server, it sends a new message event to the client. The response header is text / event-stream. The content of the data packet sent includes: eventType, which is a fixed error type, indicating that the segmented streaming client should execute the error retry mechanism.

[0069] S1.6, the client listens to the new message event, identifies eventType as error, then executes the transmission retry mechanism, clears the markdown data content that the client has saved, re-executes the current streaming content transmission, and displays a pop-up window on the page to prompt the retry. The pop-up window is closed when the retry succeeds or the maximum number of retries is reached. After waiting for the retry to succeed, save the new markdown data content resent by the segmented streaming server.

[0070] S1.7, after the server receives the entire content of the scientific field large model training corpus in markdown format, it sends a new message event to the client. The response header is text / event-stream, and the content of the data packet sent is: eventType event type, which is a fixed end type, indicating that the transmission is completed, and the segmented streaming client should disconnect.

[0071] S1.8, the client listens for new message events, identifies eventType as end, closes the persistent connection, and releases resources.

[0072] S2, parses and renders the rich text content in the received markdown format of the scientific field large model training corpus on the client to obtain the restoration result.

[0073] After receiving the scientific model training corpus in markdown format, the markdown text is accurately parsed and rendered into a restoration result consistent with the original PDF text using parsing and serialization technology, such as Figure 3 As shown, the specific steps are as follows.

[0074] S2.1, preprocess the received scientific field large model training corpus in markdown format. The preprocessing includes: filtering and deleting the identification content added by the OCR algorithm when parsing the PDF original text into markdown corpus. The identification content includes useless content, content that does not conform to the standard (such as content that does not conform to Latex writing rules), etc.

[0075] S2.2, based on the markdown parser (the embodiment uses the remark-parse plug-in), the preprocessed markdown text is parsed, including: reading the markdown text line by line, identifying identifiers such as titles, emphasis, hyperlinks, serial numbers, etc. in the markdown, and constructing a markdown abstract syntax tree represented by a JSON object layer by layer. Each node in the markdown abstract syntax tree contains four attributes: type, value, depth, and child node list. The relationship between nodes in the tree includes a sibling relationship represented by an array in JSON and a parent-child relationship represented by a parent-child key-value pair in JSON.

[0076] S2.3, based on multiple types of remark plug-ins (including remark-math plug-in for parsing mathematical formulas, remark-gfm plug-in for parsing tables, etc.), parse the rich text content such as tables, formulas, lists, pictures, references, etc. in the preprocessed markdown text, including: reading the markdown text line by line, identifying all rich text content such as inline formulas, interline formulas, tables, etc., parsing into multiple sub-markdown abstract syntax trees, each sub-markdown abstract syntax tree is consistent with the markdown abstract syntax tree representation. Traverse all sub-markdown abstract syntax trees, find the position of each sub-markdown abstract syntax tree in the markdown abstract syntax tree and replace the content of the corresponding position node to obtain the updated markdown abstract syntax tree.

[0077] S2.4, based on the HTML parser (the remark-rehype plug-in is used in the embodiment), the updated markdown abstract syntax tree is converted into an HTML abstract syntax tree, including: recursively traversing the entire updated markdown abstract syntax tree, and converting each node in the updated markdown abstract syntax tree into an HTML abstract syntax tree containing a type (type), a tag name (tagName), a value (value), a property (property) and a child node list (children).

[0078] S2.5, based on the rehype-sanitize plug-in, the tags and attributes in the HTML abstract syntax tree that may cause cross-site scripting (commonly known as XSS) are filtered and deleted to ensure that the generated HTML abstract syntax tree is safe. The tags that should be filtered include: <iframe>, tags such as <script> that may execute JavaScript scripts. Attributes that should be filtered include: event listener attributes of all tags (such as onmouseover (triggered when the mouse hovers), onclick (triggered when the mouse clicks), etc.), href attributes containing JavaScript scripts in tags, onerror (triggered when an error occurs) event listener attributes containing JavaScript scripts in tags, etc.

[0079] S2.6, Render the rich text content in the markdown text based on multiple types of rehype plugins (including the rehype-ketax plugin for rendering mathematical formulas, the rehype-toc plugin for rendering tables, the rehype-citation plugin for rendering cited references, the rehype-figure plugin for rendering images, etc.), including: Traverse the HTML abstract syntax tree recursively, detect whether the value of each node contains rich text content such as mathematical formulas, tables, chart annotations, etc. If so, construct a sub-HTML abstract syntax tree for each rich text. Traverse all sub-HTML abstract syntax trees, find the positions of each sub-HTML abstract syntax tree in the HTML abstract syntax tree and replace the content of the corresponding position nodes to obtain an updated HTML abstract syntax tree.

[0080] S2.7, Convert the updated HTML abstract syntax tree to an HTML string based on the rehype-stringify plugin.

[0081] S2.8, Render the HTML string as DOM elements on the client to obtain the restored result.

[0082] It should be noted that since the markdown content in the large model training corpus usually contains a large number of rich text contents such as formulas, tables, and charts, the parsing and rendering process of the corpus rich text content is very time-consuming. Frequent execution of this operation will cause the browser client to freeze and cannot respond to user operations in a timely manner. Therefore, in the method proposed and implemented in this embodiment of the present invention, after waiting for all markdown files to be streamed and transmitted completely, the rich text content parsing and rendering operations are performed. This avoids the problem of browser freezing caused by parsing and rendering while receiving during the streaming transmission process.

[0083] S3, Generate a comparison graph of the PDF original content and the restored result of the scientific field large model training corpus in markdown format on the client, and attach corpus metadata, user information watermark, and corpus quality score to the comparison graph.

[0084] Due to the requirements of data confidentiality, non-internal relevant personnel are usually not allowed to directly access the massive large model training corpus data. It is necessary to combine the comparison screenshots of the original corpus PDF and the markdown parsing results, as well as the corpus quality score, corpus metadata information, and user information for evaluating the corpus, into a corpus quality assessment result graph, and support sharing it with non-relevant personnel such as external experts. The implementation method for automatically generating the comparison graph of corpus quality assessment results is as shown in Figure 4, and the specific steps are as follows.

[0085] S3.1, Render the currently logged-in user information onto the canvas to form a user information watermark, and regenerate the watermark when it is detected that the user information watermark is hidden or deleted, including: obtaining user information, including user id, user name, and user unit; creating a canvas node that covers the entire page size; rendering the user information onto the canvas node; adding a monitoring mechanism to regenerate the watermark when it is detected that the watermark is hidden or deleted.

[0086] S3.2, Recursively traverse the DOM tree of the entire HTML document of the restoration result and parse to obtain the DOM node position information.

[0087] S3.3, Download and transfer the original PDF corpus from the object storage server to the client according to the method in step S1, and render the original PDF corpus onto the page. The user uses the mouse to select the region of interest (ROI) in the restoration result and the original PDF. This region usually contains rich text content such as a certain formula in the original PDF corpus, as well as the corresponding restoration result of the markdown corpus of this rich text content such as the formula.

[0088] S3.4, According to the parsed DOM node position information and the region of interest, filter out all DOM nodes whose DOM node positions are within the region of interest.

[0089] S3.5, Recursively render all the filtered DOM nodes onto the canvas in the stacking order.

[0090] S3.6, Query the corpus metadata information including corpus title, corpus type, corpus source, etc. and render it onto the canvas.

[0091] S3.7. The user draws a corpus quality score for the region of interest and converts the canvas that includes the restored result of the selected area, the region of interest in the PDF original text, the user information watermark, the corpus metadata information, and the corpus quality score into a comparison graph. It supports uploading the comparison graph to the object storage server, obtaining the comparison graph preview link from the object storage server, and sharing it with external users to view the corpus quality assessment result.

[0092] In summary, a visualization method for scientific field large model training corpus provided by an embodiment of the present invention can efficiently, accurately, and securely transmit, parse, render, and share the scientific field large model training corpus in markdown format by streaming the markdown format scientific field large model training corpus in segments to the client, identifying, parsing, and rendering the rich text in markdown, automatically generating a comparison graph of the corpus original text and the markdown parsing result, and attaching corpus metadata, user information watermark, and corpus quality score. It helps large model workers quickly evaluate the quality of training corpus online, share the corpus quality assessment result with internal and external experts, continuously collect expert feedback, and then help generate higher quality training corpus, ultimately providing a large amount of high-quality corpus for large model training and improving the performance of scientific field large models.

[0093] Based on the same inventive concept, as shown in Figure 5, an embodiment of the present invention also provides a visualization system 500 for scientific field large model training corpus, including: a corpus data transmission module 510, a content parsing and rendering module 520, and a comparison graph generation module 530.

[0094] The corpus data transmission module 510 is used to stream the markdown format scientific field large model training corpus obtained from the object storage server in segments to the client at the server side.

[0095] The content parsing and rendering module 520 is used to parse and render the rich text content in the received markdown format scientific field large model training corpus at the client to obtain the restored result.

[0096] The comparison graph generation module 530 is used to generate a comparison graph of the PDF original text content and the restored result of the markdown format scientific field large model training corpus at the client, and attach corpus metadata, user information watermark, and corpus quality score to the comparison graph.

[0097] Based on the same inventive concept, an embodiment of the present invention also provides a visualization device for large model training corpus in the scientific field, including a memory and one or more processors, the memory is used to store computer programs, and the processor is used to implement the above-mentioned visualization method for large model training corpus in the scientific field when executing the computer program.

[0098] It should be noted that the visualization system for the large model training corpus in the scientific field and the visualization device for the large model training corpus in the scientific field provided in the above embodiments belong to the same inventive concept as the visualization method for the large model training corpus in the scientific field. The specific implementation process is detailed in the embodiment of the visualization method for the large model training corpus in the scientific field, which will not be repeated here.

[0099] The specific implementation methods described above have been described in detail for the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention shall be included in the protection scope of the present invention.< / iframe>

Claims

1. A visualization method for large model training corpus in the scientific field, characterized in that: The following steps are involved: On the server side, the markdown format of the scientific field large model training corpus obtained from the object storage server is streamed to the client in segments; The client parses and renders the rich text content in the received markdown format of the scientific field large model training corpus to obtain the restoration result; Generate a comparison chart of the original PDF content and the restored result of the scientific field large model training corpus in markdown format on the client, and attach corpus metadata, user information watermark and corpus quality score to the comparison chart, including: Render the currently logged-in user information into the canvas to form a user information watermark, and regenerate the watermark when it is detected that the user information watermark is hidden or deleted; Recursively traverse the DOM tree of the entire HTML document of the restored result and parse it to obtain the DOM node position information; The original PDF is obtained and rendered to the client, and the user selects the area of ​​interest in the restored result and the original PDF; According to the DOM node position information and the area of ​​interest obtained by parsing, all DOM nodes whose DOM node positions are within the area of ​​interest are filtered out; Render all the filtered DOM nodes into the canvas; Query the corpus metadata information and render it to the canvas; The user draws a corpus quality score for the area of ​​interest, and converts the canvas including the selected restoration result and the area of ​​interest in the PDF original, user information watermark, corpus metadata information and corpus quality score into a comparison chart, and supports exporting the comparison chart for external viewing.

2. The visualization method for large-scale model training corpus in the scientific field according to claim 1, characterized in that: The server side streams the large model training corpus in the markdown format obtained from the object storage server to the client in segments, including: Establish a persistent connection for sending data from the server to the client. The server downloads the markdown format of the scientific field large model training corpus from the object storage server in a streaming manner. Whenever the server receives a streamed content, it sends a new message event based on the text / event stream protocol to the client. The client listens to the new message event and identifies the event type as a message, then merges and saves the newly received streamed content into the historical received content. When the server encounters an error in receiving streaming content from the object storage server, it sends a new message event based on the text / event stream protocol to the client. The client listens to the new message event and identifies the event type as an error. Then, the client executes the transmission retry mechanism and re-executes the streaming content transmission. After the server receives the entire content of the large scientific model training corpus in markdown format, it sends a new message event based on the text / event stream protocol to the client. The client listens to the new message event and identifies the event type as end, closes the persistent connection, and releases resources.

3. The visualization method for large-scale model training corpus in the scientific field according to claim 1, characterized in that: The client parses and renders the rich text content in the received markdown format scientific field large model training corpus to obtain the restoration result, including: Preprocess the received markdown format scientific large model training corpus; Based on the markdown parser, the preprocessed markdown text is parsed into a markdown abstract syntax tree represented by a JSON object, and the rich text content in the preprocessed markdown text is parsed into multiple sub-markdown abstract syntax trees. The position of each sub-markdown abstract syntax tree in the markdown abstract syntax tree is found and the content of the corresponding position node is replaced to obtain the updated markdown abstract syntax tree; Convert the updated markdown abstract syntax tree to an HTML abstract syntax tree based on the HTML parser, render the rich text content in the markdown text, build a child HTML abstract syntax tree for each rich text content, find the position of each child HTML abstract syntax tree in the HTML abstract syntax tree and replace the content of the corresponding position node to obtain the updated HTML abstract syntax tree; The updated HTML abstract syntax tree is converted into an HTML string, and the HTML string is rendered to the client to obtain a restored result.

4. The visualization method for large-scale model training corpus in the scientific field according to claim 3 is characterized in that: Each node in the markdown abstract syntax tree contains four attributes: type, value, depth, and child node list. The relationships between nodes in the tree include sibling relationships represented by arrays in JSON and parent-child relationships represented by parent-child key-value pairs in JSON.

5. The visualization method for large-scale model training corpus in the scientific field according to claim 3 is characterized in that: Preprocessing includes: Filter and remove the identifying content added by the OCR algorithm when parsing the PDF source into markdown corpus.

6. The visualization method for large-scale model training corpus in scientific fields according to claim 3, characterized in that: The HTML parser-based method converts the updated markdown abstract syntax tree into an HTML abstract syntax tree, including: The entire updated markdown abstract syntax tree is traversed recursively, and each node in the updated markdown abstract syntax tree is converted into an HTML abstract syntax tree containing type, tag name, value, attribute, and child node list.

7. The visualization method for large model training corpus in scientific fields according to claim 3 or 6, characterized in that: The method of converting the updated markdown abstract syntax tree into an HTML abstract syntax tree based on an HTML parser further includes: Filter and delete tags and attributes in the HTML abstract syntax tree that may lead to cross-site scripting attacks.

8. A visualization system for large model training corpus in the scientific field, implemented by the visualization method for large model training corpus in the scientific field according to any one of claims 1 to 7, characterized in that: include: Corpus data transmission module, content parsing and rendering module, and comparison chart generation module; The corpus data transmission module is used to stream the large model training corpus in the scientific field in markdown format obtained from the object storage server to the client in segments on the server side; The content parsing and rendering module is used to parse and render the rich text content in the received markdown format scientific field large model training corpus on the client to obtain a restoration result; The comparison chart generation module is used to generate a comparison chart of the PDF original content and the restored result of the scientific field large model training corpus in markdown format on the client, and add corpus metadata, user information watermark and corpus quality score to the comparison chart.

9. A visualization device for large model training corpus in the scientific field, comprising a memory and one or more processors, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the visualization method for large model training corpus in the scientific field as described in any one of claims 1 to 7 when executing the computer program.

Images