Methods, apparatuses, media, products for parsing multi-modal scientific literature

Abstract

The application provides a method, device, medium and product for parsing a multi-modal scientific literature. The method comprises: inputting the scientific literature into a preset parsing model and extracting a feature map of the scientific literature; performing dynamic reweighting on the feature map based on a domain label of the scientific literature by an adaptive attention module of the parsing model to output a domain-enhanced feature map; performing text detection, table structure recognition, formula symbol recognition and position correspondence prediction based on the domain-enhanced feature map to respectively output structured data of text, table, formula, symbol and position correspondence of the scientific literature; and performing semantic consistency verification on the table and the formula based on the text associated with the table and the formula.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, medium, or product for parsing multimodal scientific and technological literature. Background Technology

[0002] Scientific and technical literature typically includes main text paragraphs, titles, figures, tables, formulas, formula symbols, footnotes, figure captions, and explanations of references to figures or formulas within the text. Compared to ordinary text documents, figures, tables, and formulas in scientific and technical literature are more dependent on formatting and vary across different fields. For example, molecular formulas, reaction equations, and experimental data tables in chemical literature, formulas, units, and circuit symbols in physics or engineering literature, and statistical tables and diagrams in medical literature may all have different symbol systems, table hierarchies, and semantic expressions.

[0003] Existing document parsing methods can typically detect text, table areas, or formula areas on a page and output recognition results in a certain format. However, when faced with interdisciplinary scientific and technological literature, existing methods cannot output targeted recognition results based on subject attributes, resulting in poor accuracy in recognizing subject-specific features. For example, the identified molecular formulas may not conform to standards, or the circuit diagrams may not conform to circuit principles. Summary of the Invention

[0004] This application provides a method, device, medium, and product for parsing multimodal scientific and technological documents, in order to improve the problems of insufficient adaptability to scientific and technological documents of different disciplines, incomplete recovery of complex table structures, and lack of contextual semantic verification of table or formula recognition results in the prior art.

[0005] The method for parsing multimodal scientific and technological documents includes the following steps: inputting the scientific and technological documents into a preset parsing model and extracting feature maps from the scientific and technological documents; dynamically reweighting the feature maps based on the domain labels of the scientific and technological documents using the adaptive attention module of the parsing model to output domain-enhanced feature maps; performing text detection, table structure recognition, formula symbol recognition, and position correspondence prediction based on the domain-enhanced feature maps to output structured data of the text, tables, formulas, symbols, and position correspondences of the scientific and technological documents, respectively; and performing semantic consistency verification on the tables and formulas based on the text associated with the tables and formulas.

[0006] Optionally, the domain label includes a subject category label and a sub-domain feature label of the discipline to which the scientific and technological literature belongs.

[0007] Optionally, the adaptive attention module includes attention subnetworks at the coarse-grained domain level, the fine-grained domain level, and the element-level symbol level. The coarse-grained domain level is used to identify the subject category label, the fine-grained domain level is used to identify the sub-domain feature label, and the element-level symbol level is used to adjust the recognition sensitivity for the symbol type in tables and formulas.

[0008] Optionally, the table structure recognition includes: traversing cells horizontally to identify row merging relationships; traversing cells vertically to identify column merging relationships; maintaining state vectors at row and column intersections, and parsing nested header structures based on the state vectors; and constructing a multi-level header tree structure based on the row merging relationships, the column merging relationships, and the nested header structures.

[0009] Optionally, the table structure recognition further includes: generating structural features of the target table based on the number of columns, header features, and data patterns of the cross-page table; calculating the similarity between tables on adjacent pages based on the structural features; and determining that the cross-page table has continuity when the similarity is greater than a preset threshold, and performing structural splicing and semantic completion at the breaks in the cross-page table.

[0010] Optionally, the semantic consistency verification includes: identifying reference statements to charts or formulas from the text; verifying whether the descriptive content in the reference statements is consistent with the content of the charts or formulas; detecting inconsistent content and generating correction suggestions.

[0011] Optionally, the preprocessing includes: standardizing the scientific and technological documents; detecting the blur, contrast, and noise levels of the images in the scientific and technological documents; and performing at least one of denoising, sharpening, or contrast adjustment based on the blur, contrast, and noise levels.

[0012] Optionally, the method further includes: acquiring subject-based symbol classification data and legal combination patterns; and predicting the symbol type of the symbol based on the symbol's context, constrained by the symbol classification data and the legal combination patterns.

[0013] According to another aspect of this application, an electronic device is provided, the device including a memory storing computer-executable instructions and a processor; when the instructions are executed by the processor, the device causes the device to perform any of the methods described above.

[0014] According to another aspect of this application, a computer-readable storage medium is provided that stores one or more programs, which can be executed by one or more processors to implement any of the methods described above.

[0015] According to another aspect of this application, a computer program product is provided, comprising a computer program that, when executed by a processor, implements any of the methods described above.

[0016] The technical advantage of this application compared to existing technologies lies in the following: by introducing a domain-label-based adaptive attention module into the parsing model, the model can adaptively learn from scientific and technological literature from different disciplines. Through the adaptive attention module, when faced with scientific and technological literature from different disciplines, different feature weights can be applied to data associated with discipline attributes, such as text, tables, formulas, and symbols, thereby improving the adaptability of cross-disciplinary scientific and technological literature parsing.

[0017] Furthermore, by combining row-direction traversal, column-direction traversal, and row-column intersection state vectors in table structure recognition, complex table structures such as merged cells and nested headers can be recovered more accurately, reducing the probability of complex table structures being destroyed. By calculating structural feature similarity for cross-page tables and performing splicing and semantic completion, the problem of structural breakage after table cross-page can be improved. By using text associated with tables and formulas for semantic consistency verification, conflicts between reference descriptions and recognition results can be found, and correction suggestions can be generated, thereby improving the credibility and usability of the final structured data. Attached Figure Description

[0018] Figure 1 This is a schematic flowchart of a method for parsing multimodal scientific and technological documents according to an embodiment of this application.

[0019] Figure 2 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0020] The embodiments of this application will now be described with reference to the accompanying drawings. These embodiments are used to illustrate the technical solutions of this application and are not intended to limit the scope of protection of this application. Where there is no conflict, the technical features in the following embodiments can be combined with each other.

[0021] like Figure 1 The diagram shows a flowchart of a method for parsing multimodal scientific and technological documents provided in this application. The scientific and technological documents can be PDF files, scanned images, formatted documents, or files composed of multiple pages of images. The scientific and technological documents may contain main text, tables, formulas, images, figure titles, table titles, formula numbers, footnotes, and references to figures or formulas in the main text.

[0022] In step 101, the scientific and technological documents are input into the preset parsing model, and the feature maps of the scientific and technological documents are extracted.

[0023] Scientific and technological literature consists of multi-source, interdisciplinary documents in various formats. For PDF documents, feature map extraction can begin by obtaining page images; for scanned or image-formatted documents, page images can be obtained directly, and metadata can be recorded as needed for subsequent operations such as location tracking, positional prediction, and cross-page table identification.

[0024] Furthermore, since scientific and technological documents may contain images of poor quality, such as hand-drawn illustrations or low-quality scans, the analytical model also needs to detect the blurriness, contrast, and noise levels of the images in the documents. This allows for targeted corrections based on these factors to improve image quality. Those skilled in the art will understand that any appropriate method can be used to detect and determine the blurriness, contrast, and noise levels. For example, blurriness can be determined based on edge gradients, Laplacian response, etc.; contrast can be determined based on grayscale distribution, local brightness differences, etc.; and noise levels can be determined based on grayscale fluctuations in flat areas of the page, compression artifacts, or random noise density, etc.

[0025] The feature map can include page features at different scales, such as high-resolution local texture features (e.g., detailed features such as formula symbols and table lines) and low-resolution global layout features (e.g., large areas such as text blocks, chart areas, and formula areas on the page).

[0026] In one implementation, the feature map records not only visual features but also coordinate information related to the page position. For each text block, table area, or formula area, its position information on the page, such as bounding box, center point, aspect ratio, and relationship with adjacent areas, can be mapped to the feature map.

[0027] In step 102, the adaptive attention module of the parsing model dynamically reweights the feature map based on the domain labels of scientific and technological literature to output a domain-enhanced feature map.

[0028] Domain tags include a broad subject category tag for the discipline to which the scientific and technological literature belongs, and sub-domain feature tags. The broad subject category tag describes the macro-level disciplinary attributes of the literature (e.g., medicine, mechanical engineering, computer science, biology, etc.), while the sub-domain feature tags describe the specific objects within the corresponding discipline (e.g., chemical formulas, structural diagrams, circuit diagrams, biological maps, etc.). In this embodiment, domain tags can be determined by analyzing literature metadata, titles, abstracts, keywords, classification numbers, and other data. In other embodiments, they can be predicted by a parsing model based on page content analysis, or domain tags can be applied to the scientific and technological literature before it is input into the parsing model.

[0029] The adaptive attention module is positioned after the feature extraction layer. The parsing model first converts the domain labels into domain vectors. The adaptive attention module then dynamically generates corresponding weight parameters based on these domain vectors and applies these weight parameters to the feature map to dynamically adjust its weighting, thereby obtaining a domain-enhanced feature map. Thus, by reweighting the feature map based on the domain labels using the adaptive attention module, a more domain-adaptive enhanced feature map is obtained, enabling the dynamic switching of the parsing model's inference direction according to the domain of the scientific literature.

[0030] In one implementation, the adaptive attention module includes attention subnetworks at the coarse-grained domain level, fine-grained domain level, and element-level symbol level. The coarse-grained domain level identifies subject category tags and adjusts the overall features of the scientific literature page based on these tags. The fine-grained domain level identifies sub-domain feature tags and further weights explicit features related to the subject domain, such as charts, formulas, and circuit diagrams. The element-level symbol level further adjusts the recognition sensitivity for symbol types in tables and formulas. Specifically, the parsing model can determine easily confused symbol types based on the probability that the current area belongs to a table, formula, figure title, or text, combined with the domain tag. For example, in the mathematical formula area, the letter '1', the number '1', and the vertical bar symbol may be confused; in recognizing chemical formulas or unit expressions, uppercase and lowercase letters, superscripts and subscripts, and special connectors may have different meanings. The element-level symbol level can improve the feature response of the local areas containing these symbols, enabling subsequent formula symbol recognition or symbol type prediction to utilize more comprehensive local information.

[0031] Through the above processing, the domain-enhanced feature map retains the layout and texture information of the original page image while incorporating prior information provided by the domain labels. This domain-enhanced feature map can serve as a shared input for subsequent multi-task parsing, enabling text detection, table structure recognition, formula symbol recognition, and positional correspondence prediction to be performed based on the same domain-adapted features.

[0032] In step 103, based on the domain-enhanced feature map, text detection, table structure recognition, formula symbol recognition, and position correspondence prediction are performed to output structured data of text, tables, formulas, symbols, and position correspondence of scientific and technological documents, respectively.

[0033] In this step, the parsing model can simultaneously perform multi-task parsing based on the domain-enhanced feature map. The multi-task parsing can share the domain-enhanced feature map obtained in step 102, and after the adaptive attention module, there are text detection branches, table structure recognition branches, formula symbol recognition branches, and position correspondence prediction branches to perform text detection, table structure recognition, formula symbol recognition, and position correspondence prediction respectively.

[0034] The following sections will explain text detection, table structure recognition, formula symbol recognition, and positional correspondence prediction. Text detection can employ OCR technology to detect and recognize text content in scientific and technological documents, such as titles, body paragraphs, figure titles, table titles, formula numbers, and footnotes.

[0035] Table structure recognition includes table region detection, cell localization, row and column relationship analysis, and logical structure reconstruction. The analytical model can first determine the table region in the enhanced feature map of the domain and identify table structure information such as cell boundaries, row lines, column lines, blank separators, and text alignment. For three-line tables or statistical tables without obvious borders, the cell range can be inferred based on the alignment of text blocks, column spacing, and the position of the table header.

[0036] For complex tables, the parsing model can employ a recurrent neural network to traverse cells horizontally to identify row merging relationships and vertically to identify column merging relationships. Specifically, for adjacent cells located in the same row / column direction, the parsing model can compare their text baseline, boundary continuity, and whether the cell content spans multiple columns / rows. When a cell covers multiple columns horizontally, or its left and right adjacent cells share a common top and bottom boundary but lack internal column separators, it can be determined that the cell has a row merging relationship. When a cell covers multiple rows vertically, or its top and bottom adjacent cells form continuous headers or category descriptions within the same column, it can be determined that the cell has a column merging relationship.

[0037] Nested headers (also known as multi-level headers or hierarchical headers) refer to a table structure where column headings have a parent-child hierarchical relationship, meaning a main header can contain one or more child headers, used for categorizing and displaying multi-dimensional data. To handle nested header structures, the parsing model maintains state vectors at the intersections of rows and columns.

[0038] The relevant information for the row-column intersection includes: row number, column number, whether there is a cell / column to the left (Boolean value), whether there is a cell above (Boolean value), whether it is a header, which parent node it belongs to, whether there are child nodes and their row and column numbers, etc. This information forms a state vector to represent the structural state of the row-column intersection. The parsing model can parse the nested header structure based on the state vector and associate the parent nodes, child nodes, and leaf nodes in the header. Subsequently, the parsing model constructs a multi-level header tree structure based on row merging relationships, column merging relationships, and the nested header structure, so that the table data units can correspond to the correct multi-level header paths.

[0039] Furthermore, a table may span adjacent pages. To avoid recognizing a cross-page table as multiple unrelated tables, the parsing model can process cross-page tables sequentially. Specifically, the parsing model first generates the structural features of the target table (i.e., tables that may be on adjacent pages of the cross-page table) based on the number of columns, header features, and data pattern of the cross-page table. Header features may include header text, header hierarchy, header width, and header number; data patterns may include the data type, numerical range, unit representation, null value distribution, and text length distribution of each column.

[0040] After obtaining the structural features, the parsing model calculates the similarity of the structural features of tables between adjacent pages. Similarity can be determined by the consistency of column count, header similarity, column width ratio similarity, data type consistency, and the truncation status of the table at the page edge. When the similarity exceeds a preset threshold, the parsing model determines that the cross-page table is continuous and is the same table, and performs structural splicing and semantic completion at breaks in the cross-page table. Structural splicing may include unifying column widths, merging headers, continuing data rows, and maintaining the original page number information; semantic completion at breaks may include determining whether there are missing row headings or unit descriptions based on the last row of the previous page, the first row of the next page, and the header path.

[0041] Formula symbol recognition can include formula region detection, formula symbol segmentation, symbol type prediction, and formula structure recovery. The analytical model can identify formula-related information such as characters, operators, parentheses, subscripts and superscripts, square roots, integral symbols, summation symbols, matrix structures, and formula numbers within a formula region based on the enhanced feature map of the neighborhood. For formulas with hierarchical relationships, the analytical model can determine subscript / superscript relationships, fractional relationships, radical relationships, and parenthesis inclusion relationships based on the relative positions of the symbols.

[0042] In one implementation, the analytical model can acquire subject-based symbol classification data and legal combination patterns, and predict the symbol type based on the symbol's context, constrained by the symbol classification data and legal combination patterns. Symbol classification data can include categories, appearance characteristics, meanings, and easily confused symbol sets of common symbols in different subjects; legal combination patterns can include operator combination rules in mathematical formulas, element symbols and subscript combination rules in chemical formulas, device symbol connection rules in circuit diagrams, or dimensional combination rules in unit expressions. When identifying a single symbol, the analytical model not only provides candidate types based on symbol image features but also constrains candidate types by considering preceding and following symbols, subscript and subscript positions, region type, and domain label, thereby reducing identification results that do not conform to domain rules. In one example, when the scientific document to be identified belongs to the field of chemistry, chemical element symbol classification data and legal combination patterns of chemical formulas are acquired. If the OCR recognition result is "H2O", combining the legal combination pattern determines that the letter "O" is a legal element symbol, while the number "0" cannot participate in the combination as an element symbol. Therefore, the last character is corrected to the letter "O", and it is determined that numbers in chemical formulas are usually subscripts, ultimately resulting in "H2O". In another example, when the scientific and technological document to be identified belongs to the field of mathematics, if the identification results contain multiple candidate symbols such as "lowercase letter l", "number 1" and "uppercase letter I", the confidence level of each candidate symbol is determined by combining variable naming rules, formula context and legal combination patterns.

[0043] Position correspondence prediction is used to determine the page positional and referential relationships between text, tables, formulas, and symbols. The analytical model can predict the correspondence between text paragraphs and tables / formulas, or the correspondence between formula numbers and specific formula areas, based on the page coordinates, reading order, distance, numbering, caption content, and text citations of each object.

[0044] In step 104, semantic consistency is checked on the tables and formulas based on the text associated with them.

[0045] In this step, the parsing model uses the text associated with the tables and formulas to perform semantic consistency checks on the tables and formulas output in step 103. The text associated with the tables and formulas may include table titles, figure titles, descriptions near formula numbers, quotations in the main text, and paragraphs before and after the tables or formulas. The parsing model can first identify quotations for figures or formulas from the text, such as statements containing phrases like "as shown in Table 1," "see formula (2)," "as shown in Figure 3," or similar expressions, and then associate the quotations with the corresponding tables or formulas.

[0046] After establishing the association, the parsing model verifies whether the descriptions in the referenced statements are consistent with the content of the tables or formulas. For tables, the parsing model can compare whether the indicator names, values, units, row and column headings, trends, and extreme value descriptions appearing in the referenced statements can be found in the table's structured data. For example, if the text describes an indicator as having the maximum value in the table, the parsing model can calculate whether the indicator is indeed the maximum value based on the table's data columns; if the text references a table number, the parsing model can determine whether that number is consistent with the table title and table area.

[0047] For formulas, the analytical model can compare the formula number, variable name, symbol meaning, unit description, and formula expression appearing in the reference statement to see if they are consistent with the formula structured data. If the text states that a certain variable is velocity, but the formula recognition result identifies the symbol in the same position as another variable, the analytical model can determine whether there is an inconsistency based on the context, symbol classification data, and legal combination patterns. If the reference statement points to formula (3), but the location of the formula area numbered (3) on the page is inconsistent with the reference relationship prediction result, the analytical model can also mark it as an item to be verified.

[0048] When inconsistencies are detected, the parsing model generates correction suggestions. These suggestions can include correcting symbols, adjusting table cells, correcting cross-page table joins, re-associating references with referenced objects (images, formulas, tables, etc.), and indicating potential identification errors. For correction suggestions with high confidence, the parsing model can automatically update the structured data. For correction suggestions with multiple results of similar confidence, the original output can be retained along with additional prompt text for manual review.

[0049] Through the above steps, the parsing method of this application can complete page feature extraction, domain-adaptive feature enhancement, multi-task structured parsing, and contextual semantic verification in the same process. Simultaneously, page coordinates, domain labels, and structured intermediate results are retained during each step, so that subsequent verification can trace back to specific page regions and specific identified objects, facilitating further correction or review.

[0050] Now for reference Figure 2 The diagram shown is a block diagram of an electronic device 200 according to an embodiment of the present application. The electronic device 200 may include one or more processors 202, system control logic 208 connected to at least one of the processors 202, system memory 204 connected to the system control logic 208, non-volatile memory (NVM) 206 connected to the system control logic 208, and network interface 210 connected to the system control logic 208.

[0051] Processor 202 may include one or more single-core or multi-core processors. Processor 202 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In embodiments herein, processor 202 may be configured to perform one or more embodiments of various embodiments of the method for parsing multimodal scientific and technological documents according to this application.

[0052] In some embodiments, system control logic 208 may include any suitable interface controller to provide any suitable interface to at least one of the processors 202 and / or any suitable device or component communicating with system control logic 208.

[0053] In some embodiments, system control logic 208 may include one or more memory controllers to provide an interface to system memory 204. System memory 204 may be used to load and store data and / or instructions. In some embodiments, system memory 204 of electronic device 200 may include any suitable volatile memory, such as suitable dynamic random access memory (DRAM).

[0054] The nonvolatile memory 206 may include one or more tangible, non-transitory computer-readable storage media for storing data and / or instructions. In some embodiments, the nonvolatile memory 206 may include any suitable nonvolatile memory such as flash memory and / or any suitable nonvolatile storage device, such as at least one of HDD (Hard Disk Drive), CD (Compact Disc) drive, and DVD (Digital Versatile Disc) drive.

[0055] The non-volatile memory 206 may include a portion of the storage resources installed on the device of the electronic device 200, or it may be accessible by the device, but is not necessarily part of the device. For example, the non-volatile memory 206 may be accessed over a network via the network interface 210.

[0056] Specifically, system memory 204 and non-volatile memory 206 may each include a temporary copy and a permanent copy of instructions 220. Instructions 220 may include instructions that, when executed by at least one of processors 202, cause electronic device 200 to implement the methods provided in this application. In some embodiments, instructions 220, hardware, firmware, and / or their software components may additionally / alternatively reside in system control logic 208, network interface 210, and / or processor 202.

[0057] In some embodiments, network interface 210 may be integrated into other components of electronic device 200. For example, network interface 210 may be integrated into at least one of processor 202, system memory 204, non-volatile memory 206, and firmware device (not shown) with instructions that, when at least one of the processors 202 executes the instructions, electronic device 200 implements one or more embodiments of the method for parsing multimodal scientific and technological documents described herein. Network interface 210 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface.

[0058] In one embodiment, at least one of the processors 202 may be packaged together with the logic of one or more controllers for system control logic 208 to form a system package (SiP). In another embodiment, at least one of the processors 202 may be integrated on the same die with the logic of one or more controllers for system control logic 208 to form a system on chip (SoC).

[0059] Electronic device 200 may further include: input / output (I / O) device 212. Input / output (I / O) device 212 may include a user interface that enables a user to interact with electronic device 200; the design of peripheral component interfaces enables peripheral components to also interact with electronic device 200.

[0060] In some embodiments, the user interface may include, but is not limited to, a display (e.g., a liquid crystal display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., a light-emitting diode flash), and a keyboard.

[0061] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0062] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0063] Program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0064] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this paper are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0065] One or more aspects of at least one embodiment can be implemented by representational instructions stored on a computer-readable storage medium, the instructions representing various logics in a processor, which, when read by a machine, cause the machine to create logic for performing the techniques described herein. These representations, referred to as “IP cores,” can be stored on a tangible computer-readable storage medium and provided to multiple customers or production facilities for loading into manufacturing machines that actually manufacture the logic or processor.

[0066] One embodiment of this application discloses a computer-readable storage medium storing one or more programs executable by one or more processors to implement the method of parsing multimodal scientific and technological documents of this application.

[0067] One embodiment of this application discloses a computer program product, including a computer program that, when executed by a processor, implements the method of parsing multimodal scientific and technological documents of this application.

[0068] The specific embodiments described above illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Although the description of this application is presented in conjunction with preferred embodiments, this does not mean that the features of this invention are limited to these embodiments. Furthermore, to avoid confusion or obscuring the focus of this application, some specific details will be omitted in the description. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0069] Furthermore, the various operations will be described as multiple discrete operations in a manner most conducive to understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations must depend on the order. In particular, these operations do not need to be performed in the order presented.

[0070] Unless the context otherwise specifies, the terms “contains,” “has,” and “includes” are synonyms. The phrase “A / B” means “A or B.” The phrase “A and / or B” means “(A and B) or (A or B).”

[0071] As used herein, the terms “module” or “unit” may refer to, be, or include: application-specific integrated circuits (ASICs), electronic circuits, (shared, dedicated, or group) processors and / or memories that execute one or more software or firmware programs, combinational logic circuits, and / or other suitable components that provide the described functionality.

[0072] In the accompanying drawings, certain structural or methodological features are shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. In some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0073] It should be understood that although terms such as "first," "second," etc., may be used herein to describe various units or data, these units or data should not be limited by these terms. These terms are used merely to distinguish one feature from another. For example, without departing from the scope of the exemplary embodiments, a first feature may be referred to as a second feature, and similarly, a second feature may be referred to as a first feature.

[0074] It should be noted that in this specification, similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0075] Although the invention has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.

Claims

1. A method for parsing multimodal scientific and technological literature, characterized in that, include: The scientific and technological documents are input into a preset analytical model, and feature maps of the scientific and technological documents are extracted. The adaptive attention module of the analytical model dynamically reweights the feature map based on the domain labels of the scientific and technological literature to output a domain-enhanced feature map. Based on the domain-enhanced feature map, text detection, table structure recognition, formula symbol recognition, and position correspondence prediction are performed to output structured data of the text, tables, formulas, symbols, and position correspondences of the scientific and technological documents, respectively. as well as Based on the text associated with the table and the formula, a semantic consistency check is performed on the table and the formula.

2. The method according to claim 1, characterized in that, The domain tags include the subject category tag and sub-domain feature tags of the discipline to which the scientific and technological literature belongs.

3. The method according to claim 2, characterized in that, The adaptive attention module includes attention subnetworks at the coarse-grained domain level, the fine-grained domain level, and the element-level symbol level. The coarse-grained domain level is used to identify the subject category labels, the fine-grained domain level is used to identify the sub-domain feature labels, and the element-level symbol level is used to adjust the recognition sensitivity based on the symbol types in tables and formulas.

4. The method according to claim 3, characterized in that, The table structure identification includes: Traverse the cells horizontally to identify row merging relationships; Traverse the cells vertically to identify column merge relationships; Maintain the state vectors of the row and column intersections, and parse the nested header structure based on the state vectors; A multi-level header tree structure is constructed based on the row merging relationship, the column merging relationship, and the nested header structure.

5. The method according to claim 4, characterized in that, The table structure recognition also includes: Generate the structural features of the target table based on the number of columns, header features, and data pattern of the cross-page table; Based on the structural features, the similarity of tables between adjacent pages is calculated; When the similarity is greater than a preset threshold, it is determined that the cross-page table has continuity, and structural splicing and semantic completion of the breaks are performed on the cross-page table.

6. The method according to claim 5, characterized in that, The semantic consistency check includes: Identify references to charts or formulas from text; Verify that the description in the cited statement is consistent with the content of the chart or formula; Detect inconsistencies and generate correction suggestions.

7. The method according to claim 6, characterized in that, The preprocessing includes: The aforementioned scientific and technological documents are standardized. The blur, contrast, and noise levels of the images in the aforementioned scientific and technological documents were detected. Based on the blurriness, contrast, and noise level, at least one of denoising, sharpening, or contrast adjustment is performed.

8. The method according to claim 7, characterized in that, Also includes: Obtain subject-based symbol classification data and legal combination patterns; Based on the context of the symbol, and constrained by the symbol classification data and the legal combination pattern, the symbol type is predicted.

9. An electronic device, characterized in that, The device includes a memory storing computer-executable instructions and a processor; when the instructions are executed by the processor, the device performs the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the method of any one of claims 1 to 8.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images