Multimodal mathematical literature retrieval method and system based on structural semantic graph embedding

Abstract

The application belongs to the field of artificial intelligence and information retrieval, and provides a multi-modal mathematical literature retrieval method and system based on structure semantic graph embedding. The content of the literature is converted to form a triple structure, the expression of the LaTeX formula in the triple structure is extracted, an abstract syntax tree is constructed, the abstract syntax tree is converted into a structure semantic graph, a pre-trained model is used to encode the structure semantic graph, a semantic embedding vector is extracted to form a formula embedding vector; a pre-trained language model is used to perform semantic modeling on the text paragraphs of the triple structure to generate a text embedding vector; based on the formula embedding vector and the text embedding vector, a retrieval index structure is constructed; in response to a query request of a user, the query request is subjected to modal recognition, processing is performed according to the recognized modal, semantic information is generated, and a query is performed on the retrieval index structure according to the semantic information. The application improves the accuracy of mathematical literature retrieval.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence and information retrieval, and particularly relates to a multi-modal mathematical literature retrieval method and system based on structural semantic graph embedding. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] With the development of mathematical research and interdisciplinary research, the number of mathematical literature worldwide is rapidly increasing, and researchers' demand for efficient and accurate access to mathematical knowledge is rising. Mathematical literature retrieval systems play an important role in providing literature evidence for project selection, helping researchers quickly locate the research direction by integrating academic resources in the mathematical field.

[0004] However, most of the existing mainstream mathematical literature retrieval systems are based on keyword matching for text vectorization processing, which is suitable for semantic search of conventional natural language literature. However, when dealing with literature containing mathematical formulas, there are many shortcomings.

[0005] Firstly, the existing mathematical literature retrieval system cannot accurately model the semantic of mathematical formulas. Mathematical formulas are often treated as ordinary text, ignoring their structural and semantic hierarchy, which leads to the fact that the retrieval system cannot understand the mathematical meaning of the formulas and only relies on literal matching. However, if the formula representation is inaccurate or has no semantic information, it is difficult to return results with reasoning relationships or semantics.

[0006] Secondly, the existing mathematical literature retrieval system generally only supports keyword or full-text input, and cannot recognize LaTeX formula queries or the content of formulas in images, which limits the input method and interactive experience of users.

[0007] In addition, the literature sources of the existing mathematical literature retrieval system are scattered, and the models are not unified. Different types of databases are not uniform in data structure and format, which makes it difficult for the retrieval system to uniformly process, model and index.

[0008] In summary, the existing mathematical literature retrieval system cannot accurately express the semantic structure of formulas, cannot perform high-quality semantic matching and reasoning search, and lacks effective support for multi-modal input, resulting in low retrieval accuracy and poor applicability, which cannot meet the complex needs of mathematical research scenarios. SUMMARY

[0009] The present application proposes a multi-modal mathematical literature retrieval method and system based on structural semantic graph embedding, which can solve the problems of weak formula structure modeling capability, low semantic matching precision, limited query input modal, and lack of context understanding and multi-database fusion modeling in the prior art.

[0010] According to some embodiments, the present application adopts the technical solutions as follows:

[0011] A multi-modal mathematical literature retrieval method based on structural semantic graph embedding, comprising the following steps:

[0012] Obtaining multi-source heterogeneous mathematical literature, transforming the same, converting the content of mathematical literature in different formats into a unified triple structure, the triple structure including a text paragraph, a LaTeX formula and meta information;

[0013] Extracting the expression of the LaTeX formula in the triple structure, performing analysis, constructing an abstract syntax tree to represent the syntax relationship and operation level in the LaTeX formula, and converting the abstract syntax tree into a structural semantic graph, wherein the nodes of the graph in the structural semantic graph represent the respective constituent elements in the formula, and the edges represent the semantic or structural relationship between the constituent elements;

[0014] Encoding the structural semantic graph using a pre-trained model to extract a globally consistent semantic embedding vector, forming a formula embedding vector;

[0015] Performing semantic modeling on the text paragraph of the triple structure using a pre-trained language model to generate an embedding vector representation, so as to capture the implicit semantics related to the mathematical formula in the text paragraph, and generate a text embedding vector;

[0016] Based on the formula embedding vector and the text embedding vector, constructing a retrieval index structure;

[0017] In response to a query request of a user, performing modality recognition on the query request, processing according to the recognized modality, generating semantic information, querying in the retrieval index structure according to the semantic information, encapsulating the query result as a structured semantic vector, and the structured semantic vector including a text embedding, a formula embedding, a modality label and meta information.

[0018] As an optional implementation, the process of obtaining multi-source heterogeneous mathematical literature and transforming the same to convert the content of mathematical literature in different formats into a unified triple structure includes: accessing different mathematical databases, periodically grabbing the latest data documents under the mathematical partition for open-source mathematical databases; and obtaining original data documents through an interface provided by authorization for commercial databases;

[0019] For literature provided in the form of LaTeX source code, extracting the literature paragraph and the formula region, keeping the original LaTeX expression unchanged, and generating a structurally consistent triple through structure tag recognition and text position matching;

[0020] For PDF format literature, the formula region is identified by layout analysis and visual detection, and then the image or character combination is reconstructed into an equivalent LaTeX formula. The reconstructed equivalent LaTeX formula is bound with the context paragraph to generate a consistent triple structure;

[0021] For XML format literature, the structured tags are parsed, the text and MathML expression are extracted, the MathML expression is converted into a standard LaTeX formula, and a consistent triple structure is generated;

[0022] For HTML web page type content, the page paragraph and embedded formula tag are extracted and converted into a LaTeX formula to generate a consistent triple structure;

[0023] The form of the consistent triple structure is <text paragraph, LaTeX formula, meta information>.

[0024] As an optional implementation, multiple source heterogeneous mathematical literature is obtained, and the different formats of mathematical literature content are converted into a unified triple structure. During the conversion of the formula and paragraph content, the source, format type and access permission state of each piece of data are labeled.

[0025] As an optional implementation, the expression of the LaTeX formula in the triple structure is extracted and parsed to construct an abstract syntax tree to represent the syntax relationship and operation level in the LaTeX formula. The process of converting the abstract syntax tree into a structure semantic graph includes: reading the LaTeX formula in the triple, inputting and parsing one by one, and parsing the LaTeX formula into an abstract syntax tree to represent the syntax relationship and operation level in the formula.

[0026] After obtaining the abstract syntax tree, the abstract syntax tree is converted into a structure semantic graph, wherein the nodes of the graph represent the constituent elements in the formula, and the constituent elements include variables, operators, functions and constants. The edges represent the semantic or structural relationship between the constituent elements. Attributes are set for at least part of the nodes and edges, and the attributes include type, depth level and position in the original formula.

[0027] As an optional implementation, the structure semantic graph is encoded using a pre-trained model to extract a globally consistent semantic embedding vector. The process includes: using a pre-trained graph convolution network model to encode the structure semantic graph. The nodes represent the constituent elements in the mathematical formula, and the edges represent the structural or semantic relationship between them. Through layer-by-layer feature aggregation mechanism, the nested relationship, operation priority and symbol dependence of the entire formula structure are modeled, and a globally consistent semantic embedding vector is extracted.

[0028] And the structural features of the graph are introduced as input features in the training process, and a contrast learning algorithm and a self-supervised learning algorithm are used to enhance the discrimination ability of the graph convolution network model for formulas with similar structures but different semantics, or different structures but similar semantics; the structural features include node type, position embedding and depth information.

[0029] As an optional implementation, the process of using a pre-trained language model to semantically model the text passage of the triple structure to generate an embedding vector representation to capture the implicit semantics related to the mathematical formula in the text passage includes: using a pre-trained language model to encode the natural language passage, extracting the pure text information of the part containing LaTeX formula references, term chains or long sentence dependencies in the text, and constructing a composite representation jointly with the corresponding formula embedding vector; using a syntax analysis method to deconstruct the text structure and extract a syntax path as a semantic feature to assist modeling.

[0030] As an optional implementation, based on the formula embedding vector and the text embedding vector, the process of constructing a retrieval index structure includes: using a fusion strategy to fuse the formula embedding vector and the text embedding vector, uniformly normalizing the fused vectors of all documents, and constructing a vectorized retrieval index;

[0031] When constructing the vectorized retrieval index, the index is constructed at the paragraph level or multiple triple semantic vectors of the entire document are aggregated into a document-level vector to realize multi-level retrieval.

[0032] As an optional implementation, in response to a user's query request, the query request is modality-identified, processed according to the identified modality, and semantic information is generated.

[0033] The user's input is modality-identified and segmented, and it is determined whether it is pure text, LaTeX formula, image format, or a mixed query of text and formula;

[0034] If the user's input is pure text, the text is semantically modeled to generate a text embedding representation;

[0035] If the user's input is a LaTeX formula, perform syntax analysis and semantic graph construction, and generate a graph embedding representation;

[0036] If the user's input is an image format formula, convert the image to a standard LaTeX formula, and then uniformly enter the structure analysis process to realize format alignment between text and image modalities;

[0037] If the user's input is a mixed query of text and formula, perform modality segmentation, then process the content of each modality, integrate the semantic information of multiple modalities through a modality fusion strategy, and obtain a complete and structurally unified query representation.

[0038] As an alternative embodiment, the process of querying the retrieval index structure according to the semantic information comprises: according to the semantic information generated by the query, searching the retrieval index structure to obtain a candidate document set similar to the query semantics, and using a deep semantic matching model to bidirectionally model and score the matching degree of each candidate document in the candidate document set and the original query;

[0039] And in the matching process, the natural language expression, LaTeX formula and image formula transcription in the query are considered, and the text content, formula structure embedding vector and meta information in the document are integrated and matched;

[0040] Based on the obtained matching degree score, the candidate documents are sorted, and the matching degree and external features are considered for final sorting optimization, and the external features include several of the number of document citations, publication time, novelty score, author influence and field authority.

[0041] A multi-modal mathematical document retrieval system based on structure semantic graph embedding, comprising:

[0042] A multi-source data access and format analysis module is configured to obtain multi-source heterogeneous mathematical documents, convert them, and convert mathematical document content in different formats into a unified triple structure, which includes text paragraphs, LaTeX formulas and meta information;

[0043] A LaTeX structure analysis and semantic construction module is configured to extract the expression of LaTeX formulas in the triple structure, analyze and construct an abstract syntax tree to represent the syntax relationship and operation level in the LaTeX formula, and convert the abstract syntax tree into a structure semantic graph, in which the nodes of the graph represent the elements in the formula, and the edges represent the semantic or structural relationship between the elements;

[0044] A graph neural network embedding module is configured to use a pre-trained model to encode the structure semantic graph and extract a globally consistent semantic embedding vector to form a formula embedding vector;

[0045] A text semantic modeling module is configured to use a pre-trained language model to model the semantics of the text paragraphs of the triple structure and generate an embedding vector representation to capture the implicit semantics related to the mathematical formula in the text paragraphs, and generate a text embedding vector;

[0046] A semantic fusion and index module is configured to construct a retrieval index structure based on the formula embedding vector and the text embedding vector;

[0047] A multi-modal query analysis module is configured to respond to a user's query request, identify the modality of the query request, process it according to the identified modality, and generate semantic information;

[0048] The retrieval matching and sorting module is configured to query the retrieval index structure according to the semantic information, encapsulate the query result as a structured semantic vector, and the structured semantic vector includes a text embedding, a formula embedding, a modal label and meta information.

[0049] Compared with the prior art, the present application has the following beneficial effects:

[0050] The present application provides a multi-modal mathematical literature retrieval method, which accesses literature data from different types of databases and parses structured uniform literature meta information and LaTeX formulas; then, through structured semantic graph and graph neural network modeling technology, the mathematical formulas are deeply semantically modeled; meanwhile, combined with natural language paragraph semantic embedding model and multi-modal input analysis mechanism, the unified semantic representation of the query input and retrieval matching are realized, accurate matching literature paragraphs, formula positions and context explanation contents can be obtained, the formula structure modeling capability is improved, the semantic matching precision is high, and the query input modal is expanded, realizing multi-database fusion modeling.

[0051] The present application converts literature content of different formats into a unified triple structure by accessing open source and commercial databases, and classifies and processes different formats during the conversion process, realizes unified structure modeling of multi-source heterogeneous mathematical literature, and the triple structure can completely express the language content, formula structure and source information of mathematical knowledge.

[0052] The present application fully expresses the constituent elements and mutual relationships in the formula by constructing an abstract syntax tree and generating a structured semantic graph; and through a layer-by-layer feature aggregation mechanism, the modeling of the nesting relationship, operation priority and symbol dependence of the entire formula structure is realized, so as to extract a globally consistent semantic embedding vector, further improving the embedding quality and generalization ability.

[0053] In the actual modeling process, the present application not only extracts the pure text information of the part containing LaTeX formula reference, term chain or long sentence dependence in the text, but also constructs a composite representation by combining the corresponding formula embedding vector, deconstructs the structure of the text, extracts the syntactic path as a semantic feature to assist modeling, and further improves the understanding ability of complex sentence patterns and nested expressions, while ensuring the integrity of the semantic expression of natural language, laying a high-quality text foundation for the semantic fusion and retrieval sorting of the whole system.

[0054] The present application performs modal recognition on the query input of the user, processes the recognition result according to different cases, and obtains a unified query representation, which not only retains the language context, but also contains the structural features of the formula, enhances the understanding and matching ability of the system for mathematical semantic queries, and enhances the robustness and adaptability of the query.

[0055] The application not only supports semantic matching degree in the query process, but also introduces external characteristics such as literature citation times, publication time, novelty score, author influence, and field authority for final sorting optimization, thereby improving the user experience and improving the user's understanding of search intent and result interpretation.

[0056] In order to make the above objectives, characteristics and advantages of the present application more apparent, comprehensible and easy to understand, the following preferred embodiments are specifically described below, together with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0057] The drawings constituting a part of the specification of the present application are used to provide further understanding of the present application, and the schematic embodiments of the present application and the description thereof are used to explain the present application, and do not constitute undue limitation on the present application.

[0058] Figure 1 is a processing flow chart of a multi-source data access and format analysis module of an embodiment;

[0059] Figure 2 is a processing flow chart of a LaTeX structure analysis and semantic construction module of an embodiment;

[0060] Figure 3 is a processing flow chart of a graph neural network embedding module of an embodiment;

[0061] Figure 4 is a processing flow chart of a text semantic modeling module of an embodiment;

[0062] Figure 5 is a processing flow chart of a multi-modal query analysis module of an embodiment;

[0063] Figure 6 is a processing flow chart of a semantic fusion and indexing module of an embodiment;

[0064] Figure 7 is a processing flow chart of a retrieval matching and sorting module of an embodiment;

[0065] Figure 8 is a processing flow chart of a multi-modal mathematical literature retrieval method of an embodiment. DETAILED DESCRIPTION

[0066] The present application will be further described below in conjunction with the drawings and embodiments.

[0067] It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0068] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0069] The embodiments in the present application and the features in the embodiments can be combined with each other in the case of no conflict.

[0070] Embodiment one

[0071] A multi-modal mathematical literature retrieval method based on structural semantic graph embedding, as shown in Figure 8 includes the following steps:

[0072] Obtain multi-source heterogeneous mathematical literature, transform it, and convert the content of mathematical literature in different formats into a unified triple structure, which includes text paragraphs, LaTeX formulas and meta information;

[0073] Extract the expression of LaTeX formula in the triple structure, parse it, and construct an abstract syntax tree to represent the syntax relationship and operation level in the LaTeX formula. The abstract syntax tree is converted into a structural semantic graph, in which the nodes of the graph represent the elements in the formula, and the edges represent the semantic or structural relationship between the elements;

[0074] Use a pre-trained model to encode the structural semantic graph and extract a globally consistent semantic embedding vector to form a formula embedding vector;

[0075] Use a pre-trained language model to perform semantic modeling on the text paragraphs of the triple structure to generate an embedding vector representation, which captures the implicit semantics related to mathematical formulas in the text paragraphs, and generates a text embedding vector;

[0076] Based on the formula embedding vector and the text embedding vector, construct a retrieval index structure;

[0077] In response to a user's query request, the query request is modality-identified, processed according to the identified modality, semantic information is generated, the retrieval index structure is queried according to the semantic information, and the query result is encapsulated as a structured semantic vector, which includes a text embedding, a formula embedding, a modality label and meta information.

[0078] In the embodiment, similarity calculation is performed when querying, and the candidate documents of the query can be sorted according to the similarity, or the candidate documents can be sorted in multiple dimensions, that is, multi-factor sorting, for example, in addition to the similarity, the sorting factors also introduce external features such as the number of document citations, publication time, novelty score, author influence, field authority, and the like, and corresponding results are displayed.

[0079] In the embodiment, multi-source heterogeneous mathematical documents are obtained, and the mathematical document contents in different formats are converted into a unified triple structure, including: accessing different mathematical databases, periodically capturing the latest data documents under the mathematical partition for open-source mathematical databases; and obtaining original data documents through an interface provided by authorization for commercial databases.

[0080] For documents provided in the form of LaTeX source code, Abstract Syntax Tree (AST) analysis can be used to extract document paragraphs and formula regions, and the original LaTeX expression is kept unchanged, and a structure-consistent triple is generated by matching the structure label and the text position.

[0081] For documents in PDF format, formula recognition can be performed through Optical Character Recognition (OCR), or the formula region can be recognized through layout analysis and visual detection, and the image or character combination is reconstructed into an equivalent LaTeX formula, and the reconstructed equivalent LaTeX formula is bound with the context paragraph to generate a structure-consistent triple.

[0082] For documents in XML format, structured labels can be parsed, that is, label extraction is performed, the text and MathML expressions are extracted, the MathML expressions are converted into standard LaTeX formulas, and a structure-consistent triple is generated.

[0083] For HTML web page type content, page paragraphs and embedded formula labels are extracted, converted into LaTeX formulas, and a structure-consistent triple is generated.

[0084] The structure-consistent triple has a form of <text paragraph, LaTeX formula, meta information>.

[0085] In the embodiment, multi-source heterogeneous mathematical documents are obtained, and the mathematical document contents in different formats are converted into a unified triple structure, and in the process, the converted formula and paragraph content are cleaned and standardized, and the source, format type, and access permission state of each piece of data are labeled.

[0086] In this embodiment, the expression of the LaTeX formula in the triple structure is extracted, parsed, and an abstract syntax tree is constructed to represent the syntax relationship and operation level in the LaTeX formula. The process of converting the abstract syntax tree into a structural semantic graph includes: reading the LaTeX formula in the triple, inputting and parsing each piece, parsing the LaTeX formula into an abstract syntax tree for representing the syntax relationship and operation level in the formula;

[0087] After obtaining the abstract syntax tree, the abstract syntax tree is converted into a structural semantic graph, wherein the nodes of the graph represent the respective constituent elements in the formula, the constituent elements including variables, operators, functions, and constants, the edges represent the semantic or structural relationship between the constituent elements, and attributes are set for at least part of the nodes and edges, the attributes including type, depth level, and position in the original formula.

[0088] In this embodiment, the pre-trained model is used to encode the structural semantic graph to extract a globally consistent semantic embedding vector. The process includes: using a pre-trained graph convolutional network model to encode the structural semantic graph, the nodes representing the constituent elements in the mathematical formula, and the edges representing the structural or semantic relationship between them. Through layer-by-layer feature aggregation mechanism, the modeling of the nested relationship, operation priority and symbol dependence of the entire formula structure is realized, and a globally consistent semantic embedding vector is extracted.

[0089] And in the training process, the structural features of the graph are introduced as input features, and the contrast learning algorithm and the self-supervised learning algorithm are used to enhance the discrimination ability of the graph convolutional network model for structures that are similar but different in semantics, or different in structure but similar in semantics; the structural features include node type, position embedding, and depth information.

[0090] In this embodiment, the pre-trained language model is used to perform semantic modeling on the text segment of the triple structure to generate an embedding vector representation to capture the implicit semantics related to the mathematical formula in the text segment. The process includes: using a pre-trained language model to encode the natural language segment, extracting the pure text information of the part containing LaTeX formula reference, term chain or long sentence dependency in the text, and constructing a composite representation jointly with the corresponding formula embedding vector; using a syntax analysis method to deconstruct the text structure and extracting a syntax path as a semantic feature to assist modeling.

[0091] In this embodiment, based on the formula embedding vector and the text embedding vector, a retrieval index structure is constructed. The process includes: using a fusion strategy to fuse the formula embedding vector and the text embedding vector, uniformly normalizing the fused vectors of all documents, and constructing a vectorized retrieval index.

[0092] In constructing the vectorization retrieval index, the index is constructed at the paragraph level or the multiple triple semantic vectors of the whole document are aggregated into a document-level vector to realize multi-level retrieval.

[0093] In this embodiment, in response to a query request of a user, modal recognition is performed on the query request, processing is performed according to the recognized modal, and the process of generating semantic information includes:

[0094] Modal recognition and segmentation are performed on the input of the user, and it is judged whether the input is pure text, a LaTeX formula, an image format, or a composite query of text and formula mixed;

[0095] If the input of the user is pure text, a Transformer encoding can be used to perform semantic modeling on the text to generate a text embedding representation;

[0096] If the input of the user is a LaTeX formula, syntax analysis and semantic graph construction are performed, and a graph embedding representation is generated;

[0097] If the input of the user is an image format formula, the image is converted into a standard LaTeX formula, and then unified into a structural analysis process to realize format alignment between the image and text modalities;

[0098] If the input of the user is a composite query of text and formula mixed, modal segmentation is performed, and then the content of each modality is processed respectively. Through a modal fusion strategy, the semantic information of multiple modalities is integrated to obtain a complete and structurally unified query representation. Based on the query representation, a vector index query is performed.

[0099] In this embodiment, the process of querying according to the semantic information in the retrieval index structure includes: performing retrieval in the retrieval index structure according to the semantic information generated by the query, obtaining a candidate document set similar to the query semantics, and using a deep semantic matching model to perform bidirectional modeling and matching score evaluation on each candidate document in the candidate document set and the original query;

[0100] And in the matching process, the natural language expression in the query, the LaTeX formula and the image formula transcription, and the body content, formula structure embedding vector and meta information in the document are considered for integrated matching;

[0101] Based on the obtained matching score, the candidate documents are sorted, and the matching score and external features are comprehensively considered for final sorting optimization during the sorting, the external features including several of document citation times, publication time, novelty score, author influence and field authority.

[0102] Embodiment Two

[0103] A multi-modal mathematical document retrieval system based on structural semantic graph embedding includes:

[0104] The multi-source data access and format analysis module is configured to obtain multi-source heterogeneous mathematical literature, convert the mathematical literature content in different formats into a unified triple structure, and the triple structure includes a text paragraph, a LaTeX formula, and meta information.

[0105] The LaTeX structure analysis and semantic construction module is configured to extract the expression of the LaTeX formula in the triple structure, perform analysis, construct an abstract syntax tree to represent the syntax relationship and operation level in the LaTeX formula, and convert the abstract syntax tree into a structural semantic graph, wherein the nodes of the graph in the structural semantic graph represent the respective constituent elements in the formula, and the edges represent the semantic or structural relationship between the constituent elements.

[0106] The graph neural network embedding module is configured to encode the structural semantic graph by using a pre-trained model, extract a globally consistent semantic embedding vector, and form a formula embedding vector.

[0107] The text semantic modeling module is configured to perform semantic modeling on the text paragraph of the triple structure by using a pre-trained language model, generate an embedding vector representation, capture the implicit semantics related to the mathematical formula in the text paragraph, and generate a text embedding vector.

[0108] The semantic fusion and indexing module is configured to construct a retrieval index structure based on the formula embedding vector and the text embedding vector.

[0109] The multi-modal query analysis module is configured to respond to a query request of a user, perform modality recognition on the query request, process the query request according to the recognized modality, and generate semantic information.

[0110] The retrieval matching and sorting module is configured to query the retrieval index structure according to the semantic information, encapsulate the query result as a structured semantic vector, and the structured semantic vector includes a text embedding, a formula embedding, a modality label, and meta information.

[0111] Specifically, as shown in FIG. 1, the multi-source data access and format analysis module is mainly used to realize unified data access and standardized structure analysis of different mathematical literature databases, and the literature databases specifically include open source databases and commercial databases. Figure 1

[0112] Because the mathematical literature sources are diverse and the formats are complex, the design focus of the multi-source data access and format analysis module is to compatible different literature formats (such as LaTeX source code, PDF, XML, HTML, etc.) and uniformly convert them into a structured triple form that can be processed by the system.

[0113] ​First, in terms of data sources, this module supports access to multiple mainstream mathematical databases, including open source and commercial databases. For example, for the open source database arXiv.org, the latest papers in the math.* section are regularly crawled through its official API and OAI-PMH protocol; for commercial databases such as Springer, the original PDF, HTML, or XML data documents are obtained through the authorized FTP interface.

[0114] Second, to achieve unified structural modeling of multi-source heterogeneous mathematical literature, the content of literature from different formats (such as LaTeX source code, PDF, XML, HTML, etc.) is converted into a unified triple structure. The form is <text paragraph, LaTeX formula, meta information>, which can fully express the language content, formula structure, and source information of mathematical knowledge, and is the basis for subsequent semantic modeling and retrieval.

[0115] For literature provided in LaTeX source code form, such as arXiv.org, the system extracts literature paragraphs and formula areas through tools such as LaTeXML and TexSoup, and keeps the original LaTeX formula (or LaTeX mathematical expression, LaTeX structured formula, etc.) unchanged. Through structure tag recognition and text location matching, standard triple structures are directly generated.

[0116] For journal literature in PDF format (such as Springer, Elsevier), due to the lack of formula source code, the system needs to first identify the formula area (through layout analysis and visual detection), and then use the mathematical formula recognition tool MathPix to reconstruct the image or character combination into equivalent LaTeX formula. Then bind the reconstructed formula with the context paragraph to generate a triple structure with consistent structure.

[0117] For XML format literature, the system directly parses the ` ` and ` <formula>structured tags, extract the text and MathML expressions, and convert MathML to standard LaTeX formulas using tools, and then output as triples.

[0118] For HTML web page type content, such as zbMath or mathematical encyclopedia, the system extracts page paragraphs and embedded formula tags, and then converts them to LaTeX formulas and stores them uniformly.

[0119] During the entire process, the multi-source data access and format parsing module cleans and standardizes the formula and paragraph content, and labels the source, format type, and access permission state of each piece of data. Ultimately, regardless of the original format, all data will be uniformly converted into structured triples that can be used for subsequent semantic graph modeling, ensuring high compatibility and uniformity of the system.

[0120] As shown in Figure 2 LaTeX structure parsing and semantic construction module, including LaTeX structure parsing module and semantic construction module, after the multi-source data access and format parsing module runs, the system has uniformly converted all literature content into structured triple form, i.e. <text paragraph, LaTeX formula, meta information>, at this time, the LaTeX structure parsing module further processes the formula part based on the LaTeX formula field in the triple, the specific process is as follows:

[0121] LaTeX formula extraction and preparation: The LaTeX structure parsing module first reads the LaTeX formula in the triple and inputs it for parsing one by one. These LaTeX formulas are usually formulas in mathematical environments, such as \frac{a+b}{c}, \int_0^1 f(x) dx, etc. This module performs syntax-level parsing processing on these expressions.

[0122] Formula structure parsing and construction: Use LaTeXML, TexSoup or custom parser to parse LaTeX formulas into Abstract Syntax Tree (AST) for representing syntax relationships and operation levels in formulas. For example, \frac{a+b}{c} is parsed into a root node \frac, with a+b as the left child and c as the right child; for the integral \int_0^1 f(x) dx, it can be parsed into a "integral" root node with "upper and lower limits", "integrand function", "differential term" and other structure child nodes.

[0123] Structure Semantic Graph Generation: After obtaining the AST, the semantic construction module further converts it into a structure semantic graph, where the nodes of the graph represent the individual components in the formula (variables, operators, functions, constants, etc.), and the edges represent the semantic or structural relationships between the components, such as "operation", "subscript", "nesting", "function action", etc. The nodes and edges can have attributes such as type (variable / symbol), depth level, position in the original formula, etc.

[0124] As shown in Figure 3 , the core task of the graph neural network embedding module is to convert the mathematical formula structure semantic graph constructed in the previous stage into a computable and comparable high-dimensional vector representation. The system uses a graph convolutional network (GCN) model to encode the formula graph. Each node in the semantic graph represents a component in the mathematical formula, such as a variable, an operator, a function symbol, or a constant, and the edges represent the structural or semantic relationships between them, such as "subscript", "function action", "nesting structure", etc. The GNN model realizes the modeling of the nesting relationship, operation priority, and symbol dependence of the entire formula structure through layer-by-layer feature aggregation mechanism, thereby extracting a globally consistent semantic embedding vector.

[0125] To further improve the embedding quality and generalization ability, the structural features of the graph (node type, position embedding, depth information) are introduced as input features during training, and contrastive learning and self-supervised learning are combined to enhance the discrimination ability of formulas with similar structures but different semantics, or different structures but similar semantics. The final output formula embedding vector can be directly used for structural formula retrieval, semantic alignment, formula normalization, etc., and is a key bridge connecting formula representation and natural language semantics.

[0126] As shown in Figure 4 , the text semantic modeling module is used for deep semantic understanding and vector representation of the natural language content in mathematical literature, including text sections in the main text, abstracts, titles, and user queries.

[0127] The text semantic modeling module uses a pre-trained language model based on the Transformer architecture for semantic modeling, and the selected models can include BERT, SciBERT, RoBERTa, MathBERT, which have modeling capabilities sensitive to context, syntactic structure, and mathematical terminology.

[0128] By encoding the natural language paragraphs, the system can generate semantic continuous and context-sensitive vector representations, capturing the core concepts, logical arguments, and implicit semantics related to mathematical formulas in the paragraphs.

[0129] In the actual modeling process, for the part containing LaTeX formula reference, term chain or long sentence dependence in the text, the system not only extracts the pure text information, but also constructs a composite representation by combining the corresponding formula embedding vector. For example, when "can be obtained from Cauchy inequality" appears in a piece of text, the system models "Cauchy inequality" and its corresponding formula structure vector jointly, enhancing the mathematical completeness of the overall semantic representation. In addition, the system uses syntax analysis (such as dependency syntax tree, constituent structure tree, etc.) to deconstruct the text structure, extracts the syntax path as a semantic feature to assist modeling, and further improves the understanding ability of complex sentence type and nested expression.

[0130] Finally, the embedding vector output by the text semantic modeling module is not only used to construct the literature semantic index, but also as the core input for matching user query expression and literature content, realizing accurate semantic alignment and retrieval between cross-modal. The text semantic modeling module ensures the semantic integrity of natural language expression while laying a high-quality text foundation for the overall semantic fusion and retrieval sorting of the system.

[0131] As shown in Figure 5 , the multi-modal query analysis module is used to support users to submit query requests in various forms, and to parse the input content of different modalities into unified semantic representation as the entrance for semantic index and retrieval matching of the system. The multi-modal query analysis module can receive natural language text, LaTeX formula, mathematical formula image, or any combination of the three, to adapt to complex and diverse retrieval needs.

[0132] First, the input of the user is modality recognized and segmented, to determine whether it is pure text input, LaTeX formula, image format, or a composite query mixed with text and formula. In terms of natural language input, the system uses a language model based on Transformer for semantic modeling, including word segmentation, term recognition and context encoding, to extract structured semantic vectors for matching. If the user input is a LaTeX formula, it is directly handed over to the structure analysis module for syntax analysis and semantic graph construction, and a graph embedding representation is generated. For image format formula input, the system calls the OCR module MathPix to convert the image into a standard LaTeX formula, and then enters the structure analysis process uniformly, realizing the format alignment between image and text modalities.

[0133] When facing a mixed query (i.e. natural language + mathematical formula), the system first completes the modal segmentation, then processes the content of each modality respectively, and finally integrates the semantic information of multiple modalities through a modal fusion strategy to output a complete and unified query representation. This unified representation not only preserves the language context, but also contains the structural features of the formula, enhancing the system's understanding and matching ability for mathematical semantic queries.

[0134] In addition, to enhance the robustness and adaptability of the query, the module provides query normalization and semantic completion functions. For example, the system identifies "Cauchy Inequality" as a synonymous expression with "Cauchy Inequality", or automatically extends the query phrase "contains Euler formula" to a more explicit expression, and formulates it. All parsed query information is finally encapsulated as a structured semantic vector, including text embedding, formula embedding, modality label and meta information, for subsequent semantic fusion and indexing module matching and sorting processing.

[0135] As shown in Figure 6 The semantic fusion and indexing module is used to map the literature content (such as text paragraphs and mathematical formulas) from different sources and the multi-modal query (natural language and formula) of the user to a unified semantic vector space, and construct an efficient retrieval index structure to support subsequent fast matching and sorting. The core tasks of the semantic fusion and indexing module include multi-modal semantic fusion modeling and vector index construction.

[0136] In the semantic fusion stage, the semantic fusion and indexing module first obtains the high-dimensional vector representation output by the text semantic modeling module and the graph neural network embedding module, which respectively correspond to the natural language paragraphs and structured mathematical formulas of the literature. To realize the semantic alignment between multiple modalities, the semantic fusion and indexing module uses a fusion strategy to combine the vectors of the two modalities. Common fusion methods include vector splicing, weighted averaging, and modality attention mechanism. Among them, the modality attention mechanism can automatically learn the importance distribution of text and formula in different contexts, thereby realizing dynamic semantic fusion and further improving the semantic consistency of the representation.

[0137] After fusion, the semantic fusion and indexing module normalizes the fusion vectors of all literature samples and constructs a vectorized retrieval index. To meet the large-scale retrieval requirements, the Milvus system supporting vector semantic retrieval can be selected, which supports fast high-dimensional vector retrieval based on approximate nearest neighbor technology, greatly improving the response speed and concurrent processing capacity of the system.

[0138] In addition, to improve the retrieval performance and recall accuracy, the semantic fusion and indexing module introduces a multi-granularity indexing strategy. For example, the index can be constructed at the paragraph level, or multiple triple semantic vectors of the entire literature can be aggregated into a literature-level vector to realize multi-level retrieval; at the same time, different feature spaces such as formula, text, and meta information are supported to construct indexes and joint recall, improving the sensitivity of the system to modal content.

[0139] As shown in Figure 7 As shown, the retrieval matching and ranking module is used to further model deep semantic matching and relevance ranking of the candidate document set recalled by the semantic fusion and indexing module, to generate the final retrieval result list for presentation to the user. The retrieval matching and ranking module can significantly improve the accuracy of multi-modal semantic matching and the end-user experience.

[0140] Specifically, after the semantic fusion indexing module completes the preliminary vector retrieval, a candidate document set similar to the query semantics is obtained.

[0141] To overcome the limitations of shallow vector matching in expressing ambiguity, structural deformation, long-distance dependence, etc., the retrieval matching and ranking module uses a deep semantic matching model to model and match each candidate document and the original query in both directions. The retrieval matching and ranking module supports multiple matching structures, including Dual Encoder and Cross-Encoder structures. Among them, the Dual Encoder structure calculates the similarity after independent encoding of the query and the document, which is faster; the Cross-Encoder merges the query and the document into a unified model (such as BERT or RoBERTa) to obtain a more refined matching degree, which is suitable for fine ranking of small-scale candidate sets.

[0142] In terms of multi-modal matching, the retrieval matching and ranking module considers the natural language expression, LaTeX structured formula, and possible image formula transcription results in the query, as well as the text content, formula structure embedding vector, and meta information (keywords, classification labels, publication year, etc.) in the document for integrated matching. For formula structure matching, the retrieval matching and ranking module supports discriminating transformed formulas (such as equivalent transformations, symbol substitutions, and simplified versions) through graph structure similarity or contrast learning enhanced models, thereby improving the robustness of formula-level retrieval.

[0143] The retrieval matching and ranking module performs multi-dimensional weighted ranking on the candidate documents based on the matching degree scores output by the matching model. In some embodiments, in addition to the matching degree score, the ranking factor also introduces external features such as the number of document citations, publication time, novelty score, author influence, and domain authority, and performs final ranking optimization through a learning ranking model BERT Ranker. The system supports soft label supervision in the training and inference stages, such as citation pairs, click logs, and artificial similarity annotations.

[0144] Finally, the retrieval matching and ranking module returns a ranked and semantically consistent document result list to the user, and supports switching the result ranking display according to "text relevance", "formula structure similarity", or "the comprehensive weight of text relevance and formula structure similarity" in the retrieval interface, to improve the user's understanding of the retrieval intent and result interpretation.

[0145] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) having computer-usable program code embodied in the medium. CD - ROM

[0146] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0147] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0149] The above description is only preferred embodiments of the present application, and is not intended to limit the present application. The present application can have various changes and modifications within the spirit and principle thereof, and any modifications, equivalent replacements, improvements, etc. made by those skilled in the art without creative efforts should be included in the protection scope of the present application.​< / formula>

Claims

1. A multi-modal mathematical literature retrieval method based on structural semantic graph embedding, characterized in that, The method comprises the following steps: Obtaining multi-source heterogeneous mathematical literature, converting the same, and converting the content of mathematical literature in different formats into a unified triple structure, the triple structure comprising a text paragraph, a LaTeX formula and meta information; Extracting the expression of the LaTeX formula in the triple structure, performing analysis, and constructing an abstract syntax tree to represent the syntax relationship and operation level in the LaTeX formula, and converting the abstract syntax tree into a structure semantic graph, wherein the nodes of the graph represent the constituent elements in the formula, and the edges represent the semantic or structural relationship between the constituent elements; Using a pre-trained model to encode the structure semantic graph, extract a globally consistent semantic embedding vector, and form a formula embedding vector; the process comprises: using a pre-trained graph convolution network model to encode the structure semantic graph, wherein the nodes represent the constituent elements in the mathematical formula, and the edges represent the structural or semantic relationship between them, and through a layer-by-layer feature aggregation mechanism, the nested relationship, operation priority and symbol dependence of the entire formula structure are modeled, and a globally consistent semantic embedding vector is extracted; In the training process, the structural features of the graph are introduced as input features, and a contrast learning algorithm and a self-supervised learning algorithm are used to enhance the discrimination ability of the graph convolution network model for formulas with similar structures but different semantics, or different structures but similar semantics; the structural features comprise node type, position embedding and depth information; Using a pre-trained language model to perform semantic modeling on the text paragraph of the triple structure, generating an embedding vector representation to capture the implicit semantics related to the mathematical formula in the text paragraph, and generating a text embedding vector; Based on the formula embedding vector and the text embedding vector, a retrieval index structure is constructed; the process comprises: adopting a fusion strategy to fuse the formula embedding vector and the text embedding vector, uniformly normalizing the fused vectors of all documents, and constructing a vectorized retrieval index; When constructing the vectorized retrieval index, the index is constructed at the paragraph level or the multiple triple semantic vectors of the entire document are aggregated into a document-level vector to realize multi-level retrieval; In response to a user's query request, the query request is subjected to modality recognition, processed according to the recognized modality, semantic information is generated, the retrieval index structure is queried according to the semantic information, and the query result is encapsulated into a structured semantic vector, the structured semantic vector comprising a text embedding, a formula embedding, a modality label and meta information.

2. The multi-modal mathematical document retrieval method based on structural semantic graph embedding according to claim 1, characterized in that, The process of obtaining multi-source heterogeneous mathematical literature and converting the same into a unified triple structure comprises: accessing different mathematical databases, regularly grabbing the latest data documents under the mathematical partition for open-source mathematical databases; for commercial databases, obtaining original data documents through an authorized interface; For documents provided in the form of LaTeX source code, the document paragraphs and formula regions are extracted, and the original LaTeX expression is kept unchanged, the structure label recognition is matched with the text position, and a structure-consistent triple is generated; For PDF format literature, the formula region is identified by layout analysis and visual detection, and then the image or character combination is reconstructed into an equivalent LaTeX formula. The reconstructed equivalent LaTeX formula is bound with the context paragraph to generate a consistent triple structure; For XML format literature, the structured tags are parsed, the text and MathML expressions are extracted, and the MathML expressions are converted into standard LaTeX formulas to generate a consistent triple structure; For HTML web page type content, the page paragraphs and embedded formula tags are extracted and converted into LaTeX formulas to generate a consistent triple structure; The form of the consistent triple structure is <text paragraph, LaTeX formula, meta information>.

3. The multi-modal mathematical document retrieval method based on structural semantic graph embedding according to claim 1, characterized in that, After obtaining multi-source heterogeneous mathematical literature, the content of mathematical literature in different formats is converted into a unified triple structure. During the conversion process, the converted formula and paragraph content are cleaned and standardized, and the source, format type and access permission state of each piece of data are labeled.

4. The multi-modal mathematical document retrieval method based on structural semantic graph embedding according to claim 1, characterized in that, The expression of the LaTeX formula in the triple structure is extracted, parsed, and an abstract syntax tree is constructed to represent the syntax relationship and operation level in the LaTeX formula. The process of converting the abstract syntax tree into a structural semantic graph includes: reading the LaTeX formula in the triple, inputting and parsing it one by one, and parsing the LaTeX formula into an abstract syntax tree to represent the syntax relationship and operation level in the formula. After obtaining the abstract syntax tree, the abstract syntax tree is converted into a structural semantic graph, where the nodes of the graph represent the elements of the formula, including variables, operators, functions and constants, and the edges represent the semantic or structural relationship between the elements. Attributes are set for at least part of the nodes and edges, including type, depth level and position in the original formula.

5. The multi-modal mathematical document retrieval method based on structural semantic graph embedding according to claim 1, characterized in that, The process of using a pre-trained language model to perform semantic modeling on the text paragraph of the triple structure to generate an embedding vector representation to capture the implicit semantics related to the mathematical formula in the text paragraph includes: using a pre-trained language model to encode the natural language paragraph. For parts of the text that contain LaTeX formula references, term chains or long sentence dependencies, extract their pure text information and construct a composite representation jointly with the corresponding formula embedding vector; use syntax analysis method to structure the text, and extract the syntax path as a semantic feature to assist modeling.

6. The multi-modal mathematical document retrieval method based on structural semantic graph embedding according to claim 1, characterized in that, In response to a user's query request, the query request is modality recognized and processed according to the recognized modality to generate semantic information, which includes: Modal recognition and segmentation of user input to determine whether it is pure text, LaTeX formula, image format or a composite query of text and formula; If the user's input is pure text, perform semantic modeling on the text to generate a text embedding representation; If the user's input is a LaTeX formula, perform syntax analysis and semantic graph construction, and generate a graph embedding representation; If the user's input is an image format formula, convert the image into a standard LaTeX formula and then uniformly enter the structural analysis process to align the formats between text and image modalities; If the user's input is a complex query mixed with text and formula, modal segmentation is performed, and the content of each modality is processed respectively. The semantic information of multiple modalities is integrated through a modal fusion strategy to obtain a complete and structurally unified query representation.

7. The multi-modal mathematical document retrieval method based on structural semantic graph embedding according to claim 1, characterized in that, The process of querying the retrieval index structure according to the semantic information includes: querying the retrieval index structure according to the semantic information generated by the query to obtain a candidate document set similar to the query semantics, and using a deep semantic matching model to model and match the original query and each candidate document in the candidate document set in both directions and score the matching degree; In the matching process, the natural language expression in the query, the LaTeX formula and the image formula transcription, and the text content, the formula structure embedding vector and the meta information in the document are considered for integrated matching; Based on the obtained matching degree score, the candidate documents are sorted, and the matching degree and external features are considered for final sorting optimization during the sorting, wherein the external features include some of the number of document citations, publication time, novelty score, author influence and field authority.

8. A multi-modal mathematical literature retrieval system based on structural semantic graph embeddings, characterized in that, Implement a multi-modal mathematical document retrieval method based on structure semantic graph embedding as claimed in any one of claims 1-7, comprising: A multi-source data access and format analysis module configured to obtain multi-source heterogeneous mathematical documents, and convert the mathematical document content in different formats into a unified triple structure, the triple structure including text paragraphs, LaTeX formulas and meta information; A LaTeX structure analysis and semantic construction module configured to extract the expression of the LaTeX formula in the triple structure, analyze and construct an abstract syntax tree to represent the syntax relationship and operation level in the LaTeX formula, and convert the abstract syntax tree into a structure semantic graph, wherein the nodes of the graph represent the elements in the formula, and the edges represent the semantic or structural relationship between the elements; A graph neural network embedding module configured to encode the structure semantic graph using a pre-trained model to extract a globally consistent semantic embedding vector to form a formula embedding vector; A text semantic modeling module configured to use a pre-trained language model to perform semantic modeling on the text paragraphs of the triple structure to generate an embedding vector representation to capture the implicit semantics related to the mathematical formula in the text paragraphs, and generate a text embedding vector; A semantic fusion and index module configured to construct a retrieval index structure based on the formula embedding vector and the text embedding vector; A multi-modal query analysis module configured to respond to a user's query request, identify the modality of the query request, process the query request according to the identified modality, and generate semantic information; A retrieval matching and sorting module configured to query the retrieval index structure according to the semantic information, and encapsulate the query result as a structured semantic vector, wherein the structured semantic vector includes a text embedding, a formula embedding, a modality label and meta information.

Images