Multimodal patent document information processing method, apparatus, device, and medium

Abstract

The application provides a multi-modal patent document information processing method, device, equipment and medium. In the method, a multi-modal patent document containing multiple functional modules is obtained; the patent document is modularly split according to the title level characteristics of each functional module, and a title-extraction information index table is constructed for each module obtained by splitting; multi-modal information is extracted from each module based on the title-extraction information index table, wherein the multi-modal information includes text information, image information and / or formula information; after the extracted multi-modal information is associated and integrated with the title-extraction information index table, input information of a large language model is obtained, and the input information is input into the large language model to generate a structured processing result of the patent document. The scheme is used to solve the problem of poor accuracy and insufficient reliability of output results in related technologies for processing multi-modal patent document information based on a large model.

Description

Technical Field

[0001] This application relates to the field of document processing technology, and in particular to a multimodal patent document information processing method, apparatus, device and medium. Background Technology

[0002] With the development of large language models, related technologies have begun to gradually try to directly input the entire patent document into the large model, or to use search-enhanced generation (RAG) to extract and process information, in order to replace traditional manual operations.

[0003] However, patent documents have complex structures with numerous cross-module references. Directly processing entire long documents or performing only fragment searches can easily lead to excessively long contexts, making it difficult for large models to effectively identify the overall structure of the document and the semantic relationships between different modules. This can result in missing key information or generating illusions of reasoning, leading to poor processing accuracy and insufficient reliability of output results. Summary of the Invention

[0004] The multimodal patent document information processing method, apparatus, device, and medium provided in this application are intended to solve the problems of poor accuracy and insufficient reliability of output results in the related art when processing multimodal patent document information based on large models.

[0005] In a first aspect, embodiments of this application provide a multimodal patent document information processing method, including:

[0006] Obtain multimodal patent documents containing multiple functional modules;

[0007] Based on the title hierarchy characteristics of each functional module, the patent document is modularly split, and a title-extracted information index table is constructed for each module obtained from the split.

[0008] Based on the title-extracted information index table, multimodal information is extracted from each module, including text information, image information and / or formula information;

[0009] After the extracted multimodal information is associated and integrated with the title-extracted information index table, the input information of the large language model is obtained. The input information is then input into the large language model to generate the structured processing result of the patent document.

[0010] In one embodiment of this disclosure, the patent document is modularly split according to the title hierarchy characteristics of each functional module, and a title-extracted information index table is constructed. This includes: matching the hierarchical titles in the document based on a preset regular expression library, splitting the document into corresponding first-level modules, wherein the first-level modules include at least one of a specification abstract module, a claims module, a specification module, and a specification drawing module; if the first-level modules include a specification module, the specification module is further subdivided into corresponding second-level modules, wherein the second-level modules include at least one of a technical field module, a background technology module, an invention content module, a drawing description module, and a detailed implementation module; and constructing a title-extracted information index table, which is used to record the type, title, content start position, and subsequent extracted multimodal information association address of each module.

[0011] In one embodiment of this disclosure, before constructing the title-extracted information index table, the method further includes: identifying cross-module reference relationships between modules, and constructing a cross-module reference relationship graph with the decomposed modules as nodes and the reference relationships as edges, wherein the reference relationship graph contains reference chains between modules, and modules located on the same reference chain are associated modules; according to the requirements of the current processing task corresponding to the patent document, a graph traversal algorithm is used to load the associated module information on the reference chain; a graph attention mechanism is used to perform weighted aggregation of the information of different modules to obtain the corresponding aggregation result; and the aggregation result is added to the input information of the large language model.

[0012] In one embodiment of this disclosure, after extracting multimodal information from each module based on the title-extracted information index table, the method further includes: if at least one image to be detected is extracted from each module, extracting corresponding spatial and frequency domain features for the extracted at least one image to be detected; using a preset fusion weight coefficient, weighting and concatenating the spatial and frequency domain features and normalizing them to obtain a fusion feature vector of the image to be detected; if there are at least two images to be detected, calculating the cosine similarity of the fusion feature vectors of each image, comparing the similarity with a preset threshold to determine whether the images are similar; and adding the comparison result to the input information.

[0013] In one embodiment of this disclosure, if the primary module includes a specification module, after further subdividing the specification module into corresponding secondary modules, the method further includes: if the primary module includes a specification module and a specification drawing module, extracting the drawing reference relationship from the subdivided specification module using regular expression matching; based on an object detection model, identifying whether there are components and numbering features mentioned in the text description corresponding to the drawing reference relationship in the image from the corresponding specification drawing module; generating a text-image consistency score based on the identification result, and adding the text-image consistency score to the input information of the large language model.

[0014] In one embodiment of this disclosure, formula information is extracted as follows: the formula region in the patent document is detected, and the formula image is converted into semantic text in a preset format based on the formula recognition model; the converted semantic text is concatenated with the ordinary text of the paragraph containing the formula to form a mixed text containing the semantics corresponding to the formula; the mixed text replaces the original formula image at the corresponding position in the patent document and is added to the input information of the large language model.

[0015] In one embodiment of this disclosure, after extracting multimodal information from each module based on the title-extracted information index table, the method further includes: if text information is extracted from each module, calculating the text length of the text information in each module, and detecting abnormal punctuation and format features in the text information; organizing the text length, abnormal punctuation type, and format features into structured basic prior information and writing it into the title-extracted information index table; and using the basic prior information in the title-extracted information index table as context enhancement information during large language model inference.

[0016] Secondly, embodiments of this application provide a multimodal patent document information processing apparatus, comprising:

[0017] The acquisition module is used to acquire multimodal patent documents that contain multiple functional modules.

[0018] The splitting module is used to modularly split the patent document according to the title level characteristics of each functional block, and to build a title-extracted information index table for each split module;

[0019] The extraction module is used to extract multimodal information from each block based on the title-extractable information index table. The multimodal information includes text information, image information, and / or formula information.

[0020] The processing module is used to integrate the extracted multimodal information with the title-extracted information index table to obtain the input information of the large language model, and then input the input information into the large language model to generate the structured processing result of the patent document.

[0021] Optionally, the acquisition module is specifically used to: match the hierarchical headings in the document based on a preset regular expression library, and split the document into corresponding first-level modules, wherein the first-level modules include at least one of the specification abstract module, claims module, specification module, and specification drawing module; if the first-level modules include the specification module, the specification module is further subdivided into corresponding second-level modules, wherein the second-level modules include at least one of the technical field module, background technology module, invention content module, drawing description module, and detailed implementation module; and construct a title-extracted information index table, which is used to record the type, title, content start position, and subsequent multimodal information association address of each module.

[0022] Optionally, the processing module is also used to: identify cross-module reference relationships between modules before constructing the title-extracted information index table; construct a cross-module reference relationship graph with the decomposed modules as nodes and reference relationships as edges, wherein the reference relationship graph contains reference chains between modules, and modules located on the same reference chain are associated modules; load the associated module information on the reference chain using a graph traversal algorithm according to the requirements of the current processing task corresponding to the patent document; use a graph attention mechanism to perform weighted aggregation of the information of different modules to obtain the corresponding aggregation result; and add the aggregation result to the input information of the large language model.

[0023] Optionally, the extraction module is further configured to, based on the title-extraction information index table, extract multimodal information from each module respectively; if at least one image to be detected is extracted from each module, extract the corresponding spatial domain features and frequency domain features for the extracted at least one image to be detected respectively; use a preset fusion weight coefficient to weight and concatenate the spatial domain features and normalize them to obtain the fusion feature vector of the image to be detected; if there are at least two images to be detected, calculate the cosine similarity of the fusion feature vectors of each image, compare the similarity with a preset threshold to determine whether the images are similar; and add the comparison result to the input information.

[0024] Optionally, the processing module is further configured to: if the primary module includes a manual module, further subdivide the manual module into corresponding secondary modules; if the primary module includes both a manual module and a manual drawing module, extract the drawing reference relationships from the subdivided manual module using regular expression matching; based on the object detection model, identify whether there are components and numbering features mentioned in the text description corresponding to the drawing reference relationships in the image from the corresponding manual drawing module; generate a text-image consistency score based on the recognition results, and add the text-image consistency score to the input information of the large language model.

[0025] Optionally, the extraction module is specifically used to extract formula information in the following ways: detect the formula area in the patent document, and convert the formula image into semantic text in a preset format based on the formula recognition model; concatenate the converted semantic text with the ordinary text of the paragraph where the formula is located to form a mixed text containing the semantics corresponding to the formula; replace the original formula image at the corresponding position in the patent document with the mixed text, and add it to the input information of the large language model.

[0026] Optionally, the splitting module is also used for: after extracting multimodal information from each module based on the title-extracted information index table, if text information is extracted from each module, calculating the text length of the text information in each module, and detecting abnormal punctuation and format features in the text information; organizing the text length, abnormal punctuation type, and format features into structured basic prior information and writing it into the title-extracted information index table; and using the basic prior information in the title-extracted information index table as context enhancement information during large language model inference.

[0027] Thirdly, embodiments of this application provide an electronic device, including:

[0028] Processor, memory, communication interface;

[0029] Memory is used to store the processor's executable instructions;

[0030] The processor is configured to execute the multimodal patent document information processing method of any one of the first aspects by executing executable instructions.

[0031] Fourthly, embodiments of this application provide a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multimodal patent document information processing method of any one of the first aspects.

[0032] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, is used to implement the multimodal patent document information processing method of any one of the first aspects.

[0033] The multimodal patent document information processing method, apparatus, device, and medium provided in this application acquire a multimodal patent document containing multiple functional modules, modularize the patent document according to the title hierarchy features of each functional module, and construct a title-extracted information index table for each module. Then, based on this index table, multimodal information such as text, images, and formulas are extracted from each module. Finally, the extracted multimodal information is integrated with the index table and input into a large language model to generate a structured processing result. This transforms the originally lengthy and structurally ambiguous full-text input into a structured input with clear module boundaries and associated indexes. It effectively solves the technical defect of large language models that struggle to identify the overall document structure and semantic relationships between modules due to excessively long context, thereby suppressing information forgetting and inference illusions. This significantly improves the accuracy and reliability of automated information processing of multimodal patent documents, reduces computational overhead and response latency during processing, and achieves efficient and stable processing of complex structured documents. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0035] Figure 1 An application scenario diagram of the multimodal patent document information processing method provided in this application;

[0036] Figure 2 A flowchart illustrating an embodiment of the multimodal patent document information processing method provided in this application;

[0037] Figure 3 A flowchart illustrating an embodiment of a multimodal patent document information processing method provided in another embodiment of this application;

[0038] Figure 4 A schematic diagram of the structure of an embodiment of a multimodal patent document information processing device provided in another embodiment of this application;

[0039] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application.

[0040] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0041] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0042] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0043] Multimodal patent document information processing has broad and significant application value in patent examination, patent analysis, knowledge mining, and intelligent retrieval. However, the structure of patent documents is extremely complex, typically containing multiple functional modules such as a specification abstract, claims, specification, and drawings (some parts may be missing depending on the specific application, such as foreign patents that only have claims and a specification for patent analysis). There are numerous explicit or implicit cross-module references between these modules, such as claims citing technical features from the specification, and references to drawing numbers in specific embodiments.

[0044] Related automated processing solutions often directly input the entire document into a large language model, or use retrieval enhancement generation methods to perform only fragment retrieval and then rely on the model for independent analysis. These methods cannot explicitly model the hierarchical structure of the document and the logical relationships between modules. As a result, when large models process extremely long texts, key information is easily lost in the middle of the context, and frequently generate inferences that do not conform to the facts, resulting in poor accuracy and serious lack of reliability in the processing results.

[0045] The fundamental reason why related technologies struggle to overcome the aforementioned difficulties is that neither the direct input paradigm of the general large model nor the fragment recall paradigm generated by retrieval enhancement possesses the ability to proactively recognize the document structure itself.

[0046] Although the self-attention mechanism within a large model can capture local dependencies to some extent, when the document length exceeds the model's effective window, long-distance inter-module references will be diluted or even completely ignored.

[0047] While retrieval-enhanced generation methods can retrieve relevant fragments from documents, their retrieval granularity is usually at the paragraph or sentence level, failing to reflect module-level semantic boundaries and unable to build cross-module reference networks.

[0048] Moreover, none of these methods are specifically optimized for the high-density engineering line drawings, embedded formulas, and formatting constraints unique to patent documents. Furthermore, while the format of patent documents is relatively fixed, it varies across different fields. Existing methods based on fixed templates or simple rules lack generalization ability and struggle to adapt to diverse patent document structures, resulting in insufficient information processing effectiveness.

[0049] The multimodal patent document information processing method provided in this application deconstructs long documents into multiple short modules with clear boundaries through modular decomposition, and constructs a title-extracted information index table to uniformly manage the semantic content and location information of each module. This transforms the input of a large language model from an unstructured full text into a structured set of modules. Based on this, the extracted multimodal information from each module is integrated with the index table and input into the large model, enabling the model to accurately locate and call the required module information according to the index table's guidance, avoiding attention distraction and forgetting of key information due to excessively long contexts. Thus, by guiding model processing through explicit structural modeling, this method fundamentally overcomes the obstacle of existing methods' lack of ability to recognize the overall document structure and inter-module relationships, achieving highly accurate and reliable structured processing.

[0050] Figure 1 This is a schematic diagram illustrating the application scenario of the multimodal patent document information processing method provided in this application, such as... Figure 1 As shown, in the process of multimodal patent document information processing, after the server 100 obtains the multimodal patent document 110, it will parse and split it, and then output the corresponding processing result 120 through the large language model.

[0051] It should be noted that, Figure 1 The scenario shown includes a server, multimodal patent documents, and processing results, which are only illustrated by one or a specific number of examples. However, this disclosure is not limited to this. That is to say, the number of servers, multimodal patent documents, and processing results can be arbitrary.

[0052] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0053] Figure 2 A flowchart illustrating an embodiment of the multimodal patent document information processing method provided in this application is shown below. Figure 2 As shown, the method includes:

[0054] S201. Obtain a multimodal patent document containing multiple functional modules.

[0055] Specifically, this embodiment is used to illustrate the main steps of the multimodal patent document information processing method.

[0056] In this embodiment, the executing entity is one or more computers, systems, or servers with computing capabilities. This executing entity can acquire patent document data and perform information processing on multimodal patent documents according to specific processing tasks. For ease of description, it will be referred to as a server below.

[0057] In practical applications, patent documents are usually in the form of structured electronic files, such as XML files, PDF files, or text packages exported from patent databases that conform to the format requirements of the China Patent Office.

[0058] Patent documents contain multiple functional modules, which are organized according to the drafting guidelines of the Patent Law and its Implementing Regulations. Common modules include the abstract, claims, description, and drawings.

[0059] The instruction manual module can be further divided into sub-modules such as technical field, background technology, invention content, description of drawings, and specific implementation methods. The server reads the original content of the file through the document parsing interface, retaining the original information such as titles, text, figures, formulas, and their relative positions.

[0060] To facilitate subsequent processing, the server can convert the document into a standardized text representation format, such as... Encoded plain text and simultaneously recording the original coordinates and font attributes of each character.

[0061] In addition, the server also supports extracting text and layout structure from patent scans in image format using optical character recognition technology, thereby obtaining digital documents with module boundary markers.

[0062] After retrieving the document, the server will perform an integrity check to ensure that there are no basic problems such as missing pages or garbled characters, and store the document in a temporary cache or distributed file system to prepare for subsequent modular splitting.

[0063] S202. Based on the title hierarchy characteristics of each functional module, the patent document is modularly split, and a title-extracted information index table is constructed for each module obtained from the split.

[0064] Specifically, the server first traverses the document's text stream and identifies all lines that appear to be headings.

[0065] Patent documents typically have fixed wording in their titles, such as "Claims," ​​"Abstract," or "Technical Field," and often feature layout characteristics such as blank lines, font size variations, or bolding before and after the title.

[0066] The server can pre-define a title feature library containing multiple regular expressions, each corresponding to a variant of a common title, such as "claims" which may be written as "claims" or "Claims".

[0067] The server scans each line in the document order. When a line matches any regular expression in the feature library, it determines that the line is the starting boundary of a functional module.

[0068] At the same time, the server also uses the document's hierarchical indentation or numbering depth to determine the subordinate relationship between modules. For example, the second-level heading "Technical Field" is below the first-level heading "Instructions".

[0069] After completing boundary recognition, the server records the title text, start line number, end line number, and original content fragments of each module, forming a module list.

[0070] Based on this, the server assigns a unique identifier to each module and creates a title-extractable information index table. This index table uses the module identifier as the primary key, with each row corresponding to a module. The column fields include module type, title name, content start position, content end position, and a reserved "extractable information associated address" field.

[0071] The address associated with the extracted information is initially empty. After the multimodal information is extracted, this address will point to the file path or memory pointer that stores the corresponding information.

[0072] With this index table, the server can quickly locate the content area of ​​any module, avoiding the need to rescan the entire text every time it is accessed, thereby significantly improving processing efficiency.

[0073] S203. Based on the title-extracted information index table, extract multimodal information from each module respectively.

[0074] Multimodal information includes text information, image information, and / or formula information.

[0075] Specifically, for each module, the server extracts the corresponding subtext from the original document based on the start and end positions of the content recorded in the index table.

[0076] For text information, the server does not perform any substantive analysis, but only retains the original character sequence, including punctuation marks, numbers, letters and Chinese characters, while retaining paragraph separators and line breaks to maintain the original text structure.

[0077] For image information, the server detection module checks whether it contains image reference tags. In patent documents, images typically exist as standalone files or embedded data streams. The server needs to parse the document format, extract the binary data of each image, and generate a unique image identifier for it. It also records the order in which the images appear in the module and their accompanying titles or numbers, such as "...". Figure 2 "FIG.3", etc.

[0078] For formula information, the server recognizes mathematical expressions that exist in image form or formulas formatted in a special font within the recognition module.

[0079] Since the presentation of formulas varies greatly in different documents, the server can first use heuristic rules, such as finding paragraphs with text prefixes such as "Formula (1)" and "Formula 3", and then locate the image area that follows them.

[0080] For inline formulas, character subscripts and superscripts, special symbols, and specific fonts are used for identification. After extraction, the server stores the above multimodal information in separate files and fills the storage path or memory address of the files back into the "Extraction Information Association Address" field of the title-extraction information index table.

[0081] For example, in the "Detailed Implementation" module, the text information is saved as a .txt file, the image information is saved as a list of .png files, the formula information is saved as a .tex or text description file, and the access paths of these files are recorded in the index table.

[0082] In this way, the server transforms the unstructured raw documents into a structured dataset organized by modules and indexed by a table, providing clear and compact input material for subsequent processing by the large language model.

[0083] S204. After associating and integrating the extracted multimodal information with the title-extracted information index table, the input information of the large language model is obtained. The input information is then input into the large language model to generate the structured processing result of the patent document.

[0084] Specifically, the server first reads all records in the title-extract information index table and generates a complete context frame according to the natural order of the modules.

[0085] The framework describes the title and type of each module in text form and embeds reference tags pointing to specific multimodal information files in the appropriate locations.

[0086] For example, the server can construct an intermediate representation: for the "Instruction Manual Illustrations" module, write "List of Illustrations:" followed by an identifier for each image and a brief visual description placeholder in the text stream. For formulas, insert their converted semantic text directly into the corresponding position in the text stream.

[0087] The process of integration also includes explicitly marking the reference relationships between different modules, for example, when the "Detailed Implementation" mentions "such as..." Figure 2 As shown, the server will insert a link to "attached" near the text. Figure 2 "Content anchors enable large language models to align text descriptions with image content when reading input information."

[0088] After integration, the server receives a structured prompt, which includes system role settings, task descriptions, modularly organized document content, and output format requirements.

[0089] The server sends the prompt as input to a large language model inference service deployed locally or in the cloud.

[0090] The large language model analyzes the document content module by module based on the index structure and reference tags in the input information, judges the logical consistency between modules, locates possible anomalies or missing information, and generates structured processing results according to a predefined output format.

[0091] The result can be a JSON object, where each field corresponds to the detection conclusion of a module, including the anomaly type, location, and brief description.

[0092] After receiving the result, the server converts it into a user-readable report or stores it in a database for later use.

[0093] At this point, the server has completed a full multimodal patent document information processing operation based on modular splitting and index table guidance.

[0094] The multimodal patent document information processing method provided in this application acquires a multimodal patent document containing multiple functional modules, modularizes the patent document according to the title hierarchy features of each functional module, and constructs a title-extracted information index table for each module. Then, based on this index table, it extracts multimodal information such as text, images, and formulas from each module. Finally, it integrates the extracted multimodal information with the index table and inputs it into a large language model to generate a structured processing result. This transforms the originally lengthy and structurally ambiguous full-text input into a structured input with clear module boundaries and associated indexes. It effectively solves the technical defect of large language models that struggle to identify the overall document structure and semantic relationships between modules due to excessively long context, thereby suppressing information forgetting and inference illusions. This significantly improves the accuracy and reliability of automated information processing of multimodal patent documents, reduces computational overhead and response latency, and achieves efficient and stable processing of complex structured documents.

[0095] Figure 3 A flowchart illustrating an embodiment of the multimodal patent document information processing method provided in this application is shown below. Figure 3 As shown, in this embodiment... Figure 3 Based on the embodiments, the specific process of the multimodal patent document information processing method is described in detail. The method includes:

[0096] S301. Obtain a multimodal patent document containing multiple functional modules.

[0097] Specifically, this step is related to Figure 2 The corresponding steps in the illustrated embodiments are the same and will not be repeated here.

[0098] S302. Based on the preset regular expression library, match the hierarchical headings in the document and split the document into corresponding first-level modules.

[0099] The first-level module includes at least one of the following modules: abstract module, claims module, description module, and drawings module.

[0100] Specifically, the server matches hierarchical headings in the document based on a pre-defined regular expression library. This regular expression library can be configured according to common heading expressions in Chinese patent documents. For example, the regular expression corresponding to the specification abstract module could be:

[0101] (?:^|\n) Summary of Instructions\s*\n,

[0102] The regular expression corresponding to the claims module can be:

[0103] (?:^|\n) Claims\s*\n.

[0104] S303. If a primary module contains a manual module, the manual module shall be further subdivided into corresponding secondary modules.

[0105] The secondary module includes at least one of the following modules: technical field module, background technology module, invention content module, drawing description module, and detailed implementation module.

[0106] Specifically, for the instruction manual module in the first-level module, the server further identifies its internal second-level headings, such as "Technical Field", "Background Art", "Summary of the Invention", "Description of Drawings", and "Detailed Implementation" (i.e., Technical Field Module, Background Art Module, Summary of the Invention Module, Description of Drawings Module, and Detailed Implementation Module), and divides the instruction manual content into corresponding second-level modules according to the heading level.

[0107] It should be noted that when an optional module is missing from a document, the server will only process the existing modules and will not interrupt the process due to the missing module.

[0108] In some embodiments, the processing of relationships between different modules may include the following steps:

[0109] Step A1: Identify the cross-module reference relationships between modules, and construct a cross-module reference relationship graph using the modules obtained from the split as nodes and the reference relationships as edges.

[0110] The reference relationship graph contains reference chains between modules, and modules on the same reference chain are related modules.

[0111] Specifically, before building the title-extracted information index table, the server performs collaborative processing of cross-module related information.

[0112] The server scans the text content of each module and uses regular expression matching to identify explicit cross-module reference patterns, such as "as described in claim X", "see paragraph X of the specification", "as shown in Figure X", etc. Then, with each module as a node and the identified reference relationships as directed edges, a directed reference relationship graph is constructed.

[0113] At this point, each node in the graph records the type and identifier of the module, and each edge records the reference type and target module.

[0114] The reference relationship diagram naturally forms several reference chains. Modules on the same reference chain are related modules. For example, if the first item in the claim references a certain feature in the specification, then the module in the claim and the module in the specification are on the same reference chain.

[0115] Step A2: Based on the requirements of the current processing task corresponding to the patent document, use a graph traversal algorithm to load the associated module information on the reference chain.

[0116] Specifically, based on the needs of the current processing task, the server determines the starting module that needs to be analyzed in detail, and then uses a depth-first or breadth-first graph traversal algorithm to load the related module information on the reference chain as needed along the directed edges.

[0117] For example, when the task is to check the consistency between the claims and the specification, only the claims module and the specification module directly referenced by it are loaded, without loading the drawing module, thereby reducing the computational load.

[0118] Step A3: Use graph attention mechanism to weighted aggregate information from different modules to obtain the corresponding aggregation result.

[0119] Specifically, the server employs a graph attention mechanism to weighted aggregate information from different modules. Specifically, for each node in the referencing graph, the server performs attention calculations on its feature vector (which can be initialized based on the module's textual semantics or image features) and the feature vectors of its neighboring nodes to obtain the importance weight of each neighboring node to the current node.

[0120] The server then performs a weighted summation of the feature vectors of neighboring nodes to generate an aggregated cross-module feature representation. The graph attention mechanism automatically learns which inter-module reference relationships are more critical, thus highlighting important information during the aggregation process.

[0121] Step A4: Add the aggregation results to the input information of the large language model.

[0122] Specifically, the server converts the aggregated cross-module feature representations into text descriptions or embedding vectors, which are then added to the input information of the subsequent large language model.

[0123] For example, for module pairs with close referencing relationships, the input information will explicitly state that "module A references content from module B, and the two should be consistent," thereby guiding the large model to focus on the logical connections across modules.

[0124] S304. Construct a title-extracted information index table.

[0125] The index table records the type, title, starting position of the content, and associated address of the multimodal information extracted subsequently for each module.

[0126] Specifically, after completing the modular splitting, the server builds an index record for each identified module.

[0127] Each record must contain at least the module type, module title, content start position, content end position, and a reserved "multimodal information association address" field.

[0128] Module types are used to distinguish between first-level modules and second-level modules, such as "Abstract of Specification", "Claims", or "Technical Field".

[0129] The start and end positions of the content can be represented by character offsets or line number ranges, such as "start=1024, end=2048", which facilitates quick extraction of module content later.

[0130] The reserved "multimodal information association address" field is initially set to an empty value. After the S305 is executed to extract information such as text, images, and formulas, the server will fill the file path or memory pointer storing this information back into this field.

[0131] For example, in the "Detailed Implementation" module, the extracted text information is saved as "spec_text.txt", the image file list is saved in the "spec_images / " directory, and the formula semantic text is saved as "spec_formulas.tex". The corresponding associated address field in the index table can then be recorded as a path mapping in JSON format. Using this index table, the server does not need to repeatedly parse the entire document; instead, it can quickly locate all relevant data for any module by looking up the table, significantly improving processing efficiency.

[0132] S305. Based on the title-extracted information index table, extract multimodal information from each module respectively.

[0133] Specifically, the server iterates through each record in the title-extract information index table, and precisely extracts the multimodal text content of the corresponding module from the original document based on the start and end positions of the content stored in the record.

[0134] The extracted content can be organized according to module identifiers, such as "module_001 / text.txt" and "module_001 / image_001.png", and these paths can be associated with the "multimodal information association address" field of the corresponding module in the index table. This allows the originally mixed document content to be broken down into a structured dataset organized by module and separated by type.

[0135] Furthermore, the formula information can be obtained through the following steps:

[0136] Step B1: Detect the formula area in the patent document and convert the formula image into semantic text in a preset format based on the formula recognition model.

[0137] Specifically, the server detects formula regions in patent documents. For formulas that exist in image form, the server uses a deep learning-based formula detection model (such as the open-source model YOLO-Formula) to locate the formula boundaries in the image.

[0138] For formulas formatted as text, the server determines the start and end positions of the formula by recognizing special symbols (such as ∫, ∑, √), subscripts and superscripts, and specific font attributes. It then calls a formula recognition model (such as the open-source Matpix C++ inference engine or a Transformer-based model). The model converts the formula image into semantic text in a preset format.

[0139] The default format here can include LaTeX or MathML.

[0140] During the conversion process, the model outputs a sequence of labels that correspond perfectly to the formula structure. For example, “\frac{a}{b}” represents a fraction, and “{\sum}_{i=1}^{n}” represents a summation.

[0141] Step B2: Concatenate the converted semantic text with the ordinary text of the paragraph containing the formula to form a mixed text that contains the semantics corresponding to the formula.

[0142] Specifically, the server concatenates the converted semantic text with the plain text of the paragraph containing the formula.

[0143] Preserve the original paragraph order during splicing. For example, the plain text "The computational efficiency can be expressed as" is followed by the LaTeX text "\eta = \frac{P_{out}}{P_{in}}", and then the subsequent text "Where P_out is the output power".

[0144] This splicing method creates a mixed text containing the semantics corresponding to the formula, enabling the large language model to directly "understand" the mathematical meaning of the formula.

[0145] Step B3: Replace the original formula image at the corresponding position in the patent document with the mixed text and add it to the input information of the large language model.

[0146] Specifically, the server replaces the original formula image at the corresponding position in the patent document with the concatenated mixed text and adds it to the input information of the large language model.

[0147] This ensures that large models do not treat formulas as meaningless images during processing, thus avoiding inference errors caused by formulas that cannot be parsed.

[0148] In some embodiments, given the specific nature of the patent drawings, the server may perform image similarity detection to determine whether there are duplicate or highly similar drawings:

[0149] Step C1: If at least one image to be detected is extracted from each module, extract the corresponding spatial domain features and frequency domain features for the extracted at least one image to be detected.

[0150] Specifically, if at least one image to be detected is extracted from each module, the server extracts spatial and frequency domain features for each image. Spatial feature extraction uses a pre-trained visual encoder (e.g., a general image encoder based on the Transformer architecture). The image is scaled to a fixed size and then input into the encoder. The output class label vector is taken as the global spatial representation and L2 normalized.

[0151] Frequency domain feature extraction first performs a two-dimensional discrete Fourier transform on the image to convert it from the spatial domain to the frequency domain. Then, it calculates the logarithmic spectrum to enhance the discrimination of low-frequency components. The logarithmic spectrum is then treated as a new image and input into the same visual encoder. Similarly, the category label vector is taken and L2 normalized.

[0152] Step C2: Using preset fusion weight coefficients, the spatial domain features and frequency domain features are weighted, concatenated, and normalized to obtain the fusion feature vector of the image to be detected.

[0153] Specifically, the server introduces a preset fusion weight coefficient λ, which ranges from 0 to 1.

[0154] The specific method for weighted concatenation of spatial and frequency domain feature vectors is as follows: λ is multiplied by the spatial feature vector, and then concatenated with 1-λ multiplied by the frequency feature vector to form a joint vector. Then, the entire joint vector is L2 normalized to obtain the fused feature vector of the image to be detected.

[0155] The value of λ can be set based on experience. For example, λ=0.6 indicates that the spatial features are slightly dominant. It can also be dynamically adjusted according to the image content.

[0156] Step C3: If there are at least two images to be detected, calculate the cosine similarity of the fused feature vectors of each image, compare the similarity with a preset threshold, and determine whether the images are similar.

[0157] Specifically, if the number of images to be detected is at least two, the server calculates the cosine similarity of the fused feature vectors of each pair of images, which is the dot product of the two vectors divided by the product of their magnitudes.

[0158] The calculated similarity is compared with a preset threshold τ, for example, τ=0.95. If the similarity is greater than or equal to τ, the two images are considered similar; otherwise, they are not similar. The closer the similarity value is to 1, the more consistent the image content.

[0159] Step C4: Add the comparison results to the input information.

[0160] Specifically, the server adds the comparison results (including the identification of image pairs and the judgment of similarity) to the input information in the form of structured text for the large model to refer to during subsequent inference. For example, the input information may contain "attached". Figure 2 With appendix Figure 3 The similarity is 0.98, indicating a possible duplication.

[0161] In some embodiments, the server also detects non-compliant content in the patent drawings.

[0162] For example, the server can call an image understanding interface based on a multimodal large model to identify whether the attached image contains a real human face.

[0163] Specifically, the server inputs each image extracted from the patent drawings into a pre-trained face detection model. If a face region is detected and the confidence level is higher than a preset threshold, the image is marked as containing a real face.

[0164] The server also identifies whether the accompanying drawings contain unnecessary textual descriptions, such as long sentences from the instruction manual, advertising text, or explanatory text unrelated to the illustration labels.

[0165] The server adds the detected face markers and unnecessary text markers to the input information of the subsequent large language model for reference when generating structured processing results.

[0166] In some embodiments, when considering the accompanying drawings and corresponding text content, the document may also include a text-image consistency detection, which includes the following steps:

[0167] Step D1: If the first-level module contains a manual module and a manual drawing module, extract the drawing reference relationship from the split manual module using regular expression matching.

[0168] Specifically, to verify the degree of matching between the text description in the instruction manual and the content of the accompanying drawings, the server performs a consistency check between the text and the drawings.

[0169] If a first-level module contains both a specification module and a specification figure module, the server uses regular expression matching to extract the figure reference relationships from the split specification module. After matching, the referenced figure numbers are obtained.

[0170] Step D2: Based on the target detection model, identify whether there are any components and numbered features mentioned in the text description corresponding to the attached drawings in the corresponding instruction manual in the image.

[0171] Specifically, for each extracted attached figure reference relationship, the server loads the target attached figure image from the corresponding specification attached figure module. Then, it calls a pre-trained object detection model (such as the open-source YOLOv8 model) to identify whether the image contains the components and numbering features mentioned in the text description.

[0172] For example, the text description is " Figure 2 "Bolt 3 and nut 4 are mated." The target detection model needs to detect whether there is a bolt target labeled "3" and a nut target labeled "4" in the image.

[0173] The server can pre-recognize the numbers in the attached diagram using OCR and associate them with the detected component areas of the target.

[0174] Step D3: Generate a text-image consistency score based on the recognition results, and add the text-image consistency score to the input information of the large language model.

[0175] Specifically, based on the recognition results, the server generates an image-text consistency score. The scoring rules can be as follows: if all the components and numbers mentioned in the text are found in the image, the score is 1.0; if some are found, the score is 0.5; if none are found, the score is 0.0.

[0176] The server adds this consistency score as a structured field to the input information of the large language model, such as " Figure 2 Consistency score 0.0. Please check if bolt 3 and nut 4 are missing.

[0177] In some embodiments, prior information can be extracted from textual information to improve the accuracy of subsequent model recognition. Specific steps include:

[0178] Step E1: If text information is extracted from each module, the text length of the text information in each module is counted, and abnormal punctuation and format features in the text information are detected.

[0179] Specifically, to improve the large model's ability to judge the normality of text, the server performs prior extraction of text information.

[0180] If text information is extracted from each module, the server calculates the text length of the text information in each module (excluding whitespace characters such as spaces and line breaks). Simultaneously, it detects abnormal punctuation in the text information, such as multiple consecutive periods, multiple consecutive commas, mixed use of Chinese and English punctuation, and formatting features such as inconsistent paragraph indentation and discontinuous numbering.

[0181] Step E2: Organize the text length, abnormal punctuation type, and format features into structured basic prior information and write it into the title-extracted information index table.

[0182] Specifically, the server organizes the statistically obtained text length, the list of detected abnormal punctuation types, and the list of format feature anomalies into structured basic prior information, such as using JSON format:

[0183] {“module”: “Manual Summary”, “length”: 298, “abnormal_punct”: [“Duplicate Period”], “format_issue”: [“Missing First-Line Indent”]}.

[0184] Then, this structured information is written into the "Basic Prior Information" field of the corresponding module in the Title-Extracted Information Index Table.

[0185] Step E3: Use the basic prior information in the title-extracted information index table as contextual enhancement information for reasoning in the large language model.

[0186] Specifically, the server inserts the basic prior information from the title-extracted information index table as context enhancement information into the beginning of the input information of the large language model, and prompts the model "The following are the basic specification detection results of the document. Please make a judgment based on this information".

[0187] In this way, large models can obtain objective facts about text length, punctuation compliance, etc., in the early stages of inference, thereby suppressing forgetting caused by long texts and improving the accuracy of judgment on normative issues.

[0188] S306. After associating and integrating the extracted multimodal information with the title-extracted information index table, the input information of the large language model is obtained. The input information is then input into the large language model to generate the structured processing result of the patent document.

[0189] Specifically, after extracting multimodal information from all modules, the server performs an association and integration operation.

[0190] Specifically online, the server will generate an overall input suggestion framework based on the natural order of the modules in the title-extracted information index table.

[0191] The framework lists the title and type of each module in sequence, inserts the text content of the module in the corresponding position, and embeds placeholders or brief descriptions of images and formulas through reference tags.

[0192] For images, the server can generate a short placeholder description, such as "This is an attachment". Figure 1 The content; for formulas, the semantic text obtained by the server during the extraction phase will be directly inserted into the corresponding position in the text stream.

[0193] The server will also load low-level visual features of the image or complete LaTeX expressions of formulas, as needed, based on the associated addresses in the index table, and incorporate them into the input information.

[0194] After organization, the server encapsulates the entire input information into a format acceptable to the large language model, such as a message list distinguished by the roles of system and user. The system part describes the task objectives and output format requirements, while the user part contains the integrated document content mentioned above.

[0195] The server then sends the input information to the large language model via API calls or local inference interfaces, and receives the structured processing results returned by the model. The results can be a JSON array, where each element corresponds to the detection conclusion of a module, including fields such as anomaly type, location, and correction suggestions.

[0196] In some embodiments, the server employs a multi-module rule-based prompt design when generating input prompts for a large language model.

[0197] For different modules, the server customizes exclusive prompt word templates. For example, for the claims module, the prompt words emphasize checking the standardization of citation relationships (e.g., not citing non-existent claims) and the continuity of numbering; for the abstract module, the prompt words emphasize word limits and whether non-technical terms are included; for the patent title module, the prompt words emphasize whether prohibited words such as brand names and personal names are included. Through modular binding, the server associates the prompt word templates for corresponding modules with the module types in the title-extracted information index table, ensuring that the large model receives rule guidance that matches its examination focus when processing each module.

[0198] In some embodiments, the server embeds few-shot learning examples in the prompts of the large model.

[0199] When new detection rules are introduced or when encountering scenarios with few samples, the server adds a rule description, judgment criteria, and one or two positive and negative examples to the prompt.

[0200] For example, for the rule "the specific implementation should include experimental data", the server explicitly requires the model to determine whether the implementation includes experimental data such as collision speed and impact load in the prompt, and provides a positive example that includes experimental data and a negative example that does not include experimental data.

[0201] The server leverages the contextual learning capabilities of large models to enable them to quickly adapt to new rules without fine-tuning, significantly improving the generalization ability of the solution across different domains.

[0202] In some implementations, the server requires the large model to output structured detection results.

[0203] Specifically, the server explicitly specifies in the prompts that the output format is a JSON object, and each detection item must include the following four fields: non-standard location, non-standard content, description of the violated rule, and rectification suggestions. The server directly obtains structured data that can be used to generate reports by parsing the JSON text returned by the large model.

[0204] Meanwhile, the server records key information throughout the entire processing flow, including patent application number, patent name, processing time, and generated structured results, forming a complete processing log and storing it in the database.

[0205] The log supports searching and tracing by patent application number or patent name, which facilitates subsequent review and optimization of the processing results.

[0206] It should be noted that the above steps can be flexibly combined according to actual needs. The server can either execute all the slave features in sequence or execute only some of the steps.

[0207] The multimodal patent document information processing method provided in this disclosure introduces cross-module reference relationship graph construction and graph attention aggregation, spatial-frequency domain weighted fusion matching for engineering line graphs, cross-modal consistency detection of text and images, preprocessing to preserve formula semantics, and extraction of basic prior information and context enhancement. This specifically addresses several technical challenges in patent documents, such as information silos between modules leading to difficulties in cross-reference verification, insufficient discrimination of general image algorithms for high-density line graphs, inability of large models to understand formula images, and the easy forgetting of key specification information in long texts. Through the above multi-dimensional collaborative processing, the overall accuracy and reliability of the automated processing of multimodal patent documents are significantly improved, enabling large models to more completely perceive the global structure and local details of the document, achieving refined and interpretable intelligent processing of complex patent documents.

[0208] Figure 4 This is a schematic diagram of the structure of an embodiment of the multimodal patent document information processing device provided in this application, as shown below. Figure 4 As shown, the multimodal patent document information processing device 400 provided in this embodiment includes:

[0209] Module 410 is used to acquire multimodal patent documents containing multiple functional modules;

[0210] The splitting module 420 is used to modularly split the patent document according to the title level characteristics of each functional block, and to build a title-extracted information index table for each split module;

[0211] The extraction module 430 is used to extract multimodal information from each block based on the title-extractable information index table, wherein the multimodal information includes text information, image information and / or formula information;

[0212] The processing module 440 is used to integrate the extracted multimodal information with the title-extracted information index table to obtain the input information of the large language model, and then input the input information into the large language model to generate the structured processing result of the patent document.

[0213] Optionally, the acquisition module 410 is specifically used to: match the hierarchical headings in the document based on a preset regular expression library, and split the document into corresponding first-level modules, wherein the first-level modules include at least one of the specification abstract module, claims module, specification module, and specification drawing module; if the first-level modules include the specification module, the specification module is further subdivided into corresponding second-level modules, wherein the second-level modules include at least one of the technical field module, background technology module, invention content module, drawing description module, and detailed implementation module; and construct a title-extracted information index table, which is used to record the type, title, content start position, and subsequent multimodal information association address of each module.

[0214] Optionally, the processing module 440 is further configured to: identify cross-module reference relationships between modules before constructing the title-extracted information index table; construct a cross-module reference relationship graph with the decomposed modules as nodes and reference relationships as edges; wherein the reference relationship graph contains reference chains between modules, and modules located on the same reference chain are associated modules; load the associated module information on the reference chain using a graph traversal algorithm according to the requirements of the current processing task corresponding to the patent document; perform weighted aggregation of the information of different modules using a graph attention mechanism to obtain the corresponding aggregation result; and add the aggregation result to the input information of the large language model.

[0215] Optionally, the extraction module 430 is further configured to, based on the title-extraction information index table, extract multimodal information from each module respectively; if at least one image to be detected is extracted from each module, extract the corresponding spatial domain features and frequency domain features for the extracted at least one image to be detected respectively; use a preset fusion weight coefficient to weight and concatenate the spatial domain features and frequency domain features and perform normalization processing to obtain the fusion feature vector of the image to be detected; if there are at least two images to be detected, calculate the cosine similarity of the fusion feature vectors of each image, compare the similarity with a preset threshold to determine whether the images are similar; and add the comparison result to the input information.

[0216] Optionally, the processing module 440 is further configured to: if the primary module includes a manual module, further subdivide the manual module into corresponding secondary modules; if the primary module includes both a manual module and a manual drawing module, extract the drawing reference relationship from the subdivided manual module using regular expression matching; based on the object detection model, identify whether there are components and numbering features mentioned in the text description corresponding to the drawing reference relationship in the image from the corresponding manual drawing module; generate a text-image consistency score based on the recognition result, and add the text-image consistency score to the input information of the large language model.

[0217] Optionally, the extraction module 430 is specifically used to extract formula information in the following way: detect the formula area in the patent document, and convert the formula image into semantic text in a preset format based on the formula recognition model; concatenate the converted semantic text with the ordinary text of the paragraph where the formula is located to form a mixed text containing the semantics corresponding to the formula; replace the original formula image at the corresponding position in the patent document with the mixed text, and add it to the input information of the large language model.

[0218] Optionally, the splitting module 420 is also used to: after extracting multimodal information from each module based on the title-extracted information index table, if text information is extracted from each module, count the text length of the text information in each module, and detect abnormal punctuation and format features in the text information; organize the text length, abnormal punctuation type, and format features into structured basic prior information and write it into the title-extracted information index table; use the basic prior information in the title-extracted information index table as context enhancement information during large language model inference.

[0219] The multimodal patent document information processing device provided in this embodiment can execute the method provided in the above-described method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0220] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 5 As shown, the electronic device 50 includes:

[0221] Processor 51, memory 52, and communication interface 53;

[0222] Memory 52 is used to store executable instructions of processor 51;

[0223] The processor 51 is configured to execute the technical solutions in any of the foregoing method embodiments by executing the aforementioned executable instructions.

[0224] Optionally, the memory 52 can be either standalone or integrated with the processor 51.

[0225] Optionally, when the memory 52 is a device independent of the processor 51, the electronic device 50 may further include:

[0226] Bus 54, memory 52 and communication interface 53 are connected to processor 51 through bus 54 and complete communication with each other. Communication interface 53 is used to communicate with other devices.

[0227] Optionally, the communication interface 53 can be implemented using a transceiver. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write databases, and read-only databases). The memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk drive.

[0228] Bus 54 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0229] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0230] The electronic device is used to execute the technical solutions in any of the foregoing method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0231] This application also provides a readable storage medium storing a computer program thereon, which, when executed by a processor, implements the technical solutions provided in any of the foregoing method embodiments.

[0232] This application also provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the technical solutions provided in any of the foregoing method embodiments.

[0233] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0234] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for processing multimodal patent document information, characterized in that, include: Obtain multimodal patent documents containing multiple functional modules; Based on the title hierarchy characteristics of each functional module, the patent document is modularly split, and a title-extracted information index table is constructed for each module obtained from the split. Based on the title-extracted information index table, multimodal information is extracted from each module, wherein the multimodal information includes text information, image information and / or formula information; After the extracted multimodal information is associated and integrated with the title-extracted information index table, the input information of the large language model is obtained. The input information is then input into the large language model to generate the structured processing result of the patent document.

2. The method according to claim 1, characterized in that, The process of modularly splitting the patent document based on the title hierarchy characteristics of each functional module and constructing a title-extractable information index table includes: Based on a preset regular expression library, the hierarchical headings in the document are matched and the document is split into corresponding first-level modules. The first-level module includes at least one of the following: specification abstract module, claims module, specification module, and specification drawing module. If the first-level module includes a specification module, the specification module is further subdivided into corresponding second-level modules, wherein the second-level module includes at least one of a technical field module, a background technology module, an invention content module, a drawing description module, and a detailed implementation module; Construct a title-extracted information index table, which records the type, title, content start position, and associated address of the multimodal information extracted subsequently for each module.

3. The method according to claim 2, characterized in that, Before constructing the title-extracted information index table, the following steps are also included: Identify the cross-module reference relationships between modules, and construct a cross-module reference relationship graph with the modules obtained from the splitting as nodes and the reference relationships as edges. The reference relationship graph contains reference chains between modules, and modules located on the same reference chain are associated modules. Based on the requirements of the current processing task corresponding to the patent document, a graph traversal algorithm is used to load the associated module information on the reference chain; A graph attention mechanism is used to weighted aggregate information from different modules to obtain the corresponding aggregation result. The aggregation results are added to the input information of the large language model.

4. The method according to claim 2, characterized in that, After extracting multimodal information from each module based on the title-extracted information index table, the process further includes: If at least one image to be detected is extracted from each module, the corresponding spatial domain features and frequency domain features are extracted from the extracted at least one image to be detected. Using preset fusion weight coefficients, the spatial domain features and frequency domain features are weighted, concatenated, and normalized to obtain the fusion feature vector of the image to be detected. If there are at least two images to be detected, calculate the cosine similarity of the fused feature vectors of each image, compare the similarity with a preset threshold, and determine whether the images are similar. Add the comparison results to the input information.

5. The method according to claim 2, characterized in that, If the first-level module includes a specification module, after further subdividing the specification module into corresponding second-level modules, it also includes: If the first-level module includes a specification module and a specification drawing module, the drawing reference relationship is extracted from the split specification module using regular expression matching; Based on the target detection model, identify whether the image contains the components and numbered features mentioned in the text description corresponding to the reference relationship in the accompanying drawings from the corresponding specification drawing module; A text-image consistency score is generated based on the recognition results, and the text-image consistency score is added to the input information of the large language model.

6. The method according to claim 2, characterized in that, The formula information is extracted in the following way: Detect formula regions in patent documents and convert formula images into semantic text in a preset format based on a formula recognition model; The converted semantic text is then concatenated with the plain text of the paragraph containing the formula to form a mixed text that includes the semantics corresponding to the formula. The mixed text replaces the original formula image at the corresponding position in the patent document and is added to the input information of the large language model.

7. The method according to any one of claims 2 to 6, characterized in that, After extracting multimodal information from each module based on the title-extracted information index table, the process further includes: If text information is extracted from each module, the text length of the text information in each module is counted, and abnormal punctuation and formatting features in the text information are detected; Text length, abnormal punctuation types, and format features are organized into structured basic prior information and written into the title-extracted information index table; The basic prior information in the title-extracted information index table is used as contextual enhancement information during large language model inference.

8. A multimodal patent document information processing device, characterized in that, include: The acquisition module is used to acquire multimodal patent documents that contain multiple functional modules. The splitting module is used to modularly split the patent document according to the title level characteristics of each functional block, and to build a title-extracted information index table for each split module; The extraction module is used to extract multimodal information from each block based on the title-extractable information index table, wherein the multimodal information includes text information, image information and / or formula information; The processing module is used to associate and integrate the extracted multimodal information with the title-extracted information index table to obtain the input information of the large language model, and input the input information into the large language model to generate the structured processing result of the patent document.

9. An electronic device, characterized in that, include: Processor, memory, communication interface; The memory is used to store the executable instructions of the processor; The processor is configured to execute the multimodal patent document information processing method according to any one of claims 1 to 7 by executing the executable instructions.

10. A readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the multimodal patent document information processing method according to any one of claims 1 to 7.

Images