A cross-modal retrieval enhanced question answering method, apparatus and device

Abstract

This application discloses a cross-modal retrieval enhanced question answering method, apparatus, and device, relating to the field of artificial intelligence technology. It includes: acquiring a document to be processed; parsing the document's layout to extract text objects, image objects, and table objects; constructing a three-level domain ontology system consisting of a top-level product category ontology, a mid-level technology category ontology, and a bottom-level entity parameter ontology; parsing each object and generating corresponding ontology tags; establishing semantic binding relationships between modalities based on ontology tags to form multimodal knowledge units; vector encoding the multimodal knowledge units and constructing a multi-level ontology vector library; receiving user query requests and obtaining query vectors and ontology constraint information; performing progressive retrieval based on ontology constraints to determine a candidate multimodal knowledge unit set; performing fact consistency verification on the candidate multimodal knowledge unit set to obtain a target multimodal knowledge unit set; and generating and outputting question answering results based on the target multimodal knowledge unit set.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to a cross-modal retrieval enhanced question answering method, apparatus, and device. Background Technology

[0002] With the development of Retrieval Augmentation (RAG) technology, intelligent question-answering systems based on large language models have been widely applied in professional knowledge retrieval and decision-making support scenarios. Especially in high-tech fields such as photoresist, a large amount of knowledge exists in unstructured documents in the form of papers, patents, and technical reports. These documents contain not only textual information but also multimodal data such as experimental diagrams, microscopic morphology images, process flow diagrams, and parameter tables. When selecting materials, optimizing processes, analyzing performance, and diagnosing defects, technical personnel often need to combine textual descriptions, experimental data, and visual evidence for comprehensive judgment, thus placing higher demands on knowledge retrieval and question-answering systems.

[0003] Existing multimodal RAG technologies typically employ a general document parsing, unified vector encoding, and page-level association matching approach, vectorizing text, images, and tables separately to build a retrieval index. However, because the associations between different modalities mainly rely on shallow features such as page number position and page proximity, lacking unified domain knowledge constraints and semantic-level association mechanisms, they are prone to mismatches between text, images, and tables, making it difficult to form complete knowledge units. Furthermore, existing technologies mostly only perform format conversion or simple information extraction for specialized content such as experimental diagrams and process parameter tables, failing to fully explore the parameter relationships, patterns, and specialized semantic information contained within, thus affecting the accuracy of retrieval results.

[0004] Furthermore, existing technologies generally lack knowledge system constraints tailored to vertical domains, failing to effectively limit the search scope using specialized domain knowledge, resulting in significant interference from irrelevant information during the search process. In the question-and-answer generation stage, there is also a lack of effective means to verify factual consistency, addressing potential parameter conflicts, inconsistencies between text and images, and discrepancies in conclusions from different sources. This can easily lead to domain illusions, affecting the reliability and interpretability of the question-and-answer results.

[0005] Therefore, there is an urgent need to provide a cross-modal RAG question answering method that can realize semantic-level association of multimodal information, domain knowledge-constrained retrieval, and cross-modal fact consistency verification, so as to improve the retrieval accuracy, knowledge reliability, and result traceability of professional domain question answering systems. Summary of the Invention

[0006] The purpose of this invention is to provide a cross-modal retrieval-enhanced question-answering method, apparatus, and device. By constructing a three-tiered domain ontology system consisting of a top-level product category ontology, a mid-level technology category ontology, and a bottom-level entity parameter ontology, it performs unified semantic annotation on text objects, image objects, and table objects. Based on ontology tags, it establishes cross-modal semantic binding relationships, forming multimodal knowledge units. This overcomes the limitations of traditional retrieval-enhanced question-answering systems that primarily rely on text semantic matching and struggle to effectively utilize image and table information, achieving the association, fusion, and unified organization of knowledge across different modalities. Furthermore, by vector encoding the multimodal knowledge units and constructing a multi-level ontology vector library, the retrieval process simultaneously possesses vector semantic retrieval capabilities and ontology structural constraint capabilities, improving the accuracy and recall quality of knowledge retrieval. Simultaneously, by progressively narrowing the retrieval scope through an ontology-constrained retrieval mechanism and combining factual consistency verification to resolve conflicts and screen credibility of candidate knowledge, it reduces semantic drift, knowledge conflicts, and model illusion phenomena, improving the accuracy, reliability, and interpretability of question-answering results, thereby enhancing the performance of cross-modal knowledge question answering in complex professional domain scenarios.

[0007] To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a cross-modal retrieval enhanced question answering method, the method comprising: Obtain the document to be processed, parse the layout of the document to be processed, and extract text objects, image objects, and table objects; A domain three-level ontology system is constructed, consisting of a top-level product category ontology, a middle-level technology category ontology, and a bottom-level entity parameter ontology. Based on the domain three-level ontology system, deep analysis is performed on text objects, image objects, and table objects to obtain the top-level ontology tags, middle-level ontology tags, and bottom-level ontology tags corresponding to each text object, image object, and table object. Based on top-level ontology tags, mid-level ontology tags, and bottom-level ontology tags, cross-modal semantic association analysis is performed on text objects, image objects, and table objects to establish semantic binding relationships between modalities and form multimodal knowledge units composed of associated text objects, image objects, and table objects. Multimodal knowledge units are vector-encoded to generate knowledge unit vectors corresponding to the multimodal knowledge units, and the association between knowledge unit vectors and corresponding multimodal knowledge units is established to construct a multi-level ontology vector library; Receive user query requests, perform intent parsing and ontology mapping on user query requests, and obtain query vectors and ontology constraint information; Based on the query vector, ontology constraint information, and multi-level ontology vector library, perform progressive ontology constraint retrieval to obtain the target knowledge unit vector, and determine the candidate multimodal knowledge unit set according to the association between the target knowledge unit vector and the multimodal knowledge unit; Perform fact consistency verification on the candidate multimodal knowledge unit set to obtain the target multimodal knowledge unit set; Question-answering results are generated and output based on the target multimodal knowledge unit set.

[0008] Secondly, the present invention also provides a cross-modal retrieval enhanced question answering device, the device comprising: The document acquisition module is used to acquire the document to be processed, parse the layout of the document to be processed, and extract text objects, image objects, and table objects. The tag acquisition module is used to construct a domain three-level ontology system consisting of a top-level product category ontology, a middle-level technology category ontology, and a bottom-level entity parameter ontology. Based on the domain three-level ontology system, it performs in-depth analysis of text objects, image objects, and table objects to obtain the top-level ontology tags, middle-level ontology tags, and bottom-level ontology tags corresponding to each text object, image object, and table object. The unit formation module is used to perform cross-modal semantic association analysis on text objects, image objects, and table objects based on top-level ontology tags, mid-level ontology tags, and bottom-level ontology tags, establish semantic binding relationships between modalities, and form multimodal knowledge units composed of associated text objects, image objects, and table objects. The vector encoding module is used to perform vector encoding on multimodal knowledge units, generate knowledge unit vectors corresponding to the multimodal knowledge units, establish the association between knowledge unit vectors and corresponding multimodal knowledge units, and construct a multi-level ontology vector library; The request query module is used to receive user query requests, perform intent parsing and ontology mapping on user query requests, and obtain query vectors and ontology constraint information. The progressive retrieval module is used to perform progressive retrieval of ontology constraints based on query vectors, ontology constraint information and multi-level ontology vector libraries, obtain target knowledge unit vectors, and determine the candidate multimodal knowledge unit set according to the association between target knowledge unit vectors and multimodal knowledge units. The consistency verification module is used to perform fact consistency verification on the candidate multimodal knowledge unit set to obtain the target multimodal knowledge unit set; The results output module is used to generate and output question-and-answer results based on the target multimodal knowledge unit set.

[0009] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the cross-modal retrieval enhanced question answering method provided in the first aspect.

[0010] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the cross-modal retrieval enhanced question-answering method provided in the first aspect.

[0011] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the cross-modal retrieval enhanced question answering method provided in the first aspect.

[0012] The beneficial effects of this invention are as follows: This application constructs a three-level domain ontology system consisting of a top-level product category ontology, a mid-level technology category ontology, and a bottom-level entity parameter ontology. It performs unified semantic annotation on text objects, image objects, and table objects, and establishes cross-modal semantic binding relationships based on ontology tags to form multimodal knowledge units. This overcomes the problem that traditional retrieval-enhanced question-answering systems mainly rely on text semantic matching and have difficulty effectively utilizing image and table information, achieving the association, fusion, and unified organization of knowledge across different modalities. Furthermore, by vector encoding the multimodal knowledge units and constructing a multi-level ontology vector library, the retrieval process simultaneously possesses vector semantic retrieval capabilities and ontology structural constraint capabilities, improving the accuracy and recall quality of knowledge retrieval. Simultaneously, by progressively narrowing the retrieval scope through an ontology-constrained retrieval mechanism and combining factual consistency verification to resolve conflicts and screen credibility of candidate knowledge, it can reduce semantic drift, knowledge conflicts, and model illusion phenomena, improving the accuracy, reliability, and interpretability of question-answering results, thereby enhancing the performance of cross-modal knowledge question answering in complex professional domain scenarios.

[0013] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a cross-modal retrieval enhanced question answering method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a cross-modal retrieval enhancement question-answering device according to an embodiment of the present invention; Figure 3 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0015] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] It should be noted that references to "an embodiment," "embodiment," "example embodiment," etc., in this specification refer to the described embodiment including specific features, structures, or characteristics; however, not every embodiment must include these specific features, structures, or characteristics. Furthermore, such expressions do not refer to the same embodiment. Moreover, when describing specific features, structures, or characteristics in conjunction with embodiments, whether or not explicitly described, it is indicated that incorporating such features, structures, or characteristics into other embodiments is within the knowledge of those skilled in the art.

[0017] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0018] like Figure 1 As shown, this embodiment provides a cross-modal retrieval enhanced question answering method. This method is mainly applied to the question answering scenario of photoresist professional knowledge. By constructing a three-level domain ontology system, the text, images and tables in the document are organized and associated in a unified semantic way. Combined with ontology-constrained retrieval and fact consistency verification mechanism, it can realize accurate retrieval and reliable question answering of professional domain knowledge.

[0019] S101: Obtain the document to be processed, parse the layout of the document to be processed, and extract text objects, image objects, and table objects.

[0020] First, the documents to be processed are obtained. These documents refer to professional technical materials that need to be included in the knowledge base for parsing and retrieval. Their sources can include patent documents, academic papers, technical standards, experimental reports, and industry white papers in the field of photoresist. Since these documents are usually stored in PDF format and contain multiple modal information such as text content, experimental diagrams, defect morphology diagrams, process flow diagrams, and parameter tables, document structure parsing is required first.

[0021] Page layout analysis refers to using document layout analysis algorithms to identify and separate different content areas on a document page, thereby determining the actual location and organizational structure of various types of information on the page. In practice, a deep learning-based page layout detection model can be used to scan the document page, identify title areas, body text areas, image areas, and table areas, and obtain the page coordinate information corresponding to each area.

[0022] For the main text area, text information purification is performed first. Text information purification refers to removing invalid information from the original recognition results that does not contribute to knowledge expression, in order to ensure the accuracy of subsequent knowledge extraction. Specifically, based on page location features, keyword rules, and semantic discrimination logic, headers, footers, tables of contents, references, and acknowledgments can be automatically deleted, retaining only the main text content that reflects the technical solutions and experimental results.

[0023] Given the frequent presence of extremely long paragraphs in technical documents, semantic segmentation is necessary to avoid context truncation issues during subsequent semantic encoding. Semantic segmentation refers to dividing extremely long text into multiple independent semantic segments based on sentence boundaries, while maintaining semantic integrity. For example, when a paragraph simultaneously describes photoresist composition, exposure processes, and development results, it can be split into multiple sub-paragraphs according to semantic themes. After segmentation, each sub-paragraph is assigned a unique identifier, and its original paragraph identifier, preceding sub-paragraph identifier, and following sub-paragraph identifier are recorded to maintain text context. This ultimately forms a text object. A text object contains at least a document ID, page number, paragraph ID, text content, and page coordinate information.

[0024] For image regions, the image location is first determined and cropped, then the image is saved as a separate file. Subsequently, the association between the image content and the corresponding image title, caption information, and annotation information is established. The image title describes the image subject, the caption describes the experimental conditions, test objects, or test results, and the annotation information explains local structures or special markings in the image. For example, for a scanning electron microscope image, the caption might include "exposure dose 20 mJ / cm²". 2 The annotation information, which describes the line width and shape obtained under specific conditions, may indicate the exact dimensions and locations. By establishing the above relationships, the image can retain not only visual information but also semantic interpretation information. Finally, an image object is constructed, which includes the image ID, image title, caption content, annotation information, document ID, page number, image region coordinates, and image storage path.

[0025] For table areas, the layout analysis algorithm is first used to determine the table location and crop the table image. Then, a table structure recognition algorithm is used to identify the header structure, row and column relationships, and cell hierarchy, combined with an optical character recognition algorithm to extract the text, values, and unit information from each cell. For example, for a photoresist performance test table, indicators such as exposure dose, resolution, and line roughness, along with their corresponding values, can be identified. The parsed results are then converted into structured table content, and a table object is constructed. The table object includes a table ID, table title, header information, structured table data, document ID, page number, table area coordinates, and table image path.

[0026] Through the above processing, a unified decomposition and standardized expression of different modal information in heterogeneous professional documents is achieved. This transforms the originally scattered text, image, and table content into text, image, and table objects with a unified organizational form, providing a standard data foundation for subsequent cross-modal knowledge fusion. Simultaneously, by retaining metadata such as page numbers, coordinates, and contextual relationships, the original knowledge source can be traced during subsequent semantic association processes, thereby improving the integrity and interpretability of knowledge organization.

[0027] S102. Construct a domain three-level ontology system consisting of a top-level product category ontology, a middle-level technology category ontology, and a bottom-level entity parameter ontology. Based on the domain three-level ontology system, perform in-depth analysis of text objects, image objects, and table objects to obtain the top-level ontology tags, middle-level ontology tags, and bottom-level ontology tags corresponding to each text object, image object, and table object.

[0028] After obtaining text objects, image objects, and table objects, a three-level domain ontology system is further constructed, and the above objects are deeply analyzed based on the three-level domain ontology system to obtain the corresponding top-level ontology tags, middle-level ontology tags, and bottom-level ontology tags.

[0029] An ontology is a standardized knowledge system used to describe the organizational structure of domain knowledge. Its function is to unify the expression of professional terminology and establish hierarchical relationships between different knowledge. In this embodiment, a three-level ontology system is constructed for the photoresist domain, and a unified modeling of domain knowledge is achieved through a coarse-to-fine hierarchical organization.

[0030] The top-level ontology describes the major attributes of photoresist products, using exposure wavelength and imaging mechanism as the basis for classification. For example, it can be classified into KrF photoresist, ArF photoresist, EUV photoresist, and other photoresist categories, and further labeled with imaging attributes such as positive, negative, and chemical amplification. The top-level ontology is mainly used to determine the product category range to which the knowledge belongs.

[0031] The middle-layer ontology is used to describe the technical direction to which the knowledge belongs. In this embodiment, the middle-layer ontology is fixedly divided into six technical categories: raw material composition, process flow, performance indicators, defect types, detection and characterization, and industry standards. For example, "resin composition" belongs to raw material composition, "exposure and development process" belongs to process flow, "resolution" belongs to performance indicators, and "pinhole defect" belongs to defect type.

[0032] The underlying ontology describes the smallest indivisible unit of knowledge, including two categories: domain entities and quantified technical parameters. Domain entities can be specialized objects such as resins, photoacid generators, developers, and scanning electron microscopes; quantified technical parameters can be specific parameters such as exposure dose, resolution, linewidth, thickness, and temperature.

[0033] In the specific implementation process, the text object is first subjected to deep analysis. Deep analysis refers to using a specific entity recognition model in the field of photoresist to automatically identify knowledge entities in areas such as material name, process conditions, performance parameters, and defect types from the text content. For example, from "ArF photoresist at 20mJ / cm 2 Extracting "ArF photoresist", "exposure dose", and "20mJ / cm²" from the section "Achieving 45nm resolution under exposure dose". 2 The system identifies entities such as "resolution" and "45nm". These entities are then mapped to a pre-built three-level domain ontology system, and categorized layer by layer, generating corresponding top-level, middle-level, and bottom-level ontology tags. For example, the top-level ontology tag might be ArF photoresist, the middle-level tag might be performance indicators, and the bottom-level tag might be exposure dose and resolution.

[0034] Subsequently, deep analysis of the image objects is performed. Since different types of images carry significantly different information formats, a differentiated analysis strategy is required. Specifically, based on image content and technical application, images are categorized into experimental atlases, microscopic morphology images, defect morphology images, and process flow diagrams. For experimental atlases, curve peaks, trends, and key test parameters are extracted; for microscopic and defect morphology images, size features, morphological structures, and defect distribution are identified; for process flow diagrams, process nodes and their flow logic are analyzed. Then, standardized semantic description text is generated based on the analysis results and mapped to a three-level domain ontology system, generating corresponding top-level, middle-level, and bottom-level ontology tags, thereby achieving a structured expression of the image's implicit knowledge.

[0035] For table objects, semantic parsing is performed based on the table header terms and structured table content. Specifically, the table header serves as a semantic anchor to identify formulation components, process conditions, performance indicators, and comparative relationships between parameters. For example, the resolution variation patterns corresponding to different exposure doses are extracted from multiple sample comparison tables. Furthermore, the parameter entities in the table are mapped item by item to the domain's three-level ontology system, generating corresponding top-level ontology tags, mid-level ontology tags, and bottom-level ontology tags.

[0036] Through the above processing, text objects, image objects, and table objects are all mapped to a unified domain knowledge system, enabling multimodal data with different expressions to obtain a consistent semantic description framework and achieving a unified structured expression of cross-modal knowledge. Simultaneously, by introducing a three-level domain ontology system, not only can the terminology in different documents be standardized, but a unified knowledge constraint boundary can also be provided for subsequent cross-modal semantic association. This improves the accuracy of semantic matching between different modalities, reduces association bias caused by differences in terminology and expression, and lays the foundation for the subsequent construction of high-quality multimodal knowledge units.

[0037] S103, based on top-level ontology tags, mid-level ontology tags and bottom-level ontology tags, performs cross-modal semantic association analysis on text objects, image objects and table objects, establishes semantic binding relationships between modalities, and forms a multimodal knowledge unit composed of associated text objects, image objects and table objects.

[0038] After completing the ontology annotation of text objects, image objects, and table objects, cross-modal semantic association analysis is further performed on the text objects, image objects, and table objects based on the top-level ontology tags, mid-level ontology tags, and bottom-level ontology tags to establish semantic binding relationships between modalities and form multimodal knowledge units.

[0039] Cross-modal semantic association analysis refers to the process of finding information that can express the same technical fact, the same experimental object, or the same technical parameter among data from different modalities, and establishing a unified association relationship. Since knowledge in professional technical documents is often expressed in multiple forms, such as describing experimental conditions in text, recording experimental data in tables, and displaying experimental results in images, it is difficult to fully express the connotation of knowledge by relying on a single modality. It is necessary to integrate and organize the interrelated information from different modalities.

[0040] In the specific implementation process, a cross-modal candidate pairing set is first constructed based on document identifiers and page scope. The cross-modal candidate pairing set refers to a combination of objects that may have semantic relationships. For example, text objects and image objects on the same page, text objects and table objects in the same chapter, and image objects and table objects can all be candidate pairings. For image objects, the corresponding text objects can be determined using the citation relationships in the image title and caption; for table objects, initial relationships can be established based on the numbering information in the table title and the citation descriptions in the text.

[0041] After obtaining the cross-modal candidate pairing set, a top-level ontology consistency check is performed first. This check determines whether objects from different modalities belong to the same product category. For example, a text object might correspond to ArF photoresist, while an image object might correspond to EUV photoresist; these objects typically do not belong to the same knowledge topic, and no further association is established. Only when two objects share the same top-level ontology label does the analysis proceed to the next stage.

[0042] Then, a mid-level ontology consistency check is performed. This check determines whether objects of different modalities belong to the same technology category. For example, a text object describing the relationship between exposure dose and resolution has a corresponding mid-level ontology label of performance metric; a table object recording resolution test results under different exposure dose conditions also has a corresponding mid-level ontology label of performance metric. In this case, it is considered that there is a possibility of further correlation between the two.

[0043] For candidate objects that pass the consistency check, the number of overlapping underlying ontology tags is further counted. The underlying ontology tags correspond to specific entities and parameters, and are an important basis for judging the degree of knowledge association. For example, a text object may contain "exposure dose" and "20mJ / cm²". 2 The underlying ontology tags such as "resolution" and "45nm" are used, and the table object also contains the above parameters, so the two have a high degree of knowledge overlap.

[0044] When the number of overlapping underlying ontology tags reaches a preset condition, the semantic association strength is further calculated. Semantic association strength measures the degree of consistency between two objects in their knowledge representation. In practice, a pre-trained semantic encoding model can be used to generate semantic vectors for each object, and the overall similarity can be calculated by combining these vectors with the semantic vectors corresponding to the underlying ontology tags. For example, when the text object describes "20mJ / cm²",... 2 "The exposure dose achieves 45nm resolution", while the caption corresponding to the image object is "45nm linewidth topography". The semantic vector distance between the two is relatively close, thus achieving a high semantic association strength.

[0045] Specifically, a weighted average is calculated for all bottom-level labels within a single modality. The formula for calculating the single-modality weighted vector is: ; in, For a single underlying tag within the collection, For a single modality, the underlying tag set, For tags The expert weight for each mid-level category is manually assigned by combining the business importance of each mid-level technical category with the frequency of user searches. For tags The corresponding single-modal vector representation generated by the domain embedding model. The formula for calculating the semantic association strength between two modalities X and Y is: ; in and They are vectors and The L2 norm of semantic association strength has a range of values. The closer the value is to 1, the higher the semantic similarity.

[0046] When the semantic association strength exceeds a preset threshold, a semantic binding relationship between corresponding modal objects is established. The semantic binding relationship between modal objects is used to record the association path and association strength information between different modal objects. For example, a binding relationship of "text object T001-image object I001" can be established, and its association strength value, shared ontology tag, and source page information can be recorded at the same time.

[0047] After establishing semantic binding relationships between modalities, multimodal knowledge units are formed based on these relationships. The multimodal knowledge unit is the core knowledge organization carrier in this embodiment, used to uniformly encapsulate multiple modal objects describing the same technical fact. For example, for a given exposure experiment, a text object describing the experimental conditions, a scanning electron microscope image object displaying the exposure results, and a table object recording test data can be combined into a single multimodal knowledge unit. This knowledge unit not only preserves the original object content but also the top-level ontology tags, mid-level ontology tags, bottom-level ontology tags, and information on the semantic binding relationships between modalities.

[0048] For example, a multimodal knowledge unit may contain the following: a text object: ArF photoresist at 20 mJ / cm 2 Achieving 45nm resolution under the specified exposure dose conditions; Image object: Scanning electron microscope image corresponding to 45nm linewidth; Table object: Data table recording test results such as exposure dose, resolution, and line edge roughness; Body label: ArF photoresist, performance indicators, exposure dose, resolution; Binding relationship: Text-image association, text-table association.

[0049] By employing the above methods, knowledge information originally scattered across different modalities is unified and organized into multimodal knowledge units with complete semantic expression capabilities. This ensures that subsequent retrieval objects are no longer isolated texts, images, or tables, but rather knowledge entities containing complete technical facts, thereby improving the completeness of knowledge expression and question-answering reasoning capabilities. Simultaneously, establishing binding relationships through a three-level ontology system and dual constraints of semantic association strength avoids erroneous associations caused by relying solely on location proximity or keyword matching, improving the accuracy and reliability of cross-modal knowledge fusion.

[0050] S104. Vector encoding is performed on multimodal knowledge units to generate knowledge unit vectors corresponding to the multimodal knowledge units, and the association between knowledge unit vectors and corresponding multimodal knowledge units is established to construct a multi-level ontology vector library.

[0051] After forming multimodal knowledge units, the multimodal knowledge units are further vector-encoded to generate knowledge unit vectors corresponding to the multimodal knowledge units. The association between knowledge unit vectors and corresponding multimodal knowledge units is established to construct a multi-level ontology vector library.

[0052] Vector encoding refers to converting structured knowledge into a high-dimensional vector representation that can be computed by machines, so as to facilitate subsequent similarity calculation and semantic retrieval. Unlike traditional vectorization that only targets text content, the encoding object in this embodiment is a multimodal knowledge unit, thus requiring the simultaneous integration of text semantics, image semantics, tabular semantics, and ontology knowledge information.

[0053] In the specific implementation process, the text objects in the multimodal knowledge units are first encoded. Since the text objects have already undergone three-level ontology annotation, the encoding process not only considers the text content itself but also needs to incorporate the corresponding top-level, mid-level, and bottom-level ontology tags. For example, the text content can be concatenated with the corresponding ontology tags and input into the domain semantic encoding model to generate a text ontology semantic vector. By introducing ontology tag information, texts with similar semantics but different domain categories can be effectively distinguished.

[0054] The table object is then encoded. For a table object, it's necessary to encode not only the text content within the cells, but also the table header structure, parameter relationships, and correspondences between metrics. For example, for a performance test table, the correspondence between exposure dose and resolution can be encoded simultaneously, thereby obtaining a table ontology semantic vector that reflects the parameter correlation characteristics.

[0055] For image objects, visual feature extraction and semantic feature extraction are performed separately. Visual feature extraction mainly obtains texture features, edge features, structural features, and spatial distribution features in the image; semantic feature extraction utilizes image titles, captions, and annotation information to obtain the technical semantic information corresponding to the image. Subsequently, the visual and semantic features are fused and encoded to obtain a joint visual-semantic representation vector of the image.

[0056] After obtaining the semantic vectors of the text ontology, the semantic vectors of the table ontology, and the joint representation vector of the image visual semantics, knowledge unit vectors are further generated by combining the semantic binding relationships between modalities. Specifically, weights can be assigned according to the importance of different modal objects in the knowledge unit, and a unified knowledge unit vector can be generated through a vector fusion algorithm.

[0057] Subsequently, the association between knowledge unit vectors and corresponding multimodal knowledge units is established. For example, a unique knowledge unit identifier is assigned to each multimodal knowledge unit, and the mapping relationship of "knowledge unit identifier - knowledge unit vector - multimodal knowledge unit content" is stored in the vector database. Finally, all knowledge unit vectors are written into the vector database to construct a multi-level ontology vector library.

[0058] Among them, the multi-level ontology vector library not only stores knowledge unit vectors, but also stores the corresponding top-level ontology tags, middle-level ontology tags and bottom-level ontology tag index information, thus enabling simultaneous support for vector semantic retrieval and ontology constraint retrieval.

[0059] Through the above processing, a unified mapping from multimodal knowledge units to knowledge unit vectors is achieved, enabling complex cross-modal knowledge to be organized and retrieved in a unified vector form. Simultaneously, by integrating three-level ontology tags into the vector construction process, the domain knowledge constraint capability in vector representation is enhanced. This ensures that subsequent retrieval processes not only discover semantically similar knowledge but also maintain consistency between retrieval results and the target domain's knowledge structure, thereby improving the accuracy and stability of professional knowledge retrieval.

[0060] S105: Receive user query request, perform intent parsing and ontology mapping on the user query request, and obtain query vector and ontology constraint information.

[0061] After completing the construction of the multi-level ontology vector library, the system enters the question-answering service phase. When a user raises a professional question, the system first receives the user's query request. A user query request refers to a natural language question from which the user seeks an answer; it can originate from a webpage, client application, knowledge service platform, or internal enterprise knowledge system. For example, a user could ask, "Which ArF photoresist requires an exposure dose of 20 mJ / cm²?" 2 Professional questions such as "How to achieve a resolution below 45nm under certain conditions?" and "What are the main causes of bridging defects in a certain type of EUV photoresist?"

[0062] Since users ask questions using natural language, and the knowledge base organizes knowledge using an ontology system, it is necessary to first perform intent parsing and ontology mapping on the user's query request. Intent parsing refers to the process of identifying the core technical topics, retrieval goals, and constraints related to the user's question. Ontology mapping refers to the process of converting natural language expressions into standard knowledge representations within the domain's three-level ontology system.

[0063] In practice, natural language understanding models can be used to perform semantic analysis on user queries, identifying domain entities, technical parameters, and logical relationships within the question. For example, for "exposure dose 20mJ / cm²",... 2The question "What are some ArF photoresists with a resolution below 45nm under certain conditions?" can be identified as follows: the product entity is ArF photoresist; the performance indicator is resolution; and the parameter condition is an exposure dose of 20mJ / cm². 2 The screening criteria were a resolution of less than 45nm.

[0064] Further, the query intent, positive constraints, and negative constraints are obtained. The query intent describes the core technical topics the user is interested in; the positive constraints define the search criteria that must be met; and the negative constraints exclude knowledge content that does not meet the requirements.

[0065] The query intent, along with the positive and negative constraints, is then mapped to the domain's three-tier ontology system. For example, ArF photoresist is mapped to the top-level product category ontology; performance indicators are mapped to the mid-level technology category ontology; and exposure dose and resolution are mapped to the bottom-level entity parameter ontology.

[0066] After mapping is completed, the user query request is encoded into a query vector, and corresponding ontology constraint information is generated. The query vector expresses the semantic features of the question, while the ontology constraint information expresses the domain knowledge boundaries and filtering rules corresponding to the question.

[0067] Through the above processing, the user's natural language questions are transformed into domain knowledge expressions, avoiding retrieval bias caused by differences in terminology and improving the accuracy of knowledge matching in subsequent retrieval stages.

[0068] S106. Based on the query vector, ontology constraint information and multi-level ontology vector library, perform progressive ontology constraint retrieval to obtain the target knowledge unit vector, and determine the candidate multimodal knowledge unit set according to the association between the target knowledge unit vector and the multimodal knowledge unit.

[0069] After obtaining the query vector and ontology constraint information, a progressive ontology constraint retrieval is further performed based on the query vector, ontology constraint information, and multi-level ontology vector library.

[0070] Ontology-constrained progressive retrieval refers to a retrieval mechanism that utilizes a three-level domain ontology system to progressively narrow the search scope and performs semantic matching at different levels. Unlike traditional vector retrieval, which directly calculates similarity across the entire database, this embodiment filters candidate knowledge step by step through ontology levels, thereby improving retrieval efficiency and accuracy.

[0071] In the specific implementation process, the first step is to conduct a coarse recall of the top-level product category itself.

[0072] Coarse recall refers to quickly filtering knowledge units from the knowledge base that are relevant to the product categories that users are interested in. For example, when the query intent involves ArF photoresist, only knowledge units with the top-level ontology tag "ArF photoresist" are retained, while knowledge units corresponding to KrF photoresist, EUV photoresist, and other product categories are excluded.

[0073] Then, the middle-level technology category ontology filtering is performed.

[0074] For example, when users focus on performance metrics, only knowledge units whose mid-layer ontology tags belong to the performance metric category are retained, while irrelevant knowledge units such as raw material composition, process flow, and detection characterization are filtered out.

[0075] After completing the above filtering, the underlying entity parameter ontology and query vector are further used for joint matching.

[0076] Specifically, candidate knowledge units are finely screened based on underlying parameter labels such as exposure dose and resolution. At the same time, the semantic similarity between the query vector and the knowledge unit vector is calculated, and the target knowledge unit vector is obtained based on the comprehensive score.

[0077] For example, the knowledge base may contain multiple knowledge units related to the performance testing of ArF photoresist. After joint matching, only those units that simultaneously meet the exposure dose of 20 mJ / cm² are retained. 2 And the vector corresponding to the knowledge unit with the required resolution.

[0078] Subsequently, based on the pre-established association between the target knowledge unit vector and the multimodal knowledge unit, the corresponding candidate multimodal knowledge unit set is determined.

[0079] The candidate multimodal knowledge unit set refers to a collection of multiple multimodal knowledge units that are highly relevant to the user's question, obtained through progressive retrieval based on ontology constraints. These knowledge units typically include multimodal content such as text descriptions, experimental images, and test tables, and serve as input for subsequent fact-consistency verification.

[0080] By employing an ontology-constrained progressive retrieval mechanism, the system transitions from full-database retrieval to domain-constrained retrieval, enabling the retrieval process to simultaneously possess semantic matching and knowledge structure constraint capabilities. This reduces interference from irrelevant knowledge and improves the relevance and accuracy of candidate knowledge.

[0081] S107, Perform fact consistency verification on the candidate multimodal knowledge unit set to obtain the target multimodal knowledge unit set.

[0082] After obtaining the candidate multimodal knowledge unit set, a fact consistency check is further performed on the candidate multimodal knowledge unit set to obtain the target multimodal knowledge unit set.

[0083] Fact consistency verification refers to the process of verifying whether there are factual conflicts, parameter anomalies, or contradictory conclusions among multiple knowledge units. Its purpose is to improve the credibility and reliability of question-and-answer results.

[0084] In the specific implementation process, outlier detection and interval verification are performed on the quantitative parameters in the candidate multimodal knowledge units.

[0085] For example, for numerical parameters such as exposure dose, resolution, and line edge roughness, it can be determined whether they are within a reasonable range based on domain knowledge rules; for abnormal data that deviates significantly from the statistical distribution, their credibility score is reduced or they are directly removed.

[0086] For qualitative conclusions, conflict resolution is performed according to the preset credibility rules.

[0087] For example, when different knowledge units give different explanations for the same cause of a defect, factors such as the type of knowledge source, frequency of citation, publication time, and completeness of the knowledge unit can be comprehensively considered to calculate a credibility score, and conclusions with higher credibility can be retained.

[0088] Further, the candidate multimodal knowledge units are ranked and screened for consistency to obtain the target multimodal knowledge unit set.

[0089] By verifying factual consistency, the impact of knowledge conflicts, parameter errors, and model illusions on question-and-answer results can be effectively reduced, improving the authenticity and reliability of knowledge citations, thereby ensuring that the final answer has high credibility.

[0090] S108 generates and outputs question-and-answer results based on the target multimodal knowledge unit set.

[0091] After obtaining the target multimodal knowledge unit set, question-and-answer results are generated and output based on the target multimodal knowledge unit set.

[0092] In practice, the text content, image content, tabular data, and corresponding ontology labels from the target multimodal knowledge unit can be used as enhanced context input to the large language model, which then generates the answer based on the enhanced context. Simultaneously, the answer can reference the source information of the corresponding knowledge unit, experimental images, and key parameter tables to improve the interpretability and traceability of the answer.

[0093] For example, when a user inquires about the performance of a certain ArF photoresist, the system can not only output the corresponding textual conclusion, but also provide relevant scanning electron microscope images and experimental test data tables, thus forming a complete cross-modal question-and-answer result.

[0094] Through the above steps, a complete closed-loop process is achieved, from multimodal knowledge construction, multi-level ontology organization, ontology-constrained retrieval, fact consistency verification to question-answer generation. This not only makes full use of complementary knowledge information in text, images, and tables, but also uses the domain ontology system to constrain the knowledge organization and retrieval process, thereby improving the accuracy, reliability, and interpretability of the professional domain question-answering system.

[0095] In summary, this embodiment first parses the layout of the document to be processed, constructing text objects, image objects, and table objects; then, it uses a three-level domain ontology system to complete unified semantic annotation of multimodal objects and establishes cross-modal semantic binding relationships to form multimodal knowledge units; further, it constructs a multi-level ontology vector library, obtains candidate multimodal knowledge units related to the user's question through progressive retrieval based on ontology constraints, and obtains the target multimodal knowledge unit set through fact consistency verification; finally, it generates question-and-answer results based on the target multimodal knowledge unit set, thereby realizing cross-modal fusion retrieval and reliable question-and-answer of professional domain knowledge.

[0096] Based on the same inventive concept, this application also provides a cross-modal retrieval enhanced question answering device for implementing the cross-modal retrieval enhanced question answering method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more cross-modal retrieval enhanced question answering device embodiments provided below can be found in the limitations of the cross-modal retrieval enhanced question answering method described above, and will not be repeated here.

[0097] In one embodiment, such as Figure 2 As shown, a cross-modal retrieval enhanced question answering device is provided, the device comprising: The document acquisition module 30 is used to acquire the document to be processed, perform layout parsing on the document to be processed, and extract text objects, image objects, and table objects. The tag acquisition module 31 is used to construct a domain three-level ontology system consisting of a top-level product category ontology, a middle-level technology category ontology, and a bottom-level entity parameter ontology. Based on the domain three-level ontology system, it performs in-depth analysis on text objects, image objects, and table objects to obtain the top-level ontology tags, middle-level ontology tags, and bottom-level ontology tags corresponding to each text object, image object, and table object. The unit forming module 32 is used to perform cross-modal semantic association analysis on text objects, image objects and table objects based on top-level ontology tags, middle-level ontology tags and bottom-level ontology tags, establish semantic binding relationships between modalities, and form multimodal knowledge units composed of associated text objects, image objects and table objects. The vector encoding module 33 is used to perform vector encoding on multimodal knowledge units, generate knowledge unit vectors corresponding to multimodal knowledge units, establish the association between knowledge unit vectors and corresponding multimodal knowledge units, and construct a multi-level ontology vector library. The query module 34 is used to receive user query requests, perform intent parsing and ontology mapping on user query requests, and obtain query vectors and ontology constraint information. The progressive retrieval module 35 is used to perform progressive retrieval of ontology constraints based on query vectors, ontology constraint information and multi-level ontology vector library, obtain target knowledge unit vectors, and determine candidate multimodal knowledge unit set according to the association between target knowledge unit vectors and multimodal knowledge units; The consistency verification module 36 is used to perform fact consistency verification on the candidate multimodal knowledge unit set to obtain the target multimodal knowledge unit set; The result output module 37 is used to generate and output question-and-answer results based on the target multimodal knowledge unit set.

[0098] This application also provides an electronic device, in some embodiments, referring to... Figure 3 As shown, the electronic device 700 includes an input unit 710, a memory 720, a processor 730, and an output unit 740. The memory 720 stores program instructions that can be executed on the processor 730. The processor 730 can execute the cross-modal retrieval enhanced question-answering method and / or technical solution based on the foregoing embodiments by calling the program instructions. The electronic device 700 can be a mobile terminal device such as a mobile phone or a computer.

[0099] Furthermore, embodiments of this application also provide a computer-readable storage medium for storing a computer program that performs a cross-modal retrieval enhanced question-answering method. For example, computer program instructions, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. The program instructions that invoke the methods of this application may be stored in a fixed or removable storage medium, and / or transmitted via data streams in broadcast or other signal carrying media, and / or stored in a storage medium that operates according to the program instructions.

[0100] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0101] The technical features of the above embodiments can be arbitrarily integrated. For the sake of brevity, not all possible integrations of the technical features in the above embodiments are described. However, as long as the integration of these technical features does not contradict each other, they should be considered to be within the scope of this specification.

[0102] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A cross-modal retrieval enhanced question answering method, characterized in that, The method includes: Obtain the document to be processed, parse the layout of the document to be processed, and extract text objects, image objects, and table objects; A domain three-level ontology system is constructed, consisting of a top-level product category ontology, a middle-level technology category ontology, and a bottom-level entity parameter ontology. Based on the domain three-level ontology system, the text objects, image objects, and table objects are deeply analyzed to obtain the top-level ontology tags, middle-level ontology tags, and bottom-level ontology tags corresponding to each text object, image object, and table object. Based on the top-level ontology tags, mid-level ontology tags, and bottom-level ontology tags, cross-modal semantic association analysis is performed on the text objects, image objects, and table objects to establish semantic binding relationships between modalities and form a multimodal knowledge unit composed of associated text objects, image objects, and table objects. The multimodal knowledge units are vector-encoded to generate knowledge unit vectors corresponding to the multimodal knowledge units, and the association between the knowledge unit vectors and the corresponding multimodal knowledge units is established to construct a multi-level ontology vector library. Receive user query requests, perform intent parsing and ontology mapping on the user query requests, and obtain query vectors and ontology constraint information; Based on the query vector, ontology constraint information, and the multi-level ontology vector library, an ontology constraint progressive retrieval is performed to obtain the target knowledge unit vector, and a candidate multimodal knowledge unit set is determined according to the association between the target knowledge unit vector and the multimodal knowledge unit. Perform a fact consistency check on the candidate multimodal knowledge unit set to obtain the target multimodal knowledge unit set; Question-answering results are generated and output based on the target multimodal knowledge unit set.

2. The cross-modal retrieval enhanced question answering method as described in claim 1, characterized in that, The step of parsing the layout of the document to be processed and extracting text objects, image objects, and table objects includes: Use layout analysis models to identify text areas, image areas, and table areas in a document; The main text area is purified by removing headers, footers, table of contents, references and acknowledgments, and semantic segmentation is performed on excessively long paragraphs. The processed text content is then used to construct the text object. Extract the image content from the image region, establish the association between the image content and the corresponding image title, caption information and annotation information, and construct the image content into the image object; The table structure in the table area is identified, the table header, cell content and parameter data are parsed, structured table content is generated, and the structured table content is constructed into the table object.

3. The cross-modal retrieval enhanced question answering method as described in claim 2, characterized in that, The method involves deep parsing of the text objects, image objects, and table objects based on the three-level ontology system of the domain to obtain the top-level ontology tags, middle-level ontology tags, and bottom-level ontology tags corresponding to each text object, image object, and table object, including: The text object is subjected to entity recognition and parameter extraction, and mapped to the domain three-level ontology system to generate corresponding top-level ontology tags, middle-level ontology tags and bottom-level ontology tags; The image object is classified and its features are parsed to extract feature parameters, structural features or process logic from the image and mapped to the domain three-level ontology system to generate corresponding top-level ontology tags, middle-level ontology tags and bottom-level ontology tags. The table object is semantically parsed to extract parameter relationships, performance indicators, and data comparison relationships, and mapped to the domain's three-level ontology system to generate corresponding top-level ontology tags, mid-level ontology tags, and bottom-level ontology tags.

4. The cross-modal retrieval enhanced question answering method as described in claim 3, characterized in that, The establishment of semantic binding relationships between modalities includes: Construct a cross-modal candidate pairing set based on document identifiers and page ranges; Perform top-level ontology consistency checks and mid-level ontology consistency checks on different modal objects in the candidate pairing set; Count the number of overlapping underlying ontology tags between different modal objects; When the number of overlapping underlying ontology tags meets a preset condition, the semantic association strength is calculated based on the semantic vector corresponding to the underlying ontology tag. When the semantic association strength is greater than a preset threshold, a semantic binding relationship between the corresponding modal objects is established, and the multimodal knowledge unit is formed based on the semantic binding relationship between the modal objects.

5. The cross-modal retrieval enhanced question answering method as described in claim 1, characterized in that, The process involves vector encoding the multimodal knowledge units to generate knowledge unit vectors corresponding to the multimodal knowledge units, establishing associations between the knowledge unit vectors and their corresponding multimodal knowledge units, and constructing a multi-level ontology vector library, including: The text objects in the multimodal knowledge unit and their corresponding top-level ontology tags, mid-level ontology tags and bottom-level ontology tags are fused and encoded to obtain the text ontology semantic vector; The table objects in the multimodal knowledge unit and their corresponding top-level ontology tags, mid-level ontology tags and bottom-level ontology tags are fused and encoded to obtain the table ontology semantic vector; Visual and semantic features are extracted from the image objects in the multimodal knowledge unit and then fused and encoded to obtain a joint image visual-semantic representation vector. The knowledge unit vector is generated based on the text ontology semantic vector, the table ontology semantic vector, the image visual semantic joint representation vector, and the intermodal semantic binding relationship. Establish the association between the knowledge unit vector and the corresponding multimodal knowledge unit, and write the knowledge unit vector into the vector database to construct the multi-level ontology vector library.

6. The cross-modal retrieval enhanced question answering method as described in claim 1, characterized in that, The step of performing progressive ontology constraint retrieval based on the query vector, ontology constraint information, and the multi-level ontology vector library, and determining a candidate multimodal knowledge unit set based on the association between the target knowledge unit vector and multimodal knowledge units, includes: Perform intent parsing on user query requests to obtain query intent, positive constraints, and negative constraints; The query intent, positive constraints, and negative constraints are mapped to the domain's three-level ontology system to generate query vectors and ontology constraint information. The target knowledge unit vector is obtained by performing a progressive retrieval of ontology constraints in the following order: coarse recall of top-level product category ontology, filtering of mid-level technology category ontology, and joint matching of bottom-level entity parameter ontology and query vector. The candidate multimodal knowledge unit set is determined based on the correlation between the target knowledge unit vector and the multimodal knowledge unit; Outlier detection and interval verification are performed on the quantitative parameters in the candidate multimodal knowledge unit set, and conflict resolution is performed on the qualitative conclusions according to the preset credibility rules to obtain the target multimodal knowledge unit set.

7. A cross-modal retrieval enhanced question-answering device, characterized in that, The device includes: The document acquisition module is used to acquire the document to be processed, perform layout parsing on the document to be processed, and extract text objects, image objects, and table objects. The tag acquisition module is used to construct a domain three-level ontology system consisting of a top-level product category ontology, a middle-level technology category ontology, and a bottom-level entity parameter ontology. Based on the domain three-level ontology system, the module performs deep analysis on the text objects, image objects, and table objects to obtain the top-level ontology tags, middle-level ontology tags, and bottom-level ontology tags corresponding to each text object, image object, and table object. The unit formation module is used to perform cross-modal semantic association analysis on the text objects, image objects and table objects based on the top-level ontology tags, middle-level ontology tags and bottom-level ontology tags, establish semantic binding relationships between modalities, and form multimodal knowledge units composed of associated text objects, image objects and table objects; The vector encoding module is used to perform vector encoding on the multimodal knowledge units, generate knowledge unit vectors corresponding to the multimodal knowledge units, establish the association relationship between the knowledge unit vectors and the corresponding multimodal knowledge units, and construct a multi-level ontology vector library. The request query module is used to receive user query requests, perform intent parsing and ontology mapping on the user query requests, and obtain query vectors and ontology constraint information. The progressive retrieval module is used to perform progressive retrieval of ontology constraints based on the query vector, ontology constraint information and the multi-level ontology vector library to obtain the target knowledge unit vector, and to determine the candidate multimodal knowledge unit set according to the association between the target knowledge unit vector and the multimodal knowledge unit. The consistency verification module is used to perform fact consistency verification on the candidate multimodal knowledge unit set to obtain the target multimodal knowledge unit set; The result output module is used to generate and output question-and-answer results based on the target multimodal knowledge unit set.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the cross-modal retrieval enhanced question answering method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the cross-modal retrieval enhanced question-answering method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the cross-modal retrieval enhanced question answering method as described in any one of claims 1 to 6.

Images