A multimodal patent semantic retrieval method and system

Abstract

This application belongs to the field of information retrieval technology and discloses a multimodal patent semantic retrieval method and system. By establishing relation anchor points, extracting local cross-modal relation representations around the anchor points, and verifying the cross-modal consistency of these relations, reliable cross-modal relations can be identified. Furthermore, technical features / components and successfully aligned cross-modal relations are constructed into a graph structure representation, and a graph matching algorithm is used to calculate similarity. As a result, the overall logic and details of the technical solution can be understood more effectively, and solutions containing only the same technical features but with different relational structures can be distinguished, thereby improving the accuracy and relevance of multimodal patent retrieval.

Description

Technical Field

[0001] This application relates to the field of information retrieval technology, and more specifically, to a multimodal patent semantic retrieval method and system. Background Technology

[0002] Patent searching is a crucial step in obtaining prior art information, assessing novelty and inventiveness, and avoiding infringement. A complete technical solution typically consists of multiple technical features, which are related by specific structures, functions, or processes. Patent documents elaborate on these technical solutions through both textual descriptions and accompanying drawings. In patent documents, the text typically describes the function, principle, implementation method, or process steps of the technical features in natural language, and often uses reference numerals to associate specific elements or components in the drawings. The drawings, on the other hand, visually represent the form, structure, relative position, or process steps of the technical features in a graphical way, using visual symbols such as lines, arrows, labels, and dimensions to present the connection relationships, spatial layout, inclusion relationships, or sequential relationships between components. The core value of a patented technical solution often lies in the specific combination of multiple technical features and the specific relationships established between these features. When describing these solutions, the patent text not only elaborates on the details of each technical feature but also clearly describes the interaction, connection methods, relative positions, and order of the features through verbs, prepositional phrases, and conjunctions. The relationship information described in the text is usually also visually represented in the accompanying drawings.

[0003] With the explosive growth in the number of patent documents and the increasing complexity of technical solutions, traditional keyword- or classification-based retrieval methods are no longer sufficient. Multimodal retrieval methods, which combine patent text and accompanying figures, have become an important development direction. Existing multimodal patent semantic retrieval methods based on text and image modalities typically extract semantic features of technical concepts independently from the text and visual features from the image. These methods attempt to map features from different modalities to a shared representation space for fusion or alignment. However, existing methods have significant shortcomings in capturing and accurately aligning relational information in text descriptions with corresponding structural or spatial relationships in accompanying figures, exhibiting insufficient ability to capture relational information and perform cross-modal alignment. This directly leads to greater difficulties for retrieval systems in determining whether a patent document is relevant to a query containing specific combinations of technical features and complex relationships. Matching based solely on isolated technical features is prone to false positives or false negatives, failing to accurately grasp the overall logic and details of the technical solution. Especially when searching for complex devices involving multiple interacting components (such as mechanical transmission systems and electronic circuits) or complex processes containing multiple steps and branches, the key to the technical solution often lies in the precise relationship between features (such as connection method, relative position, action sequence, driving relationship, etc.).

[0004] If a retrieval system cannot effectively extract, represent, and align relational information from both text and images, it will struggle to accurately grasp the overall structure and functional logic of the technical solution, severely impacting the accuracy and relevance ranking of multimodal retrieval. For example, consider a query containing "a device that drives a shaft to rotate via gear transmission," with an accompanying diagram showing the layout of the gears and shaft. The system needs to understand the complex causal and transmission relationship in the text ("driven by gear transmission") and accurately align it with the meshing structure of the gears and shaft and the possible transmission paths in the accompanying diagram to effectively match relevant patents. Existing methods struggle to efficiently and accurately achieve this fine-grained, relational information-based cross-modal alignment.

[0005] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0006] The purpose of this application is to provide a multimodal patent semantic retrieval method and system that can improve the accuracy and relevance of multimodal patent retrieval.

[0007] In a first aspect, this application provides a multimodal patent semantic retrieval method for retrieving patent documents from a patent database based on text and images. The method includes the following steps:

[0008] A1. Obtain text data and figure data of patent documents from the patent database, parse the text data to identify technical features, relationship descriptions and icon numbers, parse the figure data to extract visual features and visual structural relationships of technical components and identify icon numbers, and construct text representation structure and image representation structure;

[0009] A2. By matching the icon numbers in the text data and the attached figures data, establish the correspondence between the technical features described in the text and the technical components in the attached figures, and determine the technical features and technical components with the corresponding relationship as relationship anchor points;

[0010] A3. For each relation anchor point, analyze the text representation structure to extract the local text relation representation centered on that relation anchor point, and analyze the image representation structure to extract the local image relation representation centered on the technical component corresponding to that relation anchor point;

[0011] A4. Verify the consistency between the local text relation representation centered on the same relation anchor point and the local image relation representation to determine the successfully aligned local relations and their corresponding confidence scores;

[0012] A5. Combine all relation anchors and cross-modal relations confirmed through local relation alignment, along with their confidence scores, to construct a multimodal relation representation of the patent document; the multimodal relation representation is a graph structure;

[0013] A6. Upon receiving query data containing text and images submitted by a user, construct a multimodal relational representation of the query data;

[0014] A7. Calculate the similarity between the query data and the multimodal relation representations of patent documents in the patent database using a graph matching algorithm;

[0015] A8. Sort the patent documents according to similarity and return the search results.

[0016] Secondly, this application provides a multimodal patent semantic retrieval system for retrieving patent documents from a patent database based on text and images. The system includes:

[0017] The data parsing module is used to acquire text data and figure data of patent documents in the patent database. It parses the text data to identify technical features, relationship descriptions and attachment marks, and parses the figure data to extract the visual features and visual structural relationships of technical components and identify attachment marks, and constructs text representation structure and image representation structure.

[0018] An anchor point determination module is used to establish a correspondence between technical features described in the text and technical components in the accompanying drawings by matching the icon numbers in the text data and the accompanying drawings, and to determine the technical features and technical components with corresponding relationships as relationship anchor points;

[0019] The relation representation extraction module is used to analyze the text representation structure for each relation anchor point to extract the local text relation representation centered on the relation anchor point, and to analyze the image representation structure to extract the local image relation representation centered on the technical component corresponding to the relation anchor point.

[0020] The relation alignment module is used to verify the consistency between local text relation representations centered on the same relation anchor point and local image relation representations, in order to determine the successfully aligned local relations and their corresponding confidence scores;

[0021] The relation representation construction module is used to combine all relation anchors and cross-modal relations confirmed by local relation alignment and their confidence scores to construct a multimodal relation representation of patent documents; the multimodal relation representation is a graph structure;

[0022] The query processing module is used to construct a multimodal relational representation of the query data when it receives query data containing text and images submitted by the user.

[0023] The similarity calculation module is used to calculate the similarity between the query data and the multimodal relationship representation of patent documents in the patent database using a graph matching algorithm.

[0024] The search and sorting module is used to sort patent documents based on similarity and return the search results.

[0025] Beneficial effects: The multimodal patent semantic retrieval method and system provided in this application solves the problems of insufficient relationship information capture and cross-modal alignment capabilities in the prior art by constructing a multimodal relationship representation of patent documents and query data and performing graph matching to calculate similarity, thereby improving the accuracy and relevance of multimodal patent retrieval. Attached Figure Description

[0026] Figure 1 A flowchart of the multimodal patent semantic retrieval method provided in the embodiments of this application.

[0027] Figure 2 This is a schematic diagram of the structure of the multimodal patent semantic retrieval system provided in the embodiments of this application.

[0028] Labeling Explanation: 1. Data Parsing Module; 2. Anchor Point Determination Module; 3. Relationship Representation Extraction Module; 4. Relationship Alignment Module; 5. Relationship Representation Construction Module; 6. Query Processing Module; 7. Similarity Calculation Module; 8. Retrieval and Ranking Module. Detailed Implementation

[0029] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0030] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] refer to Figure 1 This application proposes a multimodal patent semantic retrieval method for retrieving patent documents from a patent database based on text and images. The method includes the following steps:

[0032] A1. Obtain text data and figure data of patent documents from the patent database, parse the text data to identify technical features, relationship descriptions and icon numbers, parse the figure data to extract visual features and visual structural relationships of technical components and identify icon numbers, and construct text representation structure and image representation structure;

[0033] A2. By matching the icon numbers in the text data and the attached figures data, establish the correspondence between the technical features described in the text and the technical components in the attached figures, and determine the technical features and technical components with the corresponding relationship as relationship anchor points;

[0034] A3. For each relation anchor point, analyze the text representation structure to extract the local text relation representation centered on that relation anchor point, and analyze the image representation structure to extract the local image relation representation centered on the technical component corresponding to that relation anchor point;

[0035] A4. Verify the consistency between the local text relation representation centered on the same relation anchor point and the local image relation representation to determine the successfully aligned local relations and their corresponding confidence scores;

[0036] A5. Combine all relation anchors and cross-modal relations confirmed through local relation alignment, along with their confidence scores, to construct a multimodal relation representation of the patent document; the multimodal relation representation is a graph structure;

[0037] A6. Upon receiving query data containing text and images submitted by a user, construct a multimodal relational representation of the query data;

[0038] A7. Calculate the similarity between the query data and the multimodal relation representations of patent documents in the patent database using a graph matching algorithm;

[0039] A8. Sort the patent documents according to similarity and return the search results.

[0040] Step A1 involves preprocessing and structuring the original patent document data. Specifically, natural language processing techniques can be used to parse the text, and image processing and computer vision techniques can be used to analyze the accompanying drawings. In this scheme, this feature provides basic data and structured information for subsequent technical feature recognition, relationship extraction, and cross-modal association.

[0041] Step A2 involves associating the technical features mentioned in the text with the technical components shown in the accompanying figures using identifiers shared by the text and images (as indicated in the figure). This can be achieved using string matching or identifier lookup methods. This feature establishes a direct link between the text and the image in this scheme, providing a foundation for subsequent cross-modal relationship alignment based on these points.

[0042] Step A3 involves extracting local relation information related to the identified relation anchor point from the structured representations of both text and images. Specifically, dependency parsing can be used to extract text relations, and spatial location computation can be used to extract image relations. This feature, in this scheme, obtains local relation information surrounding key technical concepts that are to be aligned across modalities.

[0043] Step A4 involves comparing whether local relationships extracted from text and images around the same relationship anchor point are mutually supportive or consistent, and quantifying this consistency. This can be achieved by calculating the similarity of relationship vectors. This feature filters out reliable cross-modal relationships validated in both text and images, and evaluates their reliability, thus solving the accuracy problem of cross-modal alignment of relationship information.

[0044] Step A5 involves integrating technical features / components, relationship anchors, and verified cross-modal relationships and their reliability into a unified graph structure. This can be achieved by constructing nodes and edges, where nodes represent technical features or components, edges represent cross-modal relationships, and edge weights represent confidence levels. This feature constructs a structured representation in this scheme that comprehensively captures technical concepts and their complex cross-modal relationships, laying the foundation for subsequent graph matching and retrieval.

[0045] Step A6 involves performing steps similar to those used in processing patent documents on the user-input query data, converting it into a multimodal graph structure representation. This can be achieved using the same methods employed in processing patent documents. This feature in the scheme transforms the query into the same representation as the document, allowing the query and document to be compared within the same representation space.

[0046] Step A7 involves using a specialized algorithm to compare the similarity between the query graph structure and the document graph structure. This can be achieved using methods based on node embedding or subgraph matching. This feature utilizes graph structure information for similarity calculation in this scheme, which more accurately reflects the matching degree of the technical solutions at the structural and relational levels, thus improving retrieval accuracy.

[0047] Step A8 involves sorting the patent documents in the database based on the calculated similarity score between the query and the documents, and presenting the top-ranked documents to the user as search results. This can be achieved by sorting the documents in descending order of similarity score. This feature provides the final search results based on the calculated similarity in this solution.

[0048] The core innovation of this application lies in constructing a local cross-modal relation representation based on relation anchors, verifying the cross-modal consistency of these local relations, constructing a multimodal graph structure representation of patent documents and query data based on successfully aligned relations, and finally calculating similarity through a graph matching algorithm, thereby solving the shortcomings of existing methods in capturing and accurately aligning cross-modal relation information.

[0049] As a preferred embodiment, the solution of this application is specifically implemented as follows: Suppose the patent document describes a transmission device including gears and shafts, and the accompanying drawings show the connection layout of the gears and shafts.

[0050] In step A1, the system acquires the text and figure data of the patent document. The text is parsed to identify the technical features “gear” and “shaft”, the relationship descriptions “drive” and “connection”, and the reference numerals “1” (gear) and “2” (shaft). The figures are parsed to extract the visual features of the gears and shafts, identify their meshing and connection relationships, and the reference numerals “1” and “2”. A text representation structure (e.g., a syntactic tree or semantic network) and an image representation structure (e.g., a component diagram and a spatial relationship diagram) are constructed.

[0051] In step A2, the reference numeral "1" in the attached diagram is used to match the word "gear" in the text with the gear component in the attached diagram to determine "gear" / gear component as the relationship anchor point. The reference numeral "2" in the attached diagram is used to match the word "shaft" in the text with the shaft component in the attached diagram to determine "shaft" / shaft component as the relationship anchor point.

[0052] In step A3, for the relationship anchor point "gear" / gear component, the text representation structure is analyzed, and local text relationship representations surrounding "gear" are extracted, such as "gear drives shaft". The image representation structure is analyzed, and local image relationship representations surrounding the gear component are extracted, such as the meshing and adjacency relationships between the gear component and the shaft component. Similar extraction is performed for the relationship anchor point "shaft" / shaft component.

[0053] In step A4, the consistency between the local textual relation representation "gear drives shaft" and the local image relation representation (the meshing and adjacency relationship between the gear and the shaft) is verified. For example, the similarity between the semantic vector of the textual relation description and the image spatial relation vector is calculated. If the consistency reaches a certain level, the cross-modal relation "gear drives shaft" is determined to be successfully aligned, and a confidence score is calculated.

[0054] In step A5, the relationship anchors "gear" / gear component, "shaft" / shaft component, and successfully aligned cross-modal relationships (e.g., "gear drives shaft") and their confidence scores are combined. A multimodal relationship representation of the patent document is constructed, represented as a graph structure. Nodes in the graph represent technical features / components such as "gear" and "shaft," edges represent successfully aligned relationships, and the weight of each edge is its confidence score.

[0055] In step A6, when a user submits a query containing the text "gear-driven shaft device" and corresponding diagrams, the system performs a process similar to steps A1-A5 to construct a multimodal relationship representation diagram of the query data.

[0056] In step A7, a graph matching algorithm is used to calculate the similarity between the multimodal relation representation graph of the query data and the multimodal relation representation graph of the patent documents. The algorithm compares the nodes, edges, and weights of the two graphs to evaluate their structural and relational matching degree.

[0057] In step A8, the patent documents in the patent database are sorted according to the calculated similarity, and the search results are returned. Patent documents containing the relationship "gear drives shaft" and with a high graph structure matching degree will be ranked higher.

[0058] By employing the above-described approach, this application addresses the shortcomings of existing multimodal patent retrieval methods in capturing, representing, and accurately aligning the relationships between technical features described in patent texts with their corresponding visual structural or spatial relationships in accompanying drawings. This method identifies reliable cross-modal relationships by establishing relationship anchors, extracting local cross-modal relationship representations around these anchors, and verifying the cross-modal consistency of these relationships. Furthermore, it constructs a graph structure representation of technical features / components and successfully aligned cross-modal relationships, and uses a graph matching algorithm to calculate similarity. As a result, the system can more effectively understand the overall logic and details of technical solutions, distinguishing between solutions that contain only the same technical features but have different relationship structures. This improves the matching degree between search results and query intent when processing queries containing complex combinations and interrelationships of technical features, reduces false positives and false negatives, and enhances the relevance of search results.

[0059] In some implementations, step A3 includes:

[0060] A301. For each relation anchor, from the text representation structure, use dependency parsing to determine the syntactic relationship between the relation anchor and other technical features, extract the word vector representations of the syntactic relationship and other technical features, and combine them to form a local text relation representation centered on the relation anchor.

[0061] A302. For each relationship anchor point, extract the visual feature vectors of the technical component corresponding to the relationship anchor point and the adjacent technical components from the image representation structure. Calculate the spatial positional relationship vectors between the technical component and the adjacent technical components based on the visual structure relationship, and combine them to form a local image relationship representation centered on the technical component corresponding to the relationship anchor point.

[0062] Specifically, in step A301, for each relation anchor, the scheme extracts a local text relation representation from the text representation structure. Dependency parsing is particularly employed here. By utilizing dependency parsing, the syntactic relationships between the relation anchor and other technical features in the sentence can be accurately determined, such as subject-verb, verb-object, and modification relationships. These syntactic relationships directly reflect the linguistic-level associations between technical features. The scheme extracts word vector representations of these syntactic relationships and other related technical features, and combines them to form a local text relation representation centered on the relation anchor. This representation not only includes the semantic information of related technical features, but more importantly, it explicitly encodes the syntactic connections between the anchor and other features, providing a structured description of text relations, which is crucial for understanding the interactions of technical features in the text. As a possible implementation, dependency parsing can be performed using existing natural language processing tools, such as neural network-based models. The extracted syntactic relations can be represented as vectors (i.e., syntactic relation vectors), for example, through one-hot encoding or embedding vectors. The word vectors for technical features can be obtained using pre-trained word embedding models, such as Word2Vec, GloVe, or BERT. Combination methods can include concatenating syntactic relation vectors with the word vectors of relevant technical features or aggregating them using graph neural networks.

[0063] In step A302, for each relationship anchor point, the scheme extracts a local image relationship representation from the image representation structure. The scheme extracts the visual feature vector of the technical component corresponding to the relationship anchor point, which represents the appearance information of the component. Simultaneously, the scheme also extracts the visual feature vectors of adjacent technical components and incorporates these adjacent components into the consideration. Further, the scheme calculates the spatial positional relationship vector between the technical component and its adjacent technical components based on the visual structural relationships in the image. Visual structural relationships can include connectivity, containment, and relative positional relationships (e.g., above, below, left, right). The spatial positional relationship vector can be calculated based on the component's bounding box coordinates, center point distance, angle, or connectivity type. Finally, the visual feature vectors of the anchor point component and its adjacent components are combined with the spatial positional relationship vectors between them to form a local image relationship representation centered on the technical component corresponding to the relationship anchor point. As a possible implementation, the visual feature vectors of the technical components can be extracted by a convolutional neural network (CNN). The spatial positional relationship vector can be represented as a relative coordinate difference, distance, angle, or a predefined category vector. The combination method can concatenate visual feature vectors with spatial location relationship vectors or aggregate them through graph neural networks.

[0064] Through steps A301 and A302, this scheme can extract local representations of explicitly encoded relational information from text and images, respectively, around relational anchor points. Text representations focus on syntactic and semantic relations, while image representations focus on visual features and spatial relations. This targeted and structured method of extracting local relational representations, combined with the text and image representation structures constructed in the preceding steps and the identified relational anchor points, provides high-quality input for verifying the consistency of cross-modal relations in subsequent steps. Thus, the system can more accurately determine whether the relations described in the text are correspondingly reflected in the accompanying drawings, thereby improving the accuracy of multimodal patent semantic retrieval. This detailed and structured local relational representation allows subsequent cross-modal alignment (step A4) to be based on richer and more comparable information, thereby improving alignment accuracy. The accurate local relational alignment results are further used to construct a multimodal relational representation of the patent document (step A5), which captures technical features and their cross-modal relations in a graph structure. Finally, the graph matching similarity calculation based on this high-quality multimodal relation representation (step A7) can more accurately reflect the similarity of technical solutions between the query and the patent documents, solve the shortcomings of existing methods in capturing and accurately aligning relation information, and improve the relevance judgment ability of the retrieval system when dealing with complex technical solutions.

[0065] In some implementations, step A4 includes:

[0066] A401. Calculate the cosine similarity between the syntactic relation vector in the local text relation representation and the spatial position relation vector in the local image relation representation to obtain the initial alignment score;

[0067] A402. Determine whether the initial alignment score is greater than the preset similarity threshold. If so, the local relation alignment is initially determined to be successful; otherwise, the local relation alignment is determined to be unsuccessful.

[0068] A403. For local relations that are initially determined to be successfully aligned, calculate the semantic similarity between the text modality feature word vectors and the image modality component visual feature vectors to obtain a semantic consistency score;

[0069] A404. The initial alignment score and semantic consistency score are weighted and fused together, and the confidence score is obtained based on the fusion result.

[0070] A405. Compare the confidence score with the preset confidence threshold to determine whether the local text relation representation and the local image relation representation in the initially determined aligned local relation are consistent. If they are consistent, the corresponding local text relation representation and the local image relation representation are finally determined to be successfully aligned, and the corresponding confidence score is extracted.

[0071] In step A401, the syntactic relation vector in the local text relation representation can be a vector representing the syntactic structural relationship between technical features in the text, obtained through dependency parsing. The spatial position relation vector in the local image relation representation can be a vector obtained by analyzing the visual structure or relative position of technical components in the attached figure. The cosine similarity between these two types of vectors quantifies the degree of similarity between the relational structure described in the text and the spatial structure or positional relationship presented in the attached figure. For example, the syntactic relation vector can represent the structure "A connects to B," and the spatial position relation vector can represent the connection or adjacency relationship between component A and component B in the attached figure. The higher the cosine similarity value, the more similar the two relational structures or positions are.

[0072] Furthermore, in step A402, the initial alignment score obtained in step A401 is compared with a preset similarity threshold. For example, this threshold can be set to 0.6. If the initial alignment score is lower than this threshold, it is considered that the local relationships in the text and image differ significantly in structure or position, and the alignment is initially determined to be unsuccessful. Subsequent verification is then discontinued, thereby improving processing efficiency.

[0073] For local relations initially determined to be successfully aligned, step A403 further introduces semantic-level verification. The semantic similarity between the word vectors of the text modality technical features related to the local relation and the visual feature vectors of the image modality technical components is calculated to obtain a semantic consistency score. For example, if the local relation involves the technical feature "gear" in the text and the technical component "gear" in the image, the semantic similarity between the word vector representing the concept of "gear" and the feature vector representing the visual appearance of the "gear" in the attached image is calculated. This step ensures that even if the relation structure or location is similar, the associated technical features and technical components should be semantically related, compensating for potential errors arising from relying solely on structural or positional similarity.

[0074] Therefore, in step A404, the initial alignment score, reflecting the similarity of the relational structure or location, is weighted and fused with the semantic consistency score, reflecting the semantic relevance of technical features or components. For example, a linear weighting method can be used: confidence score = w1 * initial alignment score + w2 * semantic consistency score, where w1 and w2 are weight coefficients, for example, w1 = 0.7, w2 = 0.3, and w1 + w2 = 1. This fusion method comprehensively considers the relational structure / location information and the semantic information of technical features / components, enabling the obtained confidence score to more comprehensively reflect the reliability of cross-modal relational alignment.

[0075] Finally, in step A405, the fused confidence score is compared with another preset confidence threshold. For example, this threshold can be set to 0.8. Only when the confidence score is greater than or equal to this threshold is the corresponding local text relation representation and local image relation representation considered successfully aligned, and the confidence score is extracted. This ensures that only cross-modal relations that have undergone dual verification of structure / location and semantics and have high confidence are adopted. Through this fine-grained verification method, cross-modal alignment relations and their reliability can be determined more accurately, thereby laying the foundation for subsequently constructing more accurate multimodal relation representations.

[0076] Compared to methods that rely solely on general consistency verification, this approach, by combining relational structure / positional similarity with technical features / component semantic similarity, can more effectively capture subtle differences in relational expressions across different modalities and potential semantic gaps, thus improving the accuracy of relational alignment and the reliability of confidence assessment. This more accurate relational alignment result, when used to construct multimodal relational representations of patent documents, can generate graph structures that better reflect the core structure and semantic information of the technical solutions. This allows for a more precise measurement of the technical solution similarity between the query and the patent document in subsequent graph matching and similarity calculations, ultimately improving the accuracy and relevance of patent retrieval.

[0077] Preferably, step A403 may include:

[0078] Construct a semantic space containing multiple semantic dimensions and multiple semantic granularities; the semantic dimensions include topics, sentiments, and attributes, and the semantic granularities include word level, phrase level, and sentence level.

[0079] For text modality feature word vectors, based on the semantic space, the semantic representation of the text modality feature word vectors under different semantic dimensions and semantic granularities is calculated to obtain multi-level text semantic vectors;

[0080] For the visual feature vectors of image modality technology components, based on the semantic space, the semantic representation of the visual feature vectors of image modality technology components under different semantic dimensions and semantic granularities is calculated to obtain multi-level image semantic vectors;

[0081] Multi-level text semantic vectors and multi-level image semantic vectors are fused separately to obtain fused text semantic vectors and fused image semantic vectors. The cosine similarity between the fused text semantic vectors and fused image semantic vectors is calculated, and the cosine similarity is used as the semantic consistency score.

[0082] The construction of a semantic space encompassing multiple semantic dimensions and granularities provides a foundational framework for subsequent semantic representation computation. Semantic dimensions can be further refined to include aspects related to the patented technology, such as function, structure, and materials. Semantic granularity can also be further refined; for example, at the phrase level, it can distinguish between noun phrases and verb phrases. As one possible implementation, this semantic space could be a pre-trained multidimensional semantic model trained on large-scale patent text and image data to learn the semantic distribution of technical concepts across different dimensions and granularities.

[0083] For text modality feature word vectors, semantic representations at different semantic dimensions and granularities are calculated based on the constructed semantic space, resulting in multi-level text semantic vectors. Specifically, attention or gating mechanisms can be employed to ensure that when calculating the representation of a text vector at a specific semantic dimension (e.g., topic) and granularity (e.g., phrase), the model focuses on the parts of the text vector most relevant to that dimension and granularity. For example, for a word vector representing "gear," its semantic representation at the "structure" dimension and "word" granularity might emphasize its shape and composition; while at the "function" dimension and "phrase" granularity (e.g., "transmission gear"), its semantic representation might emphasize its role in transmitting power. Thus, the semantics of text modality features are decomposed and represented at multiple semantic levels.

[0084] Similarly, for the visual feature vectors of image modal technology components, semantic representations at different semantic dimensions and granularities are calculated based on the same semantic space, resulting in multi-level image semantic vectors. This step aims to map the visual information of an image into a semantic framework consistent with that of text. For example, for a visual feature vector representing a gear in an attached diagram, its semantic representation at the "structure" dimension and "component" granularity might capture visual structural information such as its tooth shape and size; while at the "function" dimension and "component" granularity, its semantic representation might capture visual cues of its role as a transmission component (such as its connection to other components). The representation of image feature vectors at different semantic dimensions and granularities can be achieved by training a cross-modal mapping network that learns how to project the visual features of an image onto different dimensions and granularities of a predefined semantic space. In this way, the visual features of image technology components are also transformed into multi-level semantic representations, which are compared with the multi-level semantic representations of text within the same semantic space.

[0085] Multi-level text semantic vectors and multi-level image semantic vectors are fused separately to obtain fused text semantic vectors and fused image semantic vectors. Fusion methods can employ various approaches, such as vector concatenation followed by linear transformation, weighted summation, or attention-based fusion. For example, text semantic vectors from different semantic dimensions and granularities can be concatenated to form a longer fused text semantic vector; alternatively, weighted summation can be performed by assigning different weights to vectors at each level based on their contribution to semantic consistency. Through fusion, semantic information from different levels is integrated to form a more comprehensive and robust semantic description of technical features or components.

[0086] The cosine similarity between the fused text semantic vector and the fused image semantic vector is calculated and used as the semantic consistency score. Since the fused vector contains semantic information extracted and integrated at multiple semantic dimensions and granularities, calculating the cosine similarity between them can more precisely measure the semantic matching degree between textual technical features and image technical components. This multi-level, multi-dimensional semantic comparison method overcomes the limitations of simple vector similarity calculation and can more accurately capture the complex and multi-level semantic correspondences in the patent technology field. Therefore, the obtained semantic consistency score is more accurate, which directly improves the accuracy of the weighted fusion in step A404, making the calculated confidence score more reliable. The more reliable confidence score further improves the accuracy of judging the success or failure of local cross-modal relationship alignment in step A4, thus laying a solid foundation for the subsequent construction of a high-quality multimodal relationship representation (step A5), and ultimately improving the accuracy of the entire patent retrieval method.

[0087] In some implementations, step A5 includes:

[0088] A501. For patent documents, extract all relationship anchor points and construct an initial graph structure, where nodes represent technical features or technical components;

[0089] A502. For each relationship anchor point, based on the confidence score, filter cross-modal relationships with confidence scores greater than a preset score threshold to obtain a set of high-confidence cross-modal relationships;

[0090] A503. Based on a set of high-confidence cross-modal relationships, add edges to the initial graph structure. The edges represent cross-modal alignment relationships, and the weights of the edges are set to the corresponding confidence scores to construct a weighted graph structure. The edges and their weights represent the strength of the cross-modal alignment relationships.

[0091] A504. For weighted graph structures, an attention-based graph embedding algorithm is adopted to dynamically adjust the information transmission between nodes according to the edge weights, and map each node to a low-dimensional vector space to obtain a node embedding vector; the node embedding vector is used to represent the semantic information of technical features or technical components.

[0092] A505. Associate the node embedding vector with the corresponding technical features or technical components to form the final multimodal relationship representation.

[0093] In step A501, all relationship anchor points are extracted from the patent document, and an initial graph structure is constructed based on these anchor points. The nodes of the graph represent technical features or technical components. These technical features and components are identified and associated from the text and drawings of the patent document in previous steps. For example, one node may represent a "gear" described in the text, and another node may represent a gear component marked "10" in the drawings. The initial graph structure establishes only these technical elements as the basic units of the graph.

[0094] Further, in step A502, for each relation anchor point, the confidence score of the local relation alignment obtained in the previous steps is used to filter out cross-modal relations with confidence scores greater than a preset score threshold, forming a set of high-confidence cross-modal relations. The preset score threshold can be set according to actual application needs, for example, it can be set to 0.7 or 0.8. This filtering process excludes cross-modal relations with low alignment confidence, ensuring that the relation representations subsequently constructed are based on information with high confidence. For example, if the confidence score of the text description "gear drives shaft to rotate" in alignment with the spatial relationship between the gear and shaft in the attached figure is lower than the threshold, then the relation will be excluded.

[0095] Then, in step A503, edges are added to the initial graph structure based on the selected set of high-confidence cross-modal relationships. These edges represent validated cross-modal alignment relationships. The weights of the edges are set to the corresponding confidence scores. This constructs a weighted graph structure. For example, if in step A502 the text description "gear and shaft connection" aligns with the gear and shaft connection structure in the attached figure, and the confidence score is 0.9, then an edge is added between the nodes representing the gear and shaft, and the weight of this edge is set to 0.9. By using the confidence scores as edge weights, the strength information of cross-modal alignment is directly encoded into the graph structure, enabling the graph structure to reflect the degree of influence of different relationships.

[0096] In step A504, an attention-based graph embedding algorithm is employed for the constructed weighted graph structure. This algorithm dynamically adjusts the information transfer between nodes based on edge weights, mapping each node to a low-dimensional vector space to obtain a node embedding vector. For example, a Graph Attention Network (GAT) or its variants can be used. When calculating a node's embedding vector, information from other nodes connected to that node is aggregated, and the attention weights applied during aggregation are influenced by the weights of the connecting edges. The larger the edge weight, the greater the influence of the connected node's information on the current node's embedding vector. Therefore, the generated node embedding vector can better capture the semantic information of technical features or components within their technical solution structure, particularly the association information between them and other features or components connected through cross-modal relationships with high confidence. This approach integrates cross-modal relationship information with confidence into the node representation, generating a representation that includes relational semantics.

[0097] Finally, in step A505, the generated node embedding vectors are associated with the corresponding technical features or technical components to form the final multimodal relationship representation of the patent document. This final representation is a set of technical features and technical components stored in vector form. Each vector contains the structural and semantic information of that feature or component within the entire technical solution, particularly its relationship with other features or components connected through high-confidence cross-modal relationships. This vectorized representation facilitates subsequent similarity calculations. Through the above steps, starting from relationship anchors and cross-modal relationships with confidence, a weighted graph is constructed, and a vectorized representation is generated using graph embedding technology, thereby solving the problem of integrating cross-modal relationship information into the patent document representation to support retrieval. This method utilizes the high-confidence relationship information and its strength obtained in previous steps to construct a technical solution representation, which contributes to the subsequent retrieval effect.

[0098] Preferably, step A504 may include:

[0099] For a weighted graph structure, calculate the degree centrality of each node in the graph; degree centrality represents how closely a node is connected to other nodes.

[0100] Based on the degree centrality of the nodes, the weighted graph structure is decomposed into multiple subgraph structures; among them, the node with high degree centrality serves as the central node of the subgraph structure, and the nodes directly connected to it constitute the other nodes of the subgraph structure.

[0101] For each subgraph structure, an attention-based graph embedding algorithm is used to dynamically adjust the information transmission between the central node and other nodes in the subgraph according to the edge weights, and to map each node to a low-dimensional vector space to obtain the node embedding subvector.

[0102] By integrating the node embedding subvectors of all subgraph structures, we obtain the final node embedding vector for each node in the weighted graph structure.

[0103] In this process, after constructing the multimodal relation representation of the patent documents and forming a weighted graph structure, the degree centrality of each node in the weighted graph structure is first calculated to optimize the graph embedding process. Degree centrality quantifies the degree of connectivity of a node in the graph; nodes with more connecting edges have higher degree centrality. These nodes correspond to technical features or components that participate extensively in cross-modal relations within the technical solution, and they typically play a core role in the technical solution. A method for calculating degree centrality is to count the number of edges directly connected to a node. For example, if a technical feature node is connected to five technical component nodes confirmed through cross-modal alignment, then the degree centrality of that technical feature node is five.

[0104] Furthermore, based on the calculated degree centrality of the nodes, the entire weighted graph structure is decomposed into multiple subgraph structures. The decomposition is based on the degree centrality of the nodes. Nodes with high degree centrality are selected as the central nodes of the subgraph structures. Each subgraph structure consists of a central node and all its directly connected neighboring nodes. This decomposition method transforms the complex global graph structure into a series of local views surrounding important nodes, allowing the subsequent embedding process to focus on the information in these local structures. For example, if the degree centrality of node A is higher than a preset degree centrality threshold, a subgraph is constructed with node A as the center, containing node A and all nodes directly connected to node A.

[0105] Next, for each decomposed subgraph structure, an attention-based graph embedding algorithm is used for processing. In each subgraph, the algorithm dynamically adjusts the information transmission intensity based on the weights of the edges connecting the central node and its neighboring nodes (these weights represent the confidence scores of cross-modal alignment relationships). Edges with higher weights indicate that the corresponding cross-modal relationship has higher confidence; therefore, when calculating the embedding vector of the central node, information from these neighboring nodes is given higher attention weights. In this way, the nodes in each subgraph are mapped to a low-dimensional vector space, resulting in node embedding subvectors. This local embedding process allows the generated subvectors to more accurately reflect the relationship information between the central node and its important related neighboring nodes. This is done on the basis of constructing a multimodal relationship representation and forming a weighted graph structure, which already includes the cross-modal relationships and their confidence scores between technical features and technical components. By applying a weighted attention mechanism to the local subgraphs, the importance differences of local relationships can be captured more precisely, which compensates for the problem of ignoring local details when directly embedding complex global graphs.

[0106] Finally, the node embedding vectors of all subgraph structures are integrated to obtain the final node embedding vector for each node in the weighted graph structure. Integration methods can include averaging, summing, or concatenating the embedding vectors of the same node obtained in different subgraphs (if it is both the center node of one subgraph and a neighbor node of another). Through integration, the final embedding vector of each node not only contains information about its role as a center node or a neighbor node in the local context, but also indirectly reflects its connection pattern and relative importance in the entire graph structure.

[0107] This method, based on constructing a multimodal relationship representation and forming a weighted graph structure, generates low-dimensional vectors that can more comprehensively and accurately represent the semantic information of technical features or components through decomposition and local embedding re-integration. These vectors can more accurately reflect the semantic information of key elements and their complex relationships in the technical solution, thereby improving the accuracy of subsequent similarity calculations and solving the problem that it is difficult to fully capture local structural information and importance differences when directly performing global embedding on complex large-scale graphs.

[0108] In some preferred embodiments, the step of employing an attention-based graph embedding algorithm for each subgraph structure, dynamically adjusting the information transfer between the center node and other nodes in the subgraph according to the edge weights, and mapping each node to a low-dimensional vector space to obtain the node embedding subvector includes:

[0109] For each subgraph structure, assign a low-dimensional vector as the initial embedding subvector to each node in the subgraph structure;

[0110] For each subgraph structure, the attention weights between the center node and other nodes are calculated based on the softmax function;

[0111] For each subgraph structure, the initial embedding subvectors of other nodes are weighted and aggregated according to the calculated attention weights to obtain the context vector of the central node;

[0112] For each subgraph structure, the initial embedding subvector of the center node is fused with its context vector to obtain the effective embedding subvector of the center node, and the initial embedding vectors of other nodes are used as the effective embedding subvectors of those other nodes.

[0113] For each subgraph structure, L2 normalization is performed on the valid embedding vectors of all nodes to obtain the final node embedding vectors.

[0114] This scheme details the specific steps of applying an attention-based graph embedding algorithm to the subgraph structure, aiming to generate node embedding subvectors that more accurately reflect the subgraph structure and relational information through refined information transmission and aggregation mechanisms.

[0115] Specifically, firstly, for each subgraph structure, a low-dimensional vector is assigned as an initial embedding vector for each node. These initial embedding vectors can be randomly initialized in a pre-trained embedding space or obtained through other methods, and their dimension is a configurable parameter, such as 128 or 256 dimensions. This lays the foundation for subsequent information learning and representation.

[0116] Furthermore, for each subgraph structure, attention weights between the central node and other nodes are calculated using the softmax function. This step is the core of the attention mechanism, providing weights for subsequent information aggregation by calculating the correlation strength between the central node and other nodes within the subgraph. The attention weights can be calculated based on the similarity between the initial embedding subvectors of the central node and its neighbors, for example, by calculating scores through dot products or feedforward neural networks, and then normalized using the softmax function, giving higher attention weights to nodes closely related to the central node. In addition, dynamically adjusting information transmission based on edge weights can be achieved by considering the weights of the edges connecting the central node and its neighbors when calculating the attention score. For example, the edge weights can be introduced as multiplication factors into the attention score calculation. Thus, the edge weight information is integrated into the determination of the attention weights, allowing the model to pay more attention to node information connected by strong connections.

[0117] Then, based on the calculated attention weights, the initial embedding subvectors of other nodes are weighted and aggregated to obtain the context vector of the central node. This step utilizes the attention weights calculated in the previous step to selectively and weightedly integrate the information of other nodes within the subgraph, forming a context representation that represents the local environment of the central node. Through weighted aggregation, the information of important neighboring nodes occupies a larger proportion in the context vector, thereby capturing the local structural information of the subgraph and the influence of neighboring nodes on the central node.

[0118] Subsequently, for each subgraph structure, the initial embedding vector of the central node is fused with its context vector to obtain the effective embedding vector of the central node. This fusion operation can be implemented in various ways, such as simple concatenation, summation, or transformation through a linear layer. This fusion combines the initial representation of the central node with the context information aggregated from its neighboring nodes, forming a more comprehensive and representative central node embedding. For other nodes in the subgraph, their initial embedding vectors are directly used as the effective embedding vectors of those other nodes. This approach highlights the special status and processing of the central node in the subgraph, while the effective embedding of neighboring nodes primarily depends on their own initial representations.

[0119] Finally, for each subgraph structure, L2 normalization is performed on the effective embedding vectors of all nodes to obtain the final node embedding vectors. Normalization helps to standardize the scale of the vectors, making the embedding vectors of different nodes comparable in the same space, and also helps with the stability of subsequent similarity calculations and model training. For example, a vector [1,2,2] becomes [1 / 3,2 / 3,2 / 3] after L2 normalization.

[0120] Therefore, through the above steps, this scheme provides a specific subgraph embedding method based on an attention mechanism. This method focuses on the information interaction between the central node and its neighboring nodes, and uses attention weights for weighted aggregation and fusion to generate node embedding sub-vectors that effectively capture the subgraph structure, the role of the central node, and the relationships between nodes. This fine-grained embedding process is performed at the subgraph level, leveraging the structural advantages of subgraph decomposition. Furthermore, through the attention mechanism and edge weight information, the generated node embedding sub-vectors can more accurately reflect the complex relationships between technical features or components and the key role of the central node, providing a high-quality underlying representation for constructing accurate multimodal relationship representations and performing similarity calculations.

[0121] In some implementations, step A7 includes:

[0122] A701. Extract all node embedding vectors from the multimodal relationship representation of the query data and patent documents, and construct the query vector set and the document vector set respectively;

[0123] A702. For each query vector in the query vector set, perform a nearest neighbor search in the document vector set to obtain several candidate document vectors that are most similar to the query vector;

[0124] A703. Based on the relationship anchor points corresponding to the candidate document vectors found in the search, calculate the matching degree between the query data and the patent documents at the relationship anchor point level;

[0125] A704. For each query vector and its corresponding candidate document vectors, calculate the cosine similarity between the query vector and each candidate document vector to obtain the vector similarity.

[0126] A705. The matching degree at the relationship anchor level and the vector similarity are weighted and fused to obtain the final similarity score;

[0127] A706. For patent documents, the final similarity scores of all query vectors and corresponding candidate document vectors are summed to obtain the overall similarity score of the patent documents, which serves as the similarity between the query data and the multimodal relationship representation of patent documents in the patent database.

[0128] Step A701 involves extracting all node embedding vectors from the multimodal relation representations of the query data and patent documents. These multimodal relation representations are graph structures constructed in previous steps, where nodes represent technical features or components. These nodes are mapped to a low-dimensional vector space using an attention-based graph embedding algorithm (e.g., as described in step A504), forming node embedding vectors. These vectors encode the semantic information of the technical features or components. By extracting these vectors, the semantic content of the nodes in the graph structure is explicitly separated, providing a foundation for subsequent vector-based similarity calculations. The extracted node embedding vectors from the query data constitute the query vector set, and the node embedding vectors from the patent documents constitute the document vector set.

[0129] Further, step A702 performs a nearest neighbor search in the document vector set for each query vector in the query vector set. This search process aims to find the document vectors in the document vector set that are closest to the current query vector in the vector space. Since node embedding vectors represent the semantic information of technical features or technical components, vector distances (e.g., Euclidean distance or cosine distance) reflect semantic similarity. Therefore, nearest neighbor search can identify the few technical features or technical components in the patent documents that are semantically most similar to a specific technical feature or technical component in the query. Nearest neighbor search can be implemented using various algorithms, such as exact search algorithms like kd-trees and ball trees, or approximate nearest neighbor search algorithms like Locality Sensitive Hashing (LSH) and graph-based search (e.g., HNSW) to improve search efficiency. For example, the search can be set to return the K most similar candidate document vectors, where K can be set to a value such as 3 or 5.

[0130] Therefore, step A703 uses the relationship anchor information corresponding to the candidate document vectors searched in step A702 to calculate the matching degree between the query data and the patent documents at the relationship anchor level. Relationship anchors are key technical features or components with cross-modal correspondences, determined in previous steps (e.g., as described in step A2) by matching text with the icon numbers in the accompanying drawings. When constructing a multimodal relationship representation, relationship anchors and their cross-modal alignment confidence scores are included (e.g., as described in step A5). By examining how many candidate document vectors found in the documents are semantically similar to the query technical features / components and how high the alignment confidence of these matching anchors is, the extent to which the key, cross-modal aligned technical features / components in the query are reflected in the documents can be assessed. Specifically, the number of relationship anchors among the searched candidate document vectors can be counted, and the confidence scores corresponding to these matching anchors can be obtained. The matching degree at the relationship anchor level can be calculated based on the number of matching anchors and their alignment confidence. For example, one can calculate the proportion of matching anchors to the total number of query anchors (query anchor coverage) and the average alignment confidence corresponding to the matching anchors, and combine the two (e.g., multiply them) as the matching degree at the relation anchor level. This matching metric provides evidence based on structural / relational correspondence, complementing matching based solely on semantic vectors.

[0131] Simultaneously, step A704 calculates the cosine similarity between each query vector and its corresponding candidate document vectors found in step A702. Cosine similarity measures the directional proximity of two vectors, ranging from -1 to 1; a higher value indicates greater directional proximity, i.e., stronger semantic similarity. Here, cosine similarity is used to quantify the semantic similarity between technical features or components in the query and similar technical features or components found in the documents. This provides a content-level matching metric.

[0132] Subsequently, step A705 weights and fuses the matching degree at the relation anchor level calculated in step A703 with the vector similarity calculated in step A704 to obtain the final similarity score. This fusion process combines structural / relational correspondence evidence based on relation anchors with semantic content similarity evidence based on node embedding vectors. The matching degree at the relation anchor level reflects whether key cross-modal alignment points in the query correspond in the literature and the reliability of that correspondence, while vector similarity reflects whether the semantic content of the technical features / components themselves is similar. By weighting and fusing these two types of information, a more comprehensive and robust similarity score can be obtained. For example, the final similarity score can be calculated as: Final similarity score = W1 * Matching degree at the relation anchor level + W2 * Vector similarity, where W1 and W2 are weight coefficients, satisfying W1 + W2 = 1, and W1 and W2 are greater than or equal to 0. The weights W1 and W2 can be set empirically, for example, W1 = 0.4, W2 = 0.6, or optimized through machine learning methods. This fusion approach allows similarity calculation to simultaneously consider the semantic similarity of technical features / components and the correspondence of key cross-modal alignment points, thus more accurately reflecting the overall relevance of complex technical solutions. Compared to using only general graph matching algorithms, this approach, by explicitly extracting and utilizing node embedding vectors and relational anchor information, can more finely measure similarity and improve the accuracy of similarity calculation.

[0133] Finally, in step A706, for each patent document, the final similarity scores of all query vectors and their corresponding candidate document vectors are summed to obtain the overall similarity score of the patent document. Specifically, for each technical feature or technical component in the query data (corresponding to a query vector), the most similar technical features or technical components (corresponding to candidate document vectors) in the document are found, and a final similarity score is calculated. Then, the final similarity scores corresponding to all technical features or technical components in the query data are summed. This summed score serves as the final similarity between the query data and the multimodal relationship representation of the patent document. For example, if the query data contains N technical features / components, the final overall document similarity score is the sum of the best matching scores (or some combination of multiple candidate matching scores) of each of these N technical features / components. This summing method ensures that the similarity score reflects the overall coverage and matching degree of the document to the multiple technical features / components and their relationships in the query, thus obtaining a comprehensive document similarity score for subsequent retrieval and ranking.

[0134] Through the above technical solution, this application can fully utilize the semantic information of technical features or components represented by node embedding vectors in the multimodal relation representation, as well as the cross-modal alignment structure information embodied by relation anchors. By combining and weighting the matching based on node semantic similarity with the matching based on relation anchor structure correspondence, the calculated similarity score can more accurately reflect the semantic relevance of complex technical solutions. This helps improve the accuracy of patent search results and optimize the relevance ranking of search results.

[0135] Preferably, step A703 may include:

[0136] Based on the candidate document vectors found, extract their corresponding relationship anchors, and count the number of the same relationship anchors between the query data and the patent documents, which is taken as the number of matching anchors.

[0137] For each matching anchor, obtain its corresponding confidence score, and calculate the sum of the confidence scores corresponding to all matching anchors as the alignment confidence sum;

[0138] The query anchor coverage is obtained by calculating the quotient of the number of matching anchors and the total number of query anchors; the total number of query anchors is the total number of relation anchors corresponding to the query data.

[0139] The average alignment confidence of the matching anchors is obtained by calculating the quotient of the sum of the alignment confidence scores and the number of matching anchors.

[0140] The product of query anchor coverage and average alignment confidence of matching anchors is calculated to obtain the matching degree between query data and patent documents at the relation anchor level.

[0141] This process involves extracting relational anchors from the candidate document vectors retrieved. These candidate document vectors, obtained in a previous step through a nearest neighbor search within the document vector set, represent document nodes similar to the query vector in the embedding space. Extracting these relational anchors identifies which technical features or components among these similar nodes are recognized as relational anchors. Subsequently, the number of identical relational anchors between the query data and the patent documents is counted, and this number is determined as the number of matching anchors. For example, if the query data has 5 relational anchors, and a candidate document has 3 identical relational anchors, then the number of matching anchors is 3. This step provides basic quantitative information on relational anchor-level matching.

[0142] Furthermore, for each anchor point identified as a match, its corresponding confidence score is obtained. These confidence scores were determined in the previous step by verifying the consistency between the local text relation representation and the local image relation representation, reflecting the reliability of the cross-modal relation alignment associated with that anchor point. For example, the confidence score can be a value between 0 and 1, where 1 indicates a highly reliable alignment and 0 indicates an unreliable alignment. The sum of the confidence scores corresponding to all these matching anchor points is calculated to obtain the total alignment confidence. For example, if the confidence scores of the three matching anchor points are 0.8, 0.9, and 0.7, respectively, then the total alignment confidence is 0.8 + 0.9 + 0.7 = 2.4. By accumulating the confidence scores, the quality information of the match is introduced into the match degree calculation, distinguishing the importance or reliability of different matching anchor points. This corresponds to the previous step of using confidence scores to filter high-confidence relations and constructing a weighted graph structure when building the multimodal relation representation, ensuring that this quality information can be utilized in the match degree calculation stage.

[0143] Based on this, the quotient of the number of matching anchors and the total number of query anchors is calculated to obtain the query anchor coverage rate. The total number of query anchors is the total number of relation anchors corresponding to the query data, which is determined when constructing the multimodal relation representation of the query data. For example, if the number of matching anchors is 3 and the total number of query anchors is 5, then the query anchor coverage rate is 3 / 5 = 0.6. Calculating the query anchor coverage rate quantifies the degree to which the literature covers the relation anchors in the query, reflecting the breadth of the match between the literature and the query in terms of the number of anchors.

[0144] Simultaneously, the average alignment confidence of the matching anchors is obtained by calculating the quotient of the sum of the alignment confidence scores and the number of matching anchors. For example, if the sum of the alignment confidence scores is 2.4 and the number of matching anchors is 3, then the average alignment confidence of the matching anchors is 2.4 / 3 = 0.8. By calculating the average confidence score, the average reliability of cross-modal alignment associated with all successfully matched relation anchors is evaluated, reflecting the matching depth between the literature and the query in terms of anchor quality.

[0145] Therefore, the calculated query anchor coverage rate is multiplied by the average alignment confidence of the matching anchors to obtain the matching degree between the query data and the patent documents at the relation anchor level. For example, if the query anchor coverage rate is 0.6 and the average alignment confidence of the matching anchors is 0.8, then the matching degree at the relation anchor level is 0.6 * 0.8 = 0.48. This product combines the quantity of matches (coverage rate) and the quality of matches (average confidence), allowing the calculated matching degree to more comprehensively and accurately reflect the similarity between the documents and the query at the relation anchor level.

[0146] In this way, the proposed scheme overcomes the shortcomings of simple quantitative matching, effectively utilizing the confidence information of cross-modal alignment and improving the accuracy of the matching degree calculation at the relation anchor level. This more accurate relation anchor level matching degree is weighted and fused with vector similarity in subsequent steps, directly affecting the final overall similarity score and retrieval ranking results, thereby improving the performance of the entire multimodal patent semantic retrieval system. For example, for two documents with the same number of matching anchors, the matching degree calculated using this scheme will differ because the confidence difference is taken into account. The document with higher confidence will receive a higher matching degree score, which helps to distinguish whether the key features of the technical solution have achieved reliable cross-modal alignment, thus more accurately assessing the relevance of the documents.

[0147] refer to Figure 2 This application provides a multimodal patent semantic retrieval system for retrieving patent documents from a patent database based on text and images. The system includes:

[0148] Data parsing module 1 is used to obtain text data and figure data of patent documents in the patent database. It parses the text data to identify technical features, relationship descriptions and attachment marks, and parses the figure data to extract the visual features and visual structural relationships of technical components and identify attachment marks. It constructs text representation structure and image representation structure (for details, refer to step A1 above).

[0149] Anchor point determination module 2 is used to establish a correspondence between the technical features described in the text and the technical components in the attached drawings by matching the icon numbers in the text data and the attached drawings, and to determine the technical features and technical components with corresponding relationships as relationship anchor points (for details, refer to step A2 above).

[0150] The relation representation extraction module 3 is used to analyze the text representation structure for each relation anchor point to extract the local text relation representation centered on the relation anchor point, and to analyze the image representation structure to extract the local image relation representation centered on the technical component corresponding to the relation anchor point (for details, refer to step A3 above).

[0151] The relation alignment module 4 is used to verify the consistency between the local text relation representation centered on the same relation anchor point and the local image relation representation, so as to determine the successfully aligned local relations and the corresponding confidence scores (for details, refer to step A4 above).

[0152] The relation representation construction module 5 is used to combine all relation anchors and cross-modal relations confirmed by local relation alignment and their confidence scores to construct a multimodal relation representation of the patent document; the multimodal relation representation is a graph structure (refer to step A5 above for details).

[0153] The query processing module 6 is used to construct a multimodal relational representation of the query data when it receives query data containing text and images submitted by the user (for details, refer to step A6 above).

[0154] Similarity calculation module 7 is used to calculate the similarity between the query data and the multimodal relationship representation of the patent documents in the patent database through a graph matching algorithm (for details, refer to step A7 above).

[0155] The search and sorting module 8 is used to sort patent documents according to similarity and return the search results (refer to step A8 above for the specific process).

[0156] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A multimodal patent semantic retrieval method for retrieving patent documents from a patent database based on text and images, characterized in that, The steps of this method include: A1. Obtain text data and figure data of patent documents from the patent database, parse the text data to identify technical features, relationship descriptions and attachment marks, parse the figure data to extract the visual features and visual structural relationships of technical components and identify attachment marks, and construct text representation structure and image representation structure. A2. By matching the icon numbers in the text data and the attached figures data, establish the correspondence between the technical features described in the text and the technical components in the attached figures, and determine the technical features and technical components with the corresponding relationship as relationship anchor points; A3. For each relation anchor point, analyze the text representation structure to extract the local text relation representation centered on the relation anchor point, and analyze the image representation structure to extract the local image relation representation centered on the technical component corresponding to the relation anchor point; A4. Verify the consistency between the local text relation representation centered on the same relation anchor point and the local image relation representation to determine the successfully aligned local relations and their corresponding confidence scores; A5. Combine all relation anchors and cross-modal relations confirmed through local relation alignment, along with their confidence scores, to construct a multimodal relation representation of the patent document; the multimodal relation representation is a graph structure; A6. Upon receiving query data containing text and images submitted by a user, construct a multimodal relational representation of the query data; A7. Calculate the similarity between the query data and the multimodal relation representations of patent documents in the patent database using a graph matching algorithm; A8. Sort the patent documents according to similarity and return the search results.

2. The multimodal patent semantic retrieval method according to claim 1, characterized in that, Step A3 includes: A301. For each relation anchor, from the text representation structure, use dependency parsing to determine the syntactic relationship between the relation anchor and other technical features, extract the word vector representations of the syntactic relationship and other technical features, and combine them to form a local text relation representation centered on the relation anchor. A302. For each relationship anchor point, extract the visual feature vectors of the technical component corresponding to the relationship anchor point and the adjacent technical components from the image representation structure. Calculate the spatial positional relationship vectors between the technical component and the adjacent technical components based on the visual structure relationship, and combine them to form a local image relationship representation centered on the technical component corresponding to the relationship anchor point.

3. The multimodal patent semantic retrieval method according to claim 1, characterized in that, Step A4 includes: A401. Calculate the cosine similarity between the syntactic relation vector in the local text relation representation and the spatial position relation vector in the local image relation representation to obtain the initial alignment score; A402. Determine whether the initial alignment score is greater than the preset similarity threshold. If so, the local relation alignment is initially determined to be successful; otherwise, the local relation alignment is determined to be unsuccessful. A403. For local relations that are initially determined to be successfully aligned, calculate the semantic similarity between the text modality feature word vectors and the image modality component visual feature vectors to obtain a semantic consistency score; A404. The initial alignment score and semantic consistency score are weighted and fused together, and the confidence score is obtained based on the fusion result. A405. Compare the confidence score with the preset confidence threshold to determine whether the local text relation representation and the local image relation representation in the initially determined aligned local relation are consistent. If they are consistent, the corresponding local text relation representation and the local image relation representation are finally determined to be successfully aligned, and the corresponding confidence score is extracted.

4. The multimodal patent semantic retrieval method according to claim 3, characterized in that, Step A403 includes: Construct a semantic space containing multiple semantic dimensions and multiple semantic granularities; the semantic dimensions include topics, sentiments, and attributes, and the semantic granularities include word level, phrase level, and sentence level. For text modality feature word vectors, based on the semantic space, the semantic representation of the text modality feature word vectors under different semantic dimensions and semantic granularities is calculated to obtain multi-level text semantic vectors; For the visual feature vectors of image modality technology components, based on the semantic space, the semantic representation of the visual feature vectors of image modality technology components under different semantic dimensions and semantic granularities is calculated to obtain multi-level image semantic vectors; Multi-level text semantic vectors and multi-level image semantic vectors are fused separately to obtain fused text semantic vectors and fused image semantic vectors. The cosine similarity between the fused text semantic vectors and fused image semantic vectors is calculated, and the cosine similarity is used as the semantic consistency score.

5. The multimodal patent semantic retrieval method according to claim 1, characterized in that, Step A5 includes: A501. For patent documents, extract all relationship anchor points and construct an initial graph structure, where nodes represent technical features or technical components; A502. For each relationship anchor point, based on the confidence score, filter cross-modal relationships with confidence scores greater than a preset score threshold to obtain a set of high-confidence cross-modal relationships; A503. Based on a set of high-confidence cross-modal relationships, add edges to the initial graph structure. The edges represent cross-modal alignment relationships, and the weights of the edges are set to the corresponding confidence scores to construct a weighted graph structure. The edges and their weights represent the strength of the cross-modal alignment relationships. A504. For weighted graph structures, an attention-based graph embedding algorithm is adopted to dynamically adjust the information transmission between nodes according to the edge weights, and map each node to a low-dimensional vector space to obtain a node embedding vector; the node embedding vector is used to represent the semantic information of technical features or technical components. A505. Associate the node embedding vector with the corresponding technical features or technical components to form the final multimodal relationship representation.

6. The multimodal patent semantic retrieval method according to claim 5, characterized in that, Step A504 includes: For a weighted graph structure, calculate the degree centrality of each node in the graph; degree centrality represents how closely a node is connected to other nodes. Based on the degree centrality of the nodes, the weighted graph structure is decomposed into multiple subgraph structures; among them, the node with high degree centrality serves as the central node of the subgraph structure, and the nodes directly connected to it constitute the other nodes of the subgraph structure. For each subgraph structure, an attention-based graph embedding algorithm is used to dynamically adjust the information transmission between the central node and other nodes in the subgraph according to the edge weights, and to map each node to a low-dimensional vector space to obtain the node embedding subvector. By integrating the node embedding subvectors of all subgraph structures, we obtain the final node embedding vector for each node in the weighted graph structure.

7. The multimodal patent semantic retrieval method according to claim 6, characterized in that, The steps of employing an attention-based graph embedding algorithm for each subgraph structure, dynamically adjusting the information transfer between the center node and other nodes in the subgraph according to the edge weights, and mapping each node to a low-dimensional vector space to obtain the node embedded subvectors include: For each subgraph structure, assign a low-dimensional vector as the initial embedding subvector to each node in the subgraph structure; For each subgraph structure, the attention weights between the center node and other nodes are calculated based on the softmax function; For each subgraph structure, the initial embedding subvectors of other nodes are weighted and aggregated according to the calculated attention weights to obtain the context vector of the central node; For each subgraph structure, the initial embedding subvector of the center node is fused with its context vector to obtain the effective embedding subvector of the center node, and the initial embedding vectors of other nodes are used as the effective embedding subvectors of those other nodes. For each subgraph structure, L2 normalization is performed on the valid embedding vectors of all nodes to obtain the final node embedding vectors.

8. The multimodal patent semantic retrieval method according to claim 5, characterized in that, Step A7 includes: A701. Extract all node embedding vectors from the multimodal relationship representation of the query data and patent documents, and construct the query vector set and the document vector set respectively; A702. For each query vector in the query vector set, perform a nearest neighbor search in the document vector set to obtain several candidate document vectors that are most similar to the query vector; A703. Based on the relationship anchor points corresponding to the candidate document vectors found in the search, calculate the matching degree between the query data and the patent documents at the relationship anchor point level; A704. For each query vector and its corresponding candidate document vectors, calculate the cosine similarity between the query vector and each candidate document vector to obtain the vector similarity. A705. The matching degree at the relationship anchor level and the vector similarity are weighted and fused to obtain the final similarity score; A706. For patent documents, the final similarity scores of all query vectors and corresponding candidate document vectors are summed to obtain the overall similarity score of the patent documents, which serves as the similarity between the query data and the multimodal relationship representation of patent documents in the patent database.

9. A multimodal patent semantic retrieval method according to claim 8, characterized in that, Step A703 includes: Based on the candidate document vectors found, extract their corresponding relationship anchors, and count the number of the same relationship anchors between the query data and the patent documents, which is taken as the number of matching anchors. For each matching anchor, obtain its corresponding confidence score, and calculate the sum of the confidence scores corresponding to all matching anchors as the alignment confidence sum; The query anchor coverage is obtained by calculating the quotient of the number of matching anchors and the total number of query anchors; the total number of query anchors is the total number of relation anchors corresponding to the query data. The average alignment confidence of the matching anchors is obtained by calculating the quotient of the sum of the alignment confidence scores and the number of matching anchors. The product of query anchor coverage and average alignment confidence of matching anchors is calculated to obtain the matching degree between query data and patent documents at the relation anchor level.

10. A multimodal patent semantic retrieval system for retrieving patent documents from a patent database based on text and images, characterized in that, The system includes: The data parsing module is used to acquire text data and figure data of patent documents in the patent database. It parses the text data to identify technical features, relationship descriptions and attachment marks, and parses the figure data to extract the visual features and visual structural relationships of technical components and identify attachment marks, and constructs text representation structure and image representation structure. An anchor point determination module is used to establish a correspondence between technical features described in the text and technical components in the accompanying drawings by matching the icon numbers in the text data and the accompanying drawings, and to determine the technical features and technical components with corresponding relationships as relationship anchor points; The relation representation extraction module is used to analyze the text representation structure for each relation anchor point to extract the local text relation representation centered on the relation anchor point, and to analyze the image representation structure to extract the local image relation representation centered on the technical component corresponding to the relation anchor point. The relation alignment module is used to verify the consistency between local text relation representations centered on the same relation anchor point and local image relation representations, in order to determine the successfully aligned local relations and their corresponding confidence scores; The relation representation construction module is used to combine all relation anchors and cross-modal relations confirmed by local relation alignment and their confidence scores to construct a multimodal relation representation of patent documents; the multimodal relation representation is a graph structure; The query processing module is used to construct a multimodal relational representation of the query data when it receives query data containing text and images submitted by the user. The similarity calculation module is used to calculate the similarity between the query data and the multimodal relationship representation of patent documents in the patent database using a graph matching algorithm. The search and sorting module is used to sort patent documents based on similarity and return the search results.

Images