Passport cross-modal retrieval method and system based on knowledge graph constraint

Abstract

This invention belongs to the field of artificial intelligence technology and discloses a passport cross-modal retrieval method and system based on knowledge graph constraints. It aims to address the problems of domain semantic gap, lack of complex semantic parsing capabilities, and black-box nature of the retrieval decision-making process in existing technologies. This invention achieves a unified approach to high-precision retrieval and interpretability of the reasoning process under complex semantic understanding by constructing a passport multimodal knowledge graph, designing spatial inductive bias, building a multi-granularity feature deep fusion mechanism, and generating a visual evidence chain.

Description

Technical Field

[0001] This invention belongs to, but is not limited to, the field of artificial intelligence technology, and particularly relates to a method and system for cross-modal passport retrieval based on knowledge graph constraints. Background Technology

[0002] With the acceleration of globalization, passports, as key documents for citizens to enter and exit countries and prove their nationality and identity abroad, are being used more frequently. Currently, more than 200 countries / regions worldwide issue passports following the International Civil Aviation Organization (ICAO) Doc 9303 standard framework, but they differ significantly in anti-counterfeiting technologies, the number of features, and implementation methods. Intelligent verification and analysis of passport samples has become a core requirement for enhancing national border security.

[0003] Existing passport retrieval technologies primarily rely on structured field queries or image similarity matching based on deep learning. While these methods can achieve sample matching at the visual level, they still have the following significant drawbacks when dealing with complex real-world assessment tasks:

[0004] (1) Domain Semantic Gap: Existing systems lack a specific ontology definition for passport anti-counterfeiting technology, resulting in cognitive gaps when the model processes technical terms. General visual models struggle to distinguish subtle physical differences between optically variable inks and ordinary printing, and also fail to understand the hierarchical relationships within the certificate version system, leading to systemic biases in high-level semantic reasoning.

[0005] (2) Lack of complex semantic parsing capabilities: Inspectors often use natural language to make requests that include multiple combinations of conditions and logical constraints. Existing systems mostly use structured metadata filtering methods, which lack the ability to parse abstract concept descriptions and complex logical constraints. This makes it impossible for users to efficiently express their inspection intentions in natural language, which seriously restricts the retrieval accuracy and interaction efficiency in complex scenarios.

[0006] (3) The retrieval decision-making process is black-boxed: Existing end-to-end retrieval models lack verifiable reasoning basis and cannot meet the stringent requirements of judicial-level judgment for transparency of results. The system cannot establish a complete chain of evidence from text query to structured logic to visual anchors, making it impossible for examiners to verify the accuracy and reliability of the model's decisions.

[0007] Based on the above analysis, the urgent technical problems that need to be solved in the existing technology are:

[0008] How to overcome the bottlenecks of fragmented passport information and single retrieval dimensions, and achieve efficient, intelligent, semantic, and cross-modal search of global passports and their anti-counterfeiting features? Summary of the Invention

[0009] To address the problems existing in the prior art, this invention provides a method and system for cross-modal passport retrieval based on knowledge graph constraints.

[0010] This invention is implemented as follows: a passport cross-modal retrieval method based on knowledge graph constraints, characterized in that the method specifically includes:

[0011] Step S10: Obtain basic information and multispectral images of passport samples, establish topological relationships between passport entities, anti-counterfeiting features and their physical spatial attributes through ontology modeling, generate fine-grained visual description text for anti-counterfeiting features, and transform unstructured anti-counterfeiting specifications into structured graph nodes and edges, and store them in a graph database.

[0012] Step S20: Receive the user's natural language query text, parse the text using a pre-set domain dictionary, identify anti-counterfeiting entities and parse Boolean logic constraints, and semantically expand the fuzzy query using the hierarchical relationship of the graph concept, and finally map the natural language into machine-executable graph path constraints.

[0013] Step S30: Based on the parsed logical constraints, perform an initial screening in the passport image library to obtain a candidate image set; if the initial screening result is empty, automatically execute the hierarchical constraint relaxation strategy to remove weak constraints such as year and format according to priority, so as to ensure that a sufficient number of candidate images are obtained for subsequent processing.

[0014] Step S40: Traverse the candidate image set, retrieve the normalized coordinates corresponding to the query keywords on each image, map them to the feature map scale through coordinate affine transformation, and adaptively generate hard attention or soft attention masks according to the physical properties of the anti-counterfeiting features; for cases where multiple features are hit or no features are hit, execute an adaptive spatial gating strategy to generate a comprehensive mask or degenerate into global retrieval.

[0015] Step S50: Construct a domain-adaptive fine-tuned visual encoder, input the candidate image into the visual encoder to extract global features, and simultaneously apply the spatial attention mask to the image feature layer to extract local features focused on the anti-counterfeiting area; then, use a residual combination network to deeply fuse the global features and local features to generate the final visual vector that takes into account both the overall layout and the anti-counterfeiting details.

[0016] Step S60: Calculate the similarity score between the query text features and the final visual vector, and generate an explanatory evidence chain that includes the graph inference path, feature location coordinates and matching score; finally, sort the results according to the similarity score and output the retrieved passport image and the visualized anti-counterfeiting feature location box to the user.

[0017] Furthermore, the steps of obtaining basic information and multispectral images of passport samples, establishing topological relationships between passport entities, anti-counterfeiting features, and their physical spatial attributes through ontology modeling, generating fine-grained visual description text for anti-counterfeiting features, and transforming unstructured anti-counterfeiting specifications into structured graph nodes and edges, and storing them in a graph database, specifically include:

[0018] (1) Conduct entity relationship analysis on the passport source data, sort out the logical relationship between passport version, anti-counterfeiting features, issuing country and physical location attributes, and construct a passport anti-counterfeiting ontology model;

[0019] (2) Use a deep learning entity recognition model to extract anti-counterfeiting terms from unstructured text and map them to entity nodes in the graph;

[0020] (3) Use prompt templates to transform graph nodes and path information into structured visual description text, which serves as a knowledge supervision signal for cross-modal alignment fine-tuning.

[0021] Furthermore, the steps of receiving the user's natural language query text, parsing the text using a pre-set domain dictionary, identifying anti-counterfeiting entities and resolving Boolean logic constraints, and semantically expanding the fuzzy query using the hierarchical relationships of graph concepts, ultimately mapping the natural language into machine-executable graph path constraints, specifically include:

[0022] (1) Input the natural language query text into the rule-based semantic parser and load the passport anti-counterfeiting domain dictionary;

[0023] (2) Perform word segmentation and dependency parsing analysis on the text to identify descriptive words belonging to anti-counterfeiting technology terms and mark them as keywords;

[0024] (3) Identify restrictive words in the text regarding the issuing country, year, and document type, and parse the Boolean logic relationships therein to generate graph path constraints;

[0025] (4) Use the concept attribution relationship in the graph to perform semantic expansion, and map the fuzzy concept to a set of specific anti-counterfeiting feature entities;

[0026] (5) Query the pre-built knowledge graph index and map the keywords to unique entity node IDs in the graph. If there are synonyms, they are uniformly mapped to standard term nodes.

[0027] Furthermore, the step of performing an initial screening in the passport image database to obtain a candidate image set based on the parsed logical constraints, and automatically executing a hierarchical constraint relaxation strategy to remove weak constraints such as year and format according to priority to ensure that a sufficient number of candidate images are obtained for subsequent processing, specifically includes:

[0028] (1) Using the parsed initial logical constraints, perform an exact match query in the metadata field of the passport image database to obtain the initial candidate image set;

[0029] (2) Real-time detection of whether the initial candidate image set is an empty set; if it is an empty set, trigger the hierarchical constraint relaxation mechanism, and remove non-critical weak constraints such as year and format in turn according to the preset priority list.

[0030] (3) Based on the relaxed constraints, the query is re-executed in the passport image database until a non-empty valid candidate image set is obtained.

[0031] Furthermore, the step of traversing the candidate image set, retrieving the normalized coordinates corresponding to the query keywords on each image, mapping them to the feature map scale through coordinate affine transformation, and adaptively generating a hard attention or soft attention mask based on the physical properties of the anti-counterfeiting features; for cases with multiple feature hits or no feature hits, executing an adaptive spatial gating strategy to generate a comprehensive mask or degenerate into a global search, specifically includes:

[0032] (1) Link the current image in the candidate image set to the corresponding passport version entity in the knowledge graph;

[0033] (2) Retrieve the region of interest data associated with the query keywords under the entity, obtain the normalized spatial coordinates, and calibrate them to the model input coordinate system through affine transformation;

[0034] (3) Adaptively select the generated mask based on the physical properties (rigid or diffuse) of the anti-counterfeiting features: generate a hard attention mask for features with clear boundaries, and generate a Gaussian-distributed soft attention mask for features such as fluorescence;

[0035] (4) Perform adaptive spatial gating: If multiple anti-counterfeiting points are hit, perform a union operation to generate a comprehensive mask; if no coordinates are hit, automatically degenerate into a global mask with full weights.

[0036] Furthermore, the step of constructing a domain-adaptive fine-tuned visual encoder, inputting candidate images into the visual encoder to extract global features, and simultaneously applying the spatial attention mask to the image feature layer to extract local features focused on the anti-counterfeiting area; subsequently, using a residual combination network to deeply fuse the global and local features to generate the final visual vector that takes into account both the overall layout and anti-counterfeiting details, specifically includes:

[0037] (1) The original image is scaled proportionally and preprocessed with zero-pixel padding, and then input into a visual encoder that has been adaptively fine-tuned in the domain; wherein, before inference, the encoder has completed the alignment fine-tuning for the passport anti-counterfeiting texture by using the symmetric contrast loss function based on the visual description text provided in S10 as positive sample pairs.

[0038] (2) Perform Hadamard product operation between the original feature map output by the fine-tuned feature extraction network and the spatial attention mask to output a local feature map focused on the anti-counterfeiting feature region;

[0039] (3) Perform global average pooling on the original feature map and the local feature map respectively to obtain the global feature vector and the local feature vector;

[0040] (4) Input the global feature vector and the local feature vector into the residual combination network, use the global feature as the identity mapping benchmark, use the local feature as the residual correction term for deep fusion, and output the final fused visual vector.

[0041] Further, the steps of calculating the similarity score between the query text features and the final visual vector, and generating an explanatory evidence chain including the graph inference path, feature location coordinates, and matching score; and finally sorting the results according to the similarity score and outputting the retrieved passport image and the visualized anti-counterfeiting feature location box to the user, include:

[0042] (1) Extract the semantic feature vector of the query text using the fine-tuned text encoder;

[0043] (2) Calculate the cosine similarity between the text semantic feature vector and the fused visual vector as the image matching score;

[0044] (3) Construct an explanatory chain of evidence that includes the graph logic reasoning path, the final effective constraint, the feature location coordinates and the matching score;

[0045] (4) Sort all candidate images in descending order based on matching scores and select the Top-K results;

[0046] (5) When outputting the results, use the region coordinates to draw a highlighted positioning box on the image and simultaneously display the corresponding explanatory evidence.

[0047] Another objective of this invention is to provide a passport cross-modal retrieval system based on knowledge graph constraints, the system specifically comprising:

[0048] The graph construction and management module is used to construct a multimodal passport anti-counterfeiting knowledge graph and transform unstructured anti-counterfeiting specifications into structured graph nodes and edges.

[0049] The semantic parsing module is used to parse the user's natural language query text, extract keywords and logical constraints, and complete entity mapping;

[0050] The candidate screening and relaxation module is used to perform initial screening of the candidate set and is equipped with an automatic relaxation unit to perform constraint relaxation strategy when the initial screening result is empty.

[0051] The spatial mask generation module is used to generate a spatial attention mask that guides visual attention based on the physical region coordinates retrieved from the knowledge graph.

[0052] The feature fusion network module is used to extract global features of the image and fuse global features with mask-weighted local features through a residual combination network;

[0053] The reasoning, decision-making, and visualization module is used to calculate similarity scores, generate explanatory evidence chains, and output ranking results and feature localization boxes.

[0054] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0055] This invention provides a cross-modal passport retrieval method based on knowledge graph constraints, comprising the following steps: First, constructing a multimodal passport anti-counterfeiting knowledge graph and establishing topological associations between entities and physical spatial attributes; then, receiving the user's natural language query text, parsing the text using a pre-set domain dictionary, separating keywords and logical constraints, and completing entity mapping; performing initial screening of the candidate set based on the parsed logical constraints, and configuring an automatic constraint relaxation strategy to ensure recall; traversing the candidate image set, retrieving the physical region coordinates corresponding to the query keywords, and generating a spatial attention mask; inputting the candidate images into a visual encoder to extract global features, while simultaneously using the spatial attention mask to extract local features focused on the anti-counterfeiting area, and fusing the two through a residual combination network; finally, calculating the similarity score, generating an explanatory evidence chain containing effective constraints and feature location coordinates, and outputting the retrieval results and a visualized anti-counterfeiting feature location box. This invention effectively overcomes the semantic gap in understanding technical terms by constructing an ontology representation in the anti-counterfeiting field, designing a spatial inductive bias mechanism, and establishing a visual reasoning evidence chain. It accurately responds to natural language commands containing anti-counterfeiting features through Gaussian masking technology. At the same time, while improving retrieval accuracy, this invention provides verifiable reasoning evidence, meeting the stringent requirements for the reliability of retrieval results in high-security scenarios.

[0056] The expected benefits and commercial value of the technical solution of this invention after transformation are as follows: By constructing the first relatively complete global mainstream passport multispectral knowledge graph and supporting retrieval system, this invention transforms fragmented expert experience into a standardized technical foundation, realizing the intelligent transformation of the verification and analysis mode from manual experience-driven to expert knowledge-enhanced, and improving the work efficiency of front-line and back-end analysts; The field of passport anti-counterfeiting has a high professional threshold. This invention, by algorithmizing complex retrieval logic, enables non-experts to quickly locate and identify the key features of various rare documents with the assistance of the system, significantly reducing the time and economic cost of talent training.

[0057] This invention fills a technological gap in the domestic and international industry: through independent research and development, it constructs the first relatively complete multispectral knowledge graph of mainstream global passports, achieving self-sufficiency and secure control over the underlying core data; existing technologies still lack the ability to quickly and directionally match specific anti-counterfeiting processes in massive image databases, but this solution, through a spatial attention mechanism guided by a knowledge graph, achieves for the first time the accurate positioning and retrieval of minute anti-counterfeiting features in complex backgrounds, filling a gap in the field of efficient anti-counterfeiting feature search tools. Furthermore, addressing concerns about the distrust and lack of transparency of black-box models in the security field, this solution is the first to achieve complete evidence chain output. This WYSIWYG white-box reasoning mode not only allows analysts to clearly understand the origin of the search results but also provides convincing logical evidence for subsequent administrative verification. Attached Figure Description

[0058] Figure 1 This is a flowchart of a passport cross-modal retrieval method based on knowledge graph constraints provided in an embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram of the passport anti-counterfeiting knowledge graph provided in an embodiment of the present invention;

[0060] Figure 3 This is a diagram of the multi-granularity feature fusion and retrieval strategy architecture provided in the embodiments of the present invention;

[0061] Figure 4 This is the knowledge graph visualization and interactive main interface provided in the embodiments of the present invention;

[0062] Figure 5 This is a timing diagram of cross-modal retrieval service processing provided in an embodiment of the present invention;

[0063] Figure 6 This is a schematic diagram of the application of the retrieval system provided in an embodiment of the present invention;

[0064] Figure 7 This is a structural block diagram of a cross-modal passport retrieval system based on knowledge graph constraints provided in an embodiment of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0066] This invention provides a passport cross-modal retrieval method and system based on knowledge graph constraints. Its core lies in constructing a multimodal knowledge graph for anti-counterfeiting verification, which uniformly describes the complex relationships between passport version, anti-counterfeiting features, physical regions and metadata, transforming unstructured anti-counterfeiting specifications into computable structured priors. At the same time, it uses spatial inductive bias to guide visual attention and fuses global layout and local anti-counterfeiting features through residual networks, ultimately outputting retrieval results with a visualized evidence chain.

[0067] like Figure 1 As shown, this embodiment of the invention provides a passport cross-modal retrieval method based on knowledge graph constraints, which specifically includes:

[0068] Step S10: Construct a multimodal passport anti-counterfeiting knowledge graph; obtain basic information and multispectral images of passport samples, establish topological relationships between passport entities, anti-counterfeiting features and their physical spatial attributes through ontology modeling, and transform unstructured anti-counterfeiting specifications into structured graph nodes and edges, and store them in a graph database;

[0069] Step S20: Receive the user's natural language query text, parse the text using a pre-built domain dictionary, separate out keywords describing passport features and logical constraints such as year and country, and map the keywords to the corresponding unique entity node ID in the knowledge graph;

[0070] Step S30: Perform initial screening and constraint relaxation of the candidate set; based on the parsed logical constraints, perform initial screening in the passport image database to obtain a candidate image set; if the initial screening result is empty, automatically execute the constraint relaxation strategy, remove weak constraints and perform a new search to ensure that a sufficient number of candidate images are obtained for subsequent processing.

[0071] Step S40: Generate a spatial attention mask; traverse the candidate image set, retrieve the physical region coordinates corresponding to the query keywords on each image based on the entity attributes recorded in the knowledge graph, and generate a spatial attention mask based on the coordinates to visually highlight the anti-counterfeiting feature areas that match the text description.

[0072] Step S50: Feature extraction and residual fusion; The candidate image is input into the visual encoder to extract global features, and the spatial attention mask is applied to the image feature layer to extract local features focused on the anti-counterfeiting area; Then, the global features and local features are weighted and fused through the residual combination network to generate the final visual vector that takes into account both the overall layout and the anti-counterfeiting details.

[0073] Step S60: Similarity Calculation and Evidence Generation; Calculate the similarity score between the query text features and the final visual vector, and generate an explanatory evidence chain containing effective constraints, feature location coordinates, and matching scores; Finally, sort the results according to the similarity scores and output the retrieved passport image and the visualized anti-counterfeiting feature location box to the user.

[0074] The schematic diagram of the passport anti-counterfeiting knowledge graph is as follows: Figure 2 As shown. The steps of obtaining basic information and multispectral images of passport samples, establishing topological relationships between passport entities, anti-counterfeiting features, and their physical spatial attributes through ontology modeling, generating fine-grained visual description text for anti-counterfeiting features, and transforming unstructured anti-counterfeiting specifications into structured graph nodes and edges, and storing them in a graph database, include:

[0075] Entity relationship analysis was performed on passport source data to sort out the logical relationships between passport version, anti-counterfeiting features, issuing country and physical location attributes, and to construct a passport anti-counterfeiting ontology model.

[0076] Using a deep learning entity recognition model, anti-counterfeiting terms are extracted from unstructured text and mapped to entity nodes in a graph;

[0077] By using prompt templates, graph node and path information is transformed into structured visual descriptive text, which serves as a knowledge supervision signal for cross-modal alignment fine-tuning.

[0078] The steps of receiving the user's natural language query text, parsing the text using a pre-built domain dictionary, identifying anti-counterfeiting entities and resolving Boolean logic constraints, semantically expanding the fuzzy query using the hierarchical relationships of graph concepts, and finally mapping the natural language into machine-executable graph path constraints include:

[0079] Input the natural language query text into a rule-based semantic parser and load a dictionary for passport anti-counterfeiting.

[0080] The text is segmented and subjected to dependency parsing analysis to identify descriptive terms belonging to anti-counterfeiting technology and mark them as keywords;

[0081] Identify restrictive words in the text regarding the issuing country, year, and document type, and parse the Boolean logic relationships within them to generate graph path constraints;

[0082] Semantic expansion is performed by utilizing the concept attribution relationships in the graph, mapping fuzzy concepts to a set of specific anti-counterfeiting feature entities;

[0083] Query the pre-built knowledge graph index and map the keywords to unique entity node IDs in the graph. If synonyms exist, they are uniformly mapped to standard term nodes.

[0084] Multi-granularity feature fusion and retrieval strategy architecture diagram as follows: Figure 3 As shown.

[0085] The step of performing an initial screening in the passport image database to obtain a candidate image set based on the parsed logical constraints, and automatically executing a hierarchical constraint relaxation strategy to remove weak constraints such as year and format according to priority to ensure that a sufficient number of candidate images are obtained for subsequent processing, includes:

[0086] Using the parsed initial logical constraints, an exact match query is performed in the metadata fields of the passport image database to obtain an initial candidate image set;

[0087] Real-time detection of whether the initial candidate image set is empty; if it is empty, a hierarchical constraint relaxation mechanism is triggered, and non-critical weak constraints such as year and format are eliminated in turn according to the preset priority list.

[0088] The query is re-executed in the passport image database based on the relaxed constraints until a non-empty set of valid candidate images is obtained.

[0089] The process of traversing the candidate image set, retrieving the normalized coordinates corresponding to the query keywords on each image, mapping them to the feature map scale through coordinate affine transformation, and adaptively generating hard or soft attention masks based on the physical properties of the anti-counterfeiting features; for cases with multiple feature hits or no feature hits, executing an adaptive spatial gating strategy to generate a comprehensive mask or degenerate into a global search step includes:

[0090] Link the current image in the candidate image set to the corresponding passport version entity in the knowledge graph;

[0091] Retrieve the region of interest data associated with the query keywords under the entity, obtain the normalized spatial coordinates, and calibrate them to the model input coordinate system through affine transformation;

[0092] The mask generation is adaptively selected based on the physical properties of the anti-counterfeiting features: a hard attention mask is generated for features with clear boundaries, and a soft attention mask with a Gaussian distribution is generated for features such as fluorescence.

[0093] Perform adaptive spatial gating: If multiple anti-counterfeiting points are hit, perform a union operation to generate a comprehensive mask; if no coordinates are hit, automatically degenerate into a fully weighted global mask.

[0094] The steps of constructing a domain-adaptive fine-tuned visual encoder, inputting candidate images into the visual encoder to extract global features, and simultaneously applying the spatial attention mask to the image feature layer to extract local features focused on the anti-counterfeiting area; subsequently, using a residual combination network to deeply fuse the global and local features to generate a final visual vector that takes into account both the overall layout and anti-counterfeiting details, include:

[0095] The original image is scaled proportionally and preprocessed with zero-pixel padding, and then input into a domain-adaptive fine-tuned visual encoder. Before inference, the encoder has already performed alignment fine-tuning for passport anti-counterfeiting textures based on the visual description text provided in S10 as positive sample pairs and through a symmetric contrast loss function.

[0096] The original feature map output by the fine-tuned feature extraction network is combined with the spatial attention mask to perform a Hadamard product operation, and the output is a local feature map focused on the anti-counterfeiting feature region.

[0097] Global average pooling is performed on the original feature map and the local feature map respectively to obtain the global feature vector and the local feature vector;

[0098] The global feature vector and local feature vector are input into the residual combination network. The global feature vector is used as the identity mapping benchmark, and the local feature vector is used as the residual correction term for deep fusion. The final fused visual vector is output.

[0099] The steps of calculating the similarity score between the query text features and the final visual vector, generating an explanatory chain of evidence including a graph inference path, feature location coordinates, and matching scores, and finally sorting the results according to the similarity score and outputting the retrieved passport image and a visualized anti-counterfeiting feature location box to the user, include:

[0100] The semantic feature vector of the query text is extracted using a fine-tuned text encoder;

[0101] Calculate the cosine similarity between the text semantic feature vector and the fused visual vector, and use it as the image matching score;

[0102] Construct an explanatory chain of evidence that includes the graph logic reasoning path, the final effective constraint, the feature location coordinates, and the matching score;

[0103] All candidate images are sorted in descending order based on matching scores, and the Top-K results are selected.

[0104] When outputting the results, highlight bounding boxes are drawn on the image using region coordinates, and corresponding explanatory evidence is displayed simultaneously.

[0105] In the specific application of the method in intelligent judgment and auxiliary analysis scenarios, the method also includes the following process:

[0106] The system receives a reference image to be analyzed and analysis instructions in natural language from the analyst; the analysis instructions include attribute association constraints and visual feature descriptions.

[0107] Visual fingerprints of reference images are extracted using a visual encoder, and the corresponding passport version entities are anchored in the knowledge graph. Multi-hop reasoning of the graph is performed in conjunction with the attribute association constraints to instantiate the logical boundaries of the retrieval space, thereby determining the target retrieval library.

[0108] The visual feature description in the parsing instruction is mapped to anti-counterfeiting feature nodes in the knowledge graph, and the physical region coordinates of the feature in the reference image version are retrieved; a Gaussian spatial attention mask is generated based on the coordinates to force the model to focus on specific anti-counterfeiting regions;

[0109] Within the locked retrieval range, the generated spatial attention mask is applied to the feature extraction process of the database samples. The fusion vector of local focus features and global layout features is used for weighted matching to recall a list of associated passport samples that meet the combination conditions.

[0110] Output a list of related samples and simultaneously display the analysis basis: use bounding boxes to highlight the common feature areas of the reference samples and the retrieved samples on the interface, and display the logical connection path between the two in the knowledge graph.

[0111] Knowledge graph visualization interactive main interface as follows Figure 4 As shown; the timing diagram for cross-modal retrieval service processing is as follows. Figure 5 As shown; a schematic diagram of the retrieval system application is shown below. Figure 6 As shown.

[0112] like Figure 7 As shown, an embodiment of the present invention provides a passport cross-modal retrieval system based on knowledge graph constraints, specifically including:

[0113] The knowledge graph construction and management module is used to build and maintain a multimodal passport anti-counterfeiting knowledge graph. This module is responsible for receiving basic information and multispectral images of passport samples, establishing topological relationships between passport entities, anti-counterfeiting features, and their physical spatial attributes through ontology modeling, and transforming unstructured anti-counterfeiting technical specifications into structured graph nodes and edges, providing underlying knowledge base support for subsequent logical reasoning.

[0114] Semantic parsing module: This module processes users' natural language queries or analysis commands. It incorporates a rule-based semantic parser and a domain-specific anti-counterfeiting dictionary. It performs word segmentation and dependency analysis on the input text, separating keywords describing passport features and logical constraints such as year and country, and accurately mapping these keywords to unique entity node IDs in the knowledge graph.

[0115] Candidate Screening and Relaxation Module: This module performs efficient initial screening and adaptive recall of the image database. It conducts initial screening of the passport image database based on the parsed logical constraints. Simultaneously, it is equipped with an automatic relaxation unit. When the system detects that the initial screening results are empty, it automatically triggers a constraint relaxation strategy, removing weak constraints according to preset priorities before re-searching, ensuring the system's robustness under fuzzy queries.

[0116] Spatial Mask Generation Module: This module introduces a spatial inductive bias mechanism. It iterates through the candidate image set, retrieves the coordinates of the region of interest (ROI) corresponding to the query keyword in each image based on the knowledge graph, and generates a Gaussian-form soft-injection spatial attention mask based on these coordinates. This mask forces the model to focus on specific anti-spoofing physical regions that match the text description at the visual level.

[0117] Feature Fusion Network Module: This module performs feature extraction and deep fusion. It utilizes a pre-trained visual encoder to extract global layout features of the image, while applying the spatial attention mask to the feature layer to extract local features focused on anti-counterfeiting details. Subsequently, through a built-in residual combination network, residual parameters are dynamically learned to weight and fuse global and local features, generating a final visual vector that takes into account both the overall picture and the details.

[0118] The Reasoning, Decision-Making, and Visualization module generates assessment results and explanatory evidence. This module calculates the similarity score between the query text features and the final visual vector, generating an explanatory evidence chain that includes effective constraints, feature location coordinates, and matching scores. Finally, the results are sorted according to the scores, and the retrieved passport images are output to the user interface. Simultaneously, anti-counterfeiting feature location boxes are highlighted on the images, and the corresponding graph reasoning path is displayed to assist personnel in completing the assessment.

[0119] This invention can be implemented through a combination of hardware and software. The hardware implementation relies on a passport acquisition device, which uses a multispectral light source assembly to illuminate the passport placed on the device's glass lens, and a dedicated camera to acquire the passport image. The multispectral light source assembly includes infrared, ultraviolet, and visible light sources. Furthermore, the passport image acquisition device can respond to control commands from a host computer to adjust certain hardware modules. The core function of the host computer software is to manage the entire system's workflow and receive data transmitted from the passport acquisition device. System communication data includes text data such as passport image data and basic passport information, as well as a passport database. This database integrates algorithm modules such as a cross-modal passport retrieval method based on knowledge graph constraints. After processing some of the data using built-in algorithms, the host computer software presents the information visually to the user on the host computer interface.

[0120] Example 1: Identity verification for large cross-border transfers.

[0121] Specifically, in this embodiment, the steps of the retrieval process constructed in this embodiment include:

[0122] Step S10: Construction of a Multimodal Passport Anti-counterfeiting Knowledge Graph

[0123] In this step, to support the integration of unified logical reasoning with downstream deep learning models, this study formalizes the Passport Multimodal Knowledge Graph as a directed multigraph containing type constraints and attribute features. Its mathematical definition is as follows:

[0124]

[0125] in, It represents a collection of entity nodes, covering core elements such as version, features, and concepts. This represents a set of relation edges, defining semantic connections between entities (such as hasFeature). A set of types representing entities and relationships, used to constrain the ontology structure of the graph. It represents a collection of attribute values, including multimodal data such as numerical values, text descriptions, and normalized spatial coordinates.

[0126] For any triplet in the graph ,in Through mapping function Determine the semantic type of the entity and define the attribute mapping. To describe the specific characteristics of an entity.

[0127] Combining the business logic of passport anti-counterfeiting with the requirements of cross-modal retrieval, this paper constructs five types of core entity elements:

[0128] Document Version: This is the main node in the graph, used to uniquely identify the version of a passport issued by a specific country in a given year. A combined code (such as ISO3_TYYYY) is used as a globally unique identifier to ensure logical consistency across modules.

[0129] Security Feature: Used to describe a specific anti-counterfeiting process unit belonging to a particular version. Its attributes focus on the textual description of visual representation, aiming to provide fine-grained textual supervision signals for the CLIP visual encoder to address the semantic gap problem in cross-modal alignment.

[0130] Physical Region of Interest (ROI): Describes the spatial distribution of anti-counterfeiting features under the standard layout. A normalized bounding box is used. This indicates that the differences in image resolution have been eliminated, providing a geometric basis for the subsequent generation of differentiable spatial attention masks.

[0131] Semantic Concept: This is an abstract generalization of anti-counterfeiting features across versions and countries (e.g., "UV_Comparison" represents ultraviolet contrast characteristics). This entity is used to aggregate features with the same technological principles, supporting knowledge transfer from concrete instances to abstract concepts in the model.

[0132] Country metadata: Includes attributes such as country code and continent, used for scope filtering and macro-statistical analysis during the retrieval phase.

[0133] In terms of graph relationship modeling, the system defines the visual association between ROI and semantic concepts, and records the association strength in the relationship attributes to explain path ordering and evidence presentation.

[0134] In this embodiment, after the graph is constructed, a standardized data serialization process is established, which supports importing into graph databases and streaming loading for deep learning training.

[0135] Step S20: Semantic parsing and entity anchoring

[0136] This step will involve unstructured natural language queries. The system transforms these constraints into machine-executable structured constraints. Considering the highly specialized and closed nature of terminology in passport anti-counterfeiting, and the lack of large-scale annotated corpora, this embodiment employs a domain ontology-based symbol parsing strategy. Utilizing the controlled vocabulary maintained during knowledge graph construction, the system accurately extracts anti-counterfeiting entity keywords using a maximum forward matching algorithm and directly anchors them to node IDs or relation types in the knowledge graph. For higher-order query requirements involved in passport verification, the system integrates a logical reasoning module. For multi-condition queries containing Boolean logic (such as "after 2019 and with infrared features"), the system transforms them into path constraints at the graph level.

[0137]

[0138] in Represents the passport entity, Represents infrared feature nodes. Furthermore, for concept-driven retrieval where users cannot accurately describe terms, semantic expansion is performed using the HasConcept relationship in the graph. Define concepts. The corresponding specific feature set is as follows:

[0139]

[0140] During retrieval, the query criteria are automatically expanded from a single concept to a set. This allows for the use of any element in the dataset, thereby achieving semantic-level fuzzy matching and enhanced recall.

[0141] Step S30: Initial screening of candidate set and relaxation of hierarchical constraints

[0142] To address the efficiency bottleneck of large-scale graph database retrieval, this step introduces a pre-screening and post-ranking mechanism. This mechanism transforms the parsed metadata constraints into a graph database query statement (Cypher), pre-selecting candidate samples before vector computation. Let the set of parsed constraints be... The initial image library is Then the candidate set Defined as the intersection of all constraint query results:

[0143]

[0144] To enhance system robustness and prevent the candidate set from becoming empty due to overly strict user constraints, this embodiment introduces a hierarchical constraint relaxation mechanism. This mechanism divides constraints into strong constraints, such as anti-counterfeiting feature entities, and weak constraints, such as year and format. When a search fails (i.e., the candidate set is empty), the system automatically removes weak constraints according to priority, retaining only conditions strongly related to the essence of anti-counterfeiting for fallback retrieval, thereby achieving a dynamic balance between precision and recall.

[0145] Step S40: Coordinate Mapping and Spatial Attention Mask Generation

[0146] This step implements a knowledge-driven, differentiable spatial inductive bias mechanism. For each image in the candidate set, it first links to the atlas entity through metadata and retrieves the associated normalized ROI coordinate set.

[0147] Because the input image preprocessing stage employs a scaling and padding strategy that maintains aspect ratio, the original normalized coordinate system stored in the atlas has a spatial offset from the actual input model's image coordinate system. Therefore, it is necessary to construct a coordinate system from the original normalized coordinate space. To the model input coordinate space The mapping relationship. For any original normalized coordinates Its calibrated coordinates The following calculation was performed using an affine transformation:

[0148]

[0149] in, , Original image size; Scaling factor to maintain aspect ratio; The horizontal and vertical offsets generated for filling. This transformation ensures that the ROI region is geometrically and precisely aligned with the visual feature map. The aligned coordinate set is represented as:

[0150]

[0151] in, This step ensures strict consistency between prior knowledge and visual input in geometric space, representing the number of associated regions of interest.

[0152] After coordinate alignment, the discrete ROI boundary constraints need to be transformed into continuous spatial inductive biases, i.e., a visual attention mask. Considering that the feature map output by the CLIP visual encoder is in a high-level semantic space and has low spatial resolution, this study will use the calibrated... Mapping to the feature map scale and based on the physical properties of the anti-counterfeiting features, this embodiment designs two mask generation strategies to adapt to different types of features: For rigid features with clear boundaries, such as fixed watermarks and laser perforations, a hard attention mask is used. A binary indicator function is constructed through coordinate projection and spatial quantization; the value is 1 when a grid point on the feature map falls within the quantized ROI region, and 0 otherwise. For features with diffuse boundaries, such as fluorescent patterns, which are affected by optical scattering, a soft attention mask is used. The mask is generated around the center of the ROI. Construct a two-dimensional Gaussian distribution for the mean. To simulate the optical halo effect with strong center and weak edge, and to avoid the introduction of high-frequency noise by hard truncation.

[0153] To address different ROI hit scenarios, this embodiment designs an adaptive spatial gating strategy: when the query involves multiple anti-counterfeiting points, a union operation is used to generate a comprehensive mask. To cover all key areas; if the query does not find any specific anti-counterfeiting entity (i.e. The system then automatically reverts to a global search mode, allowing local features to be searched. Only using global features Matching is then performed. The generated mask and feature map are then input into the residual combination network described in step S50 to generate deep fusion features containing local discriminative information. And finally calculate its relationship with the query text vector. The cosine similarity is used to complete the Top-K sorting.

[0154] Step S50: Feature Extraction and Deep Residual Fusion

[0155] This step integrates global and local features into a unified retrieval decision capability through a residual combination network. First, it should be noted that the CLIP encoder described in this embodiment is a dedicated encoder with domain-adaptive fine-tuning. Because the passport page layout is relatively fixed and the aspect ratio is stable, the center-cropping strategy used in general retrieval models can lead to the loss of key anti-counterfeiting features at the passport edges, and non-uniform scaling can destroy the geometric structure of the anti-counterfeiting texture. To mitigate this impact, this step adopts a proportional scaling and padding strategy that maintains the aspect ratio: for the original passport image... First, while maintaining the original aspect ratio, scale it to the target size with the long side aligned. Then, zero pixels are added to both sides of the shorter side to complete the square input. The preprocessed image is then input into the CLIP visual encoder. Extract its normalized global visual feature vector :

[0156]

[0157] in, This is the feature dimension. This feature vector aggregates the global visual semantics of the passport page, providing a low-level feature base for subsequent spatial attention modulation.

[0158] On the text side, to enhance the text encoder's coverage of anti-counterfeiting semantics, this step utilizes the constructed knowledge graph to transcribe structured expert knowledge into natural language prompts. Graph attributes are written into the text sequence using prompt templates; for example, a graph path containing version and features is transcribed as "A passport sample of 2020 Version containing a UV Eagle feature". The text sequence is then processed. Input CLIP text encoder Extract its normalized text feature vector :

[0159]

[0160] This strategy injects expert priors into text representations in the form of learnable natural language, which helps alleviate the difficulties of cross-modal alignment caused by the scarcity of technical terms and semantic bias.

[0161] To narrow the distance between matched passport images and descriptive text in the feature space, while simultaneously distancing mismatched negative sample pairs, this study employs an in-batch symmetric contrastive loss function for end-to-end fine-tuning of the two-stream encoder. Let a training batch contain... Image-text pairs Its visual features and text features are respectively and Define the similarity function as follows: The learnable temperature coefficient is Image-to-text contrast loss for:

[0162]

[0163] Similarly, define the contrast loss for text-to-image conversion. for:

[0164]

[0165] Final optimization goal Take the average of the two:

[0166]

[0167] By minimizing the aforementioned loss function, CLIP's visual and text embedding spaces can adaptively align to the passport anti-counterfeiting semantic distribution.

[0168] Based on the finely tuned encoder described above, feature fusion is performed. The preprocessed image is then input into the CLIP visual encoder to extract the original feature maps. For the original feature map Perform global average pooling to obtain the global visual feature vector. This preserves the macroscopic layout semantics of the image. The spatial mask generated in step S40 is then used. The feature map is explicitly spatially weighted using element-wise Hadamard product:

[0169]

[0170] This operation effectively suppresses background noise at the end of feature extraction and significantly enhances the activation response of the ROI region. Global average pooling is used to obtain local feature vectors. .

[0171] To fuse multi-granularity features in a unified metric space, this embodiment designs a residual mapping structure based on MLP. Using global features... As the identity mapping benchmark, local features This is considered a correction term for the global semantics. First, concatenation is performed along the channel dimension, then the input is fed into a multilayer perceptron for nonlinear mapping, and layer normalization and ReLU activation are introduced. The final fused features... Represented as:

[0172]

[0173] This design enables the model to adaptively degenerate into a robust retrieval form based on global features even when local information is weak or the attention mask is insufficient, thus ensuring a lower bound on retrieval performance.

[0174] Step S60: Similarity Calculation and Evidence Generation. The final step completes the retrieval reasoning and outputs the results.

[0175] CLIP text encoder with frozen weights extracts query text features To eliminate the influence of feature scale, the system uses cosine similarity to measure the degree of image-text matching. Final retrieval score. Defined as the dot product of normalized eigenvectors:

[0176]

[0177] Based on ratings Candidate images are ranked Top-K. While outputting the search results, the system simultaneously displays the analytical basis: bounding boxes are drawn on the interface using the ROI coordinates from step S40, highlighting the common feature regions of the reference samples and the searched samples; logical paths are output, showing the logical association paths in the knowledge graph. In this way, this embodiment not only achieves high-precision cross-modal retrieval but also provides reliable and visualized decision-making basis for passport anti-counterfeiting verification.

[0178] The dataset used in this invention consists of passport collection devices and a publicly available passport database, covering 1,512 passport versions from 205 countries and regions worldwide. This invention selected 500 passport versions, totaling 4,500 standard images. Each version includes three sets of the highest-quality physical samples, collected under three standard light sources: visible light (VIS), ultraviolet fluorescence (UV), and infrared reflection (IR). To verify the semantic understanding capability of the cross-modal model, this invention specifically constructed a descriptive natural language text set. While maintaining the visual modality data unchanged, the text modality data was reconstructed based on the passport anti-counterfeiting knowledge graph, making it include richer anti-counterfeiting texture descriptions (such as the Great Wall pattern watermark) and layout semantic information (such as the machine-readable code located at the bottom). During the graph annotation process, 10% of the samples were randomly selected to calculate annotation consistency, achieving an average entity alignment IoU of 0.88.

[0179] Given the significant single-sample characteristics of the dataset, this invention designs a rigorous online data augmentation strategy to prevent the model from overfitting to the flaws of a single physical sample. During the training phase, the data loader dynamically introduces random perspective transformations, random cropping, rotation, and lighting jitter to expand the equivalent sample space. The experiment adopts an open set partitioning strategy: the 500 versions are divided into a training set (350 classes) and a test set (150 classes), where the 150 passport versions in the test set have never appeared in the training set.

[0180] This invention employs two types of evaluation metrics commonly used in cross-modal retrieval tasks:

[0181] (1) Recall@K (R@K): Measures the retrieval recall capability. For a given query, it checks whether the top K images in the sorted list of retrieval results contain at least one image that belongs to the correct target version.

[0182] (2) Average Retrieval Precision (mAP): Measures the overall ranking quality of retrieval results. Considering the multimodal nature of the search, VIS, UV, and IR images belonging to the same passport version are all defined as relevant samples. mAP is used to comprehensively reflect the model's ranking robustness under multimodal objectives.

[0183] Experimental results show that the proposed method achieves an R@1 of 91.5% and an mAP of 88.4% in complex natural language combination query tasks. Compared with the standard fine-tuned CLIP, the proposed method improves the mAP by 9.8 percentage points and can output the corresponding reasoning path and response region, providing interpretability for the retrieval results.

[0184] Example 1: Construction of a Passport Anti-counterfeiting Knowledge Graph Based on Multi-Source Heterogeneous Data

[0185] This embodiment describes the specific construction process of a passport anti-counterfeiting knowledge graph. First, 200 passport samples of different versions (e.g., 2012 and 2017 versions) are collected to obtain their basic information (issuing country, validity period) and multispectral images under ultraviolet, infrared, and white light. Technicians use the ontology editing tool Protégé to construct the passport anti-counterfeiting ontology, defining classes such as "passport version," "anti-counterfeiting features," and "physical location," and establishing object attributes such as "located in" and "has." Using the BERT-BiLSTM-CRF deep learning model, entities such as "optically variable ink" and "microtext" are extracted from unstructured text such as the ICAO Doc 9303 standard and mapped to nodes in the graph. Finally, a prompt template is designed: "Under [ultraviolet light], there is a [fluorescent fiber] feature on the [issue page] of the [2007 version passport], with a shape of [random lines]," generating 5000 structured visual description texts, which are then stored in the Neo4j graph database. This embodiment transforms implicit specifications into explicit knowledge, providing a knowledge foundation for subsequent retrieval.

[0186] Example 2: Fuzzy Semantic Expansion and Precise Parsing in Natural Language Queries

[0187] This embodiment demonstrates how to parse fuzzy queries input by users. The system loads a domain dictionary containing terms such as "fluorescence" and "microprinting." When a user inputs "find the fluorescent threads on the old version of the passport," the semantic parser first performs word segmentation, identifying "old version" (fuzzy time) and "fluorescent threads" (anti-counterfeiting entity). By querying the knowledge graph, it is found that the standard term for "fluorescent threads" is "fluorescent anti-counterfeiting fiber." Simultaneously, utilizing the hierarchical relationship in the graph where both "2012 version" and "2007 version" belong to "old version," the semantics of "old version" are expanded to "2012 version OR 2007 version." Finally, the graph path constraint is generated: (passport version IN [2012 version, 2007 version]) AND (includes the anti-counterfeiting feature "fluorescent anti-counterfeiting fiber"). This embodiment bridges the gap between colloquial queries and professional terminology, improving the retrieval recall rate.

[0188] Example 3: Automatic relaxation of hierarchical constraints for empty result sets

[0189] This embodiment illustrates the handling mechanism when no results are found in the initial screening. Assume a user queries "fluorescent fibers under ultraviolet light in a 2007 Chinese passport". The candidate screening module performs a precise query and finds no image in the database that simultaneously meets the four conditions: "2007 version", "Chinese passport", "ultraviolet light", and "fluorescent fibers", returning an empty set. The system immediately triggers a tiered relaxation mechanism, relaxing images according to a preset priority (lowest being year > format > lighting conditions > feature type). First, the weak constraint "year" is removed, changing the query to "fluorescent fibers under ultraviolet light in a Chinese passport". If the result is still empty, the "format" constraint is removed, changing the query to "fluorescent fibers under ultraviolet light". After relaxation, 150 candidate images are obtained, ensuring sufficient data input for subsequent processes. This embodiment, through dynamic constraint management, avoids the system directly returning failure due to overly strict input.

[0190] Example 4: Adaptive Spatial Attention Mask Generation Based on Physical Properties

[0191] This embodiment describes the specific calculation process for mask generation. For a passport image in the candidate set, the system links to the entity "2017 Chinese Passport" in the knowledge graph via the image ID. For the query keyword "optical variable ink," it is found to be located at the "document number," with normalized coordinates [x1, y1, x2, y2]=[0.2, 0.3, 0.4, 0.5]. Through affine transformation, these coordinates are mapped to the pixel region [45, 67, 90, 112] of the 224x224 pixel input image. Since "optical variable ink" has a clear circular boundary (rigid feature), the system generates a binary hard attention mask, setting the pixel weight within the region to 1 and the weight outside the region to 0. For another query, "fluorescent fiber" (diffuse feature), the graph returns a set of coordinates for a curved path. The system generates a Gaussian heatmap based on the coordinate point set as a soft attention mask, with the center point having the highest weight and gradually decreasing weight at the edges. When both are queried simultaneously, the gating unit performs a matrix union operation on the two masks to generate a composite mask.

[0192] Example 5: Residual Combination Network Achieves Deep Fusion of Global and Local Features

[0193] This embodiment details the feature fusion process. The original passport image is first scaled to 224x224 pixels and padded to a square. The visual encoder uses ResNet50, pre-trained on ImageNet and then fine-tuned using 5000 image-text pairs generated in Example 1 through domain-adaptive symmetric contrast loss. The network outputs a feature map of size 7x7x2048. The attention mask generated in Example 4 is downsampled to 7x7 and subjected to a Hadamard product with the feature map to obtain a local feature map. Global average pooling is then performed on both the original feature map and the local feature map to obtain a 2048-dimensional global feature vector G and a local feature vector L. The residual combination network uses G as the basis and learns the residual R through a fully connected layer, ultimately fusing the vector F = G + R, where R = ReLU(W * L + b). This design ensures that the fused vector retains the overall passport layout while enhancing the response to key anti-counterfeiting details.

[0194] Example 6: Visualization of the Evidence Chain Generation and Ranking in the Output Results

[0195] This embodiment describes the final output stage. Assume a user queries "2017 passport with fluorescent fibers". The system extracts query features using a finely tuned text encoder (such as BERT) and calculates the cosine similarity with the fused vectors of all candidate images. The image with a score of 0.92 is ranked first. The explanatory evidence chain generated by the system includes: "Graph Inference Path: Query -> 2017 Passport -> Contains feature 'fluorescent anti-counterfeiting fibers' -> Located at coordinates [0.1, 0.2, 0.3, 0.4] on the 'Data Page'". On the output image, a red, semi-transparent, highlighted bounding box is drawn in the data page area based on the coordinates stored in the graph. The user interface simultaneously displays the full passport image, a magnified view of the highlighted area, and the aforementioned textual evidence. This embodiment not only provides search results but also presents the entire process of "why it was a hit" and "where the feature is" to the user in an interpretable way.

[0196] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0197] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A passport cross-modal retrieval method based on knowledge graph constraints, characterized in that, include: Step S10: Obtain basic information and multispectral images of passport samples, establish topological relationships between passport entities, anti-counterfeiting features and their physical spatial attributes through ontology modeling, generate fine-grained visual description text for anti-counterfeiting features, and transform unstructured anti-counterfeiting specifications into structured graph nodes and edges, and store them in a graph database. Step S20: Receive the user's natural language query text, parse the text using a pre-set domain dictionary, identify anti-counterfeiting entities and parse Boolean logic constraints, and semantically expand the fuzzy query using the hierarchical relationship of the graph concept, and finally map the natural language into machine-executable graph path constraints. Step S30: Based on the parsed logical constraints, perform an initial screening in the passport image library to obtain a candidate image set; if the initial screening result is empty, automatically execute the hierarchical constraint relaxation strategy to remove weak constraints on year and format according to priority, so as to ensure that a sufficient number of candidate images are obtained for subsequent processing. Step S40: Traverse the candidate image set, retrieve the normalized coordinates corresponding to the query keywords on each image, map them to the feature map scale through coordinate affine transformation, and adaptively generate hard attention or soft attention masks according to the physical properties of the anti-counterfeiting features; for cases where multiple features are hit or no features are hit, execute an adaptive spatial gating strategy to generate a comprehensive mask or degenerate into global retrieval. Step S50: Construct a domain-adaptive fine-tuned visual encoder, input the candidate image into the visual encoder to extract global features, and simultaneously apply the spatial attention mask to the image feature layer to extract local features focused on the anti-counterfeiting area; then, use a residual combination network to deeply fuse the global features and local features to generate the final visual vector that takes into account both the overall layout and the anti-counterfeiting details. Step S60: Calculate the similarity score between the query text features and the final visual vector, and generate an explanatory evidence chain that includes the graph inference path, feature location coordinates and matching score; finally, sort the results according to the similarity score and output the retrieved passport image and the visualized anti-counterfeiting feature location box to the user.

2. The method according to claim 1, characterized in that, Step S10 specifically includes: Entity relationship analysis was performed on passport source data to sort out the logical relationships between passport version, anti-counterfeiting features, issuing country and physical location attributes, and to construct a passport anti-counterfeiting ontology model. Using a deep learning entity recognition model, anti-counterfeiting terms are extracted from unstructured text and mapped to entity nodes in a graph; By using prompt templates, graph node and path information is transformed into structured visual descriptive text, which serves as a knowledge supervision signal for cross-modal alignment fine-tuning.

3. The method according to claim 1, characterized in that, Step S20 specifically includes: Input the natural language query text into a rule-based semantic parser and load a dictionary for passport anti-counterfeiting. The text is segmented and subjected to dependency parsing analysis to identify descriptive terms belonging to anti-counterfeiting technology and mark them as keywords; Identify restrictive words in the text regarding the issuing country, year, and document type, and parse the Boolean logic relationships within them to generate graph path constraints; Semantic expansion is performed by utilizing the concept attribution relationships in the graph, mapping fuzzy concepts to a set of specific anti-counterfeiting feature entities; Query the pre-built knowledge graph index and map the keywords to unique entity node IDs in the graph. If synonyms exist, they are uniformly mapped to standard term nodes.

4. The method according to claim 1, characterized in that, Step S30 specifically includes: Using the parsed initial logical constraints, an exact match query is performed in the metadata fields of the passport image database to obtain an initial candidate image set; Real-time detection of whether the initial candidate image set is empty; if it is empty, trigger the hierarchical constraint relaxation mechanism, and remove weak constraints such as year and layout that are not critical according to the preset priority list. The query is re-executed in the passport image database based on the relaxed constraints until a non-empty set of valid candidate images is obtained.

5. A passport cross-modal retrieval system based on knowledge graph constraints, used to execute the method according to any one of claims 1 to 4, characterized in that, include: The map construction and management module is used to perform the function of step S10; The semantic parsing module is used to perform the functions of step S20; The candidate screening and relaxation module is used to perform the function of step S30; The spatial mask generation module is used to perform the function of step S40; The feature fusion network module is used to perform the function of step S50; The reasoning, decision-making, and visualization module is used to perform the functions of step S60.

6. The system according to claim 5, characterized in that, The spatial mask generation module is specifically used for: Link the current image in the candidate image set to the corresponding passport version entity in the knowledge graph; Retrieve the region of interest data associated with the query keywords under the entity, obtain the normalized spatial coordinates, and calibrate them to the model input coordinate system through affine transformation; The mask generation is adaptively selected based on the physical properties of the anti-counterfeiting features: a hard attention mask is generated for features with clear boundaries, and a soft attention mask with a Gaussian distribution is generated for fluorescent features. Perform adaptive spatial gating: If multiple anti-counterfeiting points are hit, perform a union operation to generate a comprehensive mask; if no coordinates are hit, automatically degenerate into a fully weighted global mask.

7. The system according to claim 5, characterized in that, The feature fusion network module is specifically used for: The original image is scaled proportionally and preprocessed with zero-pixel padding, and then input into a domain-adaptive fine-tuned visual encoder. Before inference, the encoder has already performed alignment fine-tuning for passport anti-counterfeiting textures based on the visual description text provided in step S10 as positive sample pairs and through a symmetric contrast loss function. The original feature map output by the fine-tuned feature extraction network is combined with the spatial attention mask to perform a Hadamard product operation, and the output is a local feature map focused on the anti-counterfeiting feature region. Global average pooling is performed on the original feature map and the local feature map respectively to obtain the global feature vector and the local feature vector; The global feature vector and local feature vector are input into the residual combination network. The global feature vector is used as the identity mapping benchmark, and the local feature vector is used as the residual correction term for deep fusion. The final fused visual vector is output.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 4.

9. The storage medium according to claim 8, characterized in that, When the program is executed by the processor, in step S40, the adaptive generation of hard attention or soft attention masks based on the physical properties of the anti-counterfeiting features specifically includes: generating a hard attention mask for features with clear boundaries, and generating a Gaussian-distributed soft attention mask for fluorescence features.

10. The storage medium according to claim 8, characterized in that, When the program is executed by the processor, in step S50, the deep fusion of global features and local features through the residual combination network specifically includes: inputting the global feature vector and the local feature vector into the residual combination network, using the global features as the identity mapping benchmark, and using the local features as the residual correction term for deep fusion.

Images