A Road Defect Retrieval and Diagnosis Method Based on Multimodal Knowledge Graph

Abstract

This invention discloses a road defect retrieval and diagnosis method based on a multimodal knowledge graph, comprising the following steps: S1: constructing a multimodal knowledge graph of road defects; S2: extracting features based on visual-semantic joint embedding; S3: storing the mapped visual vectors as visual attributes of entities in the graph structure, and performing multimodal fusion alignment; S4: performing cross-modal retrieval, multi-hop reasoning, and logical error correction. The beneficial effects of this invention are: by constructing a multimodal knowledge graph of road defects with visual feature nodes, the alignment of the visual feature space and the semantic feature space is achieved. This allows for the identification of defects while simultaneously using the knowledge graph for reasoning, outputting the causes of defects and maintenance measures, and correcting false detections in visual recognition, thus achieving accurate cross-modal retrieval and intelligent diagnosis.

Description

Technical Field

[0001] This invention relates to the field of road maintenance technology, and in particular to a method for retrieval and diagnosis of road defects based on a multimodal knowledge graph. Background Technology

[0002] With the development of intelligent technology, automated detection of road surface defects (such as cracks, potholes, ruts, etc.) has become a key aspect of road maintenance. Figure 1 As shown, existing technologies mainly rely on computer vision techniques in deep learning, such as using convolutional neural networks (CNNs) or visual transformers (ViTs) to perform object detection or semantic segmentation on road images. The process typically involves an onboard camera acquiring road images → inputting them into a recognition model → outputting damage category labels and location boxes. However, existing models can only identify pixel features in images and cannot understand the underlying causes of damage (such as water damage or thermal shrinkage) or related maintenance measures, thus failing to directly guide engineering practices. Furthermore, under complex lighting conditions, shadows, road surface oil stains, or repair marks, pure visual models are prone to misclassifying non-damage features as damage, lacking the logical constraints of prior knowledge (such as "cracks are usually continuous" or "specific damages are more common in specific road conditions"). Simultaneously, road maintenance data includes unstructured image data and structured text data, such as maintenance specifications and historical repair records. Existing technologies cannot achieve cross-modal retrieval such as "image-to-text search" (uploading photos to match repair plans) or "text-to-image search" (searching for "water damage" to find relevant feature images), resulting in low utilization of historical data. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a road defect retrieval and diagnosis method based on multimodal knowledge graph.

[0004] The objective of this invention is achieved through the following technical solution: a road defect retrieval and diagnosis method based on a multimodal knowledge graph, comprising the following steps:

[0005] S1: Construct a multimodal knowledge graph of road defects;

[0006] S2: Feature extraction based on visual-semantic joint embedding;

[0007] S3: Store the mapped visual vectors as visual attributes of entities in the graph structure for multimodal fusion and alignment;

[0008] S4: Perform cross-modal retrieval, multi-hop reasoning, and logical error correction.

[0009] Preferably, step S2 further includes the following step:

[0010] S21: Extract visual and textual features;

[0011] S22: Visual vectors Mapping to text vectors Same semantic space middle,

[0012] ;

[0013] in, and For learnable weight matrix, It is the ReLU activation function. For the mapped visual features, This is the learnable bias vector of the first layer of the network. This is the learnable bias vector for the second layer of the network;

[0014] S23: Use the InfoNCE loss function to bring matching image-text pairs closer together and push away mismatched image-text pairs. (Loss function...) for:

[0015] ;

[0016] in, For cosine similarity, This is the temperature coefficient.

[0017] Preferably, in step S21, a pre-trained visual Transformer model is used as the visual encoder to extract visual features. ,

[0018] ;

[0019] in, For visual feature dimensions, The input is a road surface image. This is the extracted high-dimensional visual feature vector;

[0020] Text features were extracted using the BERT model as the text encoder. ,

[0021] ;

[0022] in, A textual description of the disease. It is a text vector.

[0023] Preferably, in step S3, the TransE algorithm is used to pre-train the graph constructed in step S1, combining text entities and relations. Mapping to the same low-dimensional vector space, such that it satisfies ,in, For the head entity, It is a tail entity;

[0024] For each type of standard disease entity, select typical image samples and pass them through the projection network in step S2 to obtain alignment vectors. Instantiate a node of type visual prototype in the graph, and... Store its attribute value, and according to the relation edge has_visual_representation defined in step S1, attach the visual prototype node to the corresponding disease entity.

[0025] Preferably, in step S4, cross-modal retrieval specifically involves the system receiving the road surface image to be diagnosed. The query vector is extracted and mapped using the visual encoder and projection network trained in step S2. ,calculate By comparing the cosine similarity with the vectors of all mounted visual prototype nodes, the most similar Top-1 prototype node is retrieved, and the disease entity node connected to it is located.

[0026] Preferably, in step S4, multi-hop reasoning specifically involves inferring causes and recommending measures starting from the disease entity node. Cause inference involves traversing along the induced_by relationship edge to find associated Cause nodes; measure recommendation involves traversing along the treated_with relationship edge to find associated Measure nodes. The retrieved disease type is then integrated with the inferred implicit information to output a natural language diagnostic result.

[0027] Preferably, in step S4, the specific steps for logical error correction are as follows:

[0028] S41: The input visual model predicts the category as follows: probability and environmental entity feature vector Among them, the environmental entity feature vector It is a vector representation of environmental entities in a knowledge graph after model mapping, and the parameters obtained include at least one of road surface type, years of service, and climate conditions;

[0029] S42: Retrieve categories in the map With the environment Relationship path, define logical consistency function ,

[0030] ;

[0031] S43: Calculate the final confidence level ,

[0032] ;

[0033] in, The Sigmoid normalization function, and For learnable weight coefficients, if the corrected If the value is below the set threshold, it is considered a false detection.

[0034] Preferably, in step S41, the environmental entity feature vector The vector representation is derived from the environmental entity instance in step S1 and then mapped by the graph embedding model TransE in step S3.

[0035] The present invention has the following advantages: By constructing a multimodal knowledge graph of road defects with visual feature nodes, the present invention achieves alignment between the visual feature space and the semantic feature space. Thus, while identifying defects, the knowledge graph can be used for reasoning to output the causes of defects and maintenance measures, and correct false detections in visual recognition, thereby achieving accurate cross-modal retrieval and intelligent diagnosis. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the existing technical process;

[0037] Figure 2 This is a schematic diagram of the process of a road defect retrieval and diagnosis method based on multimodal knowledge graph. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0039] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0040] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other.

[0041] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0042] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are only used for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0043] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0044] In this embodiment, as Figure 2 As shown, a road defect retrieval and diagnosis method based on multimodal knowledge graph includes the following steps:

[0045] S1: Construct a multimodal knowledge graph of road defects; specifically, the data includes structured and unstructured data. The structured data is sourced from current standards and specifications such as the "Highway Technical Condition Assessment Standard (JTG 5210-2018)" and the "Highway Maintenance Technical Standard (JTG 5110-2023)," extracting defect classification and grading standards, deduction weights, and maintenance countermeasure tables. The unstructured data is sourced from historical road inspection reports, defect treatment plans, maintenance and repair design plans, and expert experience databases. The following core classes and their attributes are defined using the OWL language, where the entities include: (1) Defect: Attributes include name (e.g., "longitudinal crack"), severity, and unit of measurement; (2) Visual_Prototype: A cross-modal entity used to store the high-dimensional feature vector of the defect image after encoding, and its attribute is vector_value (vector value, 768-dimensional array); (3) Cause: Attributes include type (water damage, load type, temperature shrinkage type); (4) Measure: Attributes include process name (crack sealing, milling and repaving), applicable conditions, and cost estimate; (5) Environment: Attributes include pavement type (asphalt / cement), years of service, and climate conditions. The relationships include: (1) has_visual_representation: connects Defect and Visual_Prototype to achieve an explicit association between textual concepts and visual features; (2) induced_by: connects Defect and its cause; (3) treated_with: connects Defect and measures; (4) constrain_by: connects Defect and environment; (5) frequently_occurs_in: connects Defect and environment to represent statistically high-frequency concurrent relationships (e.g., "ruts" frequently occur on "high-temperature road sections"), providing positive weight support in inference; (6) rarely_occurs_in: connects Defect and environment to represent mutually exclusive / low-frequency relationships in expert experience or standard specifications (e.g., "potholes" rarely occur on "newly paved roads"), providing negative weight penalty in inference; (7) cannot_occur_in (Impossible / mutually exclusive): Connects the defect and the environment, used to represent an absolute mutual exclusion relationship defined by standards and specifications (e.g., "asphalt pavement" cannot have "cement slab breakage"), and serves as a hard filtering condition in reasoning.

[0046] S2: Feature extraction based on visual-semantic joint embedding;

[0047] S3: Store the mapped visual vectors as visual attributes of entities in the graph structure for multimodal fusion and alignment;

[0048] S4: Perform cross-modal retrieval, multi-hop reasoning, and logical error correction. By constructing a multimodal knowledge graph of road defects using visual feature nodes, the visual feature space and semantic feature space are aligned. This allows for the identification of defects while simultaneously using the knowledge graph for reasoning, outputting the causes of defects and maintenance measures, and correcting false detections in visual recognition, thus achieving accurate cross-modal retrieval and intelligent diagnosis.

[0049] Furthermore, step S2 also includes the following steps:

[0050] S21: Extract visual and textual features; specifically, in step S21, a pre-trained visual Transformer model is used as the visual encoder to extract visual features. ,

[0051] ;

[0052] in, For visual feature dimensions, The input is a road surface image. This is the extracted high-dimensional visual feature vector;

[0053] Text features were extracted using the BERT model as the text encoder. ,

[0054] ;

[0055] in, A textual description of the disease. It is a text vector.

[0056] S22: To eliminate the differences in heterogeneous feature spaces, a visual projection head composed of multilayer perceptrons is constructed, that is, visual vectors... Mapping to text vectors Same semantic space middle,

[0057] ;

[0058] in, and For learnable weight matrix, It is the ReLU activation function. For the mapped visual features, This is the learnable bias vector of the first layer of the network. This is the learnable bias vector for the second layer of the network;

[0059] S23: Use the InfoNCE loss function to bring matching image-text pairs closer together and push away mismatched image-text pairs. (Loss function...) for:

[0060] ;

[0061] in, For cosine similarity, The temperature coefficient is used. Specifically, a dual-tower structure is used to extract visual and textual features separately, and a projection network is trained using a contrastive loss function to map the visual feature vectors to a unified semantic space of the knowledge graph. This allows maintenance personnel to retrieve corresponding feature images by using text keywords (such as "severe water damage"), or to retrieve similar historical cases and documents by uploading images, greatly promoting the reuse of maintenance experience and the accumulation of knowledge.

[0062] Furthermore, in step S3, the TransE algorithm is used to pre-train the graph constructed in step S1, combining text entities and relations. Mapping to the same low-dimensional vector space, such that it satisfies ,in, For the head entity, It is a tail entity;

[0063] For each type of standard disease entity, select typical image samples and pass them through the projection network in step S2 to obtain alignment vectors. Instantiate a node of type visual prototype in the graph, and... The attribute value is stored, and based on the relation edge `has_visual_representation` defined in step S1, the visual prototype node is attached to the corresponding disease entity. At this point, the text entity vector in the graph... With mounted visual attribute vector They are located in the same measure space, so the distance can be calculated directly.

[0064] In this embodiment, in step S4, cross-modal retrieval specifically involves the system receiving the road surface image to be diagnosed. The query vector is extracted and mapped using the visual encoder and projection network trained in step S2. ,calculate The most similar Top-1 prototype node is retrieved based on the cosine similarity with all mounted visual prototype node vectors, and its connected disease entity node is located. Specifically, cross-modal reasoning not only identifies diseases but also infers "possibly caused by subgrade settlement" based on the graph logic chain and suggests "crack sealing treatment," providing a direct disease diagnosis result. Further, in step S4, multi-hop reasoning specifically involves inferring causes and recommending measures starting from the disease entity node. Cause inference involves traversing along the induced_by relationship edge to find associated Cause nodes; measure recommendation involves traversing along the treated_with relationship edge to find associated Measure nodes. The retrieved disease type is then integrated with the inferred implicit information to output a natural language diagnosis result, such as: longitudinal cracks were detected, inferred to be caused by subgrade settlement, and crack sealing repair technology is recommended. Further, by utilizing prior engineering knowledge to correct false detections in the visual model, the specific steps of logical error correction are as follows:

[0065] S41: The input visual model predicts the category as follows: probability and environmental entity feature vector Specifically, environmental entity feature vectors The vector representation is obtained by mapping the graph embedding model TransE in step S3 directly from the environmental entity instance (e.g., asphalt pavement, 3 months after opening to traffic) directly from step S1.

[0066] S42: Retrieve categories in the map With the environment Relationship path, define logical consistency function ,

[0067] ;

[0068] S43: Calculate the final confidence level ,

[0069] ;

[0070] in, The Sigmoid normalization function, and These are learnable weight coefficients, with initial values ​​of 1 and 0.5 respectively. If the adjusted weights are... If the value is below the set threshold, it is considered a false detection.

[0071] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for retrieving and diagnosing road defects based on a multimodal knowledge graph, characterized in that: Includes the following steps: S1: Construct a multimodal knowledge graph of road defects; S2: Feature extraction based on visual-semantic joint embedding; S3: Store the mapped visual vectors as visual attributes of entities in the graph structure for multimodal fusion and alignment; S4: Perform cross-modal retrieval, multi-hop reasoning, and logical error correction; In step S1, the data includes structured data and unstructured data. The structured data is derived from current standards and specifications, including the extraction of disease classification and grading standards, deduction weights, and maintenance countermeasure tables. The unstructured data is derived from historical road inspection reports, disease treatment plans, maintenance and repair design plans, and expert experience databases. The following core classes and their attributes are defined using the OWL language. The entities include: (1) Disease: The attributes include name, severity, and unit of measurement; (2) Visual prototype: A cross-modal entity used to store the high-dimensional feature vector of the disease image after encoding. Its attributes are vector values ​​and 768-dimensional arrays; (3) Cause of disease: The attributes include type; (4) Measures: The attributes include process name, applicable conditions, and cost estimate; (5) Environment: The attributes include road surface type, years of operation, and climate conditions. The relationships include: (1) has_visual_representation: connects the disease with the visual prototype, realizing the explicit association between textual concepts and visual features; (2) induced_by: connects the disease with the cause; (3) treated_with: connects the disease with the measures; (4) constraint_by: connects the disease with the environment; (5) frequently_occurs_in: connects the disease with the environment, used to represent the high-frequency concurrent relationship in statistics, providing positive weight support in inference; (6) rarely_occurs_in: connects the disease with the environment, used to represent the mutually exclusive / low-frequency relationship in expert experience or standard specifications, providing negative weight penalty in inference; (7) cannot_occur_in: connects the disease with the environment, used to represent the absolute mutually exclusive relationship defined by standard specifications, serving as a hard filtering condition in inference. In step S4, cross-modal retrieval specifically involves the system receiving the road surface image to be diagnosed. The query vector is extracted and mapped using the visual encoder and projection network trained in step S2. ,calculate The most similar Top-1 prototype node is retrieved by cosine similarity with all mounted visual prototype node vectors, and the disease entity node connected to it is located. In step S4, multi-hop reasoning specifically involves inferring causes and recommending measures starting from the disease entity node. Cause inference involves traversing along the induced_by relationship edge to find associated Cause nodes; measure recommendation involves traversing along the treated_with relationship edge to find associated Measure nodes. The retrieved disease type is then integrated with the inferred implicit information to output a natural language diagnostic result.

2. The road defect retrieval and diagnosis method based on multimodal knowledge graph as described in claim 1, characterized in that: Step S2 further includes the following steps: S21: Extract visual and textual features; S22: Visual vectors Mapping to text vectors Same semantic space middle, ； in, and For learnable weight matrix, It is the ReLU activation function. For the mapped visual features, This is the learnable bias vector of the first layer of the network. This is the learnable bias vector for the second layer of the network; S23: Use the InfoNCE loss function to bring matching image-text pairs closer together and push away mismatched image-text pairs. (Loss function...) for: ； in, For cosine similarity, This is the temperature coefficient.

3. The road defect retrieval and diagnosis method based on multimodal knowledge graph according to claim 2, characterized in that: In step S21, the visual features are extracted using a pre-trained visual Transformer model as the visual encoder. , ； in, For visual feature dimensions, The input is a road surface image. This is the extracted high-dimensional visual feature vector; Text features were extracted using the BERT model as the text encoder. , ； in, A textual description of the disease. It is a text vector.

4. The road defect retrieval and diagnosis method based on multimodal knowledge graph according to claim 3, characterized in that: In step S3, the TransE algorithm is used to pre-train the graph constructed in step S1, combining text entities and relations. Mapping to the same low-dimensional vector space, such that it satisfies ,in, For the head entity, It is a tail entity; For each type of standard disease entity, select typical image samples and pass them through the projection network in step S2 to obtain alignment vectors. Instantiate a node of type visual prototype in the graph, and... Store its attribute value, and according to the relation edge has_visual_representation defined in step S1, attach the visual prototype node to the corresponding disease entity.

5. The road defect retrieval and diagnosis method based on multimodal knowledge graph according to claim 4, characterized in that: In step S4, the specific steps of logical error correction are as follows: S41: The input visual model predicts the category as follows: probability and environmental entity feature vector Among them, the environmental entity feature vector It is a vector representation of environmental entities in a knowledge graph after model mapping, and the parameters obtained include at least one of road surface type, years of service, and climate conditions; S42: Retrieve categories in the map With the environment Relationship path, define logical consistency function , ； S43: Calculate the final confidence level , ； in, The Sigmoid normalization function, and For learnable weight coefficients, if the corrected If the value is below the set threshold, it is considered a false detection.

Images