Road infrastructure abnormity identification method based on vision and AI intelligent agent

Abstract

The invention discloses a road infrastructure anomaly recognition method based on vision and an AI agent. The method comprises the steps that a multi-agent cooperation framework with a task coordination agent as a core is constructed; a data input agent receives and preprocesses multiple paths of video streams; the visual processing agent fuses a visual model and a language model through a cross attention mechanism to realize deep fusion of image-text features; the knowledge base intelligent agent constructs a dynamic JSON Schema structured template based on a Neo4j graph database; and the data output agent calls the large language model to carry out data enhancement and structured processing, and the data is stored in a database after validity is verified by the knowledge graph. According to the method, through multi-agent cooperation, cross-modal feature fusion and a dynamic template technology, the problems of feature splitting, template stiffness and low calculation efficiency in a traditional scheme are solved, and the efficiency of road anomaly recognition is improved.

Description

Technical Field

[0001] The present invention belongs to the intersection of computer vision and artificial intelligence, and specifically relates to a method for identifying road infrastructure anomalies based on vision and AI agents. Background Art

[0002] In the field of image scene recognition, traditional methods face multiple challenges as scenarios such as smart transportation and industrial quality inspection become increasingly complex. Existing technologies primarily rely on single-modality convolutional neural networks (CNNs) or visual Transformer models to extract image features and output unstructured data (such as probability distributions or bounding box coordinates). These models require additional conversion layers to adapt to downstream systems, resulting in low processing efficiency. Especially in scenarios with high real-time requirements, such as highway monitoring and intelligent security, existing methods face drawbacks in identifying infrastructure anomalies, including limitations on unstructured output, fragmentation of multimodal data, rigid static templates, real-time bottlenecks, and a lack of knowledge verification.

[0003] In recent years, the development of AI agent technology and multimodal large models has provided new ideas for solving the above problems. By building a cluster of collaborative agents, the whole process of task scheduling, feature fusion and knowledge verification can be optimized. For example, the roadside perception method based on the visual language multimodal model (CN202410440356.0) mainly includes: detecting and tracking traffic targets, using a bird's-eye view to convert and solve multi-dimensional information, inputting data into a large language model to generate semantics for enhancing the bird's-eye view, and inputting a large language model for dialogue interaction after prompt engineering design. This invention combines the basic data obtained by the computer vision subtask with the powerful understanding and reasoning capabilities of the visual language large model to achieve all-round perception and understanding of the intersection and support real-time dialogue interaction. However, existing AI agent solutions mostly focus on a single task (such as target detection), and have not yet formed a complete technology chain covering data input, processing, structured output and storage, and have not yet been applied in dynamic resource allocation, deep fusion of knowledge graphs, etc. Summary of the Invention

[0004] In response to the defects of existing technologies such as low efficiency of unstructured output, fragmentation of multimodal features, rigid templates with poor adaptability, insufficient real-time processing capabilities and lack of knowledge verification, the present invention proposes a road infrastructure anomaly identification method based on vision and AI intelligent agents. The purpose is to realize the full process intelligence from image input to structured output through multimodal collaboration and dynamic knowledge drive, and improve the accuracy and efficiency of infrastructure anomaly identification.

[0005] The present invention is achieved through at least one of the following technical solutions.

[0006] A method for identifying road infrastructure anomalies based on vision and AI agents, comprising the following steps:

[0007] Convert multiple video stream data into image data and perform normalization on the image data;

[0008] The normalized image is fed into a visual model to extract global semantic features. The visual model extracts features specific to road infrastructure anomalies and generates visual features related to the anomalies. The global semantic features are then fed into a language model to generate scene text features. This is combined with domain knowledge to generate semantic descriptions related to road infrastructure anomalies. The cross-attention mechanism fuses the image and text features to further enhance understanding of the anomaly scene.

[0009] The fused image and text features are input into the scene recognition model to obtain structured output results, and the structured output results are imported into the database.

[0010] Furthermore, the visual model extracts global semantic features of the image including:

[0011] (1) The input image I is divided into N blocks, and each block is mapped to a d-dimensional vector by linear projection:

[0012]

[0013] Among them, Flatten(I i ) operation is to transform the multi-dimensional input into one dimension; H is the image height; W is the image width; p i is the output feature vector; W p is the weight matrix; I i is the i-th image of the input; b p is the bias term;

[0014] (2) Add a learnable position encoding matrix to each block The output embedding vector p is:

[0015] P=[p1,p2,…,p N ]+E p

[0016] Among them, p N is the feature vector of the Nth output;

[0017] (3) The embedding vector P passes through the Transformer encoder to output the image feature vector

[0018] Furthermore, the language model generates scene text features including: transforming the image feature vector f img With the prompt text T promptInput the language model and take the last layer of hidden state mean as the text feature f text :

[0019]

[0020] Where L is the sequence length; h i is the feature vector of the i-th position, and the text feature f text The fusion of image and text features is achieved through the cross-attention mechanism.

[0021] Furthermore, the cross-attention mechanism introduces a learnable gating mechanism to balance the modes and further optimize the cross-attention fusion mechanism:

[0022] g=σ(W g [f img ,f text ])

[0023] f′=g·f img +(1-g)·f

[0024] in, σ is the sigmoid function, g is the gate weight, used to control f img With f text The fusion ratio, f text is the text feature vector; f img is the image feature vector, W g is a learnable weight matrix, for the feature [f img ,f text ] performs linear transformation; f ′ is the final fusion feature.

[0025] Furthermore, the scene recognition model includes a domain ontology model and a language model. The domain ontology model is used to generate a JSON Schema structured template containing entity relationship constraints based on inputting graphic and text features into the domain ontology model; the language model performs structured data enhancement based on the JSON Schema structured template.

[0026] Furthermore, entity relationships contain entity types and relationship types:

[0027] Entity type attributes include whether damage occurred (occurred) and the corresponding Boolean value, damage type and the corresponding integer value;

[0028] The constraint validation mechanism for JSON Schema structured templates includes:

[0029] (1) Static validation: Check data types and enumeration ranges based on the JSON Schema standard;

[0030] (2) Dynamic verification: verifying knowledge graph relationship constraints through Cypher queries;

[0031] (3) Composite check: Apply Datalog rules to field combinations.

[0032] Furthermore, the language model is trained individually using deepseek-r1:70b.

[0033] Furthermore, the scene recognition model constructs a task coordination agent and uses a bipartite graph matching algorithm to dynamically allocate computing resources to achieve a reasonable allocation of resources for each agent. Its optimization objective function is as follows:

[0034]

[0035] Where M and N are the total number of tasks to be assigned and the number of available computing agents, respectively; ω1 and ω2 are weight coefficients; x ij ∈{0,1} indicates whether task i is assigned to the jth agent j ; Agent j The maximum parallel task capacity of T is determined by its memory and number of threads; ij For task i in Agent j The estimated execution time s on T ij For task i in Agent j The estimated execution time s on is calculated as follows:

[0036]

[0037] Where FLOPs represents the computational effort of task i, and FLOPs / s represents the computational effort of the jth agent. j Real-time computing power;

[0038] E ij Assign task i to Agent j The communication energy consumption is calculated as follows:

[0039] E ij = Data volume × (α·Transmission distance + β·Number of hops)

[0040] Where α is the energy consumption coefficient of transmission distance; β is the energy consumption coefficient of hop number.

[0041] A system for implementing the method for identifying abnormalities in road infrastructure based on vision and AI agents includes:

[0042] Data input agent, used for preprocessing multi-channel video stream data;

[0043] Visual processing agent, used to extract image features and fuse image and text features;

[0044] The knowledge base agent calls the domain ontology model in the Neo4j graph database to generate a JSON Schema structured template containing entity relationship constraints;

[0045] The data output agent uses a large language model to output recognition results based on a JSON Schema structured template.

[0046] A computer device of the present invention includes: a memory and a processor and a computer program stored in the memory. When the computer program is executed on the processor, the method for identifying anomalies of road infrastructure based on vision and AI agents is implemented.

[0047] Compared with the existing technology, the beneficial effects of the present invention are:

[0048] Based on AutoGen, this paper constructs an intelligent framework from video image data input to structured output and database entry, improving resource utilization through dynamic resource scheduling. Advanced visual and language models are integrated into the AI ​​agent to improve recognition accuracy. A JSON Schema template containing entity constraint relationships is constructed and used to perform personalized training and enhancement on the output agent. Multi-channel video image input is converted into structured data output, and the data is automatically entered into the database. The entire process is intelligently completed by the AI ​​agent, reducing the cost of manual intervention and having practical promotional value. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0050] Figure 1 Schematic diagram of the processing flow of an embodiment of the present invention.

[0051] Figure 2 Schematic diagram of image preprocessing according to an embodiment of the present invention.

[0052] Figure 3 A framework diagram of a method for identifying road infrastructure anomalies based on vision and AI agents in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limiting the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art. The present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0054] This embodiment of the present invention is a road infrastructure anomaly recognition method system based on vision and AI agents, with the task coordination agent as the center, to build an AutoGen-based image scene recognition framework, including

[0055] The data input agent is used to pre-process the data of multiple video streams.

[0056] Visual processing agent, used to extract image features and fuse image and text features.

[0057] The knowledge base agent calls the domain ontology model in the Neo4j graph database to generate a JSON Schema structured template containing entity relationship constraints.

[0058] The data output agent uses a large language model to output recognition results based on a JSON Schema structured template.

[0059] The language large model adopts the deepseek-r1:70b large model, and personalized training is performed on the deepseek-r1:70b large model.

[0060] like Figure 1 、 Figure 3 As shown, a method for identifying road infrastructure anomalies based on vision and AI agents in this embodiment includes the following steps:

[0061] S1. Preprocess the multi-channel video stream data. Convert the video stream data into image data, such as Figure 2 The image data is normalized as shown, including:

[0062] (1) Define the supported multi-channel video streaming protocol list self.supported_protocols = ["Real Time Streaming Protocol (RTSP)", "Real Time Messaging Protocol (RTMP)", "HTTP Real Time Streaming Protocol (HLS)" and target size self.target_size = (640, 640).

[0063] (2) The video capture function based on the open source computer vision library (OpenCV) receives multiple video stream data and divides the video stream data into image data according to a certain number of frames.

[0064] (3) The received image data is preprocessed according to the preset target size self.target_size, and the quality of the processed image is improved using the bilinear interpolation method. The formula is as follows:

[0065]

[0066] Among them, I is the pixel value of the image, I r (x ′ ,y ′ ) is the image at (x ′ ,y ′ ) is the interpolated pixel value; m is the displacement in the x direction; n is the displacement in the y direction; W o is the original image width; H o is the height of the original image; W t H is the width of the image target size; t The height of the target image size.

[0067] The image data is then normalized to make the input image data standard uniform, and the RGB channels of each pixel are processed independently. The formula is as follows:

[0068]

[0069] Among them, C is a different channel; are the original value and normalized value of channel C respectively; μ c is the average value of pixels in each color channel, which is used to eliminate illumination differences. It is usually taken as the statistical value μ of the ImageNet dataset. c =[0.485,0.456,0.406]; σ c is the standard deviation of each channel pixel, which is used to scale the data distribution. It is usually taken as the statistical value of the ImageNet dataset σ c =[0.229,0.224,0.225].

[0070] (4) The pre-processed image data is placed in the preset storage address .. / xxx / img_file / x.jpg as input data for step S2.

[0071] As an example, this example selects three pictures as examples, and the picture attributes are as follows:

[0072] Table 1 Example Picture Attributes

[0073]

[0074] The image is preprocessed through steps (1), (2), and (3). The properties of the preprocessed image are as follows:

[0075] Table 2 Image attributes after preprocessing

[0076]

[0077] S2. Use a visual model, such as the CLIP-ViT-L / 14 model, to extract global semantic features from the image converted in step S1. The visual model extracts features of specific anomalies in road infrastructure (such as damaged roads, obstacles, missing or damaged signs, and malfunctioning traffic lights), generates anomaly-related visual features, and combines them with a language model, such as the LLaMA-3.3B language model, to generate scene description text. The cross-attention mechanism is used to achieve image-text feature fusion, including:

[0078] (1) Use the visual model CLIP-ViT-L / 14 to extract features from the input image data:

[0079] The input image Divided into 14×14 blocks, each of which is mapped to a d=768-dimensional vector by linear projection:

[0080] p i =W p Flatte

[0081] Among them, Flatten(I i ) operation is to transform the multi-dimensional input into one dimension; H is the image height; W is the image width; p i is the output feature vector; W p is the weight matrix; I i is the i-th image of the input; b p is the bias term.

[0082] Add a learnable positional encoding matrix to each block The output embedding vector P is:

[0083] P=[p1,p2,…,p N ]+E p

[0084] where p N is the feature vector of the Nth output.

[0085] The embedding vector P is passed through the Transformer encoder to obtain the output image feature vector Each layer of the Transformer encoder contains multi-head self-attention (MHA) and feed-forward network (FFN).

[0086] (2) The image feature vector f img With the prompt text T prompt Input language model such as LLaMA-3.3B, take the last layer hidden state mean as text feature f text :

[0087]

[0088] Where L is the sequence length; h i is the eigenvector at the ith position.

[0089] The prompt text is:

[0090] prompt="Please describe this image [image] based on the output of the CLIP-ViT-L / 14 model."

[0091] (3) The text feature f text The cross-attention mechanism is used to achieve image-text feature fusion. The cross-attention mechanism is mainly composed of the following:

[0092] Query-key-value mapping:

[0093]

[0094] Among them, f text is the text feature vector; f img is the image feature vector; W Q ,W K ,W V They are three learnable weight matrices; Q is the question vector of the text; K is the index vector of the image, and V is the content vector of the image.

[0095] Attention weights:

[0096]

[0097] Among them, A is the attention weight matrix; softmax() is the normalization function; d k is the scaling factor; K T is the transpose of the key matrix.

[0098] Feature fusion:

[0099] f=LayerNorm(f text +A·V))

[0100] Among them, f is the final output feature after normalization; LayerNorm() is the layer normalization function.

[0101] A learnable gating mechanism is introduced to balance the modes and further optimize the cross-attention fusion mechanism. The method is as follows:

[0102] g=σ(W g [f img ,f text ])

[0103] f′=g·f img +(1-g)·f

[0104] in, σ is the sigmoid function, g is the gate weight, used to control f img Fusion ratio with f; W g is a learnable weight matrix, for the feature [f img ,f text ] performs linear transformation; f′ is the final fusion feature.

[0105] The output text obtained based on the image data input in the embodiment is as follows:

[0106] Table 3 Output text

[0107]

[0108]

[0109] S3. Call the domain ontology model in the Neo4j graph database to generate a JSON schema structured template containing entity relationship constraints (e.g., if an accident occurs, the accident type is not "null"). It also generates inter-field dependencies based on graph path analysis and combines knowledge graph reasoning to implement real-time data legitimacy verification, including:

[0110] Establish a mapping rule based on the JSON Schema generation algorithm. The generation examples are shown in Tables 4 and 5 below.

[0111] Table 4 Entity relationship generation example

[0112]

[0113]

[0114] Table 5 Mapping rules

[0115]

[0116] The entity type attribute includes whether damage occurs (occurred) and the corresponding Boolean type value, the damage type and the corresponding integer value; if damage occurs, the damage type is greater than zero.

[0117] In Table 5, the attribute type: For example, a Boolean attribute is converted to the "type" field in JSON Schema, which is represented as "type": "Boolean type";

[0118] Enumeration value: For example, the enumeration value (0, 1) is converted to the "value range" field in JSON Schema, expressed as "value range": (0, 1);

[0119] Numeric range: For example, the numeric range [1,5] is converted to the "minimum value" and "maximum value" fields in JSON Schema, represented as ["minimum value": 1, "maximum value": 5];

[0120] Relationship constraints: Generate field dependency rules, such as "if the Boolean type is 1, then the specified type is not 0", expressed in the form of JSON Schema dependency rules.

[0121] Establish a constraint verification mechanism. The specific steps are as follows:

[0122] Static validation: Checks data types and enumeration ranges based on the JSON Schema standard.

[0123] Dynamic verification: Verify the knowledge graph relationship constraints through Cypher queries, specifically including the following: Matching conditions: For example, matching data with an entity type of "Defect" and a relationship type of "affectedBy", associated with the damage type "Defect_type".

[0124] Verification condition: For example, when the value of "whether damage has occurred" (occurred) is 1 and the value of "damage type" (Defect_type) is 0, verification is performed.

[0125] Return result: For example, if the value of "whether damage occurred" (occurred) is greater than 1 and the value of "damage type" (Defect_type) is greater than 0, the return result is valid (isValid); otherwise, the return result is invalid (false).

[0126]

[0127]

[0128] Composite validation: Apply Datalog rules to a combination of fields. The following example shows:

[0129] Rule description: For example, it is required that the value of "damage type" (Defect_type) is greater than 0.

[0130] Verification conditions: For example, condition 1: the value of damage occurred is 1, etc.

[0131]

[0132] S4. In order to output the recognition results in a structured manner for entry into the database and subsequent processing, the language model is enhanced based on the structured template generated in step S3, so that the language model can convert the recognition results of S2 into structured output results.

[0133] This embodiment uses a large language model such as deepseek-r1:70b as a data output framework agent, converts JSONSchema into instruction prompts that can be understood by the large language model, and aligns the generated results with the knowledge graph logic based on reinforcement learning.

[0134] As shown in Table 6, the conversion rules for converting JSON Schema into understandable instruction prompts include the following:

[0135] (1) Field name: directly displayed as a list item. For example, if the input field name is "occurred", the converted output is "-occurred:Type=boolean";

[0136] (2) Type constraint: Splice the Type= prefix. For example, if you input {"type":"boolean type"}, the output after conversion is Type=boolean;

[0137] (3) Enumeration value: Add Options = enumeration list. For example, if you input the enumeration value (0,1), the output after conversion is Options = [0,1];

[0138] (4) Dependency: Add Requires = list of dependent fields. For example, if you input "occurred" as the "relationship constraint", the output after conversion will be Requires = "occurred".

[0139] Table 6 Example of model instruction conversion rules

[0140]

[0141]

[0142] The instructions are as follows:

[0143]

[0144] Based on the generated instructions, deepseek-r1:70b is personalized trained. The personalized training logic is as follows:

[0145] Training data preparation: The training data is the instructions generated above. The generated instructions consist of a set of input and output pairs, where the input is a specific schema prompt and the output is valid JSON data that meets the requirements. The data is used to guide the model to learn how to generate expected output based on the input.

[0146] Personalized training: Adjust the parameters of the deepseek-r1:70b model based on the training data to optimize its generative ability.

[0147] Reinforce learning alignment: Reinforce learning alignment by designing a reward function. The reward function is designed as follows:

[0148] R=0.7·R1+0.3·R2

[0149] Among them, R1 is the syntax reward, that is, the JSON Schema verification pass rate; R2 is the semantic reward, that is, the knowledge graph query consistency score, and R is the final comprehensive reward value.

[0150] The data generation process is enhanced by a hierarchical control strategy, which consists of the following steps:

[0151] Field-level control: By adjusting the Logits Bias of the model, you can control the selection of the generated results of the field and assign different bias values ​​to strengthen the generation probability of specific options, such as:

[0152] For the field occurred, by increasing the offset value, a specific enumeration value (0, 1) is generated first;

[0153] For weather, by increasing the bias value, specific enumeration values ​​(1, 2, 3, 4) are generated first;

[0154] For the facility_type field, by increasing the offset value, a specific enumeration value (0, 1, 2, 3, 4, 5) is generated first;

[0155] For the field Defect_type, by increasing the offset value, valid options (0, 1, 2, 3, 4, 5) are generated first.

[0156] Record-level control uses related data generated by field queries to supplement the correct data. The example is as follows:

[0157] In the overall data generated by the record, the correlation and consistency between fields are ensured through logical verification and completion mechanisms.

[0158] like:

[0159] If the generated field facility_type>0, you need to ensure that the related fields such as (occurred exists and is 1);

[0160] If the generated field Defect_type>0, you need to ensure that the related fields such as (occurred exists and is 1).

[0161] The scene recognition framework is based on the AutoGen automatic generator and is centered on the task coordination agent. It includes multiple AI agents from video image data input to data structured output and then to the database entry, including:

[0162] Build a task coordination agent, initialize the list of available agents (data input agent, visual processing agent, knowledge base agent, data output agent), dynamically allocate computing resources through a bipartite graph matching algorithm to achieve a reasonable allocation of resources for each AI agent, and define an optimized resource allocation function with the following objective function:

[0163]

[0164] Where M and N are the total number of tasks to be assigned and the number of available computing agents, respectively; ω1 and ω2 are weight coefficients; x ij ∈{0,1} indicates whether task i is assigned to the jth agent j ; Agent j The maximum parallel task capacity of T is determined by its memory and number of threads; ij For task i in Agent j The estimated execution time s on the task is defined as follows:

[0165]

[0166] Where FLOPs represents the computational effort of task i, and FLOPs / s represents the computational effort of the jth agent. j Real-time computing power.

[0167] E ij Assign task i to Agent j The communication energy consumption J is defined as follows:

[0168] E ij = Data volume × (α·Transmission distance + β·Number of hops)

[0169] Where α is the energy consumption coefficient of transmission distance; β is the energy consumption coefficient of hop number.

[0170] Establish a data storage mechanism. Based on the data output framework, the output results of the agent will be input into the relevant data tables of the database. The example is as follows:

[0171] Assume the output is:

[0172]

[0173] Then generate sql code:

[0174]

[0175] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, numerous modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, so that those skilled in the art can better understand and utilize the present invention.

Claims

1. A method for identifying road infrastructure anomalies based on vision and AI agents, characterized in that: The following steps are involved: Convert multiple video stream data into image data and perform normalization on the image data; The normalized image is fed into a visual model to extract global semantic features. The visual model extracts features specific to road infrastructure anomalies and generates visual features related to the anomalies. The global semantic features are then fed into a language model to generate scene text features. This is combined with domain knowledge to generate semantic descriptions related to road infrastructure anomalies. The cross-attention mechanism fuses the image and text features to further enhance understanding of the anomaly scene. The fused image and text features are input into the scene recognition model to obtain structured output results, and the structured output results are imported into the database.

2. The method for identifying road infrastructure anomalies based on vision and AI agents according to claim 1 is characterized in that: The global semantic features of the image extracted by the visual model include: (1) The input image I is divided into N blocks, and each block is mapped to a d-dimensional vector by linear projection: Among them, Flatten(I i ) operation is to transform the multi-dimensional input into one dimension; H is the image height; W is the image width; p i is the output feature vector; W p is the weight matrix; I i is the i-th image of the input; b p is the bias term; (2) Add a learnable position encoding matrix to each block The output embedding vector P is: P=[p1,p2,…,p N ]+E p Among them, p N is the feature vector of the Nth output; (3) The embedding vector P passes through the Transformer encoder to output the image feature vector 3. The method for identifying road infrastructure anomalies based on vision and AI agents according to claim 1, characterized in that: The language model generates scene text features by converting the image feature vector f img With the prompt text T prompt Input the language model and take the last layer of hidden state mean as the text feature f text : Where L is the sequence length; h i is the feature vector of the i-th position, and the text feature f text The fusion of image and text features is achieved through the cross-attention mechanism.

4. The method for identifying road infrastructure anomalies based on vision and AI agents according to claim 3 is characterized in that: The cross-attention mechanism introduces a learnable gating mechanism to balance the modes and further optimize the cross-attention fusion mechanism: g=σ(W g [f img ,f text ]) f′=g·f img +(1-g)·f in, σ is the sigmoid function, g is the gate weight, used to control f img With f text The fusion ratio, f text is the text feature vector; f img is the image feature vector, W g is a learnable weight matrix, for the feature [f img ,f text ] performs linear transformation; f ′ is the final fusion feature.

5. The method for identifying road infrastructure anomalies based on vision and AI agents according to claim 1, characterized in that: The scene recognition model includes a domain ontology model and a language model. Using the domain ontology model, the image and text features are input into the domain ontology model to generate a JSON Schema structured template containing entity relationship constraints; the language model performs structured data enhancement based on the JSON Schema structured template.

6. The method for identifying road infrastructure anomalies based on vision and AI agents according to claim 5, characterized in that: An entity relationship consists of an entity type and a relationship type: Entity type attributes include whether damage occurred (occurred) and the corresponding Boolean value, damage type and the corresponding integer value; The constraint validation mechanism for JSON Schema structured templates includes: (1) Static validation: Check data types and enumeration ranges based on the JSON Schema standard; (2) Dynamic verification: verifying knowledge graph relationship constraints through Cypher queries; (3) Composite check: Apply Datalog rules to field combinations.

7. The method for identifying road infrastructure anomalies based on vision and AI agents according to claim 1, characterized in that: The language model uses deepseek-r1:70b for personalized training.

8. The method for identifying road infrastructure anomalies based on vision and AI agents according to claim 1, characterized in that: The scene recognition model constructs a task coordination agent and uses a bipartite graph matching algorithm to dynamically allocate computing resources to achieve a reasonable allocation of resources for each agent. Its optimization objective function is as follows: Where M and N are the total number of tasks to be assigned and the number of available computing agents, respectively; ω1 and ω2 are weight coefficients; x ij ∈{0,1} indicates whether task i is assigned to the jth agent j ; Agent j The maximum parallel task capacity of T is determined by its memory and number of threads; ij For task i in Agent j The estimated execution time s on T ij For task i in Agent j The estimated execution time s on is calculated as follows: Where FLOPs represents the computational effort of task i, and FLOPs / s represents the computational effort of the jth agent. j Real-time computing power; E ij Assign task i to Agent j The communication energy consumption is calculated as follows: E ij = Data volume × (α·Transmission distance + β·Number of hops) Where α is the energy consumption coefficient of transmission distance; β is the energy consumption coefficient of hop number.

9. A system for implementing the method for identifying road infrastructure anomalies based on vision and AI agents as described in claim 1, characterized in that: include Data input agent, used for preprocessing multi-channel video stream data; Visual processing agent, used to extract image features and fuse image and text features; The knowledge base agent calls the domain ontology model in the Neo4j graph database to generate a JSONSchema structured template containing entity relationship constraints; The data output agent uses a large language model to output recognition results based on a JSON Schema structured template.

10. A computer device, characterized in that: include: A memory, a processor, and a computer program stored in the memory, which, when executed on the processor, implements a method for identifying road infrastructure anomalies based on vision and AI agents as described in any one of claims 1 to 8.

Images