Highway disease semantic description method based on multi-modal model and multi-source data

Abstract

The application discloses a highway disease semantic description method based on a multi-modal model and multi-source data, and belongs to the specific computer model field, and comprises the following steps: constructing a data set D1 and a knowledge base D2 related to highway diseases; sequentially constructing a visual module, a GIS module, a user request module, an information retrieval module, a fusion connection module, and training the above modules and a large language model to form a semantic description network to convergence, which is used for generating highway disease semantic description. The application uniformly fuses visual semantic recognition, pixel-level disease mask generation, GIS spatial geometric quantization, industry standard knowledge retrieval, and field constraint language generation, so that accurate, standard and interpretable description of highway disease types, forms, positions, lengths, areas, directions, severity and disposal suggestions is realized. The application can guarantee the factual consistency and professional reliability of the generated results.

Description

Technical Field

[0001] This invention relates to the field of specific computer modeling technology, and in particular to a method for semantic description of highway defects based on multimodal models and multi-source data. Background Technology

[0002] Currently, highway maintenance is developing towards intelligence, automation, and precision. Image-based highway defect identification technology has become a crucial component of road inspection and maintenance decision-making. Existing methods mostly focus on defect target detection or image classification, typically outputting defect categories, detection boxes, or rough locations. While these methods can assist in defect detection to some extent, they still fall short of meeting the demands for refined descriptions and spatial quantification of defects in real-world maintenance scenarios. Specifically, existing image recognition methods often lack pixel-level characterization of defect areas, making it difficult to accurately obtain spatial parameters such as the true extension length, direction, continuity of cracks, area of ​​planar defects, center location, and crack density of network-like defects. Furthermore, the detection results in defect images often lack effective linkage with road geographic information, pavement slope information, and spatial reference information, resulting in defect descriptions that fail to accurately reflect the location, size, and geometric state of defects in the actual road space. In addition, highway defect descriptions not only require image content recognition but also the integration of professional knowledge such as industry standards, maintenance specifications, defect definitions, severity assessment criteria, and treatment recommendations. While existing multimodal large models possess strong capabilities in text and image understanding and natural language generation, their direct application to highway disease description can easily lead to inaccuracies in fields such as length, area, direction, location, regulatory basis, and treatment recommendations, making it difficult to guarantee the factual consistency and professional reliability of the generated results. Summary of the Invention

[0003] The purpose of this invention is to provide a method for semantic description of highway defects based on multimodal models and multi-source data that solves the above problems and can unify and integrate visual semantic recognition, pixel-level defect mask generation, GIS spatial geometry quantification, industry standard knowledge retrieval, and field-constrained language generation, thereby achieving accurate, standardized, and interpretable description of highway defect types, morphology, location, length, area, direction, severity, and treatment recommendations.

[0004] To achieve the above objectives, the technical solution adopted by this invention is as follows: a method for semantic description of highway defects based on multimodal models and multi-source data, comprising the following steps: S1, obtain a text set S containing description texts of diseases of category C, where the description texts of diseases of category c are s. c , 1≤c≤C, for each s c Collect N road surface images corresponding to the defects to construct dataset D1; , where xi y i M represents the i-th image in D1 and its corresponding text description of the lesion. i For x i Realistic masking of diseased areas, G i For use to x i Spatial reference information for converting pixel coordinates to geographic coordinates, u i For users to x i Request text regarding disease maintenance; Obtain the text of highway maintenance to construct a knowledge base D2, and split the text in D2 into knowledge blocks; S2, Construct a visual module for analyzing x i Candidate disease categories Extract x i about Visual semantic vectors Generate information about mask ; S3, construct a GIS module for use in accordance with... Extract x i Generate spatial information vectors from GIS information. Including S31~S32; S31, classify Class C diseases into three categories: fissure type, planar type, and network type, and pre-define the GIS information extracted for each category and the structured GIS fields constructed based on the GIS information; S32, according to Category Extraction Generate structured GIS fields from GIS information. The spatial information vector is obtained through the text encoder. ; S4, construct a user request module to process user requests for x. i Request text regarding disease maintenance u i The request code is generated by the text encoder. ; S5, Construct an information retrieval module to extract x i The standard knowledge vectors, including S51~S52; S51, , , Generate fusion query vectors using a gating fusion method. ; S52 retrieves knowledge blocks from D2 based on a sparse-dense hybrid retrieval method, concatenates them, and then generates a standardized knowledge vector through a text encoder. ; S6, construct a fusion connection module to generate a prompt sequence for the highway disease description task, including S61~S62; S61, , , , Mapping to the same dimension yields the corresponding information source. , , , ; S62, Define the set of fields to be generated based on the highway defect description task. a j For the j-th field slot, where J is the total number of fields, each a j Encoded as field query vector e j ; S63, for each a j Evidence was extracted from four types of information sources and fused into a hint vector z. j Then put all z j Concatenate into a cue sequence ; S7, obtain a large language model, which is used to determine the language model based on the requested text u. i Based on the prompt sequence Z, a structured result J0 is generated, and then a natural language description text is output based on J0; S8 trains a semantic description network consisting of a visual module, a GIS module, a user request module, an information retrieval module, a fusion connection module, and a large language model until convergence, which is then used to generate semantic descriptions of highway defects.

[0005] Preferably, S2 includes S21 to S25; S21, use the CLIP model image encoder to process x i Divide the image into P patches and generate global semantic features. Local semantic features of the p-th image patch If 1 ≤ p ≤ P, then rearrange the P local semantic features into a local feature map F. i ; S22, generate s c semantic vector t c ,calculate With t c similarity , With t c similarity and P Pooling for local response scores ; S23, based on , Calculate the combined disease category score for each category c. The highest score and its corresponding category are marked as candidate scores. and candidate disease categories ,Will The corresponding semantic vector label is ; S24, for Take its corresponding P R p,c Rearranged into a semantic heatmap The decoder is based on F i and Generate mask , then calculate Local disease semantic vector ; , S25, , , , The concatenation is then processed by an MLP to output a visual semantic vector. Generate x i Visual semantic vectors .

[0006] As a preferred option, in S22, s c semantic vector t c Generated by the text encoder of the CLIP model. , Pooling operations are calculated according to the following formulas: , In the formula, For transpose operation, Let L be the L2 norm of the vector, and Pool(∙) be the maximum pooling, average pooling, or Top-K average pooling.

[0007] As a preferred embodiment, in S23, the joint disease category score of category c is calculated. for, In the formula, α is The weight.

[0008] Preferably, in S31, the GIS information for fractures includes fracture length, direction, and endpoint location; The GIS information for the areal structure includes area, center location, major axis, minor axis, and direction; The network-like GIS information includes coverage area, total skeleton length, and fissure density.

[0009] As a preferred method, the gating fusion method in S51 is as follows: (derived from the following formula) , , Corresponding weights , and Then weighted fusion is obtained ; , , In the formula, For the splicing operation, MLP(∙) is a multilayer perceptron and Softmax(∙) is the Softmax function.

[0010] As a preferred option, S52 specifically includes Sa1 to Sa4; Sa1, from the request text u i , Corresponding disease description text and Extract keywords to form a set of keywords Q i ; , In the formula, KW(∙) represents the keyword extraction operation, and Expand() represents the keyword expansion operation; Sa2 calculates the mixed retrieval score for each knowledge block in D2, and selects multiple high-scoring knowledge blocks as candidate knowledge blocks, where the m-th knowledge block c in D2... m Mixed search score S hybrid (c m Calculate according to the following formula: , In the formula, For sparse retrieval scores, Score for dense semantic retrieval; for Weight, d m For c m Text encoding generated by a text encoder; Sa3 generates a union query text. , Concat(∙) is a concatenation operation; Sa4, will Each candidate knowledge block is cross-encoded to calculate a relevance score, and the top K candidate knowledge blocks with the highest scores are selected, concatenated, and encoded into a canonical knowledge vector. .

[0011] Preferably, in the field set A, the field slots include disease type, disease morphology, geographical coordinates, specific disease characteristics, severity, standard basis, and treatment suggestions.

[0012] As a preferred option, in S63, a is generated. j The hint vector z j Includes steps Sb1 to Sb3; Sb1, extract four types of information sources according to the following formula. , , , Corresponding evidence , , , ; , , , , In the formula, Attn (∙) represents multi-head cross-attention; Sb2 uses a gated fusion method to generate weights for four types of evidence. , , , ; , For splicing operations, For multilayer perceptrons, For the Softmax function; Sb3, calculate the hint vector z j ; .

[0013] Preferably, in S7, a large language model is selected, and the generated structured result is J0. In the formula, This is a JSON function.

[0014] In this invention: Regarding the visual module: used to extract x i Regarding candidate disease categories Visual semantic vectors and generate information about mask This invention first extracts the global semantic features and local semantic features (image features) of the image according to step S21, and then extracts the disease description text s according to step S22. c semantic vector t c (Text features) enable computation of text and image features located in the same feature space.

[0015] First, calculate the global semantic features. With t c similarity This can be considered as a global response score. In addition, a pixel-level localization process is introduced to enhance the semantics of the disease description text, and local semantic features are calculated in S22. With semantic vector t c similarity Then, pooling is performed on all similarities to obtain the local response score. , To indicate whether there is a local region in the image related to type c disease, the global response score reflects the overall disease semantics of the entire image, but is easily affected by background information such as normal road surfaces and shadows; while the local response score highlights the local area where the disease is located, but may be affected by local textures and noise. Therefore, this invention finally uses a weighted fusion of the global response score and the local response score as the joint disease category score for category c. The result of fusing the two It can take into account both overall semantics and local disease characteristics, improving the accuracy and stability of disease category judgment, and providing a unique disease category for subsequent heatmap generation, reducing interference from other categories. Then, according to S24, a semantic heatmap is generated and calculated. Local disease semantic vector , This vector represents the local visual semantic features of the candidate disease region. It primarily preserves texture, edge, morphology, continuity, and local spatial response information related to this type of disease. Finally, z-axis data is used to represent the vector. cls , , , Composition x i Visual semantic vectors ,exist In the middle, z cls Provides global disease semantics for the entire image, and provides Local visual characteristics of the diseased area Provide semantic constraints for candidate disease categories. This provides confidence information for the disease type in the current image. Through this method, each candidate disease type has an independent visual semantic representation. Furthermore, global semantic features and local semantic features are used to extract a global semantic token and a local patch token, respectively, from the OpenCLIP image encoder.

[0016] Regarding the GIS module: The objective is to first design GIS information and structured GIS fields for extracting different categories of diseases, and then extract GIS information according to the actual disease category to generate the corresponding structured fields. See also Figure 3First, for the image input to the GIS module, an affine coordinate transformation is performed in conjunction with spatial reference information. First, the pixel coordinates within the image are transformed to local road surface coordinates, and then to geographic coordinates. Since only the mask is processed, and the entire image is not required, the GIS module of this invention selects the key points to be transformed based on the type of disease. For crack-type diseases, the crack skeleton points and endpoints are mapped to geographic coordinates to calculate crack length and direction. For areal diseases, the centroid of the disease area and optional boundary points are mapped to geographic coordinates to determine the center location and spatial extent of the disease. The disease area can be directly calculated based on the number of pixels in the area and the road surface pixel resolution. Second, this invention classifies Class C diseases into three categories: crack-type, areal, and network-type. When it is a crack-type disease, a skeleton extraction method is used to extract features such as crack length and direction. Select the specific feature extraction method according to the category, and extract the corresponding GIS information, for example... For crack-type defects, methods such as skeleton extraction are used to extract features such as length and direction. For areal defects, methods such as region extraction are used to extract features such as area and center location. The same applies to network-type defects. After extracting GIS information, slope correction and projection conversion can be performed in conjunction with road slope information. After the GIS module completes the calculation, the spatial information is organized into structured results, which generates short texts of structured GIS fields. For crack-type defects, crack length, direction angle, endpoint coordinates, slope correction status, and station location can be output; for areal defects, area, center location, major axis, minor axis, main direction, and slope area can be output; for network-type defects, coverage area, crack density, and severity auxiliary indicators can be output. In addition, categories can be added as needed, such as adding depth-type defects, thus dividing Class C defects into four categories: crack-type, areal, network-type, and depth-type. The GIS information for depth-type defects includes depth and volume. For depth-type defects, depth and volume are output when three-dimensional data is available; otherwise, they are marked as not calculated.

[0017] Regarding the user request module: The request text u i Encoding as request encoding , used for subsequent fusion calculations.

[0018] Regarding the information retrieval module, it is used to extract x i The present invention retrieves normative knowledge vectors. Existing normative knowledge retrieval methods typically rely primarily on user text queries, easily overlooking the actual disease type and morphology within the image, as well as spatial information such as length, area, direction, and location calculated by the GIS module. Therefore, this invention does not solely rely on user requests for normative retrieval but also incorporates visual semantic vectors. Spatial information vector User request text u i Corresponding request code Together they form a fusion query vector Used for retrieval queries, enabling retrieval results to simultaneously match disease type, disease spatial size, and user task requirements.

[0019] Compared with the prior art, the advantages of the present invention are as follows: (1) Compared with existing highway disease identification methods based on target detection or image classification, which usually only output disease category, detection box or rough location, and are difficult to obtain fine information such as crack length, direction, area of ​​surface disease and geographical location, this invention guides pixel-level segmentation through disease text semantics and combines GIS spatial reference information to realize the location of disease area and the quantification of parameters such as length, area, direction and location, thereby improving the spatial accuracy of disease description.

[0020] (2) Compared with existing text-guided visual methods such as CLIPSeg, DenseCLIP and MaskCLIP, this invention not only uses disease text and local image blocks for semantic matching and mask generation, but also uses the generated disease mask for GIS spatial quantization, so that the visual recognition results can be converted into the size and location of the disease in the actual road space.

[0021] (3) Existing normative knowledge retrieval methods usually only rely on user input text for retrieval, which easily ignores the actual disease types identified in the image and the length, area, direction and location calculated by the GIS module. This invention constructs a retrieval query by combining user requests, visual disease semantics and GIS spatial information, so that the retrieved normative clauses can match the current disease type, spatial size and user task requirements, thereby improving the relevance and accuracy of normative knowledge retrieval.

[0022] (4) Existing multimodal connection methods usually use linear projection, MLP projection, overall splicing or overall weighting to fuse information from different sources, which makes it difficult to ensure that each generated field is constrained by the corresponding information source. This invention adopts a field slot-driven multi-source fusion mechanism, so that the disease type and morphology are mainly based on visual information, the length, area, direction and location are mainly based on GIS information, and the severity, standard basis and treatment suggestions are mainly based on the retrieved standard knowledge, thereby reducing field errors and unfounded generation, and improving the accuracy, completeness, professionalism and interpretability of the final disease description.

[0023] In summary, this invention integrates visual semantic recognition, pixel-level defect mask generation, GIS spatial geometry quantification, industry standard knowledge retrieval, and field-constrained language generation to achieve an accurate, standardized, and interpretable description of highway defects in terms of type, shape, location, length, area, direction, severity, and treatment recommendations. Attached Figure Description

[0024] Figure 1 This is a diagram of the overall architecture of the present invention; Figure 2 This is a diagram of the visual module structure. Figure 3 Flowchart for the GIS module; Figure 4 A diagram of the user request module structure; Figure 5 This is a structural diagram of the information retrieval module; Figure 6 This is a structural diagram of the fusion connection module. Detailed Implementation

[0025] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0026] Example 1: See Figures 1-6 A semantic description method for highway defects based on multimodal models and multi-source data includes the following steps: S1, obtain a text set S containing description texts of diseases of category C, where the description texts of diseases of category c are s. c , 1≤c≤C, for each s c Collect N road surface images corresponding to the defects to construct dataset D1; , where x i y i M represents the i-th image in D1 and its corresponding text description of the lesion. i For x i Realistic masking of diseased areas, G i For use to x i Spatial reference information for converting pixel coordinates to geographic coordinates, u i For users to x i Request text regarding disease maintenance; Obtain the text of highway maintenance to construct a knowledge base D2, and split the text in D2 into knowledge blocks; S2, Construct a visual module for analyzing x i Candidate disease categories Extract x i about Visual semantic vectors Generate information about mask ; S3, construct a GIS module for use in accordance with... Extract x i Generate spatial information vectors from GIS information. Including S31~S32; S31, classify Class C diseases into three categories: fissure type, planar type, and network type, and pre-define the GIS information extracted for each category and the structured GIS fields constructed based on the GIS information; S32, according to Category Extraction Generate structured GIS fields from GIS information. The spatial information vector is obtained through the text encoder. ; S4, construct a user request module to process user requests for x. i Request text regarding disease maintenance u i The request code is generated by the text encoder. ; S5, Construct an information retrieval module to extract x i The standard knowledge vectors, including S51~S52; S51, , , Generate fusion query vectors using a gating fusion method. ; S52 retrieves knowledge blocks from D2 based on a sparse-dense hybrid retrieval method, concatenates them, and then generates a standardized knowledge vector through a text encoder. ; S6, construct a fusion connection module to generate a prompt sequence for the highway disease description task, including S61~S62; S61, , , , Mapping to the same dimension yields the corresponding information source. , , , ; S62, Define the set of fields to be generated based on the highway defect description task. a j For the j-th field slot, where J is the total number of fields, each a j Encoded as field query vector e j ; S63, for each a j Evidence was extracted from four types of information sources and fused into a hint vector z. j Then put all z j Concatenate into a cue sequence ; S7, obtain a large language model, which is used to determine the language model based on the requested text u. i Based on the prompt sequence Z, a structured result J0 is generated, and then a natural language description text is output based on J0; S8 trains a semantic description network consisting of a visual module, a GIS module, a user request module, an information retrieval module, a fusion connection module, and a large language model until convergence, which is then used to generate semantic descriptions of highway defects.

[0027] In this embodiment: S2 includes S21~S25; S21, use the CLIP model image encoder to process x i Divide the image into P patches and generate global semantic features. Local semantic features of the p-th image patch If 1 ≤ p ≤ P, then rearrange the P local semantic features into a local feature map F. i ; S22, generate s c semantic vector t c ,calculate With t c similarity , With t c similarity and P Pooling for local response scores , where s c semantic vector t c Generated by the text encoder of the CLIP model. , Pooling operations are calculated according to the following formulas: , In the formula, For transpose operation, Let L be the L2 norm of the vector, and Pool(∙) be the maximum pooling, average pooling, or Top-K average pooling. S23, based on , Calculate the combined disease category score for each category c. The highest score and its corresponding category are marked as candidate scores. and candidate disease categories ,Will The corresponding semantic vector label is ;in α is The weights; S24, for Take its corresponding P R p,c Rearranged into a semantic heatmap The decoder is based on F i and Generate mask , then calculate Local disease semantic vector ; , S25, , , , The concatenation is then processed by an MLP to output a visual semantic vector. Generate x i Visual semantic vectors .

[0028] In S31, GIS information for fractures includes fracture length, direction, and endpoint location; GIS information for areal structures includes area, center location, major axis, minor axis, and direction; and GIS information for networks includes coverage area, total skeleton length, and fracture density.

[0029] The gating fusion method in S51 is as follows: (derived from the following formula) , , Corresponding weights , and Then weighted fusion is obtained ; , , In the formula, For the splicing operation, MLP(∙) is a multilayer perceptron and Softmax(∙) is the Softmax function.

[0030] S52 specifically includes Sa1~Sa4; Sa1, from the request text u i , Corresponding disease description text and Extract keywords to form a set of keywords Q i ; , In the formula, KW(∙) represents the keyword extraction operation, and Expand() represents the keyword expansion operation; Sa2 calculates the mixed retrieval score for each knowledge block in D2, and selects multiple high-scoring knowledge blocks as candidate knowledge blocks, where the m-th knowledge block c in D2... m Mixed search score S hybrid (c m Calculate according to the following formula: , In the formula, For sparse retrieval scores, Score for dense semantic retrieval; for Weight, d m For c m Text encoding generated by a text encoder; Sa3 generates a union query text. , Concat(∙) is a concatenation operation; Sa4, will Each candidate knowledge block is cross-encoded to calculate a relevance score, and the top K candidate knowledge blocks with the highest scores are selected, concatenated, and encoded into a canonical knowledge vector. .

[0031] In addition, the field set A includes field slots such as disease type, disease morphology, geographical coordinates, specific disease characteristics, severity, standard basis, and treatment suggestions.

[0032] In S63, a is generated. j The hint vector z j Includes steps Sb1 to Sb3; Sb1, extract four types of information sources according to the following formula. , , , Corresponding evidence , , , ; , , , , In the formula, Attn (∙) represents multi-head cross-attention; Sb2 uses a gated fusion method to generate weights for four types of evidence. , , , ; , For splicing operations, For multilayer perceptrons, For the Softmax function; Sb3, calculate the hint vector z j ; .

[0033] In S7, a large language model is selected, and the generated structured result is J0. In the formula, This is a JSON function.

[0034] In S8, the semantic description network is trained to converge to obtain a semantic description model, which is used to generate semantic descriptions of highway defects. Specifically, the user obtains a pavement image of the highway defect to be identified, the spatial reference information corresponding to the pavement image, and the user's request text regarding the maintenance of the defect on the pavement image. These are then input into the semantic description model, and finally, the semantic description model generates the corresponding natural language description text.

[0035] Example 2: See Figures 1-6 Based on Example 1, a complete process for training the semantic description network in step S8 is presented. The semantic description network consists of a vision module, a GIS module, a user request module, an information retrieval module, a fusion connection module, and a large language model, and is trained using a "module-by-module training - overall joint fine-tuning" approach.

[0036] First, 6,000 images of highway defects of 5 types were divided into training, validation, and test sets according to road sections, with a ratio of 70%:15%:15%. The images were uniformly scaled and cropped to 512×512.

[0037] 1. Pre-trained mask generation module: The vision module uses a pre-trained OpenCLIP-ViT-B / 16 to extract global image features, local image patch features, and disease text features. During initial training, the OpenCLIP image encoder and text encoder are frozen, and only the semantic heatmap processing part and segmentation decoder are trained. A manually annotated ground-realistic mask of the disease region M is used. i As supervisory information, the segmentation decoder generates pixel-level lesion masks based on local feature maps and semantic heatmaps. The segmentation decoder is pre-trained for 50 epochs using the AdamW optimizer, with a batch size of 8 and a learning rate of 1×10⁻⁶. -4 The mask segmentation loss is 0.5L. BCE +0.5L Dice L BCE For binary classification cross-entropy loss, L Dice This is the Dice loss. Through this stage of training, the segmentation decoder first acquires stable disease area localization and mask generation capabilities.

[0038] 2. Joint training of visual modules: After pre-training the segmentation decoder, the disease classification, mask generation, and visual-semantic fusion components were jointly trained for 50 epochs. The OpenCLIP encoder was frozen for the first 10 epochs of joint training; subsequently, the last four Transformer layers of the image encoder were unfrozen. The learning rate for both the segmentation decoder and the visual fusion network was 1×10⁻⁶. -4The learning rate of the OpenCLIP image encoder is 1×10⁻⁶. -5 The total training loss for constructing the visual module is L. vis ; L vis =0.3L cls +0.7L seg , L cls =0.4L global +0.3L local +0.3L joint , In the formula, L cls For disease classification losses, L global L is the classification loss corresponding to the global semantic features; local L represents the classification loss corresponding to the local response score. joint L is the classification loss for the combined disease category score. seg This is the cross-entropy loss for multi-class classification.

[0039] 3. Training on the standardized knowledge base construction and information retrieval module: The information retrieval module inputs user request encoding, visual semantic vectors, and GIS spatial information vectors into a gated fusion network to generate dense query vectors. Simultaneously, it extracts keywords from user requests, candidate disease texts, and GIS structured fields for BM25 sparse retrieval. The canonical knowledge base is segmented into knowledge blocks of 350 Chinese characters each, with adjacent knowledge blocks overlapping by 50 characters. The retrieval module uses a fusion of BM25 and dense retrieval, with weights of 0.4 and 0.6 respectively. First, 50 knowledge blocks are recalled, then the top 20 are reordered using a cross-encoder, and finally, the top 5 are retained. The dense retrieval model is trained for 5 epochs, and the cross-encoder for 3 epochs, with a learning rate of 2×10⁻⁶ for both. -5 During training, standard knowledge blocks that match the current disease category, spatial size, and treatment conditions are used as positive samples, while knowledge blocks that do not match other disease categories or applicable conditions are used as difficult negative samples. The dense retrieval encoder and gating fusion network are trained by contrastive loss.

[0040] 4. Integrating connection module training with supervised fine-tuning of large language models: The fusion and connection module unifies the mapping of visual semantic vectors, GIS spatial information vectors, user request codes, and standard knowledge vectors into 512-dimensional features. It sets basic field slots such as disease type, disease morphology, geographical location, severity, standard basis, and treatment recommendations, and dynamically adds specific field slots based on the disease's geometric type, including length, area, orientation angle, endpoint coordinates, major axis, minor axis, and crack density. Each field slot extracts evidence from four types of information sources through two layers and eight heads of cross-attention. After gating and fusion, it generates field hint vectors, which are then concatenated into a field-level hint sequence.

[0041] The large language model employs Meta-Llama-3.1-8B-Instruct and uses 4-bit QLoRA for supervised fine-tuning. During training, user requests and field-level prompt sequences are used as input, with manually annotated JSON structured results as the supervision target. QLoRA uses NF4 quantization and BF16 precision, with a LoRA rank of 16, a scaling factor of 32, Dropout of 0.05, a maximum input length of 2048 tokens, a batch size of 1 per card, gradient accumulators 16 times, and a learning rate of 2×10⁻⁶. -4 The system was trained for 3 rounds using the AdamW optimizer.

[0042] During training, the parameters of the Llama base model are frozen, and only the parameters of the feature mapping layer, field query vector, cross attention layer and gating network in the LoRA adapter and fusion connection module are updated. This enables different fields to extract evidence from their corresponding information sources first and generate a Chinese structured description of highway defects that meets the requirements.

[0043] 5. Overall joint fine-tuning of the semantic description network: After completing the phased training of each module, the vision module, information retrieval module, fusion connection module, and large language model are connected for overall joint fine-tuning. During joint fine-tuning, the GIS calculation method, BM25 inverted index, and canonical knowledge base are fixed, while updating the LoRA adapter parameters of some layers of the OpenCLIP image encoder, segmentation decoder, dense retrieval network, fusion connection module, and Llama model. The joint fine-tuning uses a weighted sum of the vision module loss, retrieval loss, and structured result generation loss as the total loss, and performs backpropagation updates on each trainable module.

[0044] Example 3: Based on Example 1, a method for extracting three major categories of disease features from a GIS module is presented: 1. Crack-type defects: For crack-type defects, the GIS module is based on x i Candidate disease categories Corresponding mask The crack pixel set is extracted, and connected component filtering, small region removal, and skeleton extraction are performed to obtain the crack skeleton point sequence P. i , p1~p n These are the first to nth skeleton points; Each skeleton point can be represented as local road surface coordinates or geographic coordinates. This invention does not directly use the straight-line distance between the two endpoints of the crack as the crack length, but rather accumulates the length segment by segment along the skeleton. Crack skeleton length L skel It can be represented as: , Among them, (E) j N j ) and (E j+1 N j+1 The numbers and represent the geographic coordinates of adjacent skeleton points, and the overall direction angle θ of the crack. crack It can be obtained by fitting the principal axis endpoints or the principal direction of PCA: , Where (E1,N1) and (E2,N2) represent the geographical coordinates of the two ends of the crack's main axis.

[0045] Since the inspection images are acquired perpendicular to the road surface, given the known pixel resolution of the road surface, the crack length and area of ​​planar damage can be preferentially represented as the actual slope length or actual slope area. When it is necessary to unify with GIS map coordinates, or to output both the horizontal projection and the actual slope area simultaneously, slope correction can be further performed by combining spatial reference information obtained during image acquisition or preliminary measurement. Specifically, the actual slope length L of the crack can be calculated using the following formula. surface : , Among them, the crack skeleton point sequence P i The line segment between two adjacent skeleton points is marked as a skeleton segment. θ represents the horizontal projection length of the j-th skeleton segment. j Indicates the orientation angles of the skeleton segment in the horizontal plane, β and θ. s These are the slope angle and slope direction angle of the road surface where the current defect area is located, provided by the spatial reference information.

[0046] 2. Surface-like diseases: For surface defects such as potholes, loose areas, oil seepage, peeling, and repairs, the GIS module calculates the area, center location, major axis, minor axis, and main direction of the defect region based on a defect mask. Let the pixel set of the surface defect region be: After post-processing including connected component filtering, hole filling, and small region removal, stable disease areas were obtained. Since the inspection images were acquired perpendicular to the road surface, within the known pixel resolution of the road surface... , In the case of planar diseases, the actual area of ​​the slope can be expressed as: , in, This indicates the number of pixels in the affected area. The center of the affected area can be calculated using the region's centroid and then mapped to geographic coordinates or station location. The major axis, minor axis, and principal direction of the affected area can be obtained through minimum bounding rectangle or PCA principal axis analysis.

[0047] 3. Network-like diseases: For network-like defects such as crazing, net-like cracking, and blocky cracking, they possess both a planar coverage area and an internal crack network structure. Therefore, this invention simultaneously calculates the defect coverage area and the total length of the crack skeleton within the area. Crack density D crack It can be represented as: , Among them, L net A represents the total length of the crack skeleton within the network-like disease area. surface This represents the slope area covered by the disease. Using this method, network-like diseases are no longer represented solely by area, but rather spatially quantified by combining the coverage area and the density of internal cracks, which helps in subsequent severity assessment.

[0048] 5. After the GIS module completes its calculations, the spatial information is organized into structured results. For crack-type defects, the output includes crack length, orientation angle, endpoint coordinates, slope correction status, and station location; for area-type defects, the output includes area, center location, major axis, minor axis, main direction, and slope area; for network-type defects, the output includes coverage area, crack density, and severity indicators; for depth-type defects, depth and volume are output if 3D data is available, otherwise it is marked as not calculated. Depth-type defects are optional.

[0049] Subsequently, the GIS structured fields are serialized into short text, and this GIS short text is then... Input text encoder E text To obtain spatial information vectors , .

[0050] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for semantic description of highway defects based on multimodal models and multi-source data, characterized in that, Includes the following steps: S1, obtain a text set S containing description texts of diseases of category C, where the description texts of diseases of category c are s. c , 1≤c≤C, for each s c Collect N road surface images corresponding to the defects to construct dataset D1; , where x i y i M represents the i-th image in D1 and its corresponding text description of the lesion, respectively. i For x i Realistic masking of diseased areas, G i For use to x i Spatial reference information for converting pixel coordinates to geographic coordinates, u i For users to x i Request text regarding disease maintenance; Obtain the text of highway maintenance to construct a knowledge base D2, and split the text in D2 into knowledge blocks; S2, Construct a visual module for analyzing x i Candidate disease categories Extract x i about Visual semantic vectors Generate information about mask ; S3, construct a GIS module for use in accordance with... Extract x i Generate spatial information vectors from GIS information. Including S31~S32; S31, classify Class C diseases into three categories: fissure type, planar type, and network type, and pre-define the GIS information extracted for each category and the structured GIS fields constructed based on the GIS information; S32, according to Category Extraction Generate structured GIS fields from GIS information. The spatial information vector is obtained through the text encoder. ; S4, construct a user request module to process user requests for x. i Request text regarding disease maintenance u i The request code is generated by the text encoder. ; S5, Construct an information retrieval module to extract x i The standard knowledge vectors, including S51~S52; S51, , , Generate fusion query vectors using a gating fusion method. ; S52 retrieves knowledge blocks from D2 based on a sparse-dense hybrid retrieval method, concatenates them, and then generates a standardized knowledge vector through a text encoder. ; S6, construct a fusion connection module to generate a prompt sequence for the highway disease description task, including S61~S62; S61, , , , Mapping to the same dimension yields the corresponding information source. , , , ; S62, Define the set of fields to be generated based on the highway defect description task. a j For the j-th field slot, where J is the total number of fields, each a j Encoded as field query vector e j ; S63, for each a j Evidence was extracted from four types of information sources and fused into a hint vector z. j Then put all z j Concatenate into a prompt sequence ; S7, obtain a large language model, which is used to determine the language model based on the requested text u. i Based on the prompt sequence Z, a structured result J0 is generated, and then a natural language description text is output based on J0; S8, which combines a visual module, a GIS module, a user request module, an information retrieval module, a fusion connection module, and a large language model to form a semantic description network, is trained to convergence and used to generate semantic descriptions of highway defects.

2. The method for semantic description of highway defects based on multimodal models and multi-source data according to claim 1, characterized in that, S2 includes S21~S25; S21, use the CLIP model image encoder to process x i Divide the image into P patches and generate global semantic features. Local semantic features of the p-th image patch If 1 ≤ p ≤ P, then rearrange the P local semantic features into a local feature map F. i ; S22, generate s c semantic vector t c ,calculate With t c similarity , With t c similarity and P Pooling for local response scores ; S23, based on , Calculate the combined disease category score for each category c. The highest score and its corresponding category are marked as candidate scores. and candidate disease categories ,Will The corresponding semantic vector label is ; S24, for Take its corresponding P R p,c Rearranged into a semantic heatmap The decoder is based on F i and Generate mask , then calculate Local disease semantic vector ; ， S25, , , , The concatenation is then processed by an MLP to output a visual semantic vector. Generate x i Visual semantic vectors .

3. The method for semantic description of highway defects based on multimodal models and multi-source data according to claim 2, characterized in that, In S22, s c semantic vector t c Generated by the text encoder of the CLIP model. , Pooling operations are calculated according to the following formulas: ， In the formula, For transpose operation, Let L be the L2 norm of the vector, and Pool(∙) be the maximum pooling, average pooling, or Top-K average pooling.

4. The method for semantic description of highway defects based on multimodal models and multi-source data according to claim 2, characterized in that, In S23, the joint disease category score for category c is calculated. for, In the formula, α is The weight.

5. The method for semantic description of highway defects based on multimodal models and multi-source data according to claim 1, characterized in that, In S31, the GIS information for fractures includes fracture length, direction, and endpoint location; The GIS information for the areal structure includes area, center location, major axis, minor axis, and direction; The network-like GIS information includes coverage area, total skeleton length, and fissure density.

6. The method for semantic description of highway defects based on multimodal models and multi-source data according to claim 1, characterized in that, The gating fusion method in S51 is as follows: (derived from the following formula) , , Corresponding weights , and Then weighted fusion to obtain ; ， ， In the formula, For the splicing operation, MLP(∙) is a multilayer perceptron and Softmax(∙) is the Softmax function.

7. The method for semantic description of highway defects based on multimodal models and multi-source data according to claim 1, characterized in that, S52 specifically includes Sa1~Sa4; Sa1, from the request text u i , Corresponding disease description text and Extract keywords to form a set of keywords Q i ; ， In the formula, KW(∙) represents the keyword extraction operation, and Expand() represents the keyword expansion operation; Sa2 calculates the mixed retrieval score for each knowledge block in D2, and selects multiple high-scoring knowledge blocks as candidate knowledge blocks, where the m-th knowledge block c in D2... m Mixed search score S hybrid (c m Calculate according to the following formula: ， In the formula, For sparse retrieval scores, Score for dense semantic retrieval; for Weight, d m For c m Text encoding generated by a text encoder; Sa3 generates a union query text. , Concat(∙) is a concatenation operation; Sa4, will Each candidate knowledge block is cross-encoded to calculate a relevance score, and the top K candidate knowledge blocks with the highest scores are selected, concatenated, and encoded into a canonical knowledge vector. .

8. The method for semantic description of highway defects based on multimodal models and multi-source data according to claim 1, characterized in that, In the field set A, the field slots include disease type, disease morphology, geographical coordinates, specific disease characteristics, severity, standard basis, and treatment suggestions.

9. The method for semantic description of highway defects based on multimodal models and multi-source data according to claim 1, characterized in that, In S63, a is generated. j The hint vector z j Includes steps Sb1 to Sb3; Sb1, extract four types of information sources according to the following formula. , , , Corresponding evidence , , , ; ， ， ， ， In the formula, Attn (∙) represents multi-head cross-attention; Sb2 uses a gated fusion method to generate weights for four types of evidence. , , , ; ， For splicing operations, For multilayer perceptrons, For the Softmax function Sb3, calculate the hint vector z j ; 。 10. The method for semantic description of highway defects based on multimodal models and multi-source data according to claim 1, characterized in that, In S7, a large language model is selected, and the generated structured result is J0. In the formula, This is a JSON function.

Images