Bridge component information retrieval method and system based on computer vision and BIM

Abstract

The invention discloses a bridge component information retrieval method and system based on computer vision and BIM, and relates to the technical field of bridge engineering digitization, and the method comprises the steps: obtaining image data of a to-be-retrieved bridge, and inputting the image data into a computer vision cascade model to obtain category labels and appearance defect features of candidate components; associating the candidate component with a component in the BIM model of the to-be-retrieved bridge, screening out a target component through a preset screening rule, and extracting structured data; based on the structured data, the category label of the target component and the appearance defect features, constructing a retrieval request, inputting the retrieval request into a preset vector knowledge base, and obtaining an unstructured knowledge fragment related to the retrieval request; and fusing the structured data and the unstructured knowledge fragments, and inputting the fused structured data and unstructured knowledge fragments into a large language model to generate a comprehensive decision report. According to the method, the decision report is generated through the closed loop of visual identification, BIM association and knowledge retrieval, so that the retrieval efficiency and the decision accuracy are improved.

Description

Technical Field

[0001] This application relates to the field of digital technology in bridge engineering, specifically to a method and system for retrieving bridge component information based on computer vision and BIM. Background Technology

[0002] In the traditional field of bridge engineering, the query and management of component information has long suffered from inefficiency and data fragmentation. Specifically, existing technologies have the following core pain points: reliance on manual operation, requiring engineers to manually consult paper drawings, electronic ledgers, or scattered databases, resulting in cumbersome and time-consuming retrieval processes; severe data fragmentation; unstructured knowledge such as on-site inspection data (e.g., photos), BIM model structured data, industry standards, and historical cases are stored in a scattered manner, lacking effective linkages and hindering collaborative access; low retrieval efficiency and accuracy, with errors easily occurring in manual component identification under complex conditions (e.g., high altitude, obstruction, backlighting), and traditional retrieval results are mostly single data points, unable to combine comprehensive knowledge to form decision support; and delayed emergency response, with time-consuming manual retrieval of key component information (e.g., load-bearing capacity, reinforcement schemes) during sudden disasters or malfunctions, failing to meet the real-time needs of emergency repairs.

[0003] Therefore, there is an urgent need for an intelligent retrieval technology that can integrate on-site visual information, model structured data, and industry knowledge bases. Summary of the Invention

[0004] This application provides a bridge component information retrieval method and system based on computer vision and BIM, which can solve the technical problems of traditional bridge information retrieval relying on manual labor and data fragmentation, resulting in low efficiency and insufficient decision support.

[0005] In a first aspect, embodiments of this application provide a bridge component information retrieval method based on computer vision and BIM, the bridge component information retrieval method based on computer vision and BIM including: Image data of the bridge to be searched is obtained, and the image data is input into a computer vision cascade model to obtain the category labels and appearance defect features of the candidate components. The candidate components are associated with the components in the BIM model of the bridge to be searched. The target components corresponding to the candidate components are selected from the associated components by using preset filtering rules, and the structured data of the target components are extracted. Based on the structured data, the category labels and appearance defect features of the candidate components corresponding to the target component, a retrieval request is constructed, and the retrieval request is input into a preset vector knowledge base to obtain unstructured knowledge fragments related to the retrieval request. The structured data and the unstructured knowledge fragments are fused and input into a large language model to generate a comprehensive decision report that includes a description of the target component's status, compliance analysis, and disposal recommendations. The computer vision cascaded model includes a component classification model and a defect detection model.

[0006] In conjunction with the first aspect, in one implementation, acquiring the image data of the bridge to be retrieved includes: Collect multi-angle high-definition image data covering the bridge to be searched, the image data including high-definition video streams and still images; Each frame of the high-definition video stream and each still image is bound to the spatial location information and attitude angle parameters at the time of acquisition. The spatial location information includes latitude and longitude coordinates and elevation values. Image data carrying spatial location information and attitude angle parameters will be transmitted wirelessly to an edge computing server or stored locally.

[0007] In conjunction with the first aspect, in one implementation, the image data is input into a computer vision cascade model to obtain category labels for candidate components, including: Extract single-frame images from the image data or extract images from the video stream, preprocess the obtained images to adjust to a fixed resolution and normalize the pixel values, and organize them into batch tensor format; A component classification model is used to extract and fuse multi-scale features from the preprocessed image, and bounding box regression and class prediction are performed based on the fused features. The parameters of multiple candidate components are output, including the bounding box coordinates, class label and confidence score of the candidate component in the image. The parameters of multiple candidate components are filtered according to a first preset confidence threshold. Candidate components with confidence scores higher than the first preset confidence threshold are retained, and the category labels of the filtered candidate components are output as the category labels of multiple candidate components.

[0008] In conjunction with the first aspect, in one implementation, the image data is input into a computer vision cascaded model to obtain appearance defect features of candidate components, including: Based on the bounding box coordinates of each candidate component in the image, the candidate component region corresponding to the bounding box in the image data is cropped to obtain a local image of each candidate component. Each local image is preprocessed to a fixed resolution and pixel values ​​are normalized, and then organized into a batch tensor format; Based on the category label of each candidate component, the dedicated defect detection model corresponding to the category of each candidate component is called from the model library; A dedicated defect detection model for each candidate component is used to detect defects in the local image of each candidate component, and the defect parameters of multiple candidate components are output. The defect parameters include the defect category, defect location and defect confidence score of each candidate component. The defect parameters of multiple candidate components are filtered according to the second preset confidence threshold. Candidate components with confidence scores higher than the second preset confidence threshold are retained. The defect category and defect location of the filtered candidate components are output as the appearance defect features of multiple candidate components.

[0009] In conjunction with the first aspect, in one implementation, the component classification model adopts a target detection model trained based on the YOLOv8 model, and the dedicated defect detection models all adopt defect detection models trained based on the YOLOv8 model.

[0010] In conjunction with the first aspect, in one implementation, associating candidate components with components in the BIM model of the bridge to be retrieved, filtering out the target components corresponding to the candidate components from the associated components using preset filtering rules, and extracting the structured data of the target components includes: Based on the category labels of candidate components and their spatial location information in the image, all BIM components with the same category as the current candidate component are located in the BIM model of the bridge to be searched through a preset mapping relationship, and an association relationship is established. Calculate the spatial distance between the candidate component and each of its associated BIM components, and based on a preset distance threshold, select the BIM component with the smallest Euclidean distance that is lower than the preset distance threshold as the target component; Obtain the unique component identifier of the target component, access the database associated with the BIM model, and extract the structured data of the target component based on the unique component identifier.

[0011] In conjunction with the first aspect, in one implementation, the step of locating all BIM components of the same category as the current candidate component in the BIM model of the bridge to be searched, based on the category labels of the candidate components and their spatial location information in the image, through a preset mapping relationship, includes: Based on the category label of the candidate component and the spatial location information and attitude angle parameters when the image was acquired, the direction vector of the candidate component in the camera coordinate system is obtained through the direct georeferenced photogrammetry principle. The direction vector is solved into three-dimensional geographic coordinates in the WGS-84 coordinate system through rigid transformation of coordinate rotation and translation; Based on the coordinate transformation parameters of the bridge, the three-dimensional geographic coordinates are converted into three-dimensional spatial coordinates under the local engineering coordinate system of the bridge used in the BIM model of the bridge to be retrieved. Based on the preset mapping relationship and the three-dimensional spatial coordinates, determine the spatial partitioning of the current candidate component in the BIM model; Based on the category label of the current candidate component, locate all BIM components with the same category as the current candidate component in the spatial partition corresponding to the current candidate component.

[0012] In conjunction with the first aspect, in one implementation, the step of constructing a retrieval request based on the structured data, the category labels of candidate components corresponding to the target component, and appearance defect features, inputting the retrieval request into a preset vector knowledge base, and obtaining unstructured knowledge fragments related to the retrieval request includes: The category labels of candidate components, appearance defect features, and structured data of the target component are combined to generate query statements in natural language format. The query statement is converted into a query vector using a text vector embedding model. The query vector is then input into a preset vector knowledge base, and the similarity between the query vector and each knowledge fragment vector stored in the preset vector knowledge base is calculated. Based on the similarity ranking from high to low, at least one knowledge fragment with a similarity higher than a preset threshold is selected as the output unstructured knowledge fragment related to the retrieval request. The unstructured knowledge fragments include industry standard clauses, technical manual content, and historical engineering cases.

[0013] In conjunction with the first aspect, in one implementation, the step of fusing the structured data with the unstructured knowledge fragments and inputting the resulting data into a large language model to generate a comprehensive decision report containing a description of the target component's state, compliance analysis, and handling recommendations includes: Input prompt words for a large language model are constructed based on the structured data and the unstructured knowledge fragments; The input prompts are fed into the big language model, which generates a comprehensive decision report that includes a description of the target component's status, compliance analysis, and handling recommendations. The structured data includes the ComponentID, design bearing capacity, material, construction and maintenance records of the bridge to be retrieved.

[0014] Secondly, embodiments of this application provide a bridge component information retrieval system based on computer vision and BIM, the bridge component information retrieval system based on computer vision and BIM includes: The acquisition module is used to acquire image data of the bridge to be searched, and input the image data into a computer vision cascade model to obtain the category labels and appearance defect features of the candidate components. The association module is used to associate candidate components with components in the BIM model of the bridge to be searched, filter out the target components corresponding to the candidate components from the associated components through preset filtering rules, and extract the structured data of the target components. The retrieval module is used to construct a retrieval request based on the structured data, the category labels of the candidate components corresponding to the target component, and the appearance defect features, input the retrieval request into a preset vector knowledge base, and obtain unstructured knowledge fragments related to the retrieval request. The generation module is used to fuse the structured data with the unstructured knowledge fragments and input them into the large language model to generate a comprehensive decision report that includes a description of the target component's status, compliance analysis, and disposal suggestions. The computer vision cascaded model includes a component classification model and a defect detection model.

[0015] The beneficial effects of the technical solutions provided in this application include: In this embodiment, a computer vision cascaded model automatically identifies component categories and defects, replacing manual on-site identification and recording. It automatically associates and filters BIM components through preset filtering rules, replacing manual review of drawings and ledgers, significantly improving overall retrieval efficiency and accuracy. This embodiment associates visual recognition results with unique components and their structured data in the BIM model, and further vectorizes the retrieval of unstructured knowledge, achieving the integration and collaborative use of on-site data, model data, and knowledge data at the system level. This embodiment's fully automated component identification, BIM data association, knowledge retrieval, and report generation process can quickly retrieve key structured data of target components and corresponding unstructured knowledge related to handling, rapidly generating comprehensive decision reports containing handling suggestions. This significantly shortens the time for retrieving key component information and making decision analyses, meeting the real-time needs of emergency repairs during sudden disasters or malfunctions. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the bridge component information retrieval method based on computer vision and BIM proposed in this application. Figure 2 This is a schematic diagram of the functional modules of the bridge component information retrieval system based on computer vision and BIM in this application. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0019] This application provides a bridge component information retrieval method and system based on computer vision and BIM, which can solve the technical problems of traditional bridge information retrieval relying on manual labor and data fragmentation, resulting in low efficiency and insufficient decision support.

[0020] In a first aspect, embodiments of this application provide a method for retrieving bridge component information based on computer vision and BIM.

[0021] Reference Figure 1 , Figure 1 This is a flowchart illustrating the bridge component information retrieval method based on computer vision and BIM proposed in this application. Figure 1 As shown, the bridge component information retrieval method based on computer vision and BIM specifically includes the following steps: Step S1: Obtain image data of the bridge to be searched, and input the image data into the computer vision cascade model to obtain the category labels and appearance defect features of the candidate components.

[0022] In this embodiment of the application, the step S1 of obtaining the image data of the bridge to be retrieved specifically includes the following steps: Step S111: Collect multi-angle high-definition image data covering the bridge to be searched. The image data includes high-definition video streams and still images.

[0023] Specifically, inspection personnel can use industrial-grade multi-rotor drones equipped with RTK (real-time dynamic positioning) capabilities as data acquisition terminals. For example, the DJI Matrice 350 RTK, with centimeter-level positioning accuracy, can be selected. The drone is equipped with an image acquisition unit capable of visible light and zoom. In the pre-planned flight path, the area of ​​the bridge pier supports to be inspected is divided into multiple shooting viewpoints. Each viewpoint is configured with corresponding latitude and longitude coordinates, elevation values, and gimbal attitude angle parameters. During flight, the drone performs close-range circling photography of the support area while maintaining a safe distance, according to the mission plan. The acquired image data includes 4K high-definition video streams with a resolution of 3840×2160, as well as JPG still images automatically or manually triggered at key locations.

[0024] This application embodiment ensures that the surface features of the bridge components to be searched are completely recorded by taking pictures from multiple angles, reducing the risk of missed detection due to limited perspective, and providing a more complete visual data foundation for defect identification.

[0025] Step S112: Bind the spatial location information and attitude angle parameters of the acquisition time to each frame of the high-definition video stream and each still image. The spatial location information includes latitude and longitude coordinates and elevation values.

[0026] Specifically, during the data acquisition process, the sensor modules of the acquisition device collect real-time spatial location information and attitude angle parameters of single frames of high-definition video streams and still images at the moment of acquisition. Spatial location information includes latitude and longitude coordinates and elevation values ​​in the geodetic coordinate system, while attitude angle parameters include the heading angle, pitch angle, and roll angle of the acquisition device. This provides a spatial location benchmark for the subsequent mapping of visual recognition results to the BIM model.

[0027] Step S113: Transmit the image data carrying spatial location information and attitude angle parameters to the edge computing server or store it locally via wireless communication.

[0028] Specifically, the acquisition device transmits image data, along with its associated spatial location information and attitude angle parameters at the acquisition time, in real time to an edge computing server deployed within the inspection vehicle via an onboard 5G communication module. If on-site network conditions are limited, a batch upload method after local caching can also be used. This embodiment preferably employs real-time transmission to support near real-time interpretation. Thus, data is directly transmitted from the acquisition end to nearby computing resources, significantly reducing data transmission latency.

[0029] In this embodiment of the application, step S1, which involves inputting image data into a computer vision cascade model to obtain category labels for candidate components, specifically includes the following steps: Step S121: Extract single-frame images from image data or extract images from video streams, preprocess the obtained images to adjust to a fixed resolution and normalize pixel values, and organize them into batch tensor format.

[0030] Specifically, static images are directly extracted from multi-source image data as the data to be processed. Frame extraction is performed on high-definition video streams to obtain single-frame images, forming the image set to be processed by the model. Preprocessing operations are performed on the images, including fixed-resolution adjustment to ensure the image size perfectly matches the preset input size of the computer vision cascaded model. Then, the image pixel values ​​are normalized to map them to the numerical range suitable for the model, eliminating interference from non-component features. One or more preprocessed image data are stacked along the batch dimension to organize a four-dimensional tensor structure, forming the standard input format for the model.

[0031] Step S122: Use a component classification model to extract and fuse multi-scale features from the preprocessed image, and perform bounding box regression and class prediction based on the fused features to output parameters of multiple candidate components, including the bounding box coordinates, class label and confidence score of the candidate components in the image.

[0032] Specifically, the computer vision cascaded model used in this application embodiment is composed of a component classification model and a defect detection model cascaded together. The component classification model is a target detection model trained based on the YOLOv8 model. Its core architecture includes three parts: a backbone network, a neck network, and a detection head. The specific processing flow is as follows: First, the backbone network performs multi-scale feature extraction. The preprocessed image tensor is input into the backbone network, which employs a hierarchical convolutional structure to generate multiple feature maps with decreasing spatial resolution and increasing channel number through multiple downsampling operations.

[0033] In a specific embodiment, the input preprocessed image is downsampled for the first time by an initial convolutional layer (3×3 kernel, 2 stride), and the output feature map has a dimension of B×64×320×320, thereby increasing the number of channels and halving the image resolution. Then, the feature is processed sequentially through four C2f module groups (denoted as C2f_1 to C2f_4). Each C2f module group has multiple branch convolutions and residual connections, which can achieve deep abstraction and extraction of features. After passing through modules C2f_2, C2f_3, and C2f_4, convolutional layers with a stride of 2 are connected for downsampling, ultimately generating three sets of core multi-scale feature maps: shallow feature map F1: dimension B×256×160×160, mainly preserving detailed positional information such as the edges, textures, and corners of the component; mid-layer feature map F2: dimension B×512×80×80, which integrates the local structure and basic semantic information of the component; deep feature map F3: dimension B×1024×40×40, which further aggregates global contextual information through SPPF (pooling module), focusing on the overall semantic category of the component. The final output of the backbone network consists of three sets of feature maps: F1, F2, and F3. These three sets of feature maps exhibit a pattern of decreasing spatial resolution (160×160→80×80→40×40) and increasing channel number (256→512→1024).

[0034] Secondly, the neck network performs multi-scale feature fusion. The neck network amplifies high-level semantic features and concatenates them with low-level detail features through upsampling operations, while compressing low-level features and aligning them with high-level features through downsampling operations.

[0035] In a specific embodiment, the neck network adopts a PAN-FPN (Path Aggregation Network - Feature Pyramid Network) architecture. The deep feature map F3 is compressed to 512 dimensions through a 1×1 convolution, and then its resolution is increased to 80×80 through upsampling (multiplier of 2). It is then concatenated with the middle feature map F2 in the channel dimension, resulting in a concatenated feature map with dimensions of B×1024×80×80. After the concatenated result is fused by the C2f module, its resolution is increased to 160×160 again through upsampling (multiplier of 2), and it is then concatenated with the shallow feature map F1. The concatenated feature map has dimensions of B×512×160×160. After further fusion by the C2f module, the shallow fused feature map Ff1 is obtained. Then, the shallow fusion feature map Ff1 is downsampled by a convolutional layer with a stride of 2, reducing the resolution to 80×80. It is then concatenated again with the feature map obtained by "upsampling by F3 + concatenation by F2". After fusion by the C2f module, the middle fusion feature map Ff2 (dimension B×1024×80×80) is obtained. The middle fusion feature map Ff2 is then downsampled again by a convolutional layer with a stride of 2, reducing the resolution to 40×40. It is then concatenated with the original deep feature map F3. After fusion by the C2f module, the deep fusion feature map Ff3 (dimension B×2048×40×40) is obtained. The final output of the neck network consists of three sets of fusion feature maps: Ff1 (160×160), Ff2 (80×80), and Ff3 (40×40). Each set of fusion feature maps contains both high-level semantic information and low-level detail information, providing comprehensive feature support for subsequent detection tasks.

[0036] Subsequently, the detection head performs bounding box regression and class prediction. The fused feature map is input into the detection head, which simultaneously performs bounding box regression and component class prediction. Bounding box regression locates the candidate component in the image pixel coordinate system, while class prediction determines the bridge component category corresponding to the target within the bounding box.

[0037] In a specific embodiment, the fused feature maps Ff1, Ff2, and Ff3 are input to the decoupled detection head of YOLOv8. This detection head is divided into two independent sub-networks: a bounding box regression branch and a class prediction branch, which can perform parallel inference synchronously. For the bounding box regression branch, the input is a single set of fused feature maps (e.g., Ff1: B×512×160×160). After processing through two layers of 3×3 convolutions and one layer of 1×1 convolutions, the output is a regression tensor with dimensions B×4×N×N (where N is the resolution of the corresponding fused feature map, such as 160, 80, or 40; 4 corresponds to the four core parameters of the bounding box, namely x, y, w, and h, representing the center coordinates, width, and height of the bounding box, respectively). The final output is a bounding box regression tensor based on the anchor box free mechanism, whose parameter values ​​are normalized pixel coordinates (ranging from 0 to 1). For the category prediction branch, the input is the same as that of the bounding box regression branch, both being a single set of fused feature maps. After processing through two layers of 3×3 convolutions and one layer of 1×1 convolutions, the output is a classification tensor with dimensions B×C×N×N (where C is the total number of categories of bridge components, such as piers, box girders, and supports). The final output is a category probability tensor activated by a sigmoid function. The value at each position in this tensor represents the probability that the corresponding region belongs to a certain component category (the value ranges from 0 to 1). The detection head integrates the inference results of the three sets of fused feature maps and finally outputs the original detection tensor, which contains the bounding box parameters and category probability information for all detection locations.

[0038] Finally, the component classification model integrates the inference results in a structured manner and outputs standardized parameters for multiple candidate components. Each set of parameters corresponds to a single candidate component and includes the bounding box coordinates, category label, and confidence score representing the reliability of the recognition result of the candidate component in the image.

[0039] In a specific implementation, the first step is confidence screening: setting a reasonable confidence threshold (which can be 0.25), removing detection boxes with class probabilities lower than this threshold, and retaining detection results with higher confidence; the second step is non-maximum suppression: setting an IoU threshold (which can be 0.7), deduplicating duplicate detection boxes belonging to the same component, and retaining only the detection box with the highest confidence; the third step is parameter standardization: converting the normalized bounding box parameters (x, y, w, h) into actual coordinates in the image pixel coordinate system (i.e., the coordinates of the top left corner). , and the coordinates of the bottom right corner , The specific calculation formula is as follows: , , , (W and H are the width and height of the original image, respectively).

[0040] The final output is a structured list of candidate component parameters. Each set of parameters contains three parts: (1) bounding box coordinates: , , , (All are pixel values); (2) Category labels: such as “pier”, “box girder”, “support”, etc. in text or number form; (3) Confidence score: the value ranges from 0 to 1, used to characterize the reliability of the detection result.

[0041] Step S123: Filter the parameters of multiple candidate components according to the first preset confidence threshold, retain the candidate components whose confidence scores are higher than the first preset confidence threshold, and output the category labels of the filtered candidate components as the category labels of multiple candidate components.

[0042] Specifically, the first preset confidence threshold can be between 0.5 and 0.7. Candidate components with confidence scores lower than the first preset confidence threshold are deemed to have insufficient recognition reliability and are discarded, retaining only candidate components with confidence scores higher than the first preset confidence threshold. The corresponding category labels are extracted from the filtered candidate component parameters and used as the category labels for multiple candidate components to complete the output.

[0043] In one specific embodiment, after the component classification model performs inference on the support area image, the output recognition result is: Category tag: pot bearing Confidence level: 0.96 Bounding box coordinates:

[0044] Wherein, the bounding box coordinates are the coordinates given in the image pixel coordinate system.

[0045] In this embodiment of the application, step S1, which involves inputting image data into a computer vision cascaded model to obtain the appearance defect features of candidate components, specifically includes the following steps: Step S131: Based on the bounding box coordinates of each candidate component in the image, crop the candidate component region corresponding to the bounding box in the image data to obtain a local image of each candidate component.

[0046] Specifically, based on the unique bounding box coordinates corresponding to a single candidate component, the pixel region defined by the bounding box is selected and cropped in the corresponding image of the image data. This pixel region is a local image containing only a single candidate component. For each candidate component identified by the general component classification model, an independent region cropping operation is performed, ultimately obtaining a local image corresponding to each candidate component.

[0047] Step S132: Perform preprocessing on each local image, adjusting it to a fixed resolution and normalizing the pixel values, and organize it into a batch tensor format.

[0048] Specifically, preprocessing is performed on each local impact. In the embodiments of this application, the dedicated defect detection model is a defect detection model trained based on the YOLOv8 model. First, the local image is adjusted to the preset fixed resolution of the dedicated defect detection model so that the image size completely matches the model input specifications. Then, the local image is normalized to map the pixel values ​​to the numerical range adapted to the dedicated defect detection model, eliminating pixel interference from non-defect features. Finally, according to the technical requirements of batch inference of the dedicated defect detection model, all the standardized local images are combined in the batch dimension and organized into a batch tensor format to form standardized input data that can be directly called by the dedicated defect detection model.

[0049] Step S133: Based on the category label of each candidate component, call the dedicated defect detection model corresponding to the category of each candidate component from the model library.

[0050] Specifically, the model library is a dedicated defect detection model library. This application embodiment pre-configures a dedicated component defect detection model library, training multiple sub-models for different bridge components, and extracting and identifying features of common defects in different components. The training sample set of the dedicated defect detection model consists of images labeled with typical appearance defects.

[0051] Step S134: Use a dedicated defect detection model for each candidate component to perform defect detection on the local image of each candidate component, and output the defect parameters of multiple candidate components. The defect parameters include the defect category, defect location and defect confidence score of each candidate component.

[0052] Specifically, based on the category label output in step S122 for each candidate component, a dedicated defect detection model library is accessed. This library employs a key-value storage structure, using the component category label as the index key and the corresponding defect detection model as the storage value. The model library is retrieved based on the current candidate component's category label, and the defect detection model uniquely bound to that component type is loaded. Different component categories correspond to different defect detection models. In one specific embodiment, when the component classification model outputs the label "pot bearing," the corresponding dedicated defect detection model for bearings, such as the Bearing-Defect-YOLOv8n sub-model, is automatically loaded from the model library based on the category label. This embodiment of the application uses a dedicated defect detection model library to improve the targeting and accuracy of appearance defect detection for different types of bridge components.

[0053] For each candidate component's local image, a dedicated defect detection model is invoked to perform defect detection inference. This model extracts and fuses multi-scale defect features from the local image to determine the category of the component's appearance defects, locate the defects within the component's region, and outputs a defect confidence score that characterizes the reliability of the defect detection results. The above defect detection is performed on all candidate components identified by the component classification model, ultimately outputting structured defect parameters for multiple candidate components. Each set of defect parameters uniquely corresponds to a single candidate component, and each parameter includes the defect category, defect location, and defect confidence score. The defect location is the positioning information of the defect in the local image pixel coordinate system of the candidate component or the specific region information of the corresponding bridge component entity.

[0054] Step S135: Filter the defect parameters of multiple candidate components according to the second preset confidence threshold, retain the candidate components with confidence scores higher than the second preset confidence threshold, and output the defect category and defect location of the filtered candidate components as the appearance defect features of multiple candidate components.

[0055] Specifically, the second preset confidence threshold can be between 0.6 and 0.8. Candidate components with confidence scores below the second preset confidence threshold are deemed to have insufficient detection reliability and are discarded; only candidate components with confidence scores above the second preset confidence threshold are retained. Corresponding defect parameters are extracted from the filtered candidate component parameters to complete the output as the appearance defect features of multiple candidate components.

[0056] In one specific embodiment, after the dedicated defect detection model performs inference on the local image, the output defect parameters are: Defect category: Surface rust spots Defect location: Edge area of ​​the support steel plate Confidence level: 0.91.

[0057] This application embodiment structurally integrates the identification results of the component classification model with the detection results of the dedicated defect detection model to generate standardized structured identification results for subsequent steps. The specific data structure can be: {Component category: "Pot bearing", Defect feature: "Surface has rust spots", Image coordinates:} Data collection location: (longitude, latitude, elevation), data collection time: t}.

[0058] Through the above-mentioned computer vision cascaded reasoning and recognition process, the category determination of the target component and the automatic extraction of appearance defect features are completed, while key data such as spatial location feature information associated with the component and collection time sequence information are completely retained.

[0059] Step S2: Associate the candidate components with the components in the BIM model of the bridge to be searched, filter out the target components corresponding to the candidate components from the associated components through preset filtering rules, and extract the structured data of the target components.

[0060] In this embodiment of the application, step S2 specifically includes the following steps: Step S21: Based on the category labels of the candidate components and their spatial location information in the image, locate all BIM components with the same category as the current candidate components in the BIM model of the bridge to be searched through a preset mapping relationship, and establish an association relationship.

[0061] In this embodiment of the application, step S21, which involves locating all BIM components of the same category as the current candidate component in the BIM model of the bridge to be searched based on the category labels of the candidate components and their spatial location information in the image, through a preset mapping relationship, specifically includes the following steps: Step S211: Based on the category label of the candidate component and the spatial location information and attitude angle parameters when the image was acquired, the direction vector of the candidate component in the camera coordinate system is obtained through the direct georeferenced photogrammetry principle.

[0062] Specifically, based on the direct georeference principle in the field of photogrammetry, the spatial location information and attitude angle parameters of the candidate component are extracted from the image during the acquisition stage. Then, combined with the positional characteristics of the candidate component in the image pixel coordinate system, spatial geometric calculation is performed through the collinearity condition equation of photogrammetry to finally obtain the direction vector of the candidate component in the camera coordinate system.

[0063] Step S212: Solve the direction vector into three-dimensional geographic coordinates in the WGS-84 coordinate system through rigid transformation of coordinate rotation and translation.

[0064] Specifically, based on the direction vector, combined with the spatial position, attitude angle parameters and camera exterior orientation elements output by the positioning and attitude determination system (POS) of the acquisition device, and in accordance with the principle of forward intersection in photogrammetric space, rigid transformation geometric operations of coordinate rotation and translation are performed on the direction vector. Thus, the direction vector in the camera coordinate system is solved into three-dimensional geographic coordinates in the WGS-84 geodetic coordinate system, with coordinate dimensions including longitude, latitude and elevation.

[0065] Step S213: Based on the coordinate transformation parameters of the bridge, convert the three-dimensional geographic coordinates into three-dimensional spatial coordinates under the local engineering coordinate system of the bridge used in the BIM model of the bridge to be retrieved.

[0066] Specifically, based on the coordinate system transformation parameters calibrated during the bridge design phase, the three-dimensional geographic coordinates are mapped to the three-dimensional spatial coordinates (X, Y, Z) under the bridge local engineering coordinate system adopted by the BIM model of the bridge to be retrieved through a common coordinate transformation algorithm in the surveying and mapping field, thereby achieving the matching of the spatial coordinate system of the candidate component with the coordinate system of the BIM model.

[0067] Step S214: Determine the spatial partitioning of the current candidate component in the BIM model based on the preset mapping relationship and three-dimensional spatial coordinates.

[0068] Specifically, the preset mapping relationship is the mapping of three-dimensional spatial coordinates to the spatial partitioning rules of the BIM model. The spatial partitioning rules of the BIM model are specifically based on the spatial partitioning criteria established during the bridge design phase, according to "pier number + span number + lane direction + elevation range". The three-dimensional spatial coordinates of the candidate component are matched one by one with the coordinate range of each spatial partition in the BIM model to determine the unique spatial partition to which the candidate component belongs in the BIM model, thereby determining the spatial range of the candidate component in the BIM model.

[0069] In a specific embodiment, taking the top area of ​​pier No. 3 as an example, its spatial partition attributes are defined as follows: Pier No.: P3; Span No.: K123+400~K123+500; Lane direction: Left lane; Elevation range: H1~H2. All components within this spatial partition in the BIM model are assigned corresponding spatial partition attribute information, and each BIM component is assigned a unique component code, such as the code "GZ-3-5" indicating that the component is the 5th support of pier No. 3.

[0070] Step S215: Based on the category label of the current candidate component, locate all BIM components with the same category as the current candidate component in the spatial partition corresponding to the current candidate component.

[0071] Specifically, using the category label of the candidate component as the filtering condition, all components within the spatial partition of the determined BIM model are retrieved, and all BIM components in the spatial partition that are of the same category as the candidate component are extracted. This completes the association relationship between the candidate component and the BIM component of the same category, and finally forms a set of BIM components of the same category corresponding to the candidate component.

[0072] Step S22: Calculate the spatial distance between the candidate component and each of its associated BIM components. Based on a preset distance threshold, select the BIM component with the smallest Euclidean distance that is lower than the preset distance threshold as the target component.

[0073] Specifically, based on the Euclidean distance calculation formula, the spatial geometric distance between the 3D spatial coordinates of each candidate component and the 3D spatial coordinates of each associated BIM component of the same category is calculated to obtain the Euclidean distance value between each BIM component and the candidate component. First, all BIM components with Euclidean distance values ​​lower than a preset distance threshold are filtered out. Then, the single BIM component with the smallest Euclidean distance value is selected from this filtering result and determined as the target component that precisely matches the current candidate component, achieving a one-to-one correspondence between candidate components and BIM model components. The preset distance threshold is set based on the installation accuracy of bridge engineering components and coordinate calculation errors. This improves the accuracy of matching target components with candidate components.

[0074] Step S23: Obtain the unique component identifier of the target component, access the database associated with the BIM model, and extract the structured data of the target component based on the unique component identifier.

[0075] Specifically, the component identifier is a unique code assigned to each component during the BIM model design and construction phase. The database associated with the BIM model stores structured engineering attribute data for all bridge components throughout their entire lifecycle. Using the unique component identifier of the target component as the targeted search keyword, a precise search is performed in the engineering database to extract all structured data uniquely bound to that identifier, thus automating the retrieval of the target component's structured data.

[0076] Structured data can specifically include: design attributes: component model, geometric dimensions, design elevation, design bearing capacity, material grade, concrete strength grade, reinforcement configuration parameters, anti-corrosion coating grade, etc.; construction attributes: installation date, construction team, installation supervisor, site acceptance record, concealed works acceptance record, concrete pouring date, prestressing tensioning date and tension force value, etc.; operation and maintenance attributes: most recent maintenance date, maintenance content, maintenance supervisor, historical defects record, reinforcement and renovation record, condition rating, design service life, and years of use, etc. Therefore, this embodiment eliminates the need for manual intervention in component matching and data retrieval, avoiding subjective errors caused by manual judgment and ensuring the standardization and consistency of target component matching and data extraction results.

[0077] In one specific embodiment, the GlobalId "GZ-3-5" of the target component is used as the targeted search keyword to retrieve the full lifecycle structured engineering attribute data of the component. The extracted data covers, but is not limited to, the following categories: Design parameters: Design bearing capacity 5000kN, component material is Q345qD steel; Construction record: Installation date: May 10, 2020; Construction team in charge: Group A. Maintenance record: The last maintenance operation was in August 2023, and the component maintenance status rating is Class A.

[0078] To support the unified access and integration of multi-source heterogeneous data in the future, this embodiment builds a component information intermediate library inside the association layer. This component information intermediate library can be implemented based on relational database or in-memory database technology. Its preset standardized data table structure includes the following core fields: ComponentID, LocationInfo, ImageMeta, RecognitionResult, DefectInfo, DesignParams, ConstructionRecords, MaintenanceRecords, etc.

[0079] The visual recognition results output from the computer vision cascaded model are structurally integrated with the BIM structured data extracted from the engineering database. Following the preset data table structure of the component information intermediate library, the data is written as a single record. An example of a specific data record is shown below: ComponentID="GZ-3-5"; LocationInfo="Top of Pier 3, Left Side"; RecognitionResult = "Pot bearing"; DefectInfo = "Rust spots exist on the surface"; DesignParams = "Load capacity 5000kN, Material Q345qD"; ConstructionRecords="Installation on 2020-05-10, Construction Team A"; MaintenanceRecords = "Maintenance in August 2023, Condition Rating: A".

[0080] Through the above operations, automated and accurate matching from 2D image visual data to BIM model component objects is achieved. At the same time, a standardized and unified intermediate data structure is constructed, providing unified multi-source data support for subsequent retrieval request construction and vector knowledge base similarity retrieval.

[0081] Step S3: Based on structured data, category labels of candidate components corresponding to the target component, and appearance defect features, construct a retrieval request, input the retrieval request into a preset vector knowledge base, and obtain unstructured knowledge fragments related to the retrieval request.

[0082] In this embodiment of the application, step S3 specifically includes the following steps: Step S31: Combine the category labels of candidate components, appearance defect features, and structured data of the target component to generate a query statement in natural language format.

[0083] Specifically, in accordance with the semantic expression standards of natural language, a unified format natural language query statement is generated. This query statement fully covers core attribute information such as component type, defect characteristics, design, construction, and operation and maintenance, serving as the original retrieval request for the retrieval vector knowledge base. In a specific embodiment, the original retrieval request could be "pot bearing 5000kN Q345qD steel rust treatment sales manufacturer maintenance requirements".

[0084] Step S32: Use a text vector embedding model to convert the query statement into a query vector, input the query vector into a preset vector knowledge base, and calculate the similarity between the query vector and the various knowledge fragment vectors stored in the preset vector knowledge base.

[0085] Specifically, the preset vector knowledge base stores at least: official standard documents, such as the "Specifications for Maintenance of Highway Bridges" JTG 5120-2021; design manuals, product manuals and construction method data; historical disease and treatment case reports of this bridge and other similar bridges; and textual records related to bearing corrosion and anti-corrosion treatment in the expert experience database.

[0086] The query statement is vector-encoded into a query vector with the same dimension as the knowledge fragment vectors. This query vector is then input into a pre-defined vector knowledge base. According to the vector space similarity calculation rules, the query vector is compared with each of the pre-stored knowledge fragment vectors in the vector knowledge base using cosine similarity calculation. Cosine similarity measures the directional consistency by calculating the cosine of the angle between two vectors.

[0087] Step S33: Sort the similarity from high to low, and select at least one knowledge fragment with a similarity higher than a preset threshold as an output unstructured knowledge fragment related to the retrieval request.

[0088] Specifically, knowledge fragments with similarity values ​​higher than a preset threshold are selected. At least one of these selected knowledge fragments is then used as the standardized output of unstructured knowledge fragments relevant to the retrieval request. The coverage of these unstructured knowledge fragments includes clauses from bridge engineering industry standards, content from professional technical manuals, and historical engineering cases. The preset threshold is determined based on the precision-recall curve on the validation set.

[0089] In one specific embodiment, there are three knowledge fragments that exceed a preset threshold, specifically: Excerpt 1: Clauses in the standard regarding the assessment of the severity of rust on pot bearing steel components and the requirements for rust removal and anti-corrosion coating processes; Excerpt 2: A historical case report on the rust treatment of a certain bearing on this bridge in 2019, including construction techniques and effect evaluation; Excerpt 3: Operation guidelines provided by the bearing manufacturer for replacing the sealing ring and performing anti-corrosion maintenance on this type of bearing.

[0090] Step S4: After fusing structured data with unstructured knowledge fragments, input the data into the large language model to generate a comprehensive decision report that includes a description of the target component's status, compliance analysis, and disposal recommendations.

[0091] In this embodiment of the application, step S4 specifically includes the following steps: Step S41: Construct input prompt words for a large language model based on structured data and unstructured knowledge fragments.

[0092] Specifically, during the construction process, role definitions and task instructions for the large language model can be incorporated to clarify the model's reasoning perspective and report generation requirements. This ultimately results in standardized input prompts that the large language model can directly parse, ensuring the relevance and professionalism of the model's reasoning. In a specific embodiment, the input prompts for the large language model could be: "As a bridge maintenance expert, please generate a structured report containing a current status description, compliance analysis, and handling suggestions, based on the given component attributes, regulatory clauses, and historical cases."

[0093] Step S42: Input the input prompts into the large language model, which generates a comprehensive decision report that includes a description of the target component's status, compliance analysis, and handling recommendations.

[0094] Specifically, the comprehensive decision-making report includes, but is not limited to, the following parts: Component Information Card: Displays the location, design parameters, construction date, and most recent curing time of component number GZ-3-5; Disease diagnosis: Based on visual identification results and standard clauses, determine the possible severity of corrosion and suggest whether further measurement or experimental testing is needed; Standard basis: Automatically references specific clause numbers and clause summaries related to bearing corrosion assessment and anti-corrosion treatment in JTG5120-2021; Historical Case Comparison: Briefly describe the treatment plan and effect evaluation of similar bearing defects in 2019 for maintenance personnel reference; Recommendations: Provide specific processes (e.g., sandblasting for rust removal + zinc-rich epoxy primer coating) recommended for the next maintenance window, precautions, and whether monitoring and re-inspection are required.

[0095] In this embodiment, the BIM model is exported as an IFC format file using Revit software, and the project attribute data is stored in an SQLite or relational database; the vector knowledge base can be implemented using vector databases such as Chroma, and the large language model can be implemented using GPT-4 or other large language models with natural language generation capabilities.

[0096] In this embodiment, a computer vision cascaded model automatically identifies component categories and defects, replacing manual on-site identification and recording. It automatically associates and filters BIM components through preset filtering rules, replacing manual review of drawings and ledgers, significantly improving overall retrieval efficiency and accuracy. This embodiment associates visual recognition results with unique components and their structured data in the BIM model, and further vectorizes the retrieval of unstructured knowledge, achieving the integration and collaborative use of on-site data, model data, and knowledge data at the system level. This embodiment's fully automated component identification, BIM data association, knowledge retrieval, and report generation process can quickly retrieve key structured data of target components and corresponding unstructured knowledge related to handling, rapidly generating comprehensive decision reports containing handling suggestions. This significantly shortens the time for retrieving key component information and making decision analyses, meeting the real-time needs of emergency repairs during sudden disasters or malfunctions.

[0097] Secondly, embodiments of this application also provide a bridge component information retrieval system based on computer vision and BIM.

[0098] In one embodiment, reference is made to Figure 2 , Figure 2 This is a schematic diagram of the functional modules of the bridge component information retrieval system based on computer vision and BIM proposed in this application. Figure 2 As shown, the bridge component information retrieval system based on computer vision and BIM includes: The acquisition module is used to acquire image data of the bridge to be searched and input the image data into the computer vision cascade model to obtain the category labels and appearance defect features of the candidate components. The association module is used to associate candidate components with components in the BIM model of the bridge to be searched, and to filter out the target components corresponding to the candidate components from the associated components through preset filtering rules, and extract the structured data of the target components. The retrieval module is used to construct a retrieval request based on structured data, category labels of candidate components corresponding to the target component, and appearance defect features. The retrieval request is then input into a preset vector knowledge base to obtain unstructured knowledge fragments related to the retrieval request. The generation module is used to integrate structured data with unstructured knowledge fragments and input them into a large language model to generate a comprehensive decision report that includes a description of the target component's status, compliance analysis, and disposal recommendations. The computer vision cascade model includes a component classification model and a defect detection model.

[0099] The functional implementation of each module in the above-mentioned bridge component information retrieval system based on computer vision and BIM corresponds to the steps in the above-mentioned bridge component information retrieval method embodiment based on computer vision and BIM. Their functions and implementation processes will not be described in detail here.

[0100] Thirdly, embodiments of this application provide a bridge component information retrieval method and device based on computer vision and BIM. The bridge component information retrieval method and device based on computer vision and BIM can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.

[0101] In this embodiment of the application, the bridge component information retrieval method and device based on computer vision and BIM may include a processor, a memory, a communication interface, and a communication bus.

[0102] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0103] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces. These interfaces are used for interconnecting internal components of the equipment used in the computer vision and BIM-based bridge component information retrieval method, as well as for interconnecting the equipment with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0104] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0105] The processor can be a general-purpose processor, which can call the bridge component information retrieval method program based on computer vision and BIM stored in the memory and execute the bridge component information retrieval method based on computer vision and BIM provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the bridge component information retrieval method program based on computer vision and BIM is called can refer to the various embodiments of the bridge component information retrieval method based on computer vision and BIM in this application, and will not be repeated here.

[0106] Fourthly, embodiments of this application also provide a computer-readable storage medium.

[0107] The present application stores a bridge component information retrieval method program based on computer vision and BIM on a computer-readable storage medium, wherein when the bridge component information retrieval method program based on computer vision and BIM is executed by a processor, the steps of the bridge component information retrieval method based on computer vision and BIM as described above are implemented.

[0108] The method implemented when the bridge component information retrieval method based on computer vision and BIM is executed can be referred to in the various embodiments of the bridge component information retrieval method based on computer vision and BIM in this application, and will not be repeated here.

[0109] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0110] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0111] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0112] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0113] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0115] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A bridge component information retrieval method based on computer vision and BIM, characterized in that, The bridge component information retrieval method based on computer vision and BIM includes: Image data of the bridge to be searched is obtained, and the image data is input into a computer vision cascade model to obtain the category labels and appearance defect features of the candidate components. The candidate components are associated with the components in the BIM model of the bridge to be searched. The target components corresponding to the candidate components are selected from the associated components by using preset filtering rules, and the structured data of the target components are extracted. Based on the structured data, the category labels and appearance defect features of the candidate components corresponding to the target component, a retrieval request is constructed, and the retrieval request is input into a preset vector knowledge base to obtain unstructured knowledge fragments related to the retrieval request. The structured data and the unstructured knowledge fragments are fused and input into a large language model to generate a comprehensive decision report that includes a description of the target component's status, compliance analysis, and disposal recommendations. The computer vision cascaded model includes a component classification model and a defect detection model.

2. The bridge component information retrieval method based on computer vision and BIM according to claim 1, characterized in that, The acquisition of image data of the bridge to be retrieved includes: Collect multi-angle high-definition image data covering the bridge to be searched, the image data including high-definition video streams and still images; Each frame of the high-definition video stream and each still image is bound to the spatial location information and attitude angle parameters at the time of acquisition. The spatial location information includes latitude and longitude coordinates and elevation values. Image data carrying spatial location information and attitude angle parameters will be transmitted wirelessly to an edge computing server or stored locally.

3. The bridge component information retrieval method based on computer vision and BIM according to claim 2, characterized in that, The image data is input into a computer vision cascade model to obtain category labels for candidate components, including: Extract single-frame images from the image data or extract images from the video stream, preprocess the obtained images to adjust to a fixed resolution and normalize the pixel values, and organize them into batch tensor format; A component classification model is used to extract and fuse multi-scale features from the preprocessed image, and bounding box regression and class prediction are performed based on the fused features. The parameters of multiple candidate components are output, including the bounding box coordinates, class label and confidence score of the candidate component in the image. The parameters of multiple candidate components are filtered according to a first preset confidence threshold. Candidate components with confidence scores higher than the first preset confidence threshold are retained, and the category labels of the filtered candidate components are output as the category labels of multiple candidate components.

4. The bridge component information retrieval method based on computer vision and BIM according to claim 3, characterized in that, The image data is input into a computer vision cascaded model to obtain the appearance defect features of the candidate components, including: Based on the bounding box coordinates of each candidate component in the image, the candidate component region corresponding to the bounding box in the image data is cropped to obtain a local image of each candidate component. Each local image is preprocessed to a fixed resolution and pixel values ​​are normalized, and then organized into a batch tensor format; Based on the category label of each candidate component, the dedicated defect detection model corresponding to the category of each candidate component is called from the model library; A dedicated defect detection model for each candidate component is used to detect defects in the local image of each candidate component, and the defect parameters of multiple candidate components are output. The defect parameters include the defect category, defect location and defect confidence score of each candidate component. The defect parameters of multiple candidate components are filtered according to the second preset confidence threshold. Candidate components with confidence scores higher than the second preset confidence threshold are retained. The defect category and defect location of the filtered candidate components are output as the appearance defect features of multiple candidate components.

5. The bridge component information retrieval method based on computer vision and BIM according to claim 4, characterized in that, The component classification model uses a target detection model trained based on the YOLOv8 model, and the dedicated defect detection models all use defect detection models trained based on the YOLOv8 model.

6. The bridge component information retrieval method based on computer vision and BIM according to claim 2, characterized in that, The process of associating candidate components with components in the BIM model of the bridge to be searched, filtering out the target components corresponding to the candidate components from the associated components using preset filtering rules, and extracting the structured data of the target components includes: Based on the category labels of candidate components and their spatial location information in the image, all BIM components with the same category as the current candidate component are located in the BIM model of the bridge to be searched through a preset mapping relationship, and an association relationship is established. Calculate the spatial distance between the candidate component and each of its associated BIM components, and based on a preset distance threshold, select the BIM component with the smallest Euclidean distance that is lower than the preset distance threshold as the target component; Obtain the unique component identifier of the target component, access the database associated with the BIM model, and extract the structured data of the target component based on the unique component identifier.

7. The bridge component information retrieval method based on computer vision and BIM according to claim 6, characterized in that, The step of locating all BIM components of the same category as the current candidate component in the BIM model of the bridge to be searched, based on the category labels of the candidate components and their spatial location information in the image, through a preset mapping relationship, includes: Based on the category label of the candidate component and the spatial location information and attitude angle parameters when the image was acquired, the direction vector of the candidate component in the camera coordinate system is obtained through the direct georeferenced photogrammetry principle. The direction vector is solved into three-dimensional geographic coordinates in the WGS-84 coordinate system through rigid transformation of coordinate rotation and translation; Based on the coordinate transformation parameters of the bridge, the three-dimensional geographic coordinates are converted into three-dimensional spatial coordinates under the local engineering coordinate system of the bridge used in the BIM model of the bridge to be retrieved. Based on the preset mapping relationship and the three-dimensional spatial coordinates, determine the spatial partitioning of the current candidate component in the BIM model; Based on the category label of the current candidate component, locate all BIM components with the same category as the current candidate component in the spatial partition corresponding to the current candidate component.

8. The bridge component information retrieval method based on computer vision and BIM according to claim 6, characterized in that, Based on the structured data, the category labels and appearance defect features of the candidate components corresponding to the target component, a retrieval request is constructed. The retrieval request is input into a preset vector knowledge base to obtain unstructured knowledge fragments related to the retrieval request, including: The category labels of candidate components, appearance defect features, and structured data of the target component are combined to generate query statements in natural language format. The query statement is converted into a query vector using a text vector embedding model. The query vector is then input into a preset vector knowledge base, and the similarity between the query vector and each knowledge fragment vector stored in the preset vector knowledge base is calculated. Based on the similarity ranking from high to low, at least one knowledge fragment with a similarity higher than a preset threshold is selected as the output unstructured knowledge fragment related to the retrieval request. The unstructured knowledge fragments include industry standard clauses, technical manual content, and historical engineering cases.

9. The bridge component information retrieval method based on computer vision and BIM according to claim 8, characterized in that, The process involves fusing the structured data with the unstructured knowledge fragments and inputting the result into a large language model to generate a comprehensive decision report that includes a description of the target component's state, compliance analysis, and handling recommendations. Input prompt words for a large language model are constructed based on the structured data and the unstructured knowledge fragments; The input prompts are fed into the big language model, which generates a comprehensive decision report that includes a description of the target component's status, compliance analysis, and handling recommendations. The structured data includes the ComponentID, design bearing capacity, material, construction and maintenance records of the bridge to be retrieved.

10. A bridge component information retrieval system based on computer vision and BIM, characterized in that, The bridge component information retrieval system based on computer vision and BIM includes: The acquisition module is used to acquire image data of the bridge to be searched, and input the image data into a computer vision cascade model to obtain the category labels and appearance defect features of the candidate components. The association module is used to associate candidate components with components in the BIM model of the bridge to be searched, filter out the target components corresponding to the candidate components from the associated components through preset filtering rules, and extract the structured data of the target components. The retrieval module is used to construct a retrieval request based on the structured data, the category labels of the candidate components corresponding to the target component, and the appearance defect features, input the retrieval request into a preset vector knowledge base, and obtain unstructured knowledge fragments related to the retrieval request. The generation module is used to fuse the structured data with the unstructured knowledge fragments and input them into the large language model to generate a comprehensive decision report that includes a description of the target component's status, compliance analysis, and disposal suggestions. The computer vision cascaded model includes a component classification model and a defect detection model.

Images