Steel structure bridge corrosion detection and segmentation method and device

Abstract

The invention discloses a steel structure bridge corrosion detection segmentation method and device, and relates to the technical field of bridge detection, and the method comprises the steps: constructing a steel structure bridge corrosion target detection network YOLO-World based on a text encoder, a YOLO trunk detector and RepVL-PAN; giving an image sample and a corrosion text, and training the constructed steel structure bridge corrosion target detection network YOLO-World; and based on an image encoder, a prompt encoder and a mask decoder, constructing a segmentation model OfficientSAM, thereby realizing detection and segmentation of the steel structure bridge corrosion image. The detection accuracy and the overall efficiency can be remarkably improved.

Description

Technical Field

[0001] The present application relates to the technical field of bridge detection, and in particular to a method and device for detecting and segmenting corrosion of steel structure bridges. Background Art

[0002] Steel structure bridges are widely used in various bridge constructions due to their good mechanical properties and durability. However, due to long-term exposure to the natural environment, the steel structure of the bridge is easily affected by factors such as moisture, salt, and industrial pollution, leading to rust. Rust not only affects the appearance of the bridge, but also weakens its bearing capacity and threatens traffic safety. Traditional corrosion detection methods mainly rely on manual inspection and simple instrument detection, which have problems such as strong subjectivity, low efficiency, and poor accuracy.

[0003] Since traditional bridge infrastructure disease detection is mainly manual, it leads to the following defects: 1. Low efficiency. Manual detection usually takes a lot of time to complete, especially for large structures or wide areas, the detection process may be very slow; 2. High labor costs. Since the detection process is time-consuming and requires the participation of professional technicians, the labor cost is high; 3. Large detection blind spots. It is difficult for manual inspection to cover all areas, especially for parts that are difficult to reach or have obstructed vision, which are prone to detection blind spots; 4. High subjectivity. The results of manual inspection depend largely on the experience and skills of the inspectors, and there may be deviations in subjective judgment; 5. Inconvenient data recording. Data recording and organization during manual inspection are usually cumbersome, which is not conducive to subsequent analysis and archiving. Summary of the invention

[0004] The present application provides a steel structure bridge corrosion detection and segmentation method and device, which can significantly improve the accuracy and overall efficiency of detection.

[0005] In a first aspect, an embodiment of the present application provides a steel structure bridge corrosion detection and segmentation method, and the steel structure bridge corrosion detection and processing method includes:

[0006] Based on the text encoder, YOLO backbone detector and RepVL-PAN, we build the steel structure bridge corrosion target detection network YOLO-World;

[0007] Given image samples and rust text, train the constructed steel structure bridge rust target detection network YOLO-World;

[0008] Based on the image encoder, hint encoder and mask decoder, a segmentation model EfficientSAM is constructed to realize the detection and segmentation of corroded images of steel structure bridges.

[0009] In combination with the first aspect, in one embodiment,

[0010] The text encoder is used to convert the input text sequence into a vector representation of fixed dimension;

[0011] The YOLO backbone detector is used to extract image features;

[0012] The RepVL-PAN is used to enhance the representation of text and images by leveraging cross-modal fusion between image features and text embeddings.

[0013] In combination with the first aspect, in one embodiment, the YOLO backbone detector is used to extract image features, and the specific implementation steps include:

[0014] After the stem layer and the 3*3 convolution module, the feature map is initially extracted and dimensionally reduced.

[0015] The feature map is downsampled four times through the c2f module to generate the final output feature map of the c2f module;

[0016] The SPPF module captures multi-scale information and fuses features of different scales to generate the final output feature map.

[0017] In combination with the first aspect, in one embodiment, RepVL-PAN utilizes cross-modal fusion between image features and text embedding to enhance the representation of text and images, and the specific implementation steps include:

[0018] RepVL-PAN builds a feature pyramid through multi-scale image features based on a top-down or bottom-up path;

[0019] After top-down or bottom-up fusion, an improved text-guided cross-level partial layer is adopted to add text guidance to multi-scale image features;

[0020] We use image-aware information to enhance text embeddings and introduce Image-Pooling attention to aggregate image features to update text embeddings.

[0021] In combination with the first aspect, in one embodiment, the given image sample and the rust text are used to train the constructed steel structure bridge rust target detection network YOLO-World, specifically including:

[0022] Given image samples and rust text, the constructed steel structure bridge rust target detection network YOLO-World is trained, and YOLO-World outputs prediction results and annotations;

[0023] Based on task-aligned label assignment, the prediction results are matched with the true annotations, and each positive prediction result is assigned a text index as the classification label.

[0024] In combination with the first aspect, in one embodiment,

[0025] The image encoder is used to output image embedding;

[0026] The hint encoder is used to receive the box encoding information from YOLO-World, and combine it with the image embedding output by the image encoder and send it to the mask decoder;

[0027] The mask decoder is used to convert the encoded information of the image encoder and the hint encoder into a mask output.

[0028] In combination with the first aspect, in one embodiment, for the image encoder to output image embedding, the specific implementation steps include:

[0029] For the original image to be processed, fill it into a square according to the maximum side length, and then resize it to 1024*1024 to get a 1024*1024*3 image;

[0030] For a 1024*1024*3 image, use a 16*16, stride=16, and a two-dimensional convolution with the number of output channels equal to the patch bedding dimension to obtain a 768*64*64 feature map, and then adjust the dimension to obtain a 64*64*768 feature map;

[0031] Through the global_atten_indexex parameter, specify the block that needs to use global attention and the block that needs to use local window attention;

[0032] Among them, the neck part of the image encoder consists of two convolutional layers, which are 256*768*1*1 and 256*256*3*3 respectively, and the size of the image embedding output by the image encoder is 1*256*64*64.

[0033] In combination with the first aspect, in one embodiment, the prompt encoder receives the box encoding information from YOLO-World and combines it with the image embedding output by the image encoder and then sends it to the mask decoder, specifically including:

[0034] The prompt encoder receives the box encoding information from YOLO-World and uses the embed_boxes method to embed the box prompt from YOLO-World;

[0035] The obtained embedding of the box prompt is combined with the image embedding output by the image encoder and sent to the mask decoder;

[0036] Among them, the embed_boxes method is used to embed the box prompt from YOLO-World, including:

[0037] Increase the coordinates in boxes by 0.5 and move the boxes to the center of the pixel;

[0038] Process boxes into a tensor coords with a shape of (-1, 2, 2), and perform position encoding on the tensor coords to obtain corner_embedding;

[0039] Add the weight of point_embedding[2] to the 0th dimension in corner_embedding, add the weight of point_embedding[3] to the 1st dimension in corner_embedding, and return corner_embedding as the embedding of the box prompt.

[0040] In combination with the first aspect, in one embodiment, converting the encoding information of the image encoder and the hint encoder into a mask output specifically includes:

[0041] Predict a mask based on the output of the image encoder and the output of the hint encoder;

[0042] Call the predict_masks method, and predict the mask and mask quality according to the embedding of the box prompt. If multiple masks are returned, select all slices after the first dimension of the mask. Otherwise, select the slices of the first dimension of the mask, and finally output the returned mask and mask quality.

[0043] In a second aspect, an embodiment of the present application provides a steel structure bridge corrosion detection and segmentation device, the steel structure bridge corrosion detection and segmentation device comprising:

[0044] A construction module for building a steel bridge corrosion target detection network YOLO-World based on a text encoder, a YOLO backbone detector, and RepVL-PAN;

[0045] The training module is used to train the YOLO-World steel structure bridge corrosion target detection network given image samples and corrosion text;

[0046] The execution module is used to construct a segmentation model EfficientSAM based on an image encoder, a prompt encoder, and a mask decoder, thereby realizing the detection and segmentation of corrosion images of steel structure bridges.

[0047] The beneficial effects brought by the technical solution provided by the embodiments of the present application include:

[0048] (1) Combining YOLO-World and EfficientSAM, efficient target detection and fine region segmentation can be achieved in the detection and segmentation of steel structure bridge corrosion. YOLO-World provides fast target detection capabilities and can efficiently identify corroded areas, while EfficientSAM accurately segments these areas, significantly improving the accuracy and overall efficiency of detection. It can effectively solve the problems of low efficiency, high cost, and large detection blind spots existing in traditional manual detection and simple instrument detection methods.

[0049] (2) The automated detection in this application reduces manual intervention, improves detection coverage and consistency, reduces human errors, and efficiently utilizes computing resources. By introducing a lightweight attention mechanism and an optimized network structure, it effectively reduces computational complexity while ensuring high accuracy, improves the model's operational efficiency, and makes it more suitable for large-scale data processing in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flow chart of the steel structure bridge corrosion detection and segmentation method of this application;

[0051] Figure 2 This is a functional module diagram of the steel structure bridge corrosion detection and segmentation device of this application;

[0052] Figure 3 This is a schematic diagram of the hardware structure of the steel structure bridge corrosion detection and segmentation equipment in this application. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] On the first aspect, the embodiment of the present application provides a method for detecting and segmenting corrosion of steel structure bridges. Through the efficient target detection capability of geometric YOLO-World and the precise segmentation technology of EfficientSAM, an innovative solution is provided for the intelligent detection of corrosion of steel structure bridges. The YOLO-World algorithm, with its fast detection speed and good performance, can provide accurate detection results when dealing with corrosion detection tasks in complex scenarios, which is crucial for the maintenance of key infrastructure such as bridges. At the same time, the introduction of the EfficientSAM algorithm not only enables the presence of corrosion to be identified during detection, but also enables fine segmentation of the corroded area, which is of great significance for further analysis and evaluation.

[0055] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of the steel structure bridge corrosion detection and segmentation method of this application. Figure 1 As shown in Figure 1, the steel structure bridge corrosion detection and segmentation method includes:

[0056] S1: Based on the text encoder, YOLO backbone detector and RepVL-PAN, the steel structure bridge corrosion target detection network YOLO-World is constructed;

[0057] It should be noted that YOLO, or You Only Look Once, is an object detection algorithm; RepVL-PAN, or Re-parameterizable Vision-Language Path Aggregation Network, is a core network structure in YOLO-World, which improves the performance of object detection by fusing visual information and language information; YOLO-World is an open vocabulary detector.

[0058] In this application, a text encoder is used to convert an input text sequence into a vector representation of a fixed dimension; a YOLO backbone detector is used to extract image features; and RepVL-PAN is used to enhance the representation of text and images by utilizing cross-modal fusion between image features and text embedding.

[0059] For the text encoder, for a given rusted text, a Transformer text encoder pre-trained with CLIP (Contrastive Language-Image Pre-Training) is used to extract the corresponding text embedding. Compared with the plain text language encoder, the CLIP text encoder provides better visual semantic capabilities and can connect visual objects with text.

[0060] In this application, the YOLO backbone detector includes a CSPDarknet (a deep neural network model for computer vision tasks) backbone network as an image encoder. The YOLO backbone detector extracts image features, and the specific implementation steps include:

[0061] S101: After passing through the stem layer and the 3*3 convolution module, the feature map is subjected to preliminary feature extraction and dimensionality reduction;

[0062] That is, first pass through the stem layer (the initial layer, used to extract preliminary features) and use the 3*3 convolution module to perform preliminary feature extraction and dimensionality reduction on the feature map to reduce the number of parameters in the subsequent network;

[0063] S102: downsampling the feature map four times through the c2f module (vision module) to generate the final output feature map of the c2f module;

[0064] For the four-time downsampling of the feature map through the c2f module, the specific process is as follows: first, it consists of a convolution block, which receives the input feature map and generates an intermediate feature map, and then splits the feature map. The generated intermediate feature map is split into two parts, one part is directly passed to the final Concat block, and the other part is passed to multiple Bottleneck blocks for further processing. The feature maps input to these Bottleneck blocks are processed through a series of convolution, normalization and activation operations. Finally, the generated feature map will be spliced ​​with the directly passed part of the feature map in the Concat block. The spliced ​​feature map will be input into a final convolution block for further processing to generate the final output feature map of the c2f module.

[0065] S103: Capture multi-scale information through the SPPF (Spatial Pyramid Pooling Fast, used for pooling operations at different scales) module, and fuse features at different scales to generate the final output feature map.

[0066] In this application, RepVL-PAN uses cross-modal fusion between image features and text embedding to enhance the representation of text and images. The specific implementation steps include:

[0067] S201: RepVL-PAN builds a feature pyramid through multi-scale image features based on a top-down or bottom-up path;

[0068] RepVL-PAN follows a top-down and bottom-up path to build a feature pyramid through multi-scale image features; in addition, it uses text-guided T-CSPLayer and image pooling attention mechanism to further enhance the interaction between image features and text features, thereby improving the visual semantic representation ability of open vocabulary.

[0069] S202: after top-down or bottom-up fusion, an improved text-guided cross-level partial layer is used to add text guidance to the multi-scale image features;

[0070] Specifically, given a text embedding W j and image features X l , by adding max-sigmoid attention after the last bottleneck of the original c2f module, the text features are aggregated into image features:

[0071]

[0072] in, represents the image features, X l represents the original c2f module output image features, δ represents the sigmoid function, and T represents the matrix transpose;

[0073] S203: Enhance text embedding using image-aware information, introduce Image-Pooling (an attention mechanism based on image-level pooling) attention, aggregate image features to update text embedding. Use the maximum pooling on multi-scale features to obtain a 3*3 region, resulting in a total of 27 image blocks.

[0074] S2: Given image samples and rust text, train the constructed steel structure bridge rust target detection network YOLO-World;

[0075] Furthermore, in one embodiment, given the image sample and the rust text, the constructed steel structure bridge rust target detection network YOLO-World is trained, specifically including:

[0076] S201: Given an image sample and corrosion text, the constructed steel structure bridge corrosion target detection network YOLO-World is trained, and YOLO-World outputs prediction results and annotations;

[0077] S202: Based on the task alignment label assignment, the prediction results are matched with the true annotations, and a text index is assigned to each positive prediction result as a classification label.

[0078] Specifically, given an image sample and rusted text, YOLO-World outputs prediction results and annotations, uses task alignment label assignment to match the prediction results with the true annotations, and assigns a text index as a classification label to each positive prediction result. Then, based on vocabulary, the region-text contrast loss is constructed through the cross entropy between object-text similarity and object-text assignment, and then the bounding box regression can be performed on the loss and distributed focal loss.

[0079] S3: Based on the image encoder, hint encoder, and mask decoder, the segmentation model EfficientSAM is constructed to detect and segment the corroded steel bridge image. EfficientSAM is an open source project designed to improve the efficiency of semantic segmentation. It uses a mask pre-training strategy to achieve efficient segmentation of any object. The segmentation model EfficientSAM is used to perform segmentation based on the hint box information of YOLO-World.

[0080] It should be noted that, in this application, the image encoder is used to output image embedding; the prompt encoder is used to receive the box encoding information from YOLO-World, and combine it with the image embedding output by the image encoder and send it to the mask decoder; the mask decoder is used to convert the encoding information of the image encoder and the prompt encoder into a mask output.

[0081] Furthermore, in one embodiment, for the image encoder to output image embedding, the specific implementation steps include:

[0082] S301: For the original image to be processed, fill it into a square according to the maximum side length, and then resize it to 1024*1024 to obtain a 1024*1024*3 image;

[0083] S302: For a 1024*1024*3 image, use a 16*16, stride (step length) = 16, and the output channel number is the patch embedding (an embedding method used in natural language processing tasks) dimension of the two-dimensional convolution to obtain a 768*64*64 feature map, and then adjust the dimension to obtain a 64*64*768 feature map; the dimension of the patch embedding is 768, so after the convolution, it becomes a 768*64*64 feature map, and then adjust the dimension to become a 64*64*768 feature map;

[0084] S303: specify the block (convolution block) that needs to use global attention and the block that needs to use local window attention through the global_atten_indexex parameter (a parameter used to set the global index);

[0085] Specifically, through the global_atten_indexex parameter, you can flexibly specify which blocks use global attention and which blocks use local window attention, so as to strike a balance between capturing global information and reducing computational complexity.

[0086] Among them, the neck part of the image encoder consists of two convolutional layers, which are 256*768*1*1 and 256*256*3*3 respectively, and the size of the image embedding output by the image encoder is 1*256*64*64.

[0087] Furthermore, in one embodiment, the prompt encoder receives the box encoding information from YOLO-World and combines it with the image embedding output by the image encoder and then sends it to the mask decoder, specifically including:

[0088] S311: The prompt encoder receives the box encoding information from YOLO-World, and uses the embed_boxes (marked box encoding) method to embed the box prompt from YOLO-World;

[0089] S312: The obtained embedding of the box prompt is combined with the image embedding output by the image encoder and sent to the mask decoder;

[0090] Among them, the embed_boxes method is used to embed the box prompt from YOLO-World, including:

[0091] a: Increase the coordinates in boxes by 0.5 and move the boxes to the center of the pixel;

[0092] b: Process boxes into a tensor coords of shape (-1, 2, 2), and perform position encoding on the tensor coords to obtain corner_embedding (embedding vector); After processing boxes into a tensor coords of shape (-1, 2, 2), it contains the coordinates of the upper left corner and lower right corner of the box;

[0093] c: Add the weight of point_embedding[2] to the 0th dimension (upper left corner) in corner_embedding, add the weight of point_embedding[3] to the 1st dimension (lower right corner) in corner_embedding, and return corner_embedding as the embedding of the box prompt. point_embedding means embedding each token into a vector. The basic principle is to use a low-dimensional vector to represent an object.

[0094] The above implementation of the box prompt embedding encodes the coordinates of the upper left corner and lower right corner of the box, and adds the corresponding corner point embedding to obtain a richly expressive box prompt embedding.

[0095] Furthermore, in one embodiment, converting the encoded information of the image encoder and the hint encoder into a mask output specifically includes:

[0096] S321: predicting a mask according to an output of the image encoder and an output of the hint encoder;

[0097] S322: Call the predict_masks method, and according to the predicted mask and mask quality of the embedding of the box prompt, if multiple masks are returned, select all slices after the first dimension of the mask, otherwise, select the slices of the first dimension of the mask, and finally output the returned mask and mask quality. That is, responsively select the slices with IOU confidence, and finally output the returned mask and mask quality.

[0098] The steel structure bridge corrosion detection and segmentation method of the embodiment of the present application, combined with YOLO-World and EfficientSAM, can achieve efficient target detection and fine regional segmentation in the detection and segmentation of steel structure bridge corrosion; YOLO-World provides fast target detection capabilities and can efficiently identify corroded areas, while EfficientSAM accurately segments these areas, significantly improving the accuracy and overall efficiency of detection, and can effectively solve the problems of low efficiency, high cost, and large detection blind spots existing in traditional manual detection and simple instrument detection methods. The automated detection of the present application reduces human intervention, improves detection coverage and consistency, reduces human errors, and efficiently utilizes computing resources. By introducing a lightweight attention mechanism and an optimized network structure, while ensuring high precision, it effectively reduces the computational complexity and improves the operational efficiency of the model, making it more suitable for large-scale data processing in practical applications.

[0099] In a second aspect, an embodiment of the present application also provides a steel structure bridge corrosion detection and segmentation device.

[0100] In one embodiment, referring to Figure 2 , Figure 2 This is a functional module diagram of the steel structure bridge corrosion detection and segmentation device of this application. Figure 2 As shown, the steel structure bridge corrosion detection and segmentation device includes: a construction module, a training module, and an execution module.

[0101] The construction module is used to build the steel structure bridge corrosion target detection network YOLO-World based on the text encoder, YOLO backbone detector and RepVL-PAN; the training module is used to train the constructed steel structure bridge corrosion target detection network YOLO-World given image samples and corrosion text; the execution module is used to build the segmentation model EfficientSAM based on the image encoder, prompt encoder and mask decoder, so as to realize the detection and segmentation of steel structure bridge corrosion images.

[0102] In a third aspect, an embodiment of the present application provides a steel structure bridge corrosion detection and segmentation device, which can be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0103] Reference Figure 3 , Figure 3 The hardware structure diagram of the steel structure bridge corrosion detection and segmentation device involved in the embodiment of the present application is shown in FIG. In the embodiment of the present application, the steel structure bridge corrosion detection and segmentation device may include a processor, a memory, a communication interface, and a communication bus.

[0104] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0105] The communication interface includes input / output (I / O) interface, physical interface and logical interface, etc., which are used to realize the interconnection of devices inside the steel structure bridge corrosion detection and segmentation equipment, and the interface used to realize the interconnection of the steel structure bridge corrosion detection and segmentation equipment with other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display (Display), a keyboard (Keyboard), etc.

[0106] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0107] The processor may be a general-purpose processor, which may call the steel structure bridge corrosion detection segmentation program stored in the memory and execute the steel structure bridge corrosion detection segmentation method provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the steel structure bridge corrosion detection segmentation program is called may refer to the various embodiments of the steel structure bridge corrosion detection segmentation method of the present application, which will not be repeated here.

[0108] Those skilled in the art will understand that Figure 3 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0109] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0110] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0111] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0112] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0114] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A steel structure bridge corrosion detection and segmentation method, characterized in that: The steel structure bridge corrosion detection and treatment method comprises: Based on the text encoder, YOLO backbone detector and RepVL-PAN, we build the steel structure bridge corrosion target detection network YOLO-World; Given image samples and rust text, train the constructed steel structure bridge rust target detection network YOLO-World; Based on the image encoder, hint encoder and mask decoder, a segmentation model EfficientSAM is constructed to realize the detection and segmentation of corroded images of steel structure bridges.

2. A steel structure bridge corrosion detection and segmentation method as claimed in claim 1, characterized in that: The text encoder is used to convert the input text sequence into a vector representation of fixed dimension; The YOLO backbone detector is used to extract image features; The RepVL-PAN is used to enhance the representation of text and images by leveraging cross-modal fusion between image features and text embeddings.

3. A steel structure bridge corrosion detection and segmentation method as claimed in claim 2, characterized in that: For the YOLO backbone detector to extract image features, the specific implementation steps include: After the stem layer and the 3*3 convolution module, the feature map is subjected to preliminary feature extraction and dimensionality reduction; The feature map is downsampled four times through the c2f module to generate the final output feature map of the c2f module; The SPPF module captures multi-scale information and fuses features of different scales to generate the final output feature map.

4. A steel structure bridge corrosion detection and segmentation method as claimed in claim 2, characterized in that: RepVL-PAN uses cross-modal fusion between image features and text embedding to enhance the representation of text and images. The specific implementation steps include: RepVL-PAN builds a feature pyramid through multi-scale image features based on a top-down or bottom-up path; After top-down or bottom-up fusion, an improved text-guided cross-level partial layer is adopted to add text guidance to multi-scale image features; We use image-aware information to enhance text embeddings and introduce Image-Pooling attention to aggregate image features to update text embeddings.

5. A steel structure bridge corrosion detection and segmentation method as claimed in claim 1, characterized in that: Given the image samples and the rust text, the constructed steel structure bridge rust target detection network YOLO-World is trained, specifically including: Given image samples and rust text, the constructed steel structure bridge rust target detection network YOLO-World is trained, and YOLO-World outputs prediction results and annotations; Based on task-aligned label assignment, the prediction results are matched with the true annotations, and each positive prediction result is assigned a text index as the classification label.

6. A steel structure bridge corrosion detection and segmentation method as claimed in claim 1, characterized in that: The image encoder is used to output image embedding; The hint encoder is used to receive the box encoding information from YOLO-World, and combine it with the image embedding output by the image encoder and send it to the mask decoder; The mask decoder is used to convert the encoded information of the image encoder and the hint encoder into a mask output.

7. A steel structure bridge corrosion detection and segmentation method as claimed in claim 6, characterized in that: For the image encoder to output image embedding, the specific implementation steps include: For the original image to be processed, fill it into a square according to the maximum side length, and then resize it to 1024*1024 to get a 1024*1024*3 image; For a 1024*1024*3 image, use a 16*16, stride=16, and a two-dimensional convolution with the number of output channels equal to the patch bedding dimension to obtain a 768*64*64 feature map, and then adjust the dimension to obtain a 64*64*768 feature map; Use the global_atten_indexex parameter to specify the block that needs to use global attention and the block that needs to use local window attention; Among them, the neck part of the image encoder consists of two convolutional layers, which are 256*768*1*1 and 256*256*3*3 respectively, and the size of the image embedding output by the image encoder is 1*256*64*64.

8. A steel structure bridge corrosion detection and segmentation method as claimed in claim 7, characterized in that: The prompt encoder receives the box encoding information from YOLO-World and combines it with the image embedding output by the image encoder and feeds it into the mask decoder, specifically: The prompt encoder receives the box encoding information from YOLO-World and uses the embed_boxes method to embed the box prompt from YOLO-World; The obtained embedding of the box prompt is combined with the image embedding output by the image encoder and sent to the mask decoder; Among them, the embed_boxes method is used to embed the box prompt from YOLO-World, including: Increase the coordinates in boxes by 0.5 and move the boxes to the center of the pixel; Process boxes into a tensor coords with a shape of (-1, 2, 2), and perform position encoding on the tensor coords to obtain corner_embedding; Add the weight of point_embedding[2] to the 0th dimension in corner_embedding, add the weight of point_embedding[3] to the 1st dimension in corner_embedding, and return corner_embedding as the embedding of the box prompt.

9. A steel structure bridge corrosion detection and segmentation method as claimed in claim 8, characterized in that: Converting the encoded information of the image encoder and the hint encoder into mask output specifically includes: Predict a mask based on the output of the image encoder and the output of the hint encoder; Call the predict_masks method, and predict the mask and mask quality according to the embedding of the box prompt. If multiple masks are returned, select all slices after the first dimension of the mask. Otherwise, select the slices of the first dimension of the mask, and finally output the returned mask and mask quality.

10. A steel structure bridge corrosion detection and segmentation device, characterized in that: The steel structure bridge corrosion detection and segmentation device comprises: A construction module for building a steel bridge corrosion target detection network YOLO-World based on a text encoder, a YOLO backbone detector, and RepVL-PAN; The training module is used to train the YOLO-World steel structure bridge corrosion target detection network given image samples and corrosion text; The execution module is used to construct a segmentation model EfficientSAM based on an image encoder, a prompt encoder, and a mask decoder, thereby realizing the detection and segmentation of corrosion images of steel structure bridges.

Images