A steel structure bridge corrosion detection segmentation method and device

Abstract

The application discloses a steel structure bridge corrosion detection and segmentation method and device, and relates to the technical field of bridge detection. The method comprises the following steps: constructing a steel structure bridge corrosion target detection network YOLO-World based on a text encoder, a YOLO main detector and a RepVL-PAN; training the constructed steel structure bridge corrosion target detection network YOLO-World by giving an image sample and corrosion text; and constructing a segmentation model EfficientSAM based on an image encoder, a prompt encoder and a mask decoder, so as to realize the detection and segmentation of a steel structure bridge corrosion image. The application can significantly improve the accuracy and overall efficiency of detection.

Description

Technical Field

[0001] This application relates to the field of bridge inspection technology, specifically to a method and apparatus for detecting and segmenting corrosion in steel structure bridges. Background Technology

[0002] Steel structure bridges are widely used in various bridge constructions due to their excellent mechanical properties and durability. However, due to long-term exposure to the natural environment, the steel structure of bridges is susceptible to corrosion caused by factors such as moisture, salt, and industrial pollution. Corrosion not only affects the aesthetics of bridges but also weakens their load-bearing capacity, threatening traffic safety. Traditional corrosion detection methods mainly rely on manual inspection and simple instrument testing, which suffer from problems such as high subjectivity, low efficiency, and poor accuracy.

[0003] Traditional bridge infrastructure defect detection is primarily manual, which leads to the following drawbacks: 1. Low efficiency: Manual detection usually requires a significant amount of time, especially for large structures or extensive areas, where the process can be very slow; 2. High labor costs: Due to the time-consuming process and the need for specialized technical personnel, labor costs are high; 3. Large blind spots: Manual detection struggles to cover all areas, especially in hard-to-reach or obstructed areas, resulting in blind spots; 4. High subjectivity: The results of manual detection largely depend on the experience and skills of the inspectors, potentially leading to subjective judgment biases; 5. Inconvenient data recording: Data recording and organization during manual detection are typically cumbersome, hindering subsequent analysis and archiving. Summary of the Invention

[0004] This application provides a method and apparatus for detecting and segmenting corrosion in steel structure bridges, which can significantly improve the accuracy and overall efficiency of detection.

[0005] In a first aspect, embodiments of this application provide a method for detecting and segmenting corrosion in steel structure bridges, the method comprising:

[0006] Based on a text encoder, a YOLO backbone detector, and RepVL-PAN, a YOLO-World network for detecting corrosion targets on steel bridges was constructed.

[0007] Given image samples and rusted text, train the YOLO-World network for detecting rusted targets on steel bridges.

[0008] Based on an image encoder, a cue encoder, and a mask decoder, a segmentation model called EfficientSAM is constructed to achieve the detection and segmentation of rust images of steel bridge structures.

[0009] In conjunction with the first aspect, in one embodiment,

[0010] The text encoder is used to convert the input text sequence into a vector representation of fixed dimensions;

[0011] The YOLO backbone detector is used for image feature extraction;

[0012] The RepVL-PAN is used to enhance the representation of text and images by leveraging cross-modal fusion between image features and text embeddings.

[0013] In conjunction with the first aspect, in one embodiment, the extraction of image features for the YOLO backbone detector specifically includes the following steps:

[0014] The feature map is initially extracted and reduced in dimensionality by passing through the stem layer and a 3x3 convolution module;

[0015] The feature map is downsampled four times by the c2f module to generate the final output feature map of the c2f module;

[0016] The SPPF module captures multi-scale information and fuses features from different scales to generate the final output feature map.

[0017] In conjunction with the first aspect, in one embodiment, RepVL-PAN utilizes cross-modal fusion between image features and text embeddings to enhance the representation of text and images, specifically implementing the following steps:

[0018] RepVL-PAN builds a feature pyramid based on multi-scale image features using top-down or bottom-up paths.

[0019] After top-down or bottom-up fusion, an improved text-guided cross-level partial layer is used to incorporate text guidance into multi-scale image features;

[0020] We enhance text embedding by leveraging image-aware information, introducing Image-Pooling attention, and aggregating image features to update text embedding.

[0021] In conjunction with the first aspect, in one embodiment, the YOLO-World network for detecting corrosion targets on steel bridges is trained using the given image samples and rusted text, specifically including:

[0022] Given image samples and rust text, train the YOLO-World network for detecting rust targets on steel bridges. YOLO-World outputs prediction results and annotations.

[0023] Based on task-aligned label assignment, the prediction results are matched with the ground truth labels, and a text index is assigned as a classification label for each positive prediction result.

[0024] In conjunction with the first aspect, in one embodiment,

[0025] The image encoder is used to output image embedding;

[0026] The prompt encoder is used to receive box encoding information from YOLO-World, and combine it with the image embedding output by the image encoder before sending it into the mask decoder;

[0027] The mask decoder is used to convert the encoded information of the image encoder and the cue encoder into a mask output.

[0028] In conjunction with the first aspect, in one embodiment, the specific implementation steps for the image embedding output by the image encoder include:

[0029] For the original image to be processed, it is padded into a square according to the maximum side length, and then resized to 1024*1024, resulting in a 1024*1024*3 image;

[0030] For a 1024*1024*3 image, use a 16*16 two-dimensional convolution with stride=16 and output channels equal to the patch bedding dimension to obtain a 768*64*64 feature map. Then adjust the dimensions to obtain a 64*64*768 feature map.

[0031] The global_atten_indexex parameter specifies the blocks that need to use global attention and the blocks that need to use local window attention.

[0032] The neck part of the image encoder consists of two convolutional layers, namely 256*768*1*1 and 256*256*3*3, and the size of the image embedding output by the image encoder is 1*256*64*64.

[0033] In conjunction with the first aspect, in one embodiment, the process of the prompt encoder receiving box-encoded information from YOLO-World and combining it with the image embedding output by the image encoder before sending it to the mask decoder specifically includes:

[0034] The encoder receives box encoding information from YOLO-World and uses the embed_boxes method to embed the boxes from YOLO-World prompts.

[0035] The obtained bounding box prompt embedding is combined with the image embedding output by the image encoder and then fed into the mask decoder;

[0036] Specifically, the `embed_boxes` method is used to embed the prompt boxes from YOLO-World, including:

[0037] Increase the coordinates in the boxes by 0.5 and move the boxes to the center of the pixel;

[0038] The boxes are processed into tensors of shape (-1,2,2), and the tensors of shape are encoded to obtain corner_embedding.

[0039] Add the weight of point_embedding[2] to the 0th dimension of corner_embedding, add the weight of point_embedding[3] to the 1st dimension of corner_embedding, and return corner_embedding as the embedding of the box prompt.

[0040] In conjunction with the first aspect, in one embodiment, converting the encoded information of the image encoder and the cue encoder into a mask output specifically includes:

[0041] Predict the mask based on the outputs of the image encoder and the cue encoder;

[0042] The `predict_masks` method is called to predict the mask and mask quality based on the embedding of the bounding box prompt. If multiple masks are returned, all slices after the first dimension of the mask are selected; otherwise, slices of the first dimension of the mask are selected. Finally, the returned mask and mask quality are output.

[0043] Secondly, embodiments of this application provide a steel structure bridge corrosion detection and segmentation device, the steel structure bridge corrosion detection and segmentation device comprising:

[0044] The building blocks are used to construct the YOLO-World network for detecting corrosion targets on steel bridges, based on a text encoder, the YOLO backbone detector, and RepVL-PAN.

[0045] The training module is used to train the YOLO-World network for detecting corrosion targets on steel bridges, given image samples and rusted text.

[0046] The execution module is used to construct the segmentation model EfficientSAM based on the image encoder, cue encoder, and mask decoder, thereby realizing the detection and segmentation of rust images of steel bridge structures.

[0047] The beneficial effects of the technical solutions provided in this application include:

[0048] (1) Combining YOLO-World and EfficientSAM, efficient target detection and fine area segmentation can be achieved in the detection and segmentation of corrosion in steel structure bridges. YOLO-World provides fast target detection capabilities and can efficiently identify corrosion areas, while EfficientSAM accurately segments these areas, significantly improving the accuracy and overall efficiency of detection. It can effectively solve the problems of low efficiency, high cost and large blind spots in traditional manual detection and simple instrument detection methods.

[0049] (2) The automated detection in this application reduces human intervention, improves detection coverage and consistency, reduces human error, and makes efficient use of computing resources. By introducing a lightweight attention mechanism and an optimized network structure, it effectively reduces computational complexity and improves the computational efficiency of the model while ensuring high accuracy, making it more suitable for large-scale data processing in practical applications. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating the corrosion detection and segmentation method for steel structure bridges proposed in this application.

[0051] Figure 2 This is a schematic diagram of the functional modules of the steel structure bridge corrosion detection and segmentation device of this application;

[0052] Figure 3 This is a schematic diagram of the hardware structure of the steel structure bridge corrosion detection and segmentation equipment of this application. Detailed Implementation

[0053] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0054] Firstly, this application provides a method for detecting and segmenting corrosion in steel structure bridges. By combining the efficient target detection capabilities of geometric YOLO-World with the precise segmentation technology of EfficientSAM, it offers an innovative solution for the intelligent detection of corrosion in steel structure bridges. The YOLO-World algorithm, with its fast detection speed and good performance, can provide accurate detection results when handling corrosion detection tasks in complex scenarios, which is crucial for the maintenance of critical infrastructure such as bridges. At the same time, the introduction of the EfficientSAM algorithm enables not only the identification of corrosion but also the fine segmentation of the corroded area, which is of great significance for further analysis and evaluation.

[0055] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating the corrosion detection and segmentation method for steel structure bridges described in this application. Figure 1 As shown, the segmentation method for corrosion detection of steel structure bridges includes:

[0056] S1: Based on a text encoder, YOLO backbone detector and RepVL-PAN, construct the YOLO-World network for detecting corrosion targets on steel bridges;

[0057] It should be noted that YOLO, which stands for You Only Look Once, is an object detection algorithm; RepVL-PAN, which stands for Re-parameterizableVision-Language Path Aggregation Network, is a core network structure in YOLO-World that improves object detection performance by fusing visual and linguistic information; and YOLO-World is an open vocabulary detector.

[0058] In this application, a text encoder is used to convert the input text sequence into a fixed-dimensional vector representation; a YOLO backbone detector is used to extract image features; and RepVL-PAN is used to enhance the representation of text and images by utilizing cross-modal fusion between image features and text embeddings.

[0059] For the text encoder, given rusted text, the corresponding text embeddings are extracted using a Transformer text encoder pre-trained with CLIP (Contrastive Language-Image Pre-Training). Compared to a plain text language encoder, the CLIP text encoder provides better visual semantic capabilities, enabling the connection between visual objects and text.

[0060] In this application, the YOLO backbone detector includes a CSPDarknet (a deep neural network model for computer vision tasks) backbone network as an image encoder. The specific steps for extracting image features using the YOLO backbone detector include:

[0061] S101: The feature map is initially extracted and reduced in dimensionality through the stem layer and a 3*3 convolution module;

[0062] That is, firstly, the feature map is processed through a stem layer (the initial layer used to extract preliminary features) and then through a 3*3 convolution module to perform preliminary feature extraction and dimensionality reduction, so as to reduce the number of parameters in subsequent networks;

[0063] S102: The feature map is downsampled four times by the c2f module (visual module) to generate the final output feature map of the c2f module;

[0064] The specific process for performing four downsampling operations on the feature map using the c2f module is as follows: First, a convolutional block is used to receive the input feature map and generate an intermediate feature map. Then, the feature map is split into two parts. One part is directly passed to the final concat block, and the other part is passed to multiple Bottleneck blocks for further processing. The feature maps input to these Bottleneck blocks are processed through a series of convolution, normalization, and activation operations. Finally, the generated feature map is concatenated with the directly passed feature map in the concat block. The concatenated feature map is then input to a final convolutional block for further processing to generate the final output feature map of the c2f module.

[0065] S103: The SPPF (Spatial Pyramid Pooling Fast, used for pooling operations at different scales) module captures multi-scale information and fuses features from different scales to generate the final output feature map.

[0066] In this application, RepVL-PAN utilizes cross-modal fusion between image features and text embeddings to enhance the representation of text and images. The specific implementation steps include:

[0067] S201: RepVL-PAN builds a feature pyramid based on multi-scale image features using top-down or bottom-up paths;

[0068] RepVL-PAN follows both top-down and bottom-up approaches, building a feature pyramid using multi-scale image features. Furthermore, it uses a text-guided T-CSPLayer and an image pooling attention mechanism to enhance the interaction between image and text features, thereby improving the visual semantic representation capability of open vocabularies.

[0069] S202: After top-down or bottom-up fusion, an improved text-guided cross-level partial layer is used to incorporate text guidance into multi-scale image features;

[0070] Specifically, given the text embedding W j and image features X l By adding max-sigmoid attention after the last bottleneck of the original c2f module, text features are aggregated into image features.

[0071]

[0072] in, Representing image features, X l δ represents the output image features of the original c2f module, T represents the sigmoid function, and T represents the matrix transpose.

[0073] S203: Enhance text embedding by leveraging image-aware information. Introduce Image-Pooling (an attention mechanism based on image-level pooling) to aggregate image features and update text embeddings. Max pooling on multi-scale features is used to obtain 3*3 regions, resulting in a total of 27 image patches.

[0074] S2: Given image samples and rusted text, train the YOLO-World network for detecting rusted targets on steel bridges.

[0075] Furthermore, in one embodiment, given image samples and rusted text, the YOLO-World network for detecting rusted targets on steel bridges is trained, specifically including:

[0076] S201: Given image samples and rust text, train the constructed steel bridge rust target detection network YOLO-World, and YOLO-World outputs prediction results and annotations;

[0077] S202: Based on task-aligned label assignment, the prediction results are matched with the true labels, and a text index is assigned as a classification label for each positive prediction result.

[0078] Specifically, given image samples and rusted text, YOLO-World outputs predictions and annotations. Using task-aligned label assignment, the predictions are matched with the ground truth annotations, and each positive prediction is assigned a text index as a classification label. Furthermore, based on vocabulary, a region-text contrast loss is constructed using object-text similarity and the cross-entropy between object-text assignments. Bounding box regression can then be applied to this loss and the distributed focus loss.

[0079] S3: Based on an image encoder, cue encoder, and mask decoder, a segmentation model called EfficientSAM is constructed to detect and segment images of rusted steel bridges. EfficientSAM is an open-source project designed to improve the efficiency of semantic segmentation. It achieves efficient segmentation of any object by utilizing a mask pre-training strategy. The EfficientSAM segmentation model is used for segmentation based on cue information from YOLO-World.

[0080] It should be noted that in this application, the image encoder is used to output image embedding; the cue encoder is used to receive box encoding information from YOLO-World, and combine it with the image embedding output by the image encoder before sending it to the mask decoder; the mask decoder is used to convert the encoding information of the image encoder and the cue encoder into a mask output.

[0081] Furthermore, in one embodiment, the specific implementation steps for the image embedding output by the image encoder include:

[0082] S301: For the original image to be processed, padded into a square according to the maximum side length, and then resized to 1024*1024 to get a 1024*1024*3 image;

[0083] S302: For a 1024*1024*3 image, use a 16*16 two-dimensional convolution with a stride of 16 and an output channel number equal to the patch embedding dimension to obtain a 768*64*64 feature map. Then adjust the dimension to obtain a 64*64*768 feature map. The patch embedding dimension is 768, so after convolution, it becomes a 768*64*64 feature map, and adjusting the dimension again becomes a 64*64*768 feature map.

[0084] S303: The global_atten_indexex parameter (a parameter used to set the global index) specifies the blocks that should use global attention (convolutional blocks) and the blocks that should use local window attention.

[0085] Specifically, the global_atten_indexex parameter allows for flexible specification of which blocks use global attention and which blocks use local window attention, thus achieving a balance between capturing global information and reducing computational complexity.

[0086] The image encoder's neck consists of two convolutional layers, 256*768*1*1 and 256*256*3*3, and the image embedding output by the image encoder has a size of 1*256*64*64.

[0087] Furthermore, in one embodiment, the process of the prompt encoder receiving box encoding information from YOLO-World and combining it with the image embedding output by the image encoder before sending it to the mask decoder specifically includes:

[0088] S311: The encoder is prompted to receive box encoding information from YOLO-World and to embed the boxes from YOLO-World using the embed_boxes method.

[0089] S312: Combine the obtained bounding box prompt embedding with the image embedding output by the image encoder and send it into the mask decoder;

[0090] Specifically, the `embed_boxes` method is used to embed the prompt boxes from YOLO-World, including:

[0091] a: Increase the coordinates in the bounding box by 0.5 and move the bounding box to the pixel center;

[0092] b: Process the boxes into tensors of shape (-1,2,2), and encode the tensors of shape (-1,2,2) to obtain the corner_embedding (embedding vector); after processing the boxes into tensors of shape (-1,2,2), find the coordinates of the top left and bottom right corners of the boxes.

[0093] c: Add the weight of point_embedding[2] to the 0th dimension (top left corner) of corner_embedding, add the weight of point_embedding[3] to the 1st dimension (bottom right corner) of corner_embedding, and return corner_embedding as the embedding of the box prompt. point_embedding means embedding each token into a vector. The basic principle is to use a low-dimensional vector to represent an object.

[0094] The above implementation of the box prompt embedding encodes the coordinates of the top left and bottom right corners of the box and adds corresponding corner point embeddings, resulting in a richly expressive box prompt embedding.

[0095] Furthermore, in one embodiment, converting the encoded information of the image encoder and the cue encoder into a mask output specifically includes:

[0096] S321: Predict the mask based on the output of the image encoder and the output of the cue encoder;

[0097] S322: Call the `predict_masks` method. Based on the predicted mask and mask quality of the bounding box prompt's embedding, if multiple masks are returned, select all slices after the first dimension of the mask; otherwise, select slices from the first dimension of the mask. Finally, output the returned mask and mask quality. In other words, it responsively selects slices with IOU confidence and outputs the returned mask and mask quality.

[0098] The steel structure bridge corrosion detection and segmentation method of this application combines YOLO-World and EfficientSAM to achieve efficient target detection and fine region segmentation in the detection and segmentation of steel structure bridge corrosion. YOLO-World provides fast target detection capabilities, which can efficiently identify corrosion areas, while EfficientSAM accurately segments these areas, significantly improving the accuracy and overall efficiency of detection. It can effectively solve the problems of low efficiency, high cost, and large detection blind spots in traditional manual detection and simple instrument detection methods. Moreover, the automated detection of this application reduces manual intervention, improves detection coverage and consistency, reduces human error, and makes efficient use of computing resources. By introducing a lightweight attention mechanism and optimized network structure, it effectively reduces computational complexity and improves the computational efficiency of the model while ensuring high accuracy, making it more suitable for large-scale data processing in practical applications.

[0099] Secondly, embodiments of this application also provide a steel structure bridge corrosion detection and segmentation device.

[0100] In one embodiment, reference is made to Figure 2 , Figure 2 This is a schematic diagram of the functional modules of the corrosion detection and segmentation device for steel structure bridges in this application. Figure 2 As shown, the corrosion detection and segmentation device for steel structure bridges includes: a construction module, a training module, and an execution module.

[0101] The construction module is used to build the YOLO-World network for detecting rusted targets on steel bridges based on a text encoder, a YOLO backbone detector, and RepVL-PAN. The training module is used to train the YOLO-World network for detecting rusted targets on steel bridges given image samples and rusted text. The execution module is used to build the segmentation model EfficientSAM based on an image encoder, a cue encoder, and a mask decoder, thereby realizing the detection and segmentation of rusted images of steel bridges.

[0102] Thirdly, this application provides a steel structure bridge corrosion detection and segmentation device, which can be a personal computer (PC), laptop computer, server or other device with data processing capabilities.

[0103] Reference Figure 3 , Figure 3 This is a schematic diagram of the hardware structure of the steel structure bridge corrosion detection and segmentation device involved in the embodiments of this application. In this embodiment, the steel structure bridge corrosion detection and segmentation device may include a processor, a memory, a communication interface, and a communication bus.

[0104] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0105] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting internal components of the steel structure bridge corrosion detection and segmentation equipment, as well as interfaces used for interconnecting the equipment with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0106] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0107] The processor can be a general-purpose processor, which can call the steel structure bridge corrosion detection and segmentation program stored in the memory and execute the steel structure bridge corrosion detection and segmentation method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the steel structure bridge corrosion detection and segmentation program is called can be referred to in the various embodiments of the steel structure bridge corrosion detection and segmentation method of this application, and will not be repeated here.

[0108] Those skilled in the art will understand that Figure 3 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0109] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0110] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0111] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0112] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0114] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A steel structure bridge corrosion detection segmentation method, characterized in that, The steel structure bridge corrosion detection processing method comprises: Based on the text encoder, the YOLO main detector and the RepVL-PAN, a steel structure bridge corrosion target detection network YOLO-World is constructed; Given the image sample and the corrosion text, the constructed steel structure bridge corrosion target detection network YOLO-World is trained; Based on the image encoder, the prompt encoder and the mask decoder, a segmentation model EfficientSAM is constructed, so as to realize the detection and segmentation of the steel structure bridge corrosion image; The image encoder is used to output image embedding; The prompt encoder is used to receive the box encoding information from YOLO-World, and combine the image embedding output by the image encoder to send to the mask decoder; The mask decoder is used to convert the encoding information of the image encoder and the prompt encoder into mask output; For the image encoder output image embedding, the specific implementation steps include: For the to-be-processed original image, the maximum side length is supplemented to form a square, and then resized to 1024*1024 to obtain a 1024*1024*3 picture; For the 1024*1024*3 picture, a 16*16, stride=16, output channel number is patchembedding dimension two-dimensional convolution is used to obtain a 768*64*64 feature map, and then the dimension is adjusted to obtain a 64*64*768 feature map; Through the global_atten_indexex parameter, the block requiring global attention and the block requiring local window attention are specified. The neck part of the image encoder is composed of two convolution layers, which are 256*768*1*1 and 256*256*3*3 respectively, and the size of the image embedding output by the image encoder is 1*256*64*64.

2. The steel structure bridge corrosion detection and segmentation method according to claim 1, wherein: The text encoder is used to convert the input text sequence into a fixed-dimensional vector representation; The YOLO main detector is used to extract image features; The RepVL-PAN is used to enhance the representation of text and image by cross-modal fusion between image features and text embedding.

3. The method of claim 2, wherein the steel bridge structure corrosion detection segmentation method is characterized by, For the YOLO main detector to extract image features, the specific implementation steps include: After passing through the stem layer and through the 3*3 convolution module, the feature map is preliminarily extracted and reduced in dimension; Through the c2f module, the feature map is down-sampled four times to generate the final output feature map of the c2f module; Through the SPPF module, multi-scale information is captured, and features of different scales are fused to generate the final output feature map.

4. The method of claim 2, wherein the steel bridge structure corrosion detection segmentation method is characterized by, For the RepVL-PAN to enhance the representation of text and image by cross-modal fusion between image features and text embedding, the specific implementation steps include: The RepVL-PAN is based on a top-down or bottom-up path, and establishes a feature pyramid through multi-scale image features; After top-down or bottom-up fusion, an improved text-guided cross-level partial layer is used to add text guidance to multi-scale image features; Image perception information is used to enhance text embedding, and Image-Pooling attention is introduced to aggregate image features to update text embedding.

5. The method of claim 1, wherein the method is a steel bridge corrosion detection segmentation method, characterized by, The given image sample and rust text are used to train the constructed steel structure bridge rust target detection network YOLO-World, specifically including: The given image sample and rust text are used to train the constructed steel structure bridge rust target detection network YOLO-World, and YOLO-World outputs a prediction result and an annotation; Based on task-aligned label assignment, the prediction result is matched with the real annotation, and each positive prediction result is assigned a text index as a classification label.

6. The method for corrosion detection segmentation of a steel bridge structure according to claim 1, wherein, For the prompt encoder to receive box encoding information from YOLO-World and combine image embedding output by the image encoder and send it to the mask decoder, specifically including: The prompt encoder receives box encoding information from YOLO-World, and uses the embed_boxes method to embed the box prompt from YOLO-World; The obtained embedding of the box prompt is combined with the image embedding output by the image encoder and sent to the mask decoder; Wherein, the embed_boxes method is used to embed the box prompt from YOLO-World, specifically including: Increase the coordinates in boxes by 0.5, and move boxes to the pixel center; Process boxes into a tensor coords with a shape of (-1, 2, 2), and position encode the tensor coords to obtain corner_embedding; Increase the weight of point_embedding[2] in the 0th dimension of corner_embedding, and increase the weight of point_embedding[3] in the 1st dimension of corner_embedding, and return corner_embedding as the embedding of the box prompt.

7. The method of claim 6, wherein the method further comprises: For converting the encoding information of the image encoder and the prompt encoder into mask output, specifically including: According to the output of the image encoder and the output of the prompt encoder, predict the mask; Call the predict_masks method according to the predicted mask of the embedding of the box prompt and the mask quality, if multiple masks are returned, select all slices after the 1st dimension of the mask, otherwise, select the slice of the 1st dimension of the mask, and finally output the returned mask and mask quality.

8. A steel structure bridge corrosion detection segmentation device, characterized in that, The steel structure bridge rust detection segmentation device comprises: A construction module is configured to construct a steel bridge corrosion target detection network YOLO-World based on a text encoder, a YOLO backbone detector and a RepVL-PAN; A training module is configured to train the constructed steel bridge corrosion target detection network YOLO-World given an image sample and corrosion text; An execution module is configured to construct a segmentation model EfficientSAM based on an image encoder, a prompt encoder and a mask decoder, so as to realize detection and segmentation of a steel bridge corrosion image. The image encoder is configured to output image embedding. The prompt encoder is configured to receive box encoding information from the YOLO-World and combine the image embedding output by the image encoder to send to the mask decoder. The mask decoder is configured to convert the encoding information of the image encoder and the prompt encoder into mask output. For the image encoder outputting image embedding, the specific implementation steps include: For the original image to be processed, the maximum side length is supplemented to form a square, and then resized to 1024*1024 to obtain a 1024*1024*3 picture; For the 1024*1024*3 picture, a 16*16, stride=16, output channel number is a two-dimensional convolution of patchembedding dimensions to obtain a 768*64*64 feature map, and then adjust the dimensions to obtain a 64*64*768 feature map; The global_atten_indexex parameter is used to specify the blocks that need to use global attention and the blocks that need to use local window attention. The neck part of the image encoder is composed of two convolution layers, which are 256*768*1*1 and 256*256*3*3 respectively, and the size of the image embedding output by the image encoder is 1*256*64*64.

Images