Railway vehicle image defect detection method and device

Abstract

The present application provides a kind of rail vehicle image defect detection method and device, can improve candidate detection stability and reduce repeat false alarm, comprising: obtaining rail vehicle inspection image and is divided into at least one image slice, construct different input scale multi-scale image;Input pre-training open vocabulary target detection model, based on the text prompt word corresponding to each defect category carries out candidate detection, obtains candidate detection result, back mapping, carries out fusion processing to candidate frame, obtains candidate defect object;According to the candidate defect category, the candidate review area obtained by adaptive expansion;Image features are extracted using visual language model, and the difference semantic review score is calculated based on similarity;The image features of candidate review area are respectively searched with normal sample feature library and difficult negative sample feature library, and sample library search result is obtained;Fusion detection confidence, difference semantic review score and sample library search result calculate final defect score, determine alarm level.

Description

Technical Field

[0001] This invention relates to the field of intelligent detection technology for rail transit, and in particular to a method and apparatus for detecting image defects in rail vehicles. Background Technology

[0002] The bottom, bogies and lower body of rail vehicles contain a large number of mechanical connectors, oil circuit components, cables, metal frames and rubber parts. Through-type line scanning equipment can form high-resolution long-width images when the vehicle passes through the detection area, and abnormalities such as foreign objects, oil leaks, broken or loose anti-loosening wires, and component damage can be detected by manual image reading or intelligent recognition programs.

[0003] Existing identification methods typically employ closed-class supervised object detection models. These methods rely on a large number of labeled samples and are capable of detecting known faults. However, they are prone to missed detections when the number of fault samples is small, the abnormal morphology varies greatly, or the vehicle structure changes. To improve recall, the detection confidence threshold is usually lowered in engineering practice, which in turn generates more false positives.

[0004] Another approach uses a concatenated object detection model and classification model to perform secondary classification on the candidate regions output by the object detection model. However, classification models typically still rely on fixed category labels and do not fully utilize the information in the rail vehicle inspection images, resulting in the following problems: foreign objects, oil leaks, loose wire anomalies, and component damage vary significantly in size and appearance, making it difficult for a single input scale to account for both small targets and large-scale anomalies; oil leaks are easily confused with shadows, metallic reflections, ordinary stains, rust, and normal oil marks; loose wire anomalies are easily confused with ordinary cables, bolt edges, welds, and background structures; and component damage is easily confused with normal holes, splicing seams, and rust textures; long line scan images usually require slice detection, and targets located at the edge of the slice are easily truncated, and the same target is also prone to repeated alarms in adjacent windows; historical normal samples and historical false alarm samples confirmed by manual verification are not continuously utilized, leading to repeated false alarms of the same type; and outputting alarms based solely on the confidence of a single model makes it difficult to balance candidate recall and false alarm suppression capabilities. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a method and apparatus for detecting defects in images of rail vehicles, which can improve the stability of candidate detection and reduce repeated false alarms without relying on pre-positioning of vehicle components.

[0006] The technical solution is as follows: a method for detecting defects in rail vehicle images, characterized in that the method includes: Acquire rail vehicle inspection images, and divide the rail vehicle inspection images into at least one image slice according to the sliding window method with overlapping areas. Construct at least two multi-scale images with different input scales for each image slice. The multi-scale image is input into a pre-trained open vocabulary object detection model, and candidate detection is performed based on text prompts corresponding to each defect category to obtain candidate detection results. The candidate detection results include candidate boxes, candidate defect categories, and detection confidence. Candidate boxes obtained from different image slices and different input scales are back-mapped to the coordinate system of the rail vehicle inspection image. Candidate boxes with the same defect category and similar positions are fused to obtain candidate defect objects. Based on the candidate defect category of the candidate defect object, the candidate box of the candidate defect object is adaptively expanded to obtain a candidate verification region containing the abnormal region and its surrounding structural information; Image features of the candidate verification region are extracted using a pre-trained visual language model. The similarity between the image features of the candidate verification region and the fault semantic features, normal semantic features and easily confused negative sample semantic features corresponding to the candidate defect category is calculated respectively. The differential semantic verification score is calculated based on the obtained similarity. The image features of the candidate verification region are compared with the normal sample feature library and the difficult negative sample feature library to obtain the sample library retrieval results; wherein, the normal sample feature library is constructed based on the image features of manually confirmed normal sample images, and the difficult negative sample feature library is constructed based on the image features of historical false alarm image regions that have been manually verified. By integrating the detection confidence score, differential semantic verification score, and sample library retrieval results, the final defect score of the candidate defect object is calculated, and the alarm level of the candidate defect object is determined based on the final defect score.

[0007] Furthermore, after acquiring the rail vehicle inspection image, the method further includes: performing at least one preprocessing step on the rail vehicle inspection image, including decoding, brightness correction, contrast correction, size unification, noise reduction, overexposure screening, blur screening, and rotation correction. The rail vehicle inspection image is divided into at least one image slice using a sliding window method with overlapping regions, including: for rail vehicle inspection images whose width or height exceeds a preset processing size, the image slice is divided using a sliding window method with overlapping regions, and the i-th image slice is represented as: ; In the formula, These are the original images of the rail vehicle inspection. Slice the i-th image. Let W be the coordinates of the top-left corner of the i-th image slice in the original track vehicle inspection image, and let W and H be the width and height of the sliding window, respectively. Overlap rate of adjacent sliding windows Represented as: ; In the formula, The overlap width between adjacent sliding windows is set according to the target size and computing resources.

[0008] Furthermore, for each of the image slices, at least two multi-scale images with different input scales are constructed, including: Construct a multi-scale set S = {s1, s2, ..., s} v}, v≥2, where v represents the total number of scales in the input image; the j-th input scale , Let the width of the image be at the j-th input scale. Let the image height be at the j-th input scale; The input image of the i-th image slice at the j-th input scale. Represented as: ; The i-th image slice and the k-th candidate detection result at the j-th input scale output by the open vocabulary object detection model are represented as follows: ; In the formula, Let the coordinates be those of the candidate bounding boxes. Candidate defect categories, To test the confidence level.

[0009] Furthermore, the fault semantic features, the normal semantic features, and the easily confused negative sample semantic features are constructed in the following manner: For each defect category, a set of fault warning words, a set of normal warning words, and a set of easily confused negative sample warning words are constructed; wherein, the set of fault warning words for the c-th type of defect is represented as: ; In the formula, c is the defect category number. Let n be the nth text prompt word corresponding to the c-th type of defect, where n is the number of text prompt words corresponding to the c-th type of defect; Extract the text features of each text prompt word in the fault prompt word set and perform average aggregation or weighted aggregation to obtain the fault semantic features of the c-th type of defect: ; In the formula, For text feature extraction functions, The fault semantic features are for the c-th type of defect; The normal prompt word set and the easily confused negative sample prompt word set are processed in the same way to obtain the normal semantic features of the c-th type of defect. semantic features of easily confused negative samples .

[0010] Furthermore, candidate boxes obtained from different image slices and different input scales are back-mapped to the coordinate system of the rail vehicle inspection image, and candidate boxes with the same defect category and similar locations are fused, including: Let the scaling factor of the j-th input scale be... The coordinates of the candidate boxes after backmapping are represented as follows: ; In the formula, ( , )and( , ) are the coordinates of the top left and bottom right corners of the candidate box in the j-th input scale image, respectively. , () represents the coordinates of the top-left corner of the i-th image slice in the rail vehicle inspection image; Calculate the crossover ratio (CUP) between two candidate boxes with the same defect category: ; in, This represents a function for calculating the area of ​​a region. This represents the intersection region of two candidate boxes. This represents the union region of two candidate boxes. This represents the intersection-union ratio between two candidate boxes; When the cross-union ratio is greater than a preset cross-union ratio threshold, or the distance between the center points of two candidate boxes is less than a preset distance threshold, the two candidate boxes are determined to be the same candidate object, and the candidate boxes determined to be the same candidate object are merged. The position of the merged candidate box is calculated using the average or weighted average of the coordinates of each candidate box participating in the fusion. The category of the merged candidate defect is the category corresponding to the candidate box with the highest detection confidence. The detection confidence is the average, weighted average, or maximum value of the detection confidence of each candidate box participating in the fusion.

[0011] Furthermore, when calculating the position of the fused candidate box using the average value, the fused candidate box is represented as follows: ; In the formula, The candidate boxes are the merged boxes. Let q be the l-th candidate box participating in the fusion, and q be the number of candidate boxes participating in the fusion.

[0012] Furthermore, the defect categories include foreign objects, oil leaks, abnormalities in the anti-loosening wire, and breakage; Among them, the easily confused negative sample semantics of the foreign object category include cables, normal hanging structures, and shadows; the easily confused negative sample semantics of the oil leakage category include reflections, stains, rust, and normal oil marks; the easily confused negative sample semantics of the anti-loosening wire anomaly category include ordinary cables, metal edges, and bolt textures; and the easily confused negative sample semantics of the damage category include welds, normal holes, joints, and rust textures.

[0013] Furthermore, based on the candidate defect category of the candidate defect object, the candidate bounding box of the candidate defect object is adaptively expanded, including: Let the merged candidate boxes be = , ,in , The coordinates of the top left corner of the candidate box. Let W be the coordinates of the bottom right corner of the candidate box, and let W be the width and height of the candidate box. m and H m The expanded candidate verification region is represented as follows: ; In the formula, and R represents the lateral expansion coefficient and the longitudinal expansion coefficient of the c-th defect, respectively. m This is the m-th candidate verification region; Among them, the foreign object category, oil leakage category, and damage category are expanded symmetrically around the center of the candidate frame; the anti-loosening wire abnormality category is expanded with a preset expansion ratio that can cover the bolt and wire path.

[0014] Furthermore, image features of the candidate verification regions are extracted using a visual language model. The similarity between these image features and the fault semantic features, normal semantic features, and easily confused negative sample semantic features corresponding to the candidate defect categories is calculated. A differential semantic verification score is then calculated based on the similarity, including: The candidate verification region is input into a visual language model for image feature extraction. The image features of the candidate verification region are represented as follows: ; in This represents the m-th candidate review region. Image encoding function representing a visual language model. This represents the image feature vector corresponding to the candidate verification region.

[0015] Calculate the cosine similarity between the image features and the corresponding defect category's fault semantic features, normal semantic features, and easily confused negative sample semantic features, respectively: ; ; ; Calculate the differential semantic verification score:

[0016] In the formula, , and The weights are the semantic similarity of fault samples, normal samples, and easily confused negative samples.

[0017] Furthermore, the image features are compared with the normal sample feature library and the difficult negative sample feature library respectively to obtain the sample library retrieval results, including: The normal sample feature library is represented as The difficult negative sample feature library is represented as In the formula, N is the feature library of normal samples, and H is the feature library of difficult negative samples. For the features of the Mth normal sample, Features of the U-th historical false alarm sample; Perform nearest neighbor retrieval on the image features: ; ; Alternatively, take the average distance of the Top-K points: ; ; In the formula, distance(·) is the feature distance metric function, F m This represents the image feature vector corresponding to the candidate verification region. For the normal sample feature library N and F m The feature of the j-th nearest normal sample, For the difficult negative sample feature library H and F m The feature of the j-th nearest historical false alarm sample; K in the nearest neighbor search formula is the number of nearest neighbor searches. Generate a sample library suppression flag, Suppress, based on the sample library distance: when... <τ n or <τ h When τ = 1, it indicates that the candidate region has a high similarity to normal samples or historical false alarm samples, and the alarm priority needs to be reduced; otherwise, τ = 0, indicating that no obvious sample library similarity was found, and no additional suppression is performed; where τ n τ is the threshold for judging normal sample feature library. hThe threshold for judging the feature library of difficult negative samples; When the image feature is far from both the normal sample feature library and the difficult negative sample feature library and the corresponding differential semantic verification score is higher than the preset semantic threshold, sample library suppression is not triggered and it participates in the subsequent calculation of the final defect score.

[0018] Furthermore, the final defect score is calculated using the following formula: ; In the formula, P m The detection confidence level for candidate defect objects. G(·) is the differential semantic verification score, which is a monotonic function that converts the distance from the sample library into the degree of distance from the sample library, and λ1 to λ4 are the weights. Based on the final score and suppression results, a three-level result is output using a priority-constrained hierarchical judgment strategy: First, it is determined whether the candidate defect object meets the A-level automatic alarm condition. When S_final≥τ_A and Suppress=0, the candidate defect is judged as A-level. If the A-level condition is not met, the B-level manual review condition is further judged. When τ_B≤S_final<τ_A or Suppress=1, the corresponding candidate defect is judged as B-level. If none of the above conditions are met, it is judged as C-level. C-level is automatic suppression or low-priority record. In the formula, τ_A is the first scoring threshold, τ_B is the second scoring threshold, and τ_A>τ_B. Image features are extracted from candidate verification regions that have been manually verified as false alarms and then fed back into the difficult negative sample feature library to update the difficult negative sample feature library.

[0019] The present invention has the following advantages: This invention can directly process full-image or sliding-window images, reducing reliance on vehicle model templates and component positioning results, and adapting to different vehicle models, acquisition devices, and detection areas. It utilizes an open-vocabulary target detection model and multiple semantic prompts to obtain defect candidate regions, eliminating the need to accumulate a large number of labeled samples for each type of defect, and exhibiting good adaptability to low-frequency faults and novel anomalies. Overlapping sliding windows and multi-scale input are used to process slice edge truncation and target size differences, respectively. Candidate box back-mapping and cross-sliding-window fusion merge duplicate candidates for the same physical defect into candidate defect objects, reducing edge misses and duplicate alarms, and improving the stability of candidate region positions. Expanding candidate regions according to defect categories enhances visual semantic understanding. The model not only obtains local abnormal textures but also the surrounding fasteners, connection relationships, structural edges, or oil stain diffusion range, which helps distinguish between real anomalies and areas with similar local appearances but normal structural relationships. By using differential verification of fault semantics, normal semantics, and easily confused negative sample semantics, the probability of false alarms caused by reflections, shadows, stains, cables, welds, and normal structural textures can be reduced. This invention also establishes a normal sample feature library and a difficult negative sample feature library, which can utilize existing normal samples and historical false alarm samples confirmed by manual verification to reduce the recurrence of the same type of false alarms. Finally, by overlapping sliding window segmentation, multi-scale detection, and candidate box back-mapping fusion, the probability of missed detection at slice edges and repeated alarms for the same target is reduced. Attached Figure Description

[0020] Figure 1 This is an overall flowchart of the rail vehicle image defect detection method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the overlapping sliding window segmentation and multi-scale construction of the long image of the rail vehicle in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the adaptive expansion of candidate regions for foreign objects, oil leaks, loosening of anti-loosening wires, and damage in an embodiment of the present invention. Figure 4 This is a schematic diagram illustrating the differential verification of fault semantics, normal semantics, and easily confused negative sample semantics in the visual language model in an embodiment of the present invention; Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0021] See Figure 1 The present invention provides a method for detecting defects in images of rail vehicles, the method comprising: Step S1: Acquire the inspection image of the rail vehicle, and divide the inspection image of the rail vehicle into at least one image slice according to the sliding window method with overlapping areas. Construct at least two multi-scale images with different input scales for each image slice. Step S2: Input the multi-scale image into the pre-trained open vocabulary object detection model, perform candidate detection based on the text prompt words corresponding to each defect category, and obtain the candidate detection results, which include candidate boxes, candidate defect categories and detection confidence. Step S3: Map the candidate boxes obtained from different image slices and different input scales back to the coordinate system of the rail vehicle inspection image, and perform fusion processing on candidate boxes with the same defect category and similar positions to obtain candidate defect objects; Step S4: Based on the candidate defect category of the candidate defect object, adaptively expand the candidate bounding box of the candidate defect object to obtain a candidate verification region containing information about the abnormal region and its surrounding structure; Step S5: Extract image features of candidate verification regions using a pre-trained visual language model, calculate the similarity between the image features of candidate verification regions and the fault semantic features, normal semantic features, and easily confused negative sample semantic features corresponding to the candidate defect categories, and calculate the differential semantic verification score based on the obtained similarity. Step S6: Perform nearest neighbor retrieval on the image features of the candidate verification region with the normal sample feature library and the difficult negative sample feature library respectively to obtain the sample library retrieval results; wherein, the normal sample feature library is constructed based on the image features of manually confirmed normal sample images, and the difficult negative sample feature library is constructed based on the image features of historical false alarm image regions that have been manually verified. Step S7: Integrate the detection confidence score, differential semantic review score, and sample library retrieval results to calculate the final defect score of the candidate defect object, and determine the alarm level of the candidate defect object based on the final defect score.

[0022] This embodiment provides a method for detecting image defects in rail vehicles, which can be applied to rail vehicle through-type line scanning detection systems, as well as other image inspection systems, including: In step S1, an inspection image of the rail vehicle is acquired. The inspection image can be a long-format image formed by a line scan camera, an image from an area scan camera, an image of the vehicle's underside, an image of the bogie, or an image acquired by other inspection equipment. The image is then processed by at least one of the following methods: decoding, brightness correction, contrast correction, size unification, noise reduction, overexposure screening, blur screening, or rotation correction, in order to improve the input quality of subsequent inspection steps.

[0023] For images whose width or height exceeds the preset processing size, a sliding window method with overlapping areas is used for segmentation. (Refer to...) Figure 2 The line scan image of the rail vehicle is sequentially divided into multiple image slices along the length of the vehicle body, such as slice I1, slice I2, slice I3, etc., with overlapping areas retained between adjacent slices. The i-th image slice is represented as: ; In the formula, These are the original images of the rail vehicle inspection. Slice the i-th image. Let W be the coordinates of the top-left corner of the i-th image slice in the original track vehicle inspection image, and let W and H be the width and height of the sliding window, respectively. The overlap ratio of adjacent sliding windows is expressed as:

[0024] In the formula, This represents the overlap width between adjacent sliding windows. The overlap rate can be set according to the target size and computing resources, for example, from 10% to 40%, to prevent the target from being truncated when it is located at the slice boundary. As an example, the sliding window size W×H can be 2048×2048 pixels, and the overlap rate can be 20%; when the width and height of the inspected image do not exceed the preset processing size, the entire image can be processed as a single image slice.

[0025] By setting overlapping regions, targets located at the edge of a slice can be completely covered in adjacent slices, providing an image basis for suppressing missed detections at slice edges in subsequent candidate fusion steps.

[0026] In step S2, to address the issue of significant size differences in defects such as foreign objects, oil leaks, loosened wires, and damage, at least two different input scales are constructed for each image slice. The input scale sⱼ represents the spatial dimensions of the scaled image, including both width and height, and its representation is as follows:

[0027] In the formula, Let the width of the image be at the j-th input scale; Let the image height be at the j-th input scale. The multi-scale set is represented as: S={s1,s2,...,s v}, v≥2, v represents the total number of scales in the input image; The input image of the i-th image slice at the j-th scale is represented as: ; In the formula, Resize(·) represents the image scaling operation. (See also...) Figure 2 Each image slice is scaled to its respective input scale, forming a multi-scale input image. As an example, v can be 2 or 3. For instance, s1 maintains the original resolution of the slice to accommodate small targets, s2 scales the slice to 1280×1280 pixels, and s3 scales the slice to 640×640 pixels to cover large-scale anomalies.

[0028] Multi-scale images are input into a pre-trained open-vocabulary object detection model, and candidate defect regions are obtained through textual prompts corresponding to foreign objects, oil leaks, loose wire defects, and damage. An open-vocabulary object detection model is an object detection model that can detect corresponding targets in an image based on textual semantic descriptions, without being limited by a pre-fixed set of categories. This model can be the OV-DINO model or other open-vocabulary detection models capable of detecting targets based on textual semantics.

[0029] The output of the open vocabulary object detection model, specifically the i-th slice and the k-th candidate detection result at the j-th scale, is represented as follows: ; In the formula, The coordinates of the candidate bounding box; Candidate defect categories; To assess confidence, a recall-priority initial screening threshold is used in the candidate detection stage. This means that a lower confidence threshold is used to retain as many suspected regions as possible, and subsequent modules will review and suppress false alarms. In this example, the initial screening confidence threshold can be set to 0.10–0.25.

[0030] In this embodiment, this step utilizes the text semantic generalization capability of the open vocabulary target detection model to obtain high-recall candidate regions without relying on a large number of fault-labeled samples, vehicle model templates, or component pre-location, making the method more adaptable to low-frequency faults, novel anomalies, and changes in vehicle model structure.

[0031] For each defect category, a set of fault warning words, a set of normal warning words, and a set of easily confused negative sample warning words are constructed. Taking the c-th type of defect as an example, its fault warning word set is represented as follows: ; In the formula, c is the defect category number. Let n be the nth text prompt word corresponding to the c-th type of defect, where n is the number of text prompt words corresponding to the c-th type of defect; Extract the text features of each text prompt word in the fault prompt word set and perform average aggregation or weighted aggregation to obtain the fault semantic features of the c-th type of defect: ; In the formula, This is a text feature extraction function, implemented by the text encoder of the visual language model; The fault semantic features are for the c-th type of defect.

[0032] Similarly, normal semantic features are obtained. semantic features of easily confused negative samples Among them, the negative sample semantics of the foreign object category can include cables, normal hanging structures, and shadows; the oil leakage category can include reflections, stains, rust, and normal oil marks; the anti-loose wire anomaly category can include ordinary cables, metal edges, and bolt textures; and the damage category can include welds, normal holes, joints, and rust textures.

[0033] As an example in this embodiment, fault warning words for the oil leak category may include text describing oil seepage marks, oil drips, oil diffusion, and wet reflections; normal warning words for the oil leak category may include text describing dry, clean metal surfaces and normally painted component surfaces; easily confused negative sample warning words for the oil leak category may include text describing metal surface reflections, component surface shadows, ordinary stains, rust marks, and normal oil residue from assembly. Warning words for other defect categories can be constructed in the same manner.

[0034] In step S3 of the embodiment, the candidate boxes obtained at different slices and scales are mapped back to the original inspection image coordinate system. Let the scaling factor at the j-th scale be rⱼ, then the mapped coordinates of the candidate boxes are expressed as: ; In the formula, ( , )and( , ) are the coordinates of the top left and bottom right corners of the candidate box in the j-th input scale image, respectively. , ) represents the coordinates of the top-left corner of the i-th image slice in the track vehicle inspection image; scaling factor. You can follow Calculate the ratio of the image width to the sliding window width at the j-th scale.

[0035] For candidate boxes of the same category and in close proximity, the intersection-union ratio (IUU) of two candidate boxes is expressed as: ; in, This represents a function for calculating the area of ​​a region. This represents the intersection region of two candidate boxes. This represents the union region of two candidate boxes. This represents the intersection-union ratio between two candidate boxes; When the intersection-union ratio (IU) is greater than a preset threshold, or the distance between the center points of candidate boxes is less than a preset threshold, the candidate boxes are determined to be the same candidate object. Candidate boxes that meet the matching conditions are merged, combining multiple candidate boxes into one candidate defect object. The position of the merged candidate box is calculated using the average or weighted average of the coordinates of each candidate box, and the category is the category with the highest confidence. The fusion confidence can be calculated using the average, weighted average, or maximum value. In this embodiment, the average is used for calculation, and the merged candidate box is represented as follows: ; In the formula, The candidate boxes are the merged boxes. Let q be the l-th candidate box participating in the fusion, and q be the number of candidate boxes participating in the fusion.

[0036] This step performs unified back-mapping and fusion processing on candidate boxes from different input scales and candidate boxes from overlapping regions of adjacent sliding windows. In other words, it fuses detection results from different scales and overlapping regions. Through the fusion processing in this embodiment, repeated detection results of the same target in adjacent sliding windows and repeated detection results at different scales are grouped into a single candidate defect object. This can suppress duplicate alarms caused by long image tiling and improve the stability of candidate region detection, ensuring that occasional false detections at a single scale do not dominate subsequent judgments.

[0037] Open-vocabulary detection outputs candidate boxes that often only cover local anomaly regions, lacking the surrounding structural information needed to determine anomalies. Therefore, in step S4 of this embodiment, the fused candidate boxes are adaptively expanded according to the defect category. Let the fused candidate boxes be... = , ,in , The coordinates of the top left corner of the candidate box. Let W be the coordinates of the bottom right corner of the candidate box, and let W be the width and height of the candidate box. m and H m The expanded candidate verification region is represented as follows: ; In the formula, and R represents the lateral expansion coefficient and the longitudinal expansion coefficient of the c-th defect, respectively. m This is the m-th candidate verification region.

[0038] In this embodiment, corresponding extended parameters are set according to the visual characteristics of different defect categories. (Refer to...) Figure 3 : (1) Foreign objects are usually small in size and have irregular shapes. They are expanded symmetrically around the center of the candidate box. (2) The oil leakage pattern is not fixed, and the leakage is symmetrically expanded around the center of the candidate frame. (3) The anti-loosening wire anomaly needs to include information on the wire, bolt and connection structure at the same time. Therefore, a larger context expansion along the wire direction, including the bolt and wire path, is adopted. (4) The damaged target mainly focuses on the changes in the structural edge, and adopts a symmetrical expansion around the candidate box center. Figure 3 In the diagram, solid lines represent the merged candidate bounding boxes, and dashed lines represent the expanded candidate verification regions. The expansion amount is determined by the horizontal expansion amount. and vertical expansion Confirmed. As an example, the expansion factor for the categories of foreign matter, oil leaks, and breakage. and The coefficient can be 0.2 to 0.5; the expansion coefficient of the anti-loosening wire abnormality category along the wire direction can be 0.1 to 0.3, and the expansion coefficient perpendicular to the wire direction can be 0.1.

[0039] Through adaptive expansion in the implementation steps, the candidate verification region simultaneously includes information about the abnormal area and the surrounding structure, enabling subsequent visual semantic verification to make judgments based on contextual information such as attachment relationships, connection relationships, and structural integrity. For example, judging the abnormality of the anti-loosening wire requires observing the connection relationship between the wire and the bolt. It is difficult to distinguish between a broken wire and a normal cable based solely on the local image within the candidate box. The expanded candidate verification region provides the necessary image information for this judgment.

[0040] In step S5 of this embodiment, the expanded candidate verification region is input into a visual language model to extract image features. The visual language model refers to a model that maps image features and text features to a unified feature space and can measure the semantic association between images and text through similarity. It can be the EVA-CLIP model or other models with a unified feature space for images and text. In this embodiment, the candidate verification region features are represented as follows: ; in This represents the m-th candidate review region. Image encoding function representing a visual language model. This represents the image feature vector corresponding to the candidate verification region.

[0041] Reference Figure 4 The candidate verification regions are processed by the image encoder of the visual language model to obtain image features F. m The fault prompt word set, the normal prompt word set, and the easily confused negative sample prompt word set are respectively processed by a text encoder to extract features and then aggregated to obtain fault semantic features. Normal semantic features semantic features of easily confused negative samples Calculate the similarity between the candidate verification region and the corresponding category of fault semantics, normal semantics, and easily confused negative sample semantics, respectively: ; ; ; In the formula, cos(·) represents the cosine similarity calculation function. Further, the difference semantic verification score is defined as follows: Calculate the differential semantic verification score:

[0042] In the formula, , and The weights are the semantic similarity of fault samples, normal samples, and easily confused negative samples.

[0043] The closer the candidate verification region is to the fault semantics, the higher the S_clip; the closer it is to normal semantics or easily confused negative sample semantics, the lower the S_clip. To improve adaptability in different scenarios, semantic verification weights are adjusted. , and It is not limited to a fixed value; it can be optimized and determined based on the statistical results of the verification samples, and dynamically adjusted according to different fault categories, detection areas, and false detection situations.

[0044] Compared to the method of one-way matching using only fault semantics, the differential semantic verification in this embodiment explicitly introduces normal semantics and easily confused negative sample semantics as control items. For interference content similar to the appearance of faults, such as reflections and shadows similar to oil leaks, and ordinary cables similar to anti-loosening wires, the similarity with easily confused negative sample semantics is high, and the score after differential is suppressed. Thus, false alarms caused by reflections, shadows, stains, cables, welds and normal structural textures can be suppressed at the semantic level.

[0045] In step S6 of the embodiment, During system initialization, manually verified normal sample images are input into the EVA-CLIP model or other visual language models to extract corresponding image features and establish a normal sample feature library. Simultaneously, false alarm image regions, manually verified during historical detection processes, are input into the visual language model to extract corresponding image features and establish a hard negative sample feature library. The two sample libraries are represented as follows:

[0046]

[0047] In the formula, N is the feature library of normal samples, and H is the feature library of difficult negative samples. For the features of the Mth normal sample, Features of the U-th historical false alarm sample; Nearest neighbor retrieval is performed on the features of the candidate verification region: ; ; Alternatively, the average distance of the Top-K points can be used: ; ; In the formula, distance(·) is the feature distance metric function, F m This represents the image feature vector corresponding to the candidate verification region. For the normal sample feature library N and F m The feature of the j-th nearest normal sample, For the difficult negative sample feature library H and F m The feature of the j-th nearest historical false alarm sample, K in the nearest neighbor retrieval formula is the number of nearest neighbors retrieved; as an example in the embodiment, distance(·) can be Euclidean distance or cosine distance defined by 1 minus cosine similarity; the number of nearest neighbors retrieved K can be 3 to 10.

[0048] The smaller the distance, the more similar the candidate verification region is to the corresponding sample library. A sample library suppression rule is defined as follows: when... <τ n or <τ h When τ is constant, Suppress = 1; otherwise, Suppress = 0. Where τ... n τ is the threshold for judging normal sample feature library. h τ is the threshold for determining the hard negative sample feature library. n and τ h The distance distribution of normal samples, historical false alarm samples and real defect samples on the validation set can be statistically determined, for example, by taking the distance value corresponding to the overlapping area of ​​the two distributions.

[0049] Suppress=1 indicates that the candidate region has a high similarity to normal samples or historical false alarm samples, requiring a reduction in alarm priority; Suppress=0 indicates that no significant sample library similarity was found, and no additional suppression is performed. The sample library suppression flag is used to reduce the alarm priority of candidate results, rather than directly replacing the final defect score. When a candidate verification region is far from both sample libraries and the corresponding differential semantic verification score is higher than the preset semantic threshold, sample library suppression is not triggered, and it participates in the subsequent calculation of the final defect score.

[0050] By using nearest neighbor retrieval in the sample library, both normal samples and historical false alarm samples that have been manually verified are continuously utilized. Compared with methods that rely solely on model output, once a certain type of false alarm has been manually verified, its features are entered into the hard negative sample feature library. Subsequently, candidate regions of the same type are hit and suppressed in nearest neighbor retrieval, thereby reducing the recurrence of the same type of false alarm.

[0051] In step S7 of the embodiment, the final defect score is calculated by combining the candidate detection confidence score, the differential semantic verification score, and the sample library distance score. This is represented in the embodiment as follows: ; In the formula, P m The detection confidence level for candidate defect objects. The differential semantic verification score is G(·), which is a monotonic function that converts the distance from the sample library into the degree of distance from the sample library, and λ1 to λ4 are the weights; threshold logic, decision trees or other fusion methods can also be used, and are not limited to the above linear expression.

[0052] Among them, the sample library distance term is used to evaluate the degree of difference between the candidate region and normal samples and difficult negative samples. The larger the distance, the further away it is from the corresponding sample library, and the more it tends to be a real defect.

[0053] Based on the final score and suppression results, a tiered judgment strategy with priority constraints is adopted to output three levels of results. First, it is determined whether the candidate defect meets the conditions for Level A automatic alarm. If S_final ≥ τ_A and Suppress = 0, the candidate is classified as Level A. If the Level A conditions are not met, the Level B manual review conditions are further assessed. If τ_B ≤ S_final < τ_A or Suppress = 1, the candidate is classified as Level B. If none of the above conditions are met, it is classified as Level C. By setting different priority judgment orders for each level, the judgment conditions for each level are made independent, ensuring that each candidate defect corresponds to only one alarm level.

[0054] Level A results are automatically triggered by the alarm terminal; Level B results are entered into a manual review queue for manual confirmation of their authenticity; Level C results are automatically suppressed or retained as low-priority records. False alarm patches confirmed through manual review have their features extracted and fed back into the difficult negative sample feature library, forming a continuously updated closed loop of detection, review, and feedback. As the difficult negative sample feature library accumulates, similar false alarms are more easily detected and suppressed by nearest neighbor searches in subsequent detections, and the system's false alarm suppression capability continuously improves over time.

[0055] The differentiated handling methods for each stage of the treatment process for four types of defects—foreign objects, oil leaks, abnormal anti-loosening wires, and breakage—are summarized in Table 1 below:

[0056] Table 1 Differentiated treatment methods for four types of defects In a specific application example, the input is a long image acquired by a through-type line scanning device on a rail vehicle. The system segments the long image into image slices with overlapping regions, constructing multiple input scales for each slice. The OV-DINO model is used to output candidate boxes based on textual prompts indicating foreign objects, oil leaks, loose wires, and damage. The candidate boxes are backmapped to the original image, and duplicate candidates are fused. After expanding the candidate verification region according to the defect category, the EVA-CLIP model is used to calculate the semantic scores for faults, normal samples, and easily confused negative samples. The features of the candidate verification region are then compared with the features of normal samples and difficult negative samples using a Top-K nearest neighbor search. Finally, the detection confidence, differential semantic verification score, and sample library distance are fused to calculate the final defect score and determine the alarm level. The candidate detection results generated by the multi-scale and overlapping sliding windows have already undergone fusion processing during the candidate box fusion stage.

[0057] In this embodiment, open-vocabulary high-recall detection ensures candidate coverage, candidate box back-mapping fusion ensures long-image detection stability, candidate region adaptive expansion supplements contextual information through semantic judgment, differential semantic verification and dual-sample library nearest neighbor retrieval suppress false alarms from both semantic and instance perspectives, multi-source evidence fusion and alarm grading balance automation and detection reliability, and manual verification of false alarm feedback continuously improves system capabilities. All of the above steps work together to achieve the technical effect of reducing the false alarm rate while maintaining high candidate recall.

[0058] In an embodiment of the present invention, a computer device is also provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method for detecting defects in rail vehicle images.

[0059] The computer device can be a terminal, and its internal structure diagram can be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for detecting image defects in rail vehicles. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0060] Memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). Memory stores programs, and the processor executes these programs after receiving execution instructions.

[0061] A processor can be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. The processor can also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor.

[0062] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0063] In an embodiment of the present invention, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, implements the above-described method for detecting defects in rail vehicle images.

[0064] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, computer devices, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0065] The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, computer apparatuses, or computer program products according to embodiments of the invention. These computer program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams.

[0066] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in the flowchart.

[0067] In an embodiment of the present invention, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0068] In practical applications, the aforementioned computer program products include, but are not limited to: inspection equipment, smartphones, desktop computers, laptops, tablets, host computers, and server platforms, etc., without specific limitations.

[0069] The above provides a detailed description of the application of the rail vehicle image defect detection method, computer device, computer-readable storage medium, and computer program product provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for detecting defects in images of rail vehicles, characterized in that, The method includes: Acquire rail vehicle inspection images, and divide the rail vehicle inspection images into at least one image slice according to the sliding window method with overlapping areas. Construct at least two multi-scale images with different input scales for each image slice. The multi-scale image is input into a pre-trained open vocabulary object detection model, and candidate detection is performed based on text prompts corresponding to each defect category to obtain candidate detection results. The candidate detection results include candidate boxes, candidate defect categories, and detection confidence. Candidate boxes obtained from different image slices and different input scales are back-mapped to the coordinate system of the rail vehicle inspection image. Candidate boxes with the same defect category and similar positions are fused to obtain candidate defect objects. Based on the candidate defect category of the candidate defect object, the candidate box of the candidate defect object is adaptively expanded to obtain a candidate verification region containing the abnormal region and its surrounding structural information; Image features of the candidate verification region are extracted using a pre-trained visual language model. The similarity between the image features of the candidate verification region and the fault semantic features, normal semantic features and easily confused negative sample semantic features corresponding to the candidate defect category is calculated respectively. The differential semantic verification score is calculated based on the obtained similarity. The image features of the candidate verification region are compared with the normal sample feature library and the difficult negative sample feature library to obtain the sample library retrieval results; wherein, the normal sample feature library is constructed based on the image features of manually confirmed normal sample images, and the difficult negative sample feature library is constructed based on the image features of historical false alarm image regions that have been manually verified. By integrating the detection confidence score, differential semantic verification score, and sample library retrieval results, the final defect score of the candidate defect object is calculated, and the alarm level of the candidate defect object is determined based on the final defect score.

2. The method for detecting defects in rail vehicle images according to claim 1, characterized in that, After acquiring the inspection image of the rail vehicle, the method further includes: performing at least one preprocessing step on the inspection image of the rail vehicle, including decoding, brightness correction, contrast correction, size unification, noise reduction, overexposure screening, blur screening, and rotation correction. The rail vehicle inspection image is divided into at least one image slice using a sliding window method with overlapping regions, including: for rail vehicle inspection images whose width or height exceeds a preset processing size, the image slice is divided using a sliding window method with overlapping regions, and the i-th image slice is represented as: ； In the formula, These are the original images of the rail vehicle inspection. Slice the i-th image. Let W be the coordinates of the top-left corner of the i-th image slice in the original track vehicle inspection image, and let W and H be the width and height of the sliding window, respectively. Overlap rate of adjacent sliding windows Represented as: ； In the formula, The overlap width between adjacent sliding windows is set according to the target size and computing resources.

3. The method for detecting defects in rail vehicle images according to claim 1, characterized in that, For each of the image slices, construct at least two multi-scale images with different input scales, including: Construct a multi-scale set S = {s1, s2, ..., s} v }, v≥2, where v represents the total number of scales in the input image; the j-th input scale =( , ), Let the width of the image be at the j-th input scale. Let the image height be at the j-th input scale; The input image of the i-th image slice at the j-th input scale. Represented as: ； The i-th image slice and the k-th candidate detection result at the j-th input scale output by the open vocabulary object detection model are represented as follows: ； In the formula, Let the coordinates be those of the candidate bounding boxes. Candidate defect categories, To detect confidence levels, the open vocabulary target detection model employs a recall-priority initial screening confidence threshold.

4. The method for detecting defects in rail vehicle images according to claim 1, characterized in that, The fault semantic features, the normal semantic features, and the easily confused negative sample semantic features are constructed in the following manner: For each defect category, a set of fault warning words, a set of normal warning words, and a set of easily confused negative sample warning words are constructed; wherein, the set of fault warning words for the c-th type of defect is represented as: ； In the formula, c is the defect category number. Let n be the nth text prompt word corresponding to the c-th type of defect, where n is the number of text prompt words corresponding to the c-th type of defect; Extract the text features of each text prompt word in the fault prompt word set and perform average aggregation or weighted aggregation to obtain the fault semantic features of the c-th type of defect: ； In the formula, For text feature extraction functions, The fault semantic features are for the c-th type of defect; The normal prompt word set and the easily confused negative sample prompt word set are processed in the same way to obtain the normal semantic features of the c-th type of defect. semantic features of easily confused negative samples .

5. The method for detecting image defects in rail vehicles according to claim 1, characterized in that, Candidate boxes obtained from different image slices and different input scales are back-mapped to the coordinate system of the rail vehicle inspection image, and candidate boxes with the same defect category and similar location are fused, including: Let the scaling factor of the j-th input scale be... The coordinates of the candidate boxes after backmapping are represented as follows: ; In the formula, ( , )and( , ) are the coordinates of the top left and bottom right corners of the candidate box in the j-th input scale image, respectively. , () represents the coordinates of the top-left corner of the i-th image slice in the rail vehicle inspection image; Calculate the crossover ratio (CUP) between two candidate boxes with the same defect category: ； in, This represents a function for calculating the area of ​​a region. This represents the intersection region of two candidate boxes. This represents the union region of two candidate boxes. This represents the intersection-union ratio between two candidate boxes; When the cross-union ratio is greater than a preset cross-union ratio threshold, or the distance between the center points of two candidate boxes is less than a preset distance threshold, the two candidate boxes are determined to be the same candidate object, and the candidate boxes determined to be the same candidate object are merged. The position of the merged candidate box is calculated using the average or weighted average of the coordinates of each candidate box participating in the fusion. The category of the merged candidate defect is the category corresponding to the candidate box with the highest detection confidence. The detection confidence is the average, weighted average, or maximum value of the detection confidence of each candidate box participating in the fusion.

6. The method for detecting defects in rail vehicle images according to claim 5, characterized in that, When calculating the position of the merged candidate box using the average value, the merged candidate box is represented as follows: ； In the formula, The candidate boxes are the merged boxes. Let q be the l-th candidate box participating in the fusion, and q be the number of candidate boxes participating in the fusion.

7. The method for detecting image defects in rail vehicles according to claim 1, characterized in that, The defect categories include foreign objects, oil leaks, abnormalities in the anti-loosening wire, and breakage; Among them, the easily confused negative sample semantics for the foreign object category include cables, normal hanging structures, and shadows; the easily confused negative sample semantics for the oil leak category include reflections, stains, rust, and normal oil marks; and the abnormal category for loose wires... The semantics of easily confused negative samples include ordinary cables, metal edges, and bolt textures; the semantics of easily confused negative samples in the damage category include welds, normal holes, seams, and rust textures.

8. The method for detecting defects in rail vehicle images according to claim 7, characterized in that, Based on the candidate defect category of the candidate defect object, the candidate bounding box of the candidate defect object is adaptively expanded, including: Let the merged candidate boxes be = , ,in , The coordinates of the top left corner of the candidate box. Let W be the coordinates of the bottom right corner of the candidate box, and let W be the width and height of the candidate box. m and H m The expanded candidate verification region is represented as follows: ; In the formula, and R represents the lateral expansion coefficient and the longitudinal expansion coefficient of the c-th defect, respectively. m This is the m-th candidate verification region; Among them, the foreign object category, oil leakage category, and damage category are expanded symmetrically around the center of the candidate frame; the anti-loosening wire abnormality category is expanded with a preset expansion ratio that can cover the bolt and wire path.

9. The method for detecting image defects in rail vehicles according to claim 1, characterized in that, Image features of the candidate verification regions are extracted using a visual language model. The similarity between these image features and the fault semantic features, normal semantic features, and easily confused negative sample semantic features corresponding to the candidate defect categories is calculated. A differential semantic verification score is then calculated based on the similarity, including: The candidate verification region is input into a visual language model for image feature extraction. The image features of the candidate verification region are represented as follows: ； in This represents the m-th candidate review region. Image encoding function representing a visual language model. This represents the image feature vector corresponding to the candidate kernel region; Calculate the cosine similarity between the image features and the corresponding defect category's fault semantic features, normal semantic features, and easily confused negative sample semantic features, respectively: ； ； ； Calculate the differential semantic verification score: In the formula, , and The weights are the semantic similarity of fault samples, normal samples, and easily confused negative samples.

10. The method for detecting defects in rail vehicle images according to claim 1, characterized in that, The image features are compared with the normal sample feature library and the difficult negative sample feature library respectively to perform nearest neighbor search, and the sample library search results are obtained, including: The normal sample feature library is represented as follows: The difficult negative sample feature library is represented as follows: In the formula, N is the feature library of normal samples, and H is the feature library of difficult negative samples. For the features of the Mth normal sample, Features of the U-th historical false alarm sample; Perform nearest neighbor retrieval on the image features: ； ； Alternatively, take the average distance of the Top-K points: ； ； In the formula, distance(·) is the feature distance metric function, F m This represents the image feature vector corresponding to the candidate verification region. For the normal sample feature library N and F m The feature of the j-th nearest normal sample, For the difficult negative sample feature library H and F m The feature of the j-th nearest historical false alarm sample; K in the nearest neighbor search formula is the number of nearest neighbor searches. Generate a sample library suppression flag, Suppress, based on the sample library distance: when... <τ n or <τ h When τ = 1, it indicates that the candidate region has a high similarity to normal samples or historical false alarm samples, and the alarm priority needs to be reduced; otherwise, τ = 0, indicating that no obvious sample library similarity was found, and no additional suppression is performed; where τ n τ is the threshold for judging normal sample feature library. h The threshold for judging the feature library of difficult negative samples; When the image feature is far from both the normal sample feature library and the difficult negative sample feature library and the corresponding differential semantic verification score is higher than the preset semantic threshold, sample library suppression is not triggered and it participates in the subsequent calculation of the final defect score.

11. The method for detecting defects in rail vehicle images according to claim 1, characterized in that, The final defect score is calculated using the following formula: ； In the formula, P m The detection confidence level for candidate defect objects. G(·) is the differential semantic verification score, which is a monotonic function that converts the distance from the sample library into the degree of distance from the sample library, and λ1 to λ4 are the weights. Based on the final score and suppression results, a three-level result is output using a priority-constrained hierarchical judgment strategy: First, it is determined whether the candidate defect object meets the A-level automatic alarm condition. When S_final≥τ_A and Suppress=0, the candidate defect is judged as A-level. If the A-level condition is not met, the B-level manual review condition is further judged. When τ_B≤S_final<τ_A or Suppress=1, the corresponding candidate defect is judged as B-level. If none of the above conditions are met, it is judged as C-level. C-level is automatic suppression or low-priority record. In the formula, τ_A is the first scoring threshold, τ_B is the second scoring threshold, and τ_A>τ_B. Image features are extracted from candidate verification regions that have been manually verified as false alarms and then fed back into the difficult negative sample feature library to update the difficult negative sample feature library.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the rail vehicle image defect detection method as described in any one of claims 1 to 11.

13. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by the processor, it implements the rail vehicle image defect detection method as described in any one of claims 1 to 11.

14. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 11.

Images