Passenger car part fault detection method and system based on visual prompt and visual large model

Abstract

The invention discloses a passenger car component fault detection method and system based on visual prompt and a visual large model, and the method comprises the steps: carrying out the position marking of a target in each sample image for each type of component sample image set, splicing a panoramic image, and generating a mask image; describing the exception type of each type of part and the details of each exception type through the exception description text; extracting multi-scale image features of the panoramic image and the mask image, and fusing the multi-scale image features to obtain visual prompt features; extracting a multi-scale image feature of the to-be-detected part image, fusing the multi-scale image feature with the visual prompt feature to obtain a multi-scale image feature of the to-be-detected part, inputting the multi-scale image feature into the part positioning model, and positioning a target part image; and fusing the visual features of the target part image with the text features of the corresponding abnormal description text by using a visual large model, and predicting and outputting a fault detection result. According to the invention, precise part positioning and fault detection can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of bus component fault detection, and in particular to a bus component fault detection method and system based on visual prompts and a visual large model. Background Art

[0002] With the rapid development of the railway transportation industry and the continuous increase in passenger volume, the requirements for passenger car operational safety are becoming increasingly stringent. Passenger cars are complex in structure, containing a large number of components, from the running gear, brake components, to various chassis and suspension parts. Statistics show that a typical passenger train contains nearly 100 different types of components, from the wheels and bearings of the running gear to the brake discs and cylinders of the braking system to the chassis and suspension parts. This extensive component system makes fault detection a daunting task. Furthermore, each component has a wide range of potential faults due to different operating environments and stress conditions. For example, wheels may suffer from wear, cracks, and tread peeling; brake discs may experience thermal cracking and uneven wear. This requires detection methods that can accurately identify a variety of fault types.

[0003] With the continuous development and upgrading of bus technology, new components and new fault types continue to emerge. On the one hand, to improve performance and safety, new buses incorporate components made of new materials and processes. These new components have significantly different failure modes than traditional components. On the other hand, even traditional components can develop unprecedented failure modes over long-term operation and technological improvements. Current detection methods, whether based on traditional image processing or deep learning, often struggle to quickly adapt to the emergence of new components and new faults, lacking sufficient flexibility and scalability.

[0004] In bus component fault detection, fault target feature extraction and identification are fraught with difficulties. For subtle faults in key areas, such as tiny cracks on the surface of a component, current methods rely solely on a single visual image. However, because these faults occupy very few pixels in the image and their features are weak, they are easily overlooked, making it difficult to accurately extract effective features for fault diagnosis. Even if the overall fault of a large component, such as a deformed bogie, occupies a large area in the visual image, its complex structure makes fault feature extraction based solely on visual information extremely difficult. Furthermore, the complex operating environment of buses, with factors such as lighting and weather interfering with image quality, further increases the uncertainty of visual feature extraction, making it difficult to fully capture these features through visual images.

[0005] Traditional bus component fault detection methods mainly include support vector machines, Faster R-CNN, and YOLO algorithms. However, these methods face the challenges of dealing with nearly 100 types of bus components and complex backgrounds. These methods require repeated training for detecting new components, have poor adaptability to complex scenarios, and are difficult to accurately locate and analyze component anomalies from a large amount of redundant image information. Summary of the Invention

[0006] The present invention provides a bus component fault detection method and system based on visual prompts and visual large models to solve the problem that the existing technology is difficult to accurately and efficiently detect bus component faults in complex operating environments.

[0007] In a first aspect, a bus component fault detection method based on visual cues and a visual large model is provided, comprising the following steps: S1. For each type of component sample image set, mark the location of the target component in each component sample image, stitch the marked component sample image set to generate a panoramic image, and generate a mask image corresponding to the panoramic image; for each type of component, describe the abnormality type of the component and the details of each abnormality type in text to form an abnormality description text; S2. Use a convolutional neural network model to extract multi-scale image features of the panoramic image and mask image of each type of component, and then fuse the multi-scale image features of the panoramic image and mask image of each type of component to obtain the visual cue features of each type of component; S3. Use a convolutional neural network model to extract multi-scale image features of the image of the component to be inspected, and fuse them with the visual cue features of the same type of components to obtain the final multi-scale image features of the component to be inspected; S4, inputting the multi-scale image features of the component to be inspected into the component positioning model to locate and obtain the target component image; S5. Use a pre-trained visual large model to extract the visual features of the target component image and the text features of the abnormal description text related to the target component, and fuse the visual features with the text features to predict and output the fault detection results of the target component.

[0008] Furthermore, in step S1, stitching the labeled component sample image set to generate a panoramic image includes: Assume that the parts sample image set includes n parts sample images, and splice the n parts sample images into a panoramic image according to the row and column layout. The number of rows e and the number of columns f of the panoramic image satisfy ,and minimum; if , then fill the empty areas in the panoramic image.

[0009] Furthermore, in step S2, the multi-scale image features of the panoramic image and the mask image of each type of parts are fused to obtain the visual cue features of each type of parts, specifically including: For each type of component, the multi-scale image features of the panoramic image and the multi-scale image features of the mask image are multiplied bit by bit at different scales to generate image fusion features at different scales. The image fusion features at different scales are transposed, dimensionally transformed and spliced, and the feature channels are compressed to 1 to obtain the visual cue features of the current type of parts.

[0010] Furthermore, step S3 specifically includes: A convolutional neural network model is used to extract multi-scale image features of the image of the component to be inspected; The multi-scale image features of the image of the part to be inspected are transposed and dimensionally transformed, and the multi-scale image features of the image of the part to be inspected obtained by the transformation are fused with the visual cue features of the parts of the same type to obtain the fused features of the part to be inspected at multiple scales; The fused features of the parts to be inspected at multiple scales are transposed and dimensionally transformed, and finally a multi-scale image feature of the parts to be inspected with the same dimension as the multi-scale image feature of the initial images of the parts to be inspected is obtained.

[0011] Furthermore, the component localization model includes a path aggregation network, multiple C2f modules, multiple SPPF modules, and multiple detection heads; After the multi-scale image features of the parts to be inspected are input into the path aggregation network, the top-down and bottom-up image features are fused at each scale. Each fusion is performed using the C2f module for multi-scale fusion. After obtaining the fused image features of the parts to be inspected at each scale, they are input into various corresponding SPPF modules to obtain the enhanced fused image features of the parts to be inspected at each scale; The enhanced fused image features of the parts to be detected at each scale are input into the corresponding detection head to complete the positioning and extraction of the target parts.

[0012] Furthermore, in the training phase of the component positioning model, the total loss function used is the weighted sum of the classification loss and the positioning loss. The total loss function is expressed as follows: ; Where, represents the total loss, and Represents the classification loss and the weight of the positioning loss, which includes the bounding box distribution loss and geometric constraint loss .

[0013] Furthermore, before step S5, the method further includes: The located target component images are merged with overlapping target component images through non-maximum suppression to avoid repeated calculations.

[0014] Furthermore, in step S5, the process of fusing visual features with text features includes: S51: First, the self-attention mechanism is used to enhance the visual features and text features respectively; S52: Enhanced visual features and text features Fusion via Cross-Attention Mechanism: S521: First, construct the query vector based on the enhanced visual features, construct the key vector and value vector based on the enhanced text features, and introduce three learnable projection matrices 、 and , then the query vector is obtained by the following linear projection operation , key vector Sum vector : ; ; ; in, is the dimension of visual features, is the dimension of text features, The query vector projection matrix Dimensions, is the key vector projection matrix Dimensions, is the projection matrix of the value vector Dimensions; Calculate the first attention score matrix S: ; Where, Indicates that the softmax function is calculated along the column feature direction; The first fusion feature is obtained by multiplying the first attention score matrix S with the value vector V1 : ; S522: Constructing query vectors based on enhanced text features , construct key vector based on enhanced visual features Sum vector , using the same method as above S521, the second fusion feature is obtained : ;in According to the query vector and key vector The calculated second attention score matrix; S523: Fusing the first fusion feature with the second fusion feature through a gating mechanism to obtain a bidirectional fusion feature : ; in, is the learnable weight parameter.

[0015] Furthermore, in step S5, the fault detection result is output in the form of structured text, which includes the abnormality type, credibility, and whether the component type in the image of the component to be detected is consistent with the component type in the abnormality description text.

[0016] In a second aspect, a bus component fault detection system based on visual cues and a visual large model is provided, comprising: The visual prompt and exception description text construction module is used to mark the location of the target parts in each part sample image set for each type of part sample image, stitch the marked part sample image set together to generate a panoramic image, and generate a mask image corresponding to the panoramic image; for each type of part, the abnormal type of the part and the details of each abnormal type are described in text to form the abnormal description text; A visual cue feature extraction module is used to extract multi-scale image features of the panoramic image and mask image of each type of component using a convolutional neural network model, and then fuse the multi-scale image features of the panoramic image and mask image of each type of component to obtain the visual cue features of each type of component; The image feature extraction and fusion module for testing is used to extract the multi-scale image features of the image of the component to be tested using a convolutional neural network model, and fuse them with the visual cue features of the same type of components to obtain the final multi-scale image features of the component to be tested; The target component positioning module is used to input the multi-scale image features of the component to be inspected into the component positioning model to locate the target component image; The component fault detection module uses a pre-trained visual large model to extract the visual features of the target component image and the text features of the abnormal description text related to the target component, and fuses the visual features with the text features to predict and output the fault detection results of the target component.

[0017] The present invention proposes a bus component fault detection method and system based on visual prompts and visual large models, which has the following beneficial effects: 1. Compared with current object detection algorithms such as closed set detection and open word detection, this invention introduces visual cue information, which has the following advantages: (1) Enhanced adaptability to new component detection: The closed set detection method is limited to preset categories and cannot detect new components and new fault types. In addition, new components require redefining the category set and retraining the model, which is a cumbersome and inefficient process. The present invention constructs a visual cue component library and uses multi-view sample images to construct a "component-feature" mapping relationship library, which can cope with the emergence of new components, break through the limitations of closed set detection, expand the detection range, and enhance the adaptability and flexibility of new component detection. (2) Improved positioning accuracy: Open vocabulary detection does not fully extract the features of target parts in complex backgrounds, making it difficult to accurately locate target parts. The present invention enhances the features of specific parts by marking the positions of parts and constructing a mask composed of visual prompt information. This allows accurate positioning of bus parts even in complex backgrounds, thereby improving detection accuracy. 2. Compared with existing anomaly detection methods, this invention introduces anomaly corpus and a large visual model, which has the following advantages: (1) Compensate for the limitations of single modal information: Existing anomaly detection models mainly rely on single visual information and cannot fully extract subtle and complex fault features. This invention builds an anomaly description text library, combines image and text information, and uses text descriptions to supplement the deficiencies of visual information, which can more deeply extract fault features and accurately judge subtle and complex faults. (2) Improved adaptability to complex scenarios: The complex bus operating environment will interfere with the detection accuracy of the existing anomaly detection model. The present invention, with the help of a well-trained visual large model, makes full use of visual images and anomaly description text information, thereby enhancing the adaptability to complex scenarios. It can also stably and accurately detect bus component failures under interference such as lighting and weather. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is a flow chart of a bus component fault detection method based on visual cues and visual large models provided by an embodiment of the present invention; Figure 2This is a flow chart of the fusion of visual cue features and the output of positioning results for bus parts provided by an embodiment of the present invention; Figure 3 This is a flowchart of the bus parts image and text feature fusion and detection result output provided by an embodiment of the present invention; Figure 4 This is an example diagram of the visual inspection of bus parts and the structured output of the results provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the present invention more apparent, the technical solutions of the present invention will be described in detail below. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other implementations obtained by those of ordinary skill in the art without inventive effort are within the scope of protection of the present invention.

[0021] In the description of the present invention, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or order. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" is at least two.

[0022] For dynamic image detection of bus operating faults, traditional methods have the disadvantages of requiring repeated training for new component detection, poor adaptability to complex scenes, and difficulty in accurately locating and analyzing component anomalies from a large amount of redundant image information, etc., in the face of nearly a hundred types of bus components and complex backgrounds. In order to solve the problem of difficulty in accurately and efficiently detecting bus component faults in complex operating environments, the present invention proposes a bus component fault detection method and system based on visual cues and large visual models. By constructing a visual cue component library and using multi-view sample images to construct a "component-feature" mapping relationship library, it can cope with the emergence of a large number of running gear components and new components, and accurately locate components even in complex backgrounds. At the same time, by constructing an abnormality description text library, the image and text information are organically integrated to enhance adaptability to complex scenes, make up for the limitations of single visual information, deeply explore fault characteristics, and achieve accurate detection of subtle and complex faults of bus components. The technical solution of the present invention is described in detail below in conjunction with specific embodiments.

[0023] An embodiment of the present invention provides a bus component fault detection method based on visual cues and a visual large model, comprising the following steps: S1. Build a visual prompt parts library and an abnormality description text library.

[0024] For each type of parts sample image set, the position of the target parts in each part sample image is marked, and the marked parts sample image set is stitched to generate a panoramic image, and a mask image corresponding to the panoramic image is generated. The mask images corresponding to the global image set of all types of parts constitute a visual prompt parts library; for each type of parts, the abnormal type of the part and the details of each abnormal type are described in text to form an abnormal description text. The abnormal description texts of all types of parts constitute an abnormal description text library.

[0025] The following embodiment uses the example of using a rectangular frame to mark the position of the target component. , the position of the target component is manually determined using a rectangular position frame To accurately mark the position of parts, Clearly corresponds to the coordinates of the upper left corner of the rectangular position box, and The coordinates of the lower right corner of the rectangular position box are accurately represented, providing position information for subsequent image stitching and mask generation tasks.

[0026] All sample images of parts of a certain type Arrange them in rows and columns and stitch them together in an orderly manner to form a complete panoramic image Assume that the parts sample image set includes n parts sample images, and the splicing function is , The nth component sample image; first, the number of rows must be determined by precise calculation With the number of columns , number of rows With the number of columns Need to meet ,and Minimum, that is, give priority to making the number of rows and columns as close as possible, for example, by gradually increasing the number of columns or rows until the total number of images is met, thereby accurately determining the number of rows and columns; of course, if , then the vacant areas in the panoramic image need to be filled. The width of each component sample image in the stitched panoramic image is , the height is , here 、 are the total width and total height of the pre-set panoramic image respectively. sample images, their starting horizontal coordinates in the stitched image , starting vertical coordinate , Indicates the remainder operation, Indicates rounding down.

[0027] During the image stitching process, the position box in the original part sample image It is necessary to perform corresponding transformations according to the splicing rules to obtain the transformed position frame . Assume that the translation transformation in the splicing process is ,in , , then the transformed position frame coordinates are , maintain the consistency of component position information before and after splicing.

[0028] Based on the transformed position box , and further generate the corresponding mask image The size of the mask image is similar to the size of the panoramic image. Exactly the same, in the location box Within the covered area, the pixel value is set to 1, and in other locations, the pixel value is set to 0, which can be expressed mathematically as: ; Where, Representing coordinates The pixel value of the pixel at .

[0029] The mask images corresponding to the global image set of all types of parts are obtained through the above method, thereby forming a visual cue parts library.

[0030] For each annotated component type, a textual description of the possible anomalies that may occur with that component is provided. , and add corresponding detailed descriptions for each exception type, Represents the set of possible exception types that may occur for the p-th type of component, where Represents the total number of exception types, The pth type of component These exception description texts provide semantic information for the identification of specific fault types. The exception description texts of all types of parts constitute the exception description text library.

[0031] It should be noted that using a rectangular position frame to mark the position of the target component is only an example. In other embodiments, other polygonal frames can also be used for position marking. The coordinate transformation of the position frame during the global image stitching process is relatively easy to derive based on the above embodiment description and will not be further elaborated here.

[0032] S2. Use a convolutional neural network model to extract the multi-scale image features of the panoramic image and mask image of each type of parts respectively, and then fuse the multi-scale image features of the panoramic image and mask image of each type of parts to obtain the visual cue features of each type of parts.

[0033] The multi-scale image features of the panoramic image and mask image of each type of parts are fused to obtain the visual cue features of each type of parts, including: For each type of component, the multi-scale image features of the panoramic image and the multi-scale image features of the mask image are bit-wise multiplied at different scales to generate image fusion features at different scales. The image fusion features at different scales are transposed, dimensionally transformed and spliced, and the feature channels are compressed to 1 to obtain the visual cue features of the current type of parts.

[0034] The following uses the ResNet50 convolutional neural network model as an example for explanation.

[0035] A convolutional neural network model is used to perform multi-scale feature extraction on the panoramic image of the parts. The image features are extracted into feature tensors at three different scales. ,Right now 、 and .here, Represents batch size, is the feature channel, and Represent the height and width of the feature map respectively. Taking ResNet50 as an example, the image features are extracted by multi-layer convolution operation. The convolution output of the layer is , ,in is the activation function, and They are The weights and biases of the layers, Represents the convolution operation.

[0036] For mask images of the same size, the same convolutional neural network is used for multi-scale feature extraction. The mask image is converted into feature tensors of three scales corresponding to the feature tensors of the component image, namely 、 and The mask image only contains 0 and 1 values. Therefore, the mask feature tensor is actually a set of 0 and non-0 values, which can be regarded as a set of False and True. Based on this, bit-wise operations are performed on the feature tensor of the component image and the mask feature tensor. Let the feature tensor of the component image be , the mask feature tensor is ( Corresponding to the above three scales respectively), the image fusion features at different scales after operation Expressed as: ; Where, The feature tensor representing the component image is located at The eigenvalues of the feature points at Represents the mask feature tensor located at The eigenvalues of the feature points at Indicates that the image fusion feature is located at The eigenvalues of the feature points at .

[0037] After the operation is completed, three new image fusion features are obtained. Subsequently, these three image fusion features are transposed and dimensionally transformed into 、 、 . Let the transformation function be , for the feature map , , from Transform to For example, the dimension and The calculation of is determined according to the specific transformation rules. Then, the feature transformation is performed through three linear layers, and the feature dimension after transformation is 、 、 . Let the weight matrix of the linear layer be , the bias is , for the input features , output After concat operation on the first dimension, the size is After that, expand the last dimension to get Finally, through a convolution layer with a convolution kernel size of 1x1, the feature channel is compressed to 1, and we get , and finally the visual cue features are regularized into . Let the weight of the 1x1 convolution layer be , the bias is , for the input features , the convolutional layer output ,in It is a 1x1 convolution operation.

[0038] S3. Use a convolutional neural network model to extract multi-scale image features of the image of the component to be inspected, and fuse them with the visual cue features of the same type of components to obtain the final multi-scale image features of the component to be inspected. Specifically including: A convolutional neural network model is used to extract multi-scale image features of the image of the component to be inspected; The multi-scale image features of the image of the part to be inspected are transposed and dimensionally transformed, and the multi-scale image features of the image of the part to be inspected obtained by the transformation are fused with the visual cue features of the parts of the same type to obtain the fused features of the part to be inspected at multiple scales; The fused features of the parts to be inspected at multiple scales are transposed and dimensionally transformed, and finally a multi-scale image feature of the parts to be inspected with the same dimension as the multi-scale image feature of the initial images of the parts to be inspected is obtained.

[0039] In this embodiment, a convolutional neural network model is used to extract multi-scale features from the image of the component to be inspected, and features at three scales are extracted, namely 、 as well as Subsequently, the features of these three different scales are transposed and dimensionally transformed, and then feature transformation is performed with the help of three linear layers. After the transformation, the features of the three scales are transformed into , and Next, the size obtained in step S2 is The visual cue features are fused with the features of three different scales of the image to be detected. First, the visual cue features are processed by the sigmoid function. Let the visual cue features be , after the sigmoid function After processing, . Assume that the three different scale features of the image to be detected are 、 、 (correspond , and ), the fused features ( ) can be expressed as Finally, the three fused features are transposed and transformed again through three linear layers and Reshape operations. For example, first transform it into , then transpose to , and finally perform the Reshape operation to obtain The same operation is applied to the other two features, so that the features of the three scales are finally transformed back to 、 and , which is the final multi-scale image feature of the component to be detected.

[0040] S4. Input the multi-scale image features of the component to be inspected into the component positioning model to locate and obtain the target component image.

[0041] The component localization model includes a path aggregation network (PANet), multiple C2f modules, multiple SPPF modules, and multiple detection heads; After the multi-scale image features of the parts to be inspected are input into the path aggregation network, the top-down and bottom-up image features are fused at each scale. Each fusion is performed using the C2f module for multi-scale fusion. After obtaining the fused image features of the parts to be inspected at each scale, they are input into various corresponding SPPF modules to obtain the enhanced fused image features of the parts to be inspected at each scale; The enhanced fused image features of the parts to be detected at each scale are input into the corresponding detection head to complete the positioning and extraction of the target parts.

[0042] In this embodiment, first, the image features of the three scales of the parts to be inspected obtained in step S3, namely 、 and , input into PANet. In the top-down path of PANet, high-level features Upsampled to Resolution, get feature map , and then with the middle-level features Perform splicing in the channel dimension, and set the splicing operation function to be , the concatenated feature map is .then, After processing by a C2f module, the C2f module can be expressed as a function , the feature map after processing is . Continue upsampling to Resolution, get , and the underlying features Splicing is performed on the channel dimension, and the feature map after splicing is , and then processed by a C2f module to obtain .

[0043] In the bottom-up approach, the underlying features The feature map is obtained by downsampling with a convolution step size of 2. , and the middle-level features Splicing is performed on the channel dimension, and the feature map after splicing is , processed by a C2f module . Then downsample to Resolution, get , and high-level features Splicing is performed on the channel dimension, and the feature map after splicing is , and then processed by a C2f module to obtain .

[0044] At the intermediate scale, high-level features are upsampled and concatenated with middle-level features, and then processed by a C2f module to obtain , and then compared with the feature map after downsampling of the underlying features Splicing is performed on the channel dimension and processed by a C2f module to obtain .

[0045] Therefore, the fusion image features of the parts to be detected at the three scales are 、 、 .

[0046] In the feature enhancement process, the SPPF module is introduced into the fusion path. Taking high-level features as an example, After processing by SPPF module, let SPPF module function be , the feature map after processing is , which enhances the receptive field. Similarly, 、 Also enhanced by the SPPF module, we get 、 Therefore, the enhanced fusion image features of the parts to be detected at the three scales are 、 、 .

[0047] After the above processing, the fusion image features of the parts to be detected at three scales are used to detect targets of different scales. For the prediction stage, the output dimension of each detection head is ,in Represents the bounding box coordinates (taking a rectangular position box as an example), and 1 represents the probability of predicting the corresponding component type. Taking large target detection as an example, based on To make a prediction, let the prediction function be , the prediction result is Medium-range targets are based on feature maps Prediction, the result is Small targets are based on feature maps Prediction, the result is The total number of predicted boxes obtained is That is, the prediction information dimension of the component location is .

[0048] During the training phase of the component localization model, the total loss function used is the weighted sum of the classification loss and the localization loss. The total loss function is expressed as follows: ; Where, represents the total loss, and Represents the classification loss and the weight of the positioning loss, which includes the bounding box distribution loss and geometric constraint loss In this embodiment, , .

[0049] Since there are only two categories, background and parts to be located, the classification loss adopts binary classification loss. It is expressed as follows: ; Where, , used to control the intensity of difficult sample mining; , is the dynamic positive sample weight, For the The intersection of the predicted position frame and the true position frame of the positive sample, here ; To predict the foreground (component) probability (Sigmoid output), To predict the background probability; 、 represent the number of positive samples and the number of negative samples respectively.

[0050] Bounding Box Distribution Loss It is expressed as follows: ; Where, The 16-bin discrete distribution for prediction (the main purpose of the 16-bin discrete distribution is to transform the continuous coordinate prediction problem of the bounding box into a classification problem, dividing the continuous value into multiple discrete intervals, and then letting the model output the probability distribution of these intervals, and finally restoring more accurate coordinate values), Label distribution generated by bilinear interpolation; represents cross entropy; and Respectively represent the horizontal and vertical coordinates of the center point of the bounding box, and denote the width and height of the bounding box, respectively, and these values are normalized to within the scope; Geometric constraint loss It is expressed as follows: ; ; Where, Predicted location box With real position frame The intersection-over-union ratio, Represents the predicted location box With real position frame The square of the distance, is the diagonal length of the minimum enclosing rectangle containing the predicted location box and the true location box, 、 are the width and height of the predicted location box, 、 are the width and height of the real position box respectively.

[0051] Get the output of the detection head , and use these outputs to perform regression operations to determine the precise location of the target. The bounding box coordinate parameters output by the detection head are as follows The first four values of the second dimension in are used as the regression input. This section describes the component localization model. For details about the Path Aggregation Network (PANet), C2f module, and SPPF module, see the implementations of the Path Aggregation Network (PANet), C2f module, and SPPF module in YOLOv8.

[0052] The previous steps S1 to S4 can be classified as the first stage, the purpose of which is to realize the detection of parts. The process is as follows: Figure 2 As shown; the following steps are classified as the second stage, the purpose of which is to realize the fault detection of components. The process is as follows Figure 3 shown.

[0053] S5. Use a pre-trained visual large model to extract the visual features of the target component image and the text features of the abnormal description text related to the target component, and fuse the visual features with the text features to predict and output the fault detection results of the target component.

[0054] In specific implementation, it is preferred that before step S5, the method further includes: merging the located target component images with overlapping target component images by non-maximum suppression to avoid repeated calculations.

[0055] In step S5, the process of fusing visual features with text features includes: S51: First, the self-attention mechanism is used to enhance the visual features and text features respectively.

[0056] The large visual model uses the Transformer architecture, which uses multi-layer Transformer modules to patch the image into multiple small blocks and convert them into sequence form for processing, thereby extracting high-dimensional visual features. The component image uses the self-attention mechanism to enhance the visual features. Given an input sequence (query vector), (key vector), (value vector), here we use the self-attention mechanism, so 、 as well as are all visual features of parts, and the self-attention calculation is .in, Calculate the similarity between the query vector and all key vectors, Used for scaling to prevent gradients from vanishing or exploding, refers to the image feature dimension in the attention mechanism, The function converts the similarity into a probability distribution to determine the weight of each position, and finally adds it to the value vector The weighted sum is used to obtain the output.

[0057] The large visual model retrieves the abnormal description text related to the component to be inspected from the abnormal description text library built in step S1; uses a tokenizer to segment the abnormal text and encode it into text features, then uses a self-attention mechanism to semantically enhance these text features, fully mining the key semantic information in the text, including the fault type and fault details; and performs feature enhancement for professional terminology and domain knowledge to ensure that the semantic space of text features and visual features are aligned.

[0058] Specific domains contain a wealth of specialized terminology and background knowledge. This terminology and knowledge often require explicit modeling for the model to understand its semantics. The goal of feature enhancement is to prioritize semantic information related to domain knowledge over general semantics when extracting text or visual features. Based on a general visual model, secondary pre-training is performed using text-image pairs specific to bus parts anomaly detection (e.g., industrial defect descriptions of specific parts combined with part images). This strengthens the semantic associations between domain terms and guides the model's focus on specific domain tasks.

[0059] The training process of the large visual model is as follows: Taking the Qwen2.5-VL multimodal vision model as an example, we use LoRA (Low-Rank Adaptation) training (fine-tuning). Each training sample contains two inputs (part image and abnormality description text) and one annotation information: (1) Component image: an image with a component (e.g., a spring); (2) Abnormal description text: Describe the abnormal type that may occur in the current component, for example: { "Component Type": "Spring", "Possible abnormalities": ["deformation", "fracture", "corrosion"], "Detect the fields that need to be output by this component": { "Whether the component type corresponds to the provided component": "Boolean", "confidence": "floating point number", "Is it abnormal": "Boolean value", "Exception Type": "String (empty string if no exception)" } }; (3) Annotation information (i.e., information that you want the large model to answer): { "Whether to match components": true, "Confidence": 1, "Is it abnormal": true, "Exception Type": "Deformation" }; In the training and reasoning stage, the component images and abnormality description texts are combined to form the model input of the visual big model, and the annotation information is used as the output of the visual big model.

[0060] S52: Enhanced visual features and text features Fusion via Cross-Attention Mechanism: S521: First, construct the query vector based on the enhanced visual features, construct the key vector and value vector based on the enhanced text features, and introduce three learnable projection matrices 、 and , then the query vector is obtained by the following linear projection operation , key vector Sum vector : ; ; ; in, is the dimension of visual features, is the dimension of text features, The query vector projection matrix Dimensions, is the key vector projection matrix Dimensions, is the projection matrix of the value vector Dimensions; Calculate the first attention score matrix S: ; Where, Indicates that the softmax function is calculated along the column feature direction; The first fusion feature is obtained by multiplying the first attention score matrix S with the value vector V1 : ; S522: Constructing query vectors based on enhanced text features , construct key vector based on enhanced visual features Sum vector , using the same method as above S521, the second fusion feature is obtained : ;in According to the query vector and key vector The calculated second attention score matrix; S523: Fusing the first fusion feature with the second fusion feature through a gating mechanism to obtain a bidirectional fusion feature : ; in, is the learnable weight parameter.

[0061] In step S5, the fault detection result is output in the form of structured text, which includes the anomaly type, credibility, and whether the component type in the image of the component to be detected is consistent with the component type in the anomaly description text.

[0062] The large-scale visual model integrates visual and textual features, leveraging the multimodal capabilities of the Transformer architecture to analyze failure modes and anomaly characteristics, determining whether a component is in an abnormal state. The model outputs structured text for detected faults, including the fault type and description. Furthermore, through the model's internal uncertainty estimation mechanism, it outputs a numerical value reflecting the credibility of the fault detection result, presented as a probability. This value is based on the feature distribution and decision boundary learned during model training, providing a quantitative indicator of the reliability of the detection result during inference. To prevent the possibility of a mismatch between the output component and the anomaly description text in the first stage, the model also outputs additional information recording whether the input component matches the component in the anomaly description text. Figure 4 The figure shows an example of visual inspection of bus parts and the structured output of the results.

[0063] The above embodiment provides a bus component fault detection method based on visual cues and visual large models, which has the following advantages: 1. To address the large number and variety of bus parts, as well as the constant emergence of new ones, a visually cued parts library was constructed. A component-feature mapping library was constructed using multi-view sample images. This enables precise component location even when dealing with a large number of running gear components and new parts, resolving the issue of traditional methods requiring repeated training and being unable to quickly adapt to new parts.

[0064] 2. Given the challenges of extracting and identifying fault target features, particularly those associated with subtle faults that are subtle, the complex structures of large components, and the impact of operating environments on image quality, a text library of anomaly descriptions was constructed, organically integrating image and text information. This textual description complements the limitations of single visual information, allowing for deeper analysis of fault characteristics and helping to accurately identify both subtle and complex faults in critical areas.

[0065] 3. During the specific inspection steps, multi-scale feature extraction, encoding, fusion, and transformation are performed on the component cue image and the image to be inspected. A convolutional neural network is used to extract multi-scale features. In particular, bit-wise operations, dimensionality transformations, and linear layer transformations are performed on the original features in conjunction with visual cues, enhancing the model's ability to capture specific component features. A component localization model is constructed using the YOLOv8 network architecture. Through the rational design of the loss function, the localization accuracy and detection efficiency of components of different scales are improved, addressing the issues with traditional algorithms that affect detection accuracy in complex scenarios and lack sufficient processing of redundant information.

[0066] 4. A large Transformer-based visual model processes the located component images, enhancing visual features through a self-attention mechanism. Simultaneously, the retrieved textual features of the anomaly description are encoded and semantically enhanced before being fused with the visual features through a cross-attention mechanism. A gating mechanism is employed to enhance the depth of multimodal interaction. This not only discovers potential failure modes but also outputs structured text and credibility values, avoiding mismatches between components and anomaly text, enabling accurate detection and reliable judgment of bus component faults.

[0067] The embodiment of the present invention further provides a bus component fault detection system based on visual prompts and a visual large model, comprising: The visual prompt and exception description text construction module is used to mark the location of the target parts in each part sample image set for each type of part sample image, stitch the marked part sample image set together to generate a panoramic image, and generate a mask image corresponding to the panoramic image; for each type of part, the abnormal type of the part and the details of each abnormal type are described in text to form the abnormal description text; A visual cue feature extraction module is used to extract multi-scale image features of the panoramic image and mask image of each type of component using a convolutional neural network model, and then fuse the multi-scale image features of the panoramic image and mask image of each type of component to obtain the visual cue features of each type of component; The image feature extraction and fusion module for testing is used to extract the multi-scale image features of the image of the component to be tested using a convolutional neural network model, and fuse them with the visual cue features of the same type of components to obtain the final multi-scale image features of the component to be tested; The target component positioning module is used to input the multi-scale image features of the component to be inspected into the component positioning model to locate the target component image; The component fault detection module uses a pre-trained visual large model to extract the visual features of the target component image and the text features of the abnormal description text related to the target component, and fuses the visual features with the text features to predict and output the fault detection results of the target component.

[0068] It should be understood that the functional unit modules in various embodiments of the present invention can be concentrated in one processing unit, or each unit module can exist physically separately, or two or more unit modules can be integrated into one unit module, and can be implemented in the form of hardware or software.

[0069] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0070] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A bus component fault detection method based on visual cues and visual large models, characterized in that: The steps include: S1. For each type of component sample image set, mark the location of the target component in each component sample image, stitch the marked component sample image set to generate a panoramic image, and generate a mask image corresponding to the panoramic image; for each type of component, describe the abnormality type of the component and the details of each abnormality type in text to form an abnormality description text; S2. Use a convolutional neural network model to extract multi-scale image features of the panoramic image and mask image of each type of component, and then fuse the multi-scale image features of the panoramic image and mask image of each type of component to obtain the visual cue features of each type of component; S3. Use a convolutional neural network model to extract multi-scale image features of the image of the component to be inspected, and fuse them with the visual cue features of the same type of components to obtain the final multi-scale image features of the component to be inspected; S4, inputting the multi-scale image features of the component to be inspected into the component positioning model to locate and obtain the target component image; S5. Use a pre-trained visual large model to extract the visual features of the target component image and the text features of the abnormal description text related to the target component, and fuse the visual features with the text features to predict and output the fault detection results of the target component.

2. The bus component fault detection method based on visual prompts and visual large models according to claim 1 is characterized in that: In step S1, stitching the labeled component sample image set to generate a panoramic image includes: Assume that the parts sample image set includes n parts sample images, and splice the n parts sample images into a panoramic image according to the row and column layout. The number of rows e and the number of columns f of the panoramic image satisfy ,and minimum; if , then fill the empty areas in the panoramic image.

3. The bus component fault detection method based on visual prompts and visual large models according to claim 1 is characterized in that: In step S2, the multi-scale image features of the panoramic image and the mask image of each type of parts are fused to obtain the visual cue features of each type of parts, specifically including: For each type of component, the multi-scale image features of the panoramic image and the multi-scale image features of the mask image are multiplied bit by bit at different scales to generate image fusion features at different scales. The image fusion features at different scales are transposed, dimensionally transformed and spliced, and the feature channels are compressed to 1 to obtain the visual cue features of the current type of parts.

4. The bus component fault detection method based on visual prompts and visual large models according to claim 1 is characterized in that: Step S3 specifically includes: A convolutional neural network model is used to extract multi-scale image features of the image of the component to be inspected; The multi-scale image features of the image of the part to be inspected are transposed and dimensionally transformed, and the multi-scale image features of the image of the part to be inspected obtained by the transformation are fused with the visual cue features of the parts of the same type to obtain the fused features of the part to be inspected at multiple scales; The fused features of the parts to be inspected at multiple scales are transposed and dimensionally transformed, and finally a multi-scale image feature of the parts to be inspected with the same dimension as the multi-scale image feature of the initial images of the parts to be inspected is obtained.

5. The bus component fault detection method based on visual prompts and visual large models according to claim 1 is characterized in that: The component localization model includes a path aggregation network, multiple C2f modules, multiple SPPF modules, and multiple detection heads; After the multi-scale image features of the parts to be inspected are input into the path aggregation network, the top-down and bottom-up image features are fused at each scale. Each fusion is performed using the C2f module for multi-scale fusion. After obtaining the fused image features of the parts to be inspected at each scale, they are input into various corresponding SPPF modules to obtain the enhanced fused image features of the parts to be inspected at each scale; The enhanced fused image features of the parts to be detected at each scale are input into the corresponding detection head to complete the positioning and extraction of the target parts.

6. The bus component fault detection method based on visual prompts and visual large models according to claim 1 is characterized in that: During the training phase of the component localization model, the total loss function used is the weighted sum of the classification loss and the localization loss. The total loss function is expressed as follows: ； Where, represents the total loss, and Represents the classification loss and the weight of the positioning loss, which includes the bounding box distribution loss and geometric constraint loss .

7. The bus component fault detection method based on visual prompts and visual large models according to claim 1 is characterized in that: Before step S5, the method further includes: The located target component images are merged with overlapping target component images through non-maximum suppression to avoid repeated calculations.

8. The bus component fault detection method based on visual prompts and visual large models according to claim 1 is characterized in that: In step S5, the process of fusing visual features with text features includes: S51: First, the self-attention mechanism is used to enhance the visual features and text features respectively; S52: Enhanced visual features and text features Fusion via Cross-Attention Mechanism: S521: First, construct the query vector based on the enhanced visual features, construct the key vector and value vector based on the enhanced text features, and introduce three learnable projection matrices 、 and , then the query vector is obtained by the following linear projection operation , key vector Sum vector : ； ； ； in, is the dimension of visual features, is the dimension of text features, The query vector projection matrix Dimensions, is the key vector projection matrix Dimensions, is the projection matrix of the value vector Dimensions; Calculate the first attention score matrix S: ； Where, Indicates that the softmax function is calculated along the column feature direction; The first fusion feature is obtained by multiplying the first attention score matrix S with the value vector V1 : ; S522: Constructing query vectors based on enhanced text features , construct key vector based on enhanced visual features Sum vector , using the same method as above S521, the second fusion feature is obtained : ;in According to the query vector and key vector The calculated second attention score matrix; S523: Fusing the first fusion feature with the second fusion feature through a gating mechanism to obtain a bidirectional fusion feature : ； in, is the learnable weight parameter.

9. The bus component fault detection method based on visual prompts and visual large models according to claim 1 is characterized in that: In step S5, the fault detection result is output in the form of structured text, which includes the anomaly type, credibility, and whether the component type in the image of the component to be detected is consistent with the component type in the anomaly description text.

10. A bus component fault detection system based on visual prompts and visual large models, characterized in that: include: The visual prompt and exception description text construction module is used to mark the location of the target parts in each part sample image set for each type of part sample image, stitch the marked part sample image set together to generate a panoramic image, and generate a mask image corresponding to the panoramic image; for each type of part, the abnormal type of the part and the details of each abnormal type are described in text to form the abnormal description text; A visual cue feature extraction module is used to extract multi-scale image features of the panoramic image and mask image of each type of component using a convolutional neural network model, and then fuse the multi-scale image features of the panoramic image and mask image of each type of component to obtain the visual cue features of each type of component; The image feature extraction and fusion module for testing is used to extract the multi-scale image features of the image of the component to be tested using a convolutional neural network model, and fuse them with the visual cue features of the same type of components to obtain the final multi-scale image features of the component to be tested; The target component positioning module is used to input the multi-scale image features of the component to be inspected into the component positioning model to locate the target component image; The component fault detection module uses a pre-trained visual large model to extract the visual features of the target component image and the text features of the abnormal description text related to the target component, and fuses the visual features with the text features to predict and output the fault detection results of the target component.

Images