A Method and System for Fault Detection of Bus Components Based on Visual Cueing and Visual Large Model

Abstract

This invention discloses a method and system for bus component fault detection based on visual cues and a large visual model. The method includes: for each type of component sample image set, marking the location of the target in each sample image, then stitching them into a panoramic image and generating a mask image; describing the anomaly type and details of each anomaly type of component using anomaly description text; extracting multi-scale image features from the panoramic image and the mask image and fusing them to obtain visual cues features; extracting multi-scale image features from the image of the component to be detected and fusing them with the visual cues features to obtain multi-scale image features of the component to be detected, inputting them into a component localization model to locate the target component image; and using a large visual model to fuse the visual features of the target component image with the textual features of the corresponding anomaly description text to predict and output the fault detection result. This invention can achieve accurate component localization and fault detection.

Description

Technical Field

[0001] This invention relates to the field of bus component fault detection technology, and in particular to a bus component fault detection method and system based on visual cues and large visual models. Background Technology

[0002] With the rapid development of the railway transportation industry and the continuous increase in passenger volume, the requirements for passenger train operation safety are becoming increasingly stringent. Passenger trains themselves have a complex structure, containing a large number of components, from the running gear and braking parts to various underframe suspension components, in vast quantities and diverse types. Statistics show that a typical passenger train contains nearly a hundred different types of components, from the wheels and bearings of the running gear to the brake discs and cylinders of the braking system, and then to the underframe suspension components. Such a large system of components makes fault detection a heavy task. Furthermore, each component, due to different working environments and stress conditions, can experience a wide variety of faults. For example, wheels may experience wear, cracks, and tread peeling; brake discs may suffer from thermal cracks and uneven wear. This necessitates that the detection methods possess the ability to accurately identify multiple types of faults.

[0003] With the continuous development and upgrading of bus technology, new components and new types of failures are constantly emerging. On the one hand, new buses use many new materials and processes to manufacture parts to improve performance and safety, and the failure modes of these new components are completely different from those of traditional components. On the other hand, even traditional components may experience unprecedented failure modes during long-term operation and technological improvements. Current detection methods, whether based on traditional image processing or deep learning, are mostly unable to quickly adapt to the emergence of new components and new failures, lacking sufficient flexibility and scalability.

[0004] In bus component fault detection, fault target feature extraction and identification face numerous challenges. For minor faults in critical areas, such as tiny cracks on component surfaces, current methods rely solely on single visual images. However, these images occupy very few pixels and have weak features, making them easily overlooked and difficult to accurately extract effective features for fault diagnosis. Furthermore, for overall faults in large components, such as bogie deformation, even if they occupy a large area in the visual image, their complex structure makes feature extraction extremely difficult based solely on visual information. In addition, the complex operating environment of buses, with factors such as lighting and weather interfering with image quality, further increases the uncertainty of visual feature extraction, making it difficult to comprehensively capture faults through visual images.

[0005] Traditional methods for detecting faults in bus components mainly include the use of support vector machines, Faster R-CNN, and YOLO algorithms. However, traditional methods have shortcomings when facing nearly a hundred types of bus components and complex backgrounds, such as the need for repeated training to detect new components, poor adaptability to complex scenes, and difficulty in accurately locating and analyzing component anomalies from a large amount of redundant image information. Summary of the Invention

[0006] This invention provides a method and system for detecting bus component faults based on visual cues and large visual models, in order to solve the problem that existing technologies are difficult to accurately and efficiently detect bus component faults in complex operating environments.

[0007] Firstly, a method for detecting faults in bus components based on visual cues and a large visual model is provided, including the following steps:

[0008] S1. For each type of component sample image set, mark the location of the target component in each component sample image, and stitch the marked component sample image set to generate a panoramic image, and generate a mask image corresponding to the panoramic image; for each type of component, describe the anomaly type of the component and the details of each anomaly type through text to form an anomaly description text.

[0009] S2. A convolutional neural network model is used to extract multi-scale image features from panoramic and mask images of each type of component. Then, the multi-scale image features from panoramic and mask images of each type of component are fused to obtain visual cue features of each type of component.

[0010] S3. Use a convolutional neural network model to extract multi-scale image features of the parts to be detected, and fuse them with the visual cue features of similar parts to obtain the final multi-scale image features of the parts to be detected.

[0011] S4. Input the multi-scale image features of the component to be detected into the component localization model to obtain the target component image;

[0012] S5. Using a pre-trained large visual model, extract the visual features of the target component image and the text features of the abnormal description text related to the target component, and fuse the visual features and text features to predict and output the fault detection result of the target component.

[0013] Further, in step S1, the process of stitching together the labeled component sample image set to generate a panoramic image includes:

[0014] Suppose a set of component sample images consists of n component sample images. These n images are stitched together in a row-column layout to form a panoramic image, where the number of rows e and columns f of the panoramic image satisfy... ,and minimum; if Then, the missing areas in the panoramic image are filled.

[0015] Further, in step S2, the multi-scale image features of the panoramic images and mask images of each type of component are fused to obtain the visual cue features of each type of component, specifically including:

[0016] For each type of component, the multi-scale image features of its panoramic image and the multi-scale image features of its mask image are multiplied bitwise according to different scales to generate image fusion features at different scales;

[0017] By transposing, transforming, and stitching the fusion features of images at different scales, and compressing the feature channels to 1, visual cue features of the current type of component are obtained.

[0018] Furthermore, step S3 specifically includes:

[0019] A convolutional neural network model is used to extract multi-scale image features from the images of the parts to be detected.

[0020] The multi-scale image features of the component to be detected are transposed and dimensionally transformed. The multi-scale image features of the transformed component to be detected are then fused with the visual cue features of the same type of component to obtain the multi-scale fused features of the component to be detected.

[0021] The multi-scale fusion features of the parts to be detected are transposed and dimensionally transformed to obtain multi-scale image features of the parts to be detected with the same dimensionality as the initial multi-scale image features of the parts to be detected.

[0022] Furthermore, the component localization model includes a path aggregation network, multiple C2f modules, multiple SPPF modules, and multiple detection heads;

[0023] After inputting the multi-scale image features of the parts to be detected into the path aggregation network, the top-down and bottom-up image features are fused at each scale. The C2f module is used for multi-scale fusion each time.

[0024] After obtaining the fused image features of the parts to be detected at each scale, they are input into various corresponding SPPF modules to obtain the enhanced fused image features of the parts to be detected at each scale.

[0025] The enhanced image features of the component to be detected at each scale are input into the corresponding detection head to complete the localization and extraction of the target component.

[0026] Furthermore, during the training phase of the component localization model, the total loss function used is a weighted sum of the classification loss and the localization loss, expressed as follows:

[0027] ;

[0028] In the formula, Indicates the total loss. and Representing classification loss respectively The weights of the localization loss, which includes the bounding box distribution loss, are also considered. and geometric constraint loss .

[0029] Furthermore, prior to step S5, the following is also included:

[0030] The images of the located target components are merged by non-maximum suppression to avoid redundant calculations.

[0031] Furthermore, in step S5, the process of fusing visual features with textual features includes:

[0032] S51: First, enhance the visual features and text features separately using a self-attention mechanism;

[0033] S52: Enhanced visual features and text features Fusion is achieved through a cross-attention mechanism:

[0034] S521: First, construct a query vector based on the enhanced visual features, then construct key and value vectors based on the enhanced text features, and finally introduce three learnable projection matrices. , and The query vector is then obtained through the following linear projection operation. Key vector Sum value vector :

[0035] ;

[0036] ;

[0037] ;

[0038] in, For the dimensions of visual features, For the dimensions of text features, For query vector projection matrix Dimensions Projection matrix of key vectors Dimensions Projection matrix of value vector The dimension;

[0039] Calculate the first attention score matrix S:

[0040] ;

[0041] In the formula, This indicates that the softmax function is calculated along the column feature direction;

[0042] The first fused feature is obtained by multiplying the first attention score matrix S with the value vector V1. : ;

[0043] S522: Constructing query vectors based on enhanced text features Key vectors are constructed based on the enhanced visual features. Sum value vector Using the same method as S521 above, the second fusion feature is obtained. : ;in To be based on the query vector and key vector The calculated second attention score matrix;

[0044] S523: The first and second fusion features are fused using a gating mechanism to obtain a bidirectional fusion feature. :

[0045] ;

[0046] in, These are learnable weight parameters.

[0047] Furthermore, in step S5, the fault detection results are output in structured text, which includes the anomaly type, confidence level, and whether the component type in the image of the component to be detected is consistent with the component type in the anomaly description text.

[0048] Secondly, a bus component fault detection system based on visual cues and a large visual model is provided, including:

[0049] The visual cues and anomaly description text construction module is used to mark the location of target parts in each part sample image for each type of part sample image set, stitch the marked part sample image set to generate a panoramic image, and generate a mask image corresponding to the panoramic image; for each type of part, the module describes the anomaly type of the part and the details of each anomaly type through text to form an anomaly description text.

[0050] The visual cue feature extraction module is used to extract multi-scale image features of panoramic images and mask images of various types of parts using a convolutional neural network model. Then, the multi-scale image features of panoramic images and mask images of various types of parts are fused to obtain the visual cue features of various types of parts.

[0051] The image feature extraction and fusion module is used to extract multi-scale image features of the parts to be detected using a convolutional neural network model, and fuse them with visual cue features of similar parts to obtain the final multi-scale image features of the parts to be detected.

[0052] The target component localization module is used to input the multi-scale image features of the component to be detected into the component localization model to obtain the target component image.

[0053] The component fault detection module uses a pre-trained large visual model to extract visual features from the target component image and text features from the abnormal description text related to the target component. It then fuses the visual and text features to predict and output the fault detection result of the target component.

[0054] This invention proposes a method and system for detecting faults in bus components based on visual cues and a large visual model, which has the following beneficial effects:

[0055] 1. Compared with current object detection algorithms such as closed set detection and open vocabulary detection, this invention introduces visual cue information, which has the following advantages:

[0056] (1) Enhance the adaptability of new component detection: Closed set detection methods are limited to preset categories and cannot detect new components and new fault types. When facing new components, it is necessary to redefine the category set and retrain the model, which is cumbersome and inefficient. This invention constructs a visual cue component library and uses multi-view sample images to construct a "component-feature" mapping relationship library, which can cope with the emergence of new components, break through the limitations of closed set detection, expand the detection range, and enhance the adaptability and flexibility of new component detection.

[0057] (2) Improved positioning accuracy: In complex backgrounds, open vocabulary detection is insufficient in extracting the features of target components and is difficult to accurately locate the target components; This invention enhances the features of specific components by constructing a mask composed of visual cues by marking the position of the components, and can accurately locate bus components even in complex backgrounds, thus improving the accuracy of detection.

[0058] 2. Compared with existing anomaly detection methods, this invention introduces anomaly corpus and a large visual model, which has the following advantages:

[0059] (1) Overcoming the limitations of single-modal information: Existing anomaly detection models mainly rely on single visual information, which cannot fully extract subtle and complex fault features; This invention constructs an anomaly description text library, combines image and text information, and uses text description to supplement the lack of visual information, which can extract fault features more deeply and accurately judge subtle and complex faults.

[0060] (2) Improve adaptability to complex scenarios: The complex operating environment of buses can interfere with the detection accuracy of existing anomaly detection models. This invention uses a well-trained visual large model to make full use of visual images and anomaly description text information, which enhances the adaptability to complex scenarios and can stably and accurately detect bus component failures under interference such as lighting and weather. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a flowchart of a bus component fault detection method based on visual cues and a large visual model provided in an embodiment of the present invention;

[0063] Figure 2 This is a flowchart of the visual cue feature fusion and positioning result output for bus components provided in this embodiment of the invention;

[0064] Figure 3 This is a flowchart of the process for fusing images and text features of bus components and outputting detection results, provided in an embodiment of the present invention.

[0065] Figure 4 This is an example diagram of the visual inspection and structured output of bus components provided in an embodiment of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0067] In the description of this invention, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0068] Traditional methods for detecting dynamic image faults in passenger buses suffer from drawbacks, including the need for repeated training to detect new components, poor adaptability to complex scenarios, and difficulty in accurately locating and analyzing component anomalies from a large amount of redundant image information. To address the challenge of accurately and efficiently detecting passenger bus component faults in complex operating environments, this invention proposes a method and system for passenger bus component fault detection based on visual cues and a large visual model. By constructing a visual cues component library and utilizing multi-view sample images to build a "component-feature" mapping relationship library, it can handle situations involving a large number of running gear components and the appearance of new components, achieving accurate component location even in complex backgrounds. Simultaneously, by constructing an anomaly description text library, image and text information are organically integrated, enhancing adaptability to complex scenarios, overcoming the limitations of single visual information, and deeply mining fault features to achieve accurate detection of subtle and complex faults in passenger bus components. The technical solution of this invention will be specifically described below with reference to specific embodiments.

[0069] This invention provides a method for detecting faults in bus components based on visual cues and a large visual model, comprising the following steps:

[0070] S1. Build a visual cue component library and an anomaly description text library.

[0071] For each type of component sample image set, the target components in each component sample image are labeled with their positions. The labeled component sample image sets are then stitched together to generate a panoramic image, and a mask image corresponding to the panoramic image is generated. The mask images corresponding to the global image sets of all types of components constitute a visual cue component library. For each type of component, the abnormality type of the component and the details of each abnormality type are described by text to form an abnormality description text. The abnormality description texts of all types of components constitute an abnormality description text library.

[0072] The following example illustrates how to use rectangular bounding boxes to label the location of target parts. For a sample image of a certain type of part... The position of the target component is manually determined using a rectangular location box. To accurately mark the location of components, among which, Specify the coordinates of the top-left corner of the corresponding rectangle, and This accurately represents the coordinates of the lower right corner of the rectangular bounding box, providing positional information for subsequent image stitching and mask generation tasks.

[0073] Image sample of all parts of a certain type Arranged according to rows and columns, and stitched together in an orderly manner to form a complete panoramic image. Suppose the component sample image set includes n component sample images, and the stitching function is... , For the nth component sample image, the number of rows needs to be determined through precise calculation. With column number , number of rows With column number Must meet ,and Minimize, meaning prioritize making the number of rows and columns as close as possible. This can be achieved by gradually increasing the number of columns or rows until the total number of images is satisfied, thus accurately determining the number of rows and columns. Of course, if... Then, the missing areas in the panoramic image need to be filled. The width of each component sample image in the stitched panoramic image is... The height is , here , These are the pre-defined total width and total height of the panoramic image, respectively. For the first... The starting x-coordinate of each sample image in the stitched image. Starting ordinate , This represents the remainder operation. This indicates rounding down to the nearest integer.

[0074] During image stitching, the position boxes in the original component sample images The corresponding transformation needs to be performed according to the splicing rules to obtain the transformed position box. Assume the translation transformation during the splicing process is as follows: ,in , The coordinates of the transformed bounding box are To maintain the consistency of component position information before and after assembly.

[0075] Based on the transformed location box Further generate the corresponding mask image. The size of the mask image compared to the panoramic image. Exactly the same, in the position box Within the covered area, the pixel value is set to 1, while in other locations, the pixel value is set to 0. Mathematically, this can be represented as:

[0076] ;

[0077] In the formula, Representing coordinates The pixel value of the pixel at that location.

[0078] The mask images corresponding to the global image set of all types of parts are obtained by the above method, and then a visual cueing parts library is constructed.

[0079] For each labeled component type, describe in text the possible types of anomalies that may occur with that component. And add corresponding detailed descriptions for each exception type. This represents the set of possible exception types for the p-th type of component. Represents the total number of exception types. This represents the p-th type of component. There are several anomaly types, and these anomaly description texts provide semantic information for identifying specific fault types. The anomaly description texts of all types of components constitute an anomaly description text library.

[0080] It should be noted that using a rectangular bounding box to mark the location of the target component is just an example. In other embodiments, other polygonal bounding boxes can also be used for location marking. The coordinate transformation of the bounding box during the stitching of the global image is relatively easy to derive based on the above embodiments, and will not be elaborated here.

[0081] S2. A convolutional neural network model is used to extract multi-scale image features from panoramic and mask images of each type of component. Then, the multi-scale image features from panoramic and mask images of each type of component are fused to obtain visual cue features for each type of component.

[0082] Specifically, the visual cue features of each type of component are obtained by fusing multi-scale image features from panoramic and masked images, including:

[0083] For each type of component, the multi-scale image features of its panoramic image and the multi-scale image features of its mask image are bit-wise (multiplied bit by bit) according to different scales to generate image fusion features at different scales;

[0084] By transposing, transforming, and stitching the fusion features of images at different scales, and compressing the feature channels to 1, visual cue features of the current type of component are obtained.

[0085] The following explanation uses ResNet50 as an example of a convolutional neural network model.

[0086] A convolutional neural network model was used to perform multi-scale feature extraction on panoramic images of parts, and the image features were extracted into feature tensors of three different scales. ,Right now , and .here, Represents batch, For characteristic channels, and Let represent the height and width of the feature map, respectively. Taking ResNet50 as an example, image features are extracted using multi-layer convolution operations. Let the th... The output of the convolutional layer is , ,in For activation function, and They are the first Layer weights and biases This represents the convolution operation.

[0087] Mask images of the same size are used for multi-scale feature extraction using the same convolutional neural network. The mask image is transformed into three-scale feature tensors corresponding to the feature tensors of the component images, i.e. , and The mask image contains only 0 and 1 values; therefore, the mask feature tensor is essentially a set of 0 and non-zero values, which can be considered as a set of False and True. Based on this, bit-wise operations are performed on the feature tensor of the component image and the mask feature tensor. Let the feature tensor of the component image be... The mask feature tensor is ( (corresponding to the three scales mentioned above respectively) Image fusion features at different scales after operation Represented as:

[0088] ;

[0089] In the formula, The feature tensor representing the component image is located in The eigenvalues ​​of the feature points. This indicates that the mask feature tensor is located in The eigenvalues ​​of the feature points. Indicating the location of image fusion features The eigenvalues ​​of the feature points.

[0090] After the operation, three new image fusion features are obtained. Subsequently, these three image fusion features are transposed and dimensionality transformed, respectively, into... , , Let the transformation function be... For feature maps , , in order to Transform to For example, dimension and The calculation is determined based on specific transformation rules. Next, feature transformation is performed through three linear layers, resulting in a transformed feature dimension of... , , Let the weight matrix of the linear layer be... , bias is For input features Output After performing a concat operation on the first dimension, the resulting size is... The feature map. Then, the last dimension is expanded to obtain... Finally, a convolutional layer with a kernel size of 1x1 is used to compress the feature channels to 1, thus obtaining... Ultimately, the visual cue features are regularized into Let the weights of the 1x1 convolutional layer be... , bias is For input features Convolutional layer output ,in This is a 1x1 convolution operation.

[0091] S3. A convolutional neural network model is used to extract multi-scale image features from the image of the component to be detected, and these features are fused with visual cue features of similar components to obtain the final multi-scale image features of the component to be detected. Specifically, this includes:

[0092] A convolutional neural network model is used to extract multi-scale image features from the images of the parts to be detected.

[0093] The multi-scale image features of the component to be detected are transposed and dimensionally transformed. The multi-scale image features of the transformed component to be detected are then fused with the visual cue features of the same type of component to obtain the multi-scale fused features of the component to be detected.

[0094] The multi-scale fusion features of the parts to be detected are transposed and dimensionally transformed to obtain multi-scale image features of the parts to be detected with the same dimensionality as the initial multi-scale image features of the parts to be detected.

[0095] In this embodiment, a convolutional neural network model is used to perform multi-scale feature extraction on the image of the component to be detected, extracting features at three scales, namely: , as well as Subsequently, the features at these three different scales are transposed and their dimensions transformed, and then feature transformation is performed using three linear layers. After the transformation, the features at the three scales are transformed into... , and Next, the dimensions obtained in step S2 will be... The visual cue features are fused with features at three different scales of the image to be detected. First, the visual cue features are processed using the sigmoid function. Let the visual cue features be... After passing through the sigmoid function After processing, the result is Let the three different scale features of the image to be detected be as follows: , , (correspond , and Features after fusion ( ) can be represented as Finally, the three fused features are transposed and then transformed again through three linear layers and a reshape operation. Using one set of features... For example, first transform it through a linear layer to... Then transpose to become Finally, the Reshape operation is performed to obtain... The same operation is applied to the other two features, causing the features at the final three scales to revert to their original states. , and This is the final multi-scale image feature of the component to be inspected.

[0096] S4. Input the multi-scale image features of the component to be detected into the component localization model to obtain the target component image.

[0097] The component localization model includes a path aggregation network (PANet), multiple C2f modules, multiple SPPF modules, and multiple detection heads;

[0098] After inputting the multi-scale image features of the parts to be detected into the path aggregation network, the top-down and bottom-up image features are fused at each scale. The C2f module is used for multi-scale fusion each time.

[0099] After obtaining the fused image features of the parts to be detected at each scale, they are input into various corresponding SPPF modules to obtain the enhanced fused image features of the parts to be detected at each scale.

[0100] The enhanced image features of the component to be detected at each scale are input into the corresponding detection head to complete the localization and extraction of the target component.

[0101] In this embodiment, firstly, the image features of the component to be detected at three scales obtained in step S3 are analyzed, namely... , and The input is then fed into PANet. In PANet's top-down path, high-level features... Upsampled to bilinear interpolation Resolution, to obtain feature maps Then with mid-layer features Perform concatenation along the channel dimension, and let the concatenation operation function be... The spliced ​​feature map is .then, After processing by a C2f module, the C2f module can be represented as a function. The processed feature map is . Continue upsampling to Resolution, obtained , and underlying features The feature map is concatenated along the channel dimension. Then it is processed by a C2f module to obtain .

[0102] In a bottom-up path, the underlying features The feature map is obtained by downsampling with a convolution stride of 2. , and mid-layer features The feature maps are concatenated along the channel dimension, and the resulting concatenated feature map is... After being processed by a C2f module, it becomes . Downsample to Resolution, obtained , and high-level characteristics The feature map is concatenated along the channel dimension. Then it is processed by a C2f module to obtain .

[0103] At intermediate scales, high-level features are upsampled and concatenated with mid-level features, then processed by a C2f module to obtain... Then, it is compared with the feature map after downsampling of the underlying features. The concatenation is performed along the channel dimension, and then processed by a C2f module to obtain... .

[0104] Therefore, the fused image features of the components to be detected at the three scales are as follows: , , .

[0105] During feature enhancement, an SPPF module is introduced into the fusion path. Taking high-level features as an example, After processing by the SPPF module, let the SPPF module function be... The processed feature map is This enhances the receptive field. Similarly, , It also underwent enhancement processing via the SPPF module, resulting in the following: , Therefore, the enhanced fused image features of the detected components at the three scales are as follows: , , .

[0106] After the above processing, the fused image features of the parts to be detected at three scales are used for the detection of targets at different scales. For the prediction stage, the output dimension of each detection head is... ,in This represents the bounding box coordinates (using a rectangular bounding box as an example), where 1 indicates the probability of predicting the corresponding component type. Taking large target detection as an example, based on... To make a prediction, let the prediction function be... The prediction result is Medium-sized targets are based on feature maps. The prediction is as follows: Small targets are based on feature maps. The prediction is as follows: The total number of predicted bounding boxes obtained is The dimension of the predicted information for the location of components is [number], that is, [number] dimensions. .

[0107] During the training phase of the component localization model, the total loss function used is a weighted sum of the classification loss and the localization loss, expressed as follows:

[0108] ;

[0109] In the formula, Indicates the total loss. and Representing classification loss respectively The weights of the localization loss, which includes the bounding box distribution loss, are also considered. and geometric constraint loss In this embodiment, , .

[0110] Since there are only two categories—background and parts to be located—the classification loss uses binary classification loss. It is expressed as follows:

[0111] ;

[0112] In the formula, This is used to control the mining intensity of difficult samples; , which are dynamic positive sample weights. For the first The intersection-union ratio (IoU) of the predicted bounding boxes and the ground truth bounding boxes for each positive sample, here ; To predict the probability of the foreground (components) (Sigmoid output). To predict background probability; , These represent the number of positive samples and the number of negative samples, respectively.

[0113] Bounding box distribution loss It is expressed as follows:

[0114] ;

[0115] In the formula, The 16-bin discrete distribution is used for prediction (the main purpose of the 16-bin discrete distribution is to transform the problem of predicting the continuous coordinates of the bounding box into a classification problem, dividing the continuous values ​​into multiple discrete intervals, and then letting the model output the probability distribution of these intervals, ultimately restoring more accurate coordinate values). Label distribution generated by bilinear interpolation; Represents cross-entropy; and These represent the x and y coordinates of the center point of the bounding box, respectively. and These represent the width and height of the bounding box, respectively, and these values ​​are normalized to... Within the range;

[0116] Geometric constraint loss It is expressed as follows:

[0117] ;

[0118] ;

[0119] In the formula, For predicting location boxes With the actual location box The intersection and union ratio, Represents the predicted location box With the actual location box The square of the distance, Let be the diagonal length of the smallest closure rectangle containing both the predicted and ground truth bounding boxes. , These are the width and height of the predicted location box, respectively. , These are the width and height of the actual location bounding box, respectively.

[0120] Get the output of the detection head These outputs are used for regression operations to determine the precise location of the target. The bounding box coordinate parameters output by the detection head, such as... The first four values ​​of the second dimension are used as inputs for the regression. This section explains the component localization model. For further details regarding the Path Aggregation Network (PANet), C2f module, and SPPF module, please refer to the implementation of the Path Aggregation Network (PANet), C2f module, and SPPF module in YOLOv8.

[0121] The preceding steps S1 to S4 can be categorized as the first stage, the purpose of which is to achieve component inspection. The process is as follows: Figure 2 As shown; the subsequent steps constitute the second stage, the purpose of which is to achieve fault detection of components, and the process is as follows. Figure 3 As shown.

[0122] S5. Using a pre-trained large visual model, extract the visual features of the target component image and the text features of the abnormal description text related to the target component, and fuse the visual features and text features to predict and output the fault detection result of the target component.

[0123] In specific implementation, it is preferable to further include, before step S5: merging overlapping target component images by non-maximum suppression to avoid redundant calculations.

[0124] Step S5, the process of fusing visual features with text features, includes:

[0125] S51: First, enhance the visual features and text features using a self-attention mechanism respectively.

[0126] The large-scale visual model employs a Transformer architecture, using multiple Transformer modules to patch images into smaller patches and convert them into a sequence for processing, thereby extracting high-dimensional visual features. Component images utilize a self-attention mechanism to enhance visual features, given an input sequence. (Query vector) (key vector) (Value vector), here a self-attention mechanism is used, so , as well as These are all visual features of the components, and self-attention is calculated as follows: .in, Calculate the similarity between the query vector and all key vectors. Used for scaling to prevent gradient vanishing or exploding. This refers to the image feature dimension in the attention mechanism. The function transforms similarity into a probability distribution to determine the weight of each position, and finally combines it with the value vector. The weighted sum is used to obtain the output.

[0127] The visual big model retrieves abnormal description texts related to the component to be detected from the abnormal description text library constructed in step S1; it uses a tokenizer to segment and encode the abnormal texts into text features, and then uses a self-attention mechanism to semantically enhance these text features, fully mining the key semantic information in the text, including fault type, fault details, etc.; it also performs feature enhancement for technical terms and domain knowledge to ensure semantic space alignment between text features and visual features.

[0128] In specific domains, there exists a large amount of specialized terminology and background knowledge. This terminology and knowledge typically require explicit modeling for the model to understand its semantics. The purpose of feature augmentation is to allow the model to prioritize semantic information related to domain knowledge, rather than general semantics, when extracting textual or visual features. Based on a general visual model, secondary pre-training is performed using text-image pairs (such as industrial defect descriptions of specific parts + part images) from the bus component anomaly detection domain. This strengthens the semantic associations of domain terms and guides the model to focus on specific domain tasks.

[0129] The training process for the large visual model is as follows:

[0130] Taking the Qwen2.5-VL multimodal vision large model as an example, LoRA (Low-Rank Adaptation) is used for training (fine-tuning). Each training sample contains 2 inputs (part images and anomaly description text) + 1 annotation information:

[0131] (1) Component image: An image containing components (such as a spring);

[0132] (2) Exception description text: Describes the types of exceptions that may occur in the current component, for example:

[0133] {

[0134] Component Type: Spring

[0135] Possible anomalies: ["deformation", "fracture", "corrosion"],

[0136] "Fields that need to be output for this component": {

[0137] "Does the component type match the provided component?": "Boolean value",

[0138] "Confidence level": "Floating-point number",

[0139] "Is it abnormal?": "Boolean value",

[0140] "Exception type": "String (or an empty string if no exception is found)"

[0141] }

[0142] };

[0143] (3) Labeled information (i.e., information that the large model wants to answer):

[0144] {

[0145] "Does it match the component?": true,

[0146] Confidence level: 1,

[0147] "Is it abnormal?": true,

[0148] "Abnormal type": "Deformation"

[0149] };

[0150] During the training and inference phase, component images and anomaly description text are combined to form the input of the visual big model, and the labeled information is used as the output of the visual big model.

[0151] S52: Enhanced visual features and text features Fusion is achieved through a cross-attention mechanism:

[0152] S521: First, construct a query vector based on the enhanced visual features, then construct key and value vectors based on the enhanced text features, and finally introduce three learnable projection matrices. , and The query vector is then obtained through the following linear projection operation. Key vector Sum value vector :

[0153] ;

[0154] ;

[0155] ;

[0156] in, For the dimensions of visual features, For the dimensions of text features, For query vector projection matrix Dimensions Projection matrix of key vectors Dimensions Projection matrix of value vector The dimension;

[0157] Calculate the first attention score matrix S:

[0158] ;

[0159] In the formula, This indicates that the softmax function is calculated along the column feature direction;

[0160] The first fused feature is obtained by multiplying the first attention score matrix S with the value vector V1. : ;

[0161] S522: Constructing query vectors based on enhanced text features Key vectors are constructed based on the enhanced visual features. Sum value vector Using the same method as S521 above, the second fusion feature is obtained. : ;in To be based on the query vector and key vector The calculated second attention score matrix;

[0162] S523: The first and second fusion features are fused using a gating mechanism to obtain a bidirectional fusion feature. :

[0163] ;

[0164] in, These are learnable weight parameters.

[0165] In step S5, the fault detection results are output in structured text, which includes the anomaly type, confidence level, and whether the component type in the image of the component to be detected is consistent with the component type in the anomaly description text.

[0166] The visual big data model integrates visual and textual features, leveraging the multimodal capabilities of the Transformer architecture to analyze fault modes and anomaly characteristics, determining whether components are in an abnormal state. The model outputs structured text for detected faults, covering fault type and description information. Simultaneously, through an internal uncertainty estimation mechanism, it outputs a numerical value reflecting the reliability of the fault detection results, presented in probabilistic form. This value is based on the feature distribution and decision boundary learned during model training, providing a quantitative indicator of the reliability of the detection results during inference. To avoid potential discrepancies between the output component and the anomaly description text in the first stage, the model also outputs additional information recording whether the input component matches the component in the anomaly description text. Figure 4 The image shown is an example of a visual inspection of a bus component and its structured output.

[0167] The bus component fault detection method based on visual cues and a large visual model provided in the above embodiments has the following advantages:

[0168] 1. To address the large number and diverse types of bus components, and the continuous emergence of new parts, a visual cueing component library is constructed, utilizing multi-view sample images to build a "component-feature" mapping relationship library. This enables accurate component location when faced with a large number of running gear components and new components, solving the problem of traditional methods requiring repeated training for new component detection and difficulty in quickly adapting to the emergence of new components.

[0169] 2. Given the difficulties in extracting and identifying fault features, especially the weak features of subtle faults, the complex structures of large components, and the interference of the operating environment with image quality, an anomaly description text library is constructed to organically integrate image and text information. Text descriptions supplement the limitations of single visual information, allowing for deeper analysis of fault features and facilitating the accurate identification of subtle and complex faults in critical components.

[0170] 3. In the specific detection steps, multi-scale feature extraction, encoding, fusion, and transformation are performed on the component prompt images and the images to be detected. A convolutional neural network is used to extract multi-scale features, especially by combining visual cues with bit-wise operations, dimensionality transformations, and linear layer transformations on the original features, enhancing the model's ability to capture specific component features. A component localization model is constructed using the YOLOv8 network structure. By rationally designing the loss function, the localization accuracy and detection efficiency for components of different scales are improved, solving the problems of traditional algorithms being affected by detection accuracy in complex scenes and insufficient processing of redundant information.

[0171] 4. A large visual model based on the Transformer architecture is used to process the localized component images, and a self-attention mechanism is employed to enhance visual features. Simultaneously, the retrieved anomaly description text features are encoded, semantically enhanced, and then fused with visual features through a cross-attention mechanism. A gating mechanism is also used to enhance the depth of multimodal interaction. This not only uncovers potential fault patterns but also outputs structured text and confidence scores, avoiding mismatches between components and anomaly text, thus achieving accurate detection and reliable judgment of bus component faults.

[0172] This invention also provides a bus component fault detection system based on visual cues and a large visual model, comprising:

[0173] The visual cues and anomaly description text construction module is used to mark the location of target parts in each part sample image for each type of part sample image set, stitch the marked part sample image set to generate a panoramic image, and generate a mask image corresponding to the panoramic image; for each type of part, the module describes the anomaly type of the part and the details of each anomaly type through text to form an anomaly description text.

[0174] The visual cue feature extraction module is used to extract multi-scale image features of panoramic images and mask images of various types of parts using a convolutional neural network model. Then, the multi-scale image features of panoramic images and mask images of various types of parts are fused to obtain the visual cue features of various types of parts.

[0175] The image feature extraction and fusion module is used to extract multi-scale image features of the parts to be detected using a convolutional neural network model, and fuse them with visual cue features of similar parts to obtain the final multi-scale image features of the parts to be detected.

[0176] The target component localization module is used to input the multi-scale image features of the component to be detected into the component localization model to obtain the target component image.

[0177] The component fault detection module uses a pre-trained large visual model to extract visual features from the target component image and text features from the abnormal description text related to the target component. It then fuses the visual and text features to predict and output the fault detection result of the target component.

[0178] It should be understood that the functional unit modules in the various embodiments of the present invention can be concentrated in one processing unit, or each unit module can exist physically separately, or two or more unit modules can be integrated into one unit module, and can be implemented in hardware or software.

[0179] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0180] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for detecting faults in bus components based on visual cues and a large visual model, characterized in that, Includes the following steps: S1. For each type of component sample image set, mark the location of the target component in each component sample image, and stitch the marked component sample image set to generate a panoramic image, and generate a mask image corresponding to the panoramic image; for each type of component, describe the anomaly type of the component and the details of each anomaly type through text to form an anomaly description text. S2. A convolutional neural network model is used to extract multi-scale image features from panoramic and mask images of each type of component. Then, the multi-scale image features from panoramic and mask images of each type of component are fused to obtain visual cue features of each type of component. S3. Use a convolutional neural network model to extract multi-scale image features of the parts to be detected, and fuse them with the visual cue features of similar parts to obtain the final multi-scale image features of the parts to be detected. S4. Input the multi-scale image features of the component to be detected into the component localization model to obtain the target component image; S5. Using a pre-trained large visual model, extract the visual features of the target component image and the text features of the abnormal description text related to the target component, and fuse the visual features and text features to predict and output the fault detection result of the target component.

2. The bus component fault detection method based on visual cues and a large visual model according to claim 1, characterized in that, In step S1, the process of stitching together the labeled component sample image set to generate a panoramic image includes: Suppose a set of component sample images consists of n component sample images. These n images are stitched together in a row-column layout to form a panoramic image, where the number of rows e and columns f of the panoramic image satisfy... ,and minimum; if Then, the missing areas in the panoramic image are filled.

3. The bus component fault detection method based on visual cues and a large visual model according to claim 1, characterized in that, In step S2, the multi-scale image features of the panoramic images and mask images of each type of component are fused to obtain the visual cue features of each type of component, specifically including: For each type of component, the multi-scale image features of its panoramic image and the multi-scale image features of its mask image are multiplied bitwise according to different scales to generate image fusion features at different scales; By transposing, transforming, and stitching the fusion features of images at different scales, and compressing the feature channels to 1, visual cue features of the current type of component are obtained.

4. The bus component fault detection method based on visual cues and large visual model according to claim 1, characterized in that, Step S3 specifically includes: A convolutional neural network model is used to extract multi-scale image features from the images of the parts to be detected. The multi-scale image features of the component to be detected are transposed and dimensionally transformed. The multi-scale image features of the transformed component to be detected are then fused with the visual cue features of the same type of component to obtain the multi-scale fused features of the component to be detected. The multi-scale fusion features of the parts to be detected are transposed and dimensionally transformed to obtain multi-scale image features of the parts to be detected with the same dimensionality as the initial multi-scale image features of the parts to be detected.

5. The bus component fault detection method based on visual cues and a large visual model according to claim 1, characterized in that, The component localization model includes a path aggregation network, multiple C2f modules, multiple SPPF modules, and multiple detection heads; After inputting the multi-scale image features of the parts to be detected into the path aggregation network, the top-down and bottom-up image features are fused at each scale. The C2f module is used for multi-scale fusion each time. After obtaining the fused image features of the parts to be detected at each scale, they are input into various corresponding SPPF modules to obtain the enhanced fused image features of the parts to be detected at each scale. The enhanced image features of the component to be detected at each scale are input into the corresponding detection head to complete the localization and extraction of the target component.

6. The bus component fault detection method based on visual cues and a large visual model according to claim 1, characterized in that, During the training phase of the component localization model, the total loss function used is a weighted sum of the classification loss and the localization loss, expressed as follows: ； In the formula, Indicates the total loss. and Representing classification loss respectively The weights of the localization loss, which includes the bounding box distribution loss, are also considered. and geometric constraint loss .

7. The bus component fault detection method based on visual cues and large visual model according to claim 1, characterized in that, The steps preceding step S5 also include: The images of the located target components are merged by non-maximum suppression to avoid redundant calculations.

8. The bus component fault detection method based on visual cues and large visual model according to claim 1, characterized in that, Step S5, the process of fusing visual features with text features, includes: S51: First, enhance the visual features and text features separately using a self-attention mechanism; S52: Enhanced visual features and text features Fusion is achieved through a cross-attention mechanism: S521: First, construct a query vector based on the enhanced visual features, then construct key and value vectors based on the enhanced text features, and finally introduce three learnable projection matrices. , and The query vector is then obtained through the following linear projection operation. Key vector Sum value vector : ； ； ； in, For the dimensions of visual features, For the dimensions of text features, For query vector projection matrix Dimensions Projection matrix of the key vector Dimensions Projection matrix of value vector The dimension; Calculate the first attention score matrix S: ； In the formula, This indicates that the softmax function is calculated along the column feature direction; The first fused feature is obtained by multiplying the first attention score matrix S with the value vector V1. : ; S522: Constructing query vectors based on enhanced text features Key vectors are constructed based on the enhanced visual features. Sum value vector Using the same method as S521 above, the second fusion feature is obtained. : ;in Based on the query vector and key vector The calculated second attention score matrix; S523: The first and second fusion features are fused using a gating mechanism to obtain a bidirectional fusion feature. : ； in, These are learnable weight parameters.

9. The method for detecting bus component faults based on visual cues and a large visual model according to claim 1, characterized in that, In step S5, the fault detection results are output in structured text, which includes the anomaly type, confidence level, and whether the component type in the image of the component to be detected is consistent with the component type in the anomaly description text.

10. A bus component fault detection system based on visual cues and a large visual model, characterized in that, include: The visual cues and anomaly description text construction module is used to mark the location of target parts in each part sample image for each type of part sample image set, stitch the marked part sample image set to generate a panoramic image, and generate a mask image corresponding to the panoramic image; for each type of part, the module describes the anomaly type of the part and the details of each anomaly type through text to form an anomaly description text. The visual cue feature extraction module is used to extract multi-scale image features of panoramic images and mask images of various types of parts using a convolutional neural network model. Then, the multi-scale image features of panoramic images and mask images of various types of parts are fused to obtain the visual cue features of various types of parts. The image feature extraction and fusion module is used to extract multi-scale image features of the parts to be detected using a convolutional neural network model, and fuse them with the visual cue features of similar parts to obtain the final multi-scale image features of the parts to be detected. The target component localization module is used to input the multi-scale image features of the component to be detected into the component localization model to obtain the target component image. The component fault detection module uses a pre-trained large visual model to extract visual features from the target component image and text features from the abnormal description text related to the target component. It then fuses the visual and text features to predict and output the fault detection result of the target component.

Images