Vehicle bottom part anomaly detection method and system based on machine vision

Abstract

The invention provides a vehicle bottom part anomaly detection method and system based on machine vision, and the method comprises the steps: a classification model construction step: carrying out the classification training strategy of historical images of various parts, and obtaining a classification model; a part image acquisition step: acquiring train bottom continuous images shot by a line scanning camera on a rail as a train bottom part image; a primary image registration step: inputting the vehicle bottom part image into a classification model and outputting a template image; an image secondary registration step: expanding the vehicle bottom part image outwards for a certain pixel distance, performing secondary registration on the vehicle bottom part image and the template image, and performing the image primary registration step or the feature comparison output step again according to the registration result output; a feature comparison and output step: extracting feature vectors from the vehicle bottom part image, performing Euclidean distance calculation and comparison on the feature vectors and feature vectors in the template image, and outputting a part fault instruction or a part normal instruction according to a comparison result; the method has the advantage that the accuracy and reliability of component matching are improved through two times of registration.

Description

Technical Field

[0001] The present invention relates to the technical field of train bottom component detection, and more particularly to a machine vision-based method and system for detecting abnormalities of train bottom components. Background Art

[0002] With the rapid development of the railway transportation industry, the speed and frequency of trains are constantly increasing, and the requirements for train operation safety and maintenance efficiency are becoming higher and higher. The bottom parts of the train are subjected to complex mechanical loads and environmental influences during the operation of the train, and are prone to loosening, wear and even falling off. Abnormalities in the bottom parts of the train will not only affect the operating performance of the train, but may even lead to serious safety accidents. Therefore, timely and accurate detection of the status of the bottom parts of the train is of great significance to ensure the safe operation of the train.

[0003] Current inspections include traditional manual inspections and train underbody inspections based on machine vision and deep learning.

[0004] Among them, the limitations of traditional detection methods mainly include low efficiency: manual inspection is time-consuming and labor-intensive, especially for trains that require frequent inspections, the efficiency is low; high cost: a large number of professionals are required for manual inspection, resulting in high labor costs; high missed detection rate: manual inspection is limited by the experience and attention of personnel, and is prone to missed detection and false detection; safety risks: during the manual inspection process, maintenance personnel need to enter the bottom of the train to work, which poses certain safety risks.

[0005] Among them, there are still some technical difficulties in deep learning-based train bottom component detection: complex image acquisition conditions: the environment at the bottom of the train is complex, with poor lighting conditions and frequent vibrations, which may affect the quality of the collected images; high data annotation costs: training high-performance deep learning models requires a large amount of annotated data, and obtaining high-quality annotated data usually requires a lot of manpower and time; complex model training: the training of deep learning models requires a lot of computing resources and expertise, and training and optimizing models has high technical difficulties. Summary of the invention

[0006] In view of the deficiencies in the prior art, an object of the present invention is to provide a method and system for detecting abnormalities of vehicle underbody components based on machine vision.

[0007] To achieve the above object, the present invention provides the following technical solutions: A method for detecting abnormalities of vehicle bottom parts based on machine vision comprises the following steps: The classification model building step is to obtain the classification model by using the historical images of various components through the classification training strategy; A component image acquisition step, acquiring continuous images of the bottom of the train taken by a line scan camera on the rail as a bottom component image of the train; The first image registration step is to input the vehicle bottom component image into the classification model to output a template image; The image secondary registration step is to expand the vehicle bottom component image outward by a certain pixel distance and then perform secondary registration with the template image, and perform the image primary registration step again or perform the feature comparison output step according to the registration result output; The feature comparison and output step is to extract the feature vector from the vehicle bottom component image and compare it with the feature vector in the template image by Euclidean distance calculation, and output the component failure instruction or the component normal instruction according to the comparison result.

[0008] Furthermore, the classification training strategy includes a component labeling step, The component labeling step labels the historical images of various components with characteristic information to obtain a data set, wherein the characteristic information includes component size and shape, component installation position, and component function.

[0009] Furthermore, the classification training strategy also includes a classification training step. The classification training step is to obtain a target detection model by training, verifying and testing according to a certain ratio based on the data set, outputting a point graph of each type of annotated component historical images through the target detection model, and obtaining a classification model by training, verifying and testing according to the point graph through a certain ratio.

[0010] Furthermore, the classification training strategy also includes a feature extraction model generation step, The feature extraction model generation step removes the last FullConnection layer in the classification model to obtain a feature extraction model of the vehicle bottom component.

[0011] Furthermore, the classification training step includes a component classification sub-step, The component classification sub-step selects any several different types of components in the data set as reference components, and classifies other component images in the data set with the reference components through a component classification algorithm to obtain various sets of data sets for target detection model training, verification and testing.

[0012] Furthermore, the component classification algorithm is configured as follows: , , , , in, is the size and shape characteristics of the component, is the installation position feature of the component, is the functional characteristic of the component, is a function of size and shape, is the weight of the i-th dimension, is the measurement function of the i-th dimension, To install the position function, is a function of distance, is the integrated volume, is a functional function, is the weight of the jth function, is the evaluation function of the jth function, is a step function.

[0013] Furthermore, the classification training step also includes a verification submodule. The verification submodule retrieves the component name corresponding to each underbody component diagram in the preset expert experience library, aligns the component name with the underbody component diagram in each set of data sets, and then determines whether there is any classification anomaly based on semantic analysis of the underbody component diagram in each set of data sets. If so, a removal mark instruction is output, otherwise, an ignore instruction is output.

[0014] Furthermore, the secondary registration includes aligning the underbody component image after being expanded outward by a certain pixel distance with the template image through affine transformation, and the affine transformation formula is configured as: , in, are the transformed coordinates, is the transformation parameter, are the original coordinates.

[0015] Furthermore, the method further comprises a component image processing step, wherein the component image processing step processes the vehicle bottom component image through Gaussian filtering and then performs a primary image registration step.

[0016] A vehicle bottom parts abnormality detection system based on machine vision, comprising: The classification model building module uses the historical images of various components to obtain the classification model through the classification training strategy; A component image acquisition module, which acquires continuous images of the bottom of the train taken by a line scan camera on the rail as a bottom component image of the train; An image primary registration module inputs the vehicle bottom component image into a classification model to output a template image; The image secondary registration module further expands the underbody component image outward by a certain pixel distance and performs secondary registration with the template image, and re-performs the image primary registration step or the feature comparison output step according to the registration result output; The feature comparison output module extracts a feature vector from the vehicle bottom component image and compares it with the feature vector in the template image by Euclidean distance calculation, and outputs a component failure instruction or a component normal instruction according to the comparison result.

[0017] The beneficial effects of the present invention are as follows: a classification model is obtained by classifying and training various types of components, wherein the classification model can be a model of each individual component or a group of similar models. A group of similar models can improve the efficiency of model training and reduce the time of model training. In addition, the accuracy and reliability of component matching are improved through two registrations, which can significantly improve the efficiency and accuracy of train bottom component detection, reduce the workload of manual inspection, reduce operating and maintenance costs, and improve the safety and reliability of railway transportation. The promotion and application of this method will provide important technical support for the development of intelligent railway monitoring systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is the overall flow chart of the present invention; Figure 2 is a schematic diagram of secondary registration in the present invention; Figure 3 It is a schematic diagram of feature comparison in the present invention; Figure 4 It is a flow chart of the same type grouping in the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. The same parts are represented by the same reference numerals. It should be noted that the words "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to directions in the accompanying drawings, and the words "bottom surface" and "top surface", "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0020] Embodiment 1: Due to the complex image acquisition conditions at present: the train bottom environment is complex, the lighting conditions are poor, and the vibration is frequent, the quality of the collected images may be affected; the data annotation cost is high: training a high-performance deep learning model requires a large amount of annotated data, and obtaining high-quality annotated data usually requires a lot of manpower and time; model training is complex: the training of deep learning models requires a lot of computing resources and professional knowledge, and training and optimizing models has high technical difficulty. Therefore, the present invention designs this underbody component abnormality detection method based on machine vision, such as Figure 1 As shown, the following steps are included: The steps for building the classification model are as follows: first, data collection is carried out. Historical images of various components are collected from multiple sources (including factory databases and public data sets). The diversity and representativeness of the collected historical image data are ensured, including different angles and lighting conditions. The collected images must contain all the listed components.

[0021] Secondly, data organization is carried out. The collected image data is cleaned to remove blurred, repeated or irrelevant images, and the component labeling step in the classification training strategy is applied. The acquired vehicle bottom images are labeled with components using labeling software such as LabelImage. The components in the historical images include key components such as bolts, keyholes, pipe clamps and locks. That is, the labeled objects include but are not limited to the key components mentioned above. During the labeling process, the characteristic information of each component is accurately marked and the labeling results are saved as a TXT file. The characteristic information includes component size and shape, component installation position and component function. The labeled data will be used as a training data set for subsequent target detection model training.

[0022] Finally, the classification training steps in the classification training strategy are applied to perform underbody parts target detection training. The YOLOv8-M model is used for underbody parts target detection training. The labeled image data set is input into the YOLOv8-M model for training, so that the model can identify and locate the position and category of underbody parts. The YOLOv8-M model continuously optimizes its detection performance through multiple iterative training, and finally can achieve efficient and high-accuracy target detection. Among them, the loss function of the YOLOv8 model consists of three parts, and the specific configuration is: , in, is the coordinate regression loss, which is used to measure the difference between the predicted box and the real box. is the confidence loss, which is used to measure the accuracy of whether the predicted box contains the target. It is the classification loss, which is used to measure the accuracy of the category of the target in the predicted box.

[0023] On the basis of the target detection model training, the vehicle bottom parts are further classified and trained, and the image classification technology in deep learning is used (the application of image classification technology is to classify the input images by training convolutional neural network (CNN), which can identify different types of parts at the bottom of the train and judge their status. For component loss detection, image classification technology can be used to determine whether there are specific parts in the image, and whether these parts are abnormal (such as loose, damaged, etc.)). Specifically, the historical images of various annotated parts are output into item point graphs through the target detection model (for example, the item point graph output by the target detection model includes 6-7 item points, and each item point represents a different meaning. For example, item point 1 represents a bolt, and item point 2 represents a fastener. Therefore, the classification model is obtained by classification training through the item point graph), and then the item point graph is divided into training set, validation set, and test set according to 7:2:1. The image input size is unified as 224*224, the loss function adopts cross entropy loss, the optimizer adopts SGD (stochastic gradient descent method), and the learning rate is set to 0.001; The cross entropy loss calculation formula is: , in, is the true label, is the predicted probability, is the sample size; The classification model obtained by the above training is modified again, and the last layer of Full Connection is removed to obtain a feature extraction model (feature extractor) for the underbody parts. The deep learning model is used to extract the features of the image. Compared with the traditional solution of extracting feature points, the quality of feature expression is greatly improved. The output of the feature extractor is a 1*2048 vector, which means that each image will be extracted into a 2048-dimensional feature vector. This feature extractor gradually extracts local and global information and learns its main features from the input data, such as edges, textures, shapes, and complex semantic patterns. The feature vector is the output of these layers, encoding the high-level semantic information of the input data. Compared with the shallow network extraction, the features extracted by the shallow network are basically some local basic features, such as edges and some simple textures, while the output of the deeper network contains more abstract semantic features, describing the more essential features of the object.

[0024] Part image acquisition step: Install a high-resolution 2D line scan camera on the rails. The line scan camera scans along the direction of the train. When the train passes by, it scans line by line to obtain continuous images of the bottom of the train. The line scan camera has high sensitivity and high resolution, and can obtain clear images of the bottom of the train under complex environmental conditions. Each scanned image is formed into a complete bottom component image through stitching technology, providing basic data for subsequent image processing and analysis; In addition, it also includes the component image processing step, which pre-processes the collected vehicle bottom component images, including de-exposure and denoising. The de-exposure process is used to adjust the brightness and contrast of the image to ensure the consistency of the image under different lighting conditions. The denoising process is used to remove noise and interference in the image to improve the image quality. The pre-processed image provides high-quality input data for subsequent feature extraction and matching. Gaussian filtering is applied in the image denoising process, which is specifically manifested as follows: , in, is the denoised image, is the standard deviation of the Gaussian kernel, is the kernel size.

[0025] In the first image registration step, the trained classification model is applied to the template image and the image to be inspected (the vehicle bottom component map) in the actual inspection process to identify and locate the vehicle bottom components in the image. The template image is a standard image of a consistent component state, which contains the characteristic information of the component. The vehicle bottom component map is an image of the bottom of the train obtained during the actual inspection process, which contains the component information to be inspected. That is, the two are registered for the first time. The image secondary registration step is as follows: Figure 2 As shown, the underbody component image is expanded outward by a certain pixel distance, that is, 200*200 pixels are intercepted outward, and then secondary matching is performed to ensure that the position and posture of the components in the image are consistent. The secondary registration uses image transformation technology, and the image transformation technology applies affine transformation to accurately align the component position of the underbody component image with the standard position in the template image, thereby improving the accuracy and reliability of component matching; Among them, the affine transformation formula is configured as: , in, are the transformed coordinates, is the transformation parameter, are the original coordinates.

[0026] Feature comparison output step, such as Figure 3As shown, the feature vector extracted by the feature extractor in the classification model in the vehicle bottom component image is compared with the feature vector pre-stored in the template image, and the distance between the feature vectors is calculated using the Euclidean distance, which can intuitively reflect the difference between the two vectors in the high-dimensional space. When the calculated Euclidean distance value is less than or equal to 0.4, it indicates that the features of the component to be detected are highly similar to those of the template component. It can be considered that the component has not changed significantly and the output component is a normal instruction. On the contrary, when the Euclidean distance value is greater than 0.4, it indicates that there is a large difference between the features of the component to be detected and the features of the template component. The system determines that the component has changed and outputs a component fault instruction. This method can not only accurately detect subtle changes in components, but also improve the robustness and reliability of the system, and effectively avoid false alarms and missed alarms.

[0027] Embodiment 2: Since there are many types of parts at the bottom of the train and the number is large, the key parts include bolts, lock holes, pipe clamps, locks and pipelines. Take bolts as an example. The types of bolts include hexagonal bolts, square bolts, and can also include cross screws and plum screws in a larger range. The size of each bolt is also different. Therefore, if the model training is performed for each component based on the traditional method, it will lead to problems such as large training volume, poor training results, and excessive data processing volume. That is, based on Example 1, the classification training is further improved, wherein the classification training step includes a component classification sub-step, such as Figure 4 As shown, In the component classification substep, any several different types of components are selected in the data set as reference components. Specifically, based on the shape, size, function, and frequency of occurrence of the components, the reference components are determined by combining expert evaluation, statistical analysis, or clustering algorithms. For example, a bolt with a certain size and function appears the most times. Based on the screening, the bolt A is used as the reference component, and a certain pipe B is also used as the reference component. Therefore, the reference components can include bolt A, pipe B, keyhole C, etc. The other component images in the data set are classified with the reference components through the component classification algorithm to obtain the target detection model for training, verification, and testing. Each group of data sets, including bolt groups, pipe groups, keyhole groups, etc., also ensures that each group of data sets (i.e., each benchmark component category) contains a sufficient number of samples to meet the needs of model training. At the same time, in order to verify the generalization ability of the model, it should be ensured that the data in the validation set and the test set have certain differences and representativeness from the training set. The division of the data set can be optimized through cross-validation, stratified sampling and other methods. Compared with the traditional single component training, this group training method can reduce the number of trained models, expand the application range of the model, save training time, and make the model call faster during actual detection.

[0028] Specifically, the component classification algorithm is configured as follows: , , , , in, is the size and shape characteristics of the component, is the installation position feature of the component, is the functional characteristic of the component, is a size and shape function that can combine geometric parameters such as length, width, and height. It also includes some normalization steps to ensure that parts of different sizes can be compared fairly, for example, using a weighted sum to take into account the importance of different dimensions. is the weight of the i-th dimension, is the measurement function of the i-th dimension, To install the position function, you can involve points in the spatial coordinate system and use advanced integral functions to calculate the position relationship of components in a specific area. For example, you can use integration in three-dimensional space to determine the distance between the center point of a component and a reference point. is a function of distance, is the integrated volume, is a functional function, which can be the result of a logistic regression model or other machine learning algorithm, used to evaluate the functional similarity of parts, and can include linear combinations of multiple features and possible nonlinear transformations. is the weight of the jth function, is the evaluation function of the jth function, is a step function. The final classification model can be expressed as the weighted sum of the above three functions, where the weight reflects the importance of each attribute in the classification process. To determine whether two components are classified into the same category, specifically, it is only necessary to compare other components with the reference components respectively, and determine which reference component is most similar to the other components based on the weighted sum.

[0029] The classification training step also includes a verification submodule, which establishes an expert experience library containing the bottom vehicle component images and their corresponding component names. The expert experience library is the basis of the verification submodule and is used to provide accurate component names and image information. The expert experience library contains image samples of various components on the bottom of the train. Each sample should be labeled with an accurate component name. Image processing and matching algorithms, such as feature point matching and template matching, are used to compare the images in the expert experience library with the images in the data set. By calculating the similarity or matching degree, it is determined which component name in the expert experience library best matches the image in the data set, so that the names of the components in each group of data sets can be known, and then the semantic analysis is used to analyze the names of the components in each group of data sets. For example, one group of data sets includes hexagonal bolts 1, four-corner bolts 2, hexagonal bolts 3, cross screws 2, keyholes 3 and plum screws 3. At this time, keyhole 3 can be directly filtered out according to semantic analysis. Keyhole 3 does not match other bolts and screws in this group of data sets. Therefore, it can be determined that this group of data sets is classified abnormally, and the filtered keyhole 3 needs to be moved out of this group of data sets and marked. After that, keyhole 3 can be directly classified into the keyhole group according to semantics. If the semantic analysis classifies this group of data sets correctly, an ignore instruction is output.

[0030] According to the designed detection method, the corresponding vehicle bottom parts abnormality detection system based on machine vision is designed, including The classification model building module uses the historical images of various components to obtain the classification model through the classification training strategy; A component image acquisition module, which acquires continuous images of the bottom of the train taken by a line scan camera on the rail as a bottom component image of the train; The image primary registration module inputs the vehicle bottom component image into the classification model and outputs the template image; The image secondary registration module further expands the underbody component image outward by a certain pixel distance and performs secondary registration with the template image, and performs the image primary registration step again or the feature comparison output step according to the registration result output; The feature comparison output module extracts feature vectors from the vehicle bottom component image and compares them with feature vectors in the template image by Euclidean distance calculation, and outputs component fault instructions or component normal instructions based on the comparison results.

[0031] The above are only preferred embodiments of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for detecting abnormalities of vehicle bottom parts based on machine vision, characterized in that: The steps include: The classification model building step is to obtain the classification model by using the historical images of various components through the classification training strategy; A component image acquisition step, acquiring continuous images of the bottom of the train taken by a line scan camera on the rail as a bottom component image of the train; The first image registration step is to input the vehicle bottom component image into the classification model to output a template image; The image secondary registration step is to expand the vehicle bottom component image outward by a certain pixel distance and then perform secondary registration with the template image, and perform the image primary registration step again or perform the feature comparison output step according to the registration result output; The feature comparison and output step is to extract the feature vector from the vehicle bottom component image and compare it with the feature vector in the template image by Euclidean distance calculation, and output the component failure instruction or the component normal instruction according to the comparison result.

2. The method for detecting abnormalities of vehicle bottom parts based on machine vision according to claim 1, characterized in that: The classification training strategy includes a component labeling step, The component labeling step labels the historical images of various components with characteristic information to obtain a data set, wherein the characteristic information includes component size and shape, component installation position, and component function.

3. The method for detecting abnormalities of vehicle bottom parts based on machine vision according to claim 2, characterized in that: The classification training strategy also includes a classification training step, The classification training step is to obtain a target detection model by training, verifying and testing according to a certain ratio based on the data set, outputting a point graph of each type of annotated component historical images through the target detection model, and obtaining a classification model by training, verifying and testing according to the point graph through a certain ratio.

4. The method for detecting abnormalities of vehicle bottom parts based on machine vision according to claim 3, characterized in that: The classification training strategy also includes a feature extraction model generation step, The feature extraction model generation step removes the last Full Connection layer in the classification model to obtain a feature extraction model of the vehicle bottom component.

5. The method for detecting abnormalities of vehicle bottom parts based on machine vision according to claim 3, characterized in that: The classification training step includes a component classification sub-step, The component classification sub-step selects any several different types of components in the data set as reference components, and classifies other component images in the data set with the reference components through a component classification algorithm to obtain various sets of data sets for target detection model training, verification and testing.

6. The method for detecting abnormalities of vehicle bottom parts based on machine vision according to claim 5, characterized in that: The component classification algorithm is configured as follows: ， ， ， ， in, is the size and shape characteristics of the component, is the installation position feature of the component, is the functional characteristic of the component, is a function of size and shape, is the weight of the i-th dimension, is the measurement function of the i-th dimension, To install the position function, is a function of distance, is the integrated volume, is a functional function, is the weight of the jth function, is the evaluation function of the jth function, is a step function.

7. The method for detecting abnormalities of vehicle bottom parts based on machine vision according to claim 6, characterized in that: The classification training step also includes a verification submodule, The verification submodule retrieves the component name corresponding to each underbody component diagram in the preset expert experience library, aligns the component name with the underbody component diagram in each set of data sets, and then determines whether there is any classification anomaly based on semantic analysis of the underbody component diagram in each set of data sets. If so, a removal mark instruction is output, otherwise, an ignore instruction is output.

8. The method for detecting abnormalities of vehicle bottom parts based on machine vision according to claim 1, characterized in that: The secondary registration includes aligning the underbody component image after being expanded outward by a certain pixel distance with the template image through affine transformation, and the affine transformation formula is configured as: ， in, are the transformed coordinates, is the transformation parameter, are the original coordinates.

9. The method for detecting abnormalities of vehicle bottom parts based on machine vision according to claim 1, characterized in that: The method further comprises a component image processing step, wherein the component image processing step processes the vehicle bottom component image through Gaussian filtering and then performs a primary image registration step.

10. A vehicle bottom parts abnormality detection system based on machine vision, characterized in that: include The classification model building module uses the historical images of various components to obtain the classification model through the classification training strategy; A component image acquisition module, which acquires continuous images of the bottom of the train taken by a line scan camera on the rail as a bottom component image of the train; An image primary registration module inputs the vehicle bottom component image into a classification model to output a template image; The image secondary registration module further expands the underbody component image outward by a certain pixel distance and performs secondary registration with the template image, and re-performs the image primary registration step or the feature comparison output step according to the registration result output; The feature comparison output module extracts a feature vector from the vehicle bottom component image and compares it with the feature vector in the template image by Euclidean distance calculation, and outputs a component failure instruction or a component normal instruction according to the comparison result.

Images